THE DEVELOPMENT OF SIC-IR TO ASSIST WITH DIAGNOSING INFECTIONS IN CRITICALLY ILL TRAUMA PATIENTS: MOVING BEYOND THE FEVER WORKUP

Transcription

1 THE DEVELOPMENT OF SIC-IR TO ASSIST WITH DIAGNOSING INFECTIONS IN CRITICALLY ILL TRAUMA PATIENTS: MOVING BEYOND THE FEVER WORKUP SIC-IR : The Surgical Intensive Care Infection Registry by JEFFREY A. CLARIDGE, MD Submitted in partial fulfillment of the requirements For the degree of Master of Science Clinical Research Scholars Program CASE WESTERN RESERVE UNIVERSITY August, 2008

2 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES We hereby approve the thesis/dissertation of Jeffrey A. Claridge, candidate for the Master of Science degree *. (signed) Randall D. Cebul (chair of the committee) Alfred F. Connors Mark A. Malangoni (date) 6/3/2008 *We also certify that written approval has been obtained for any proprietary material contained therein. 2

3 The Development of SIC-IR to Assist with Diagnosing Infections in Critically Ill Trauma Patients: Moving Beyond the Fever Workup SIC-IR : The Surgical Intensive Care Infection Registry Abstract By JEFFREY A. CLARIDGE, MD The current work-up of infections in the surgical and trauma intensive care unit (STICU) is inefficient and often based on fever and leukocytosis. Using retrospective data we evaluated our practice of obtaining cultures in the STICU and concluded that physicians order cultures based on fever and/or leukocytosis, yet fever and/or leukocytosis were not associated with positive culture results in critically injured patients within the first 14 days of injury. To challenge and change conventional practice we developed a medical informatics system, SIC-IR (Surgical Intensive Care Infection Registry) to be a tool to allow us to further study the infection work-up in the STICU. We validated SIC-IR to be a successful medical informatics tool that archived accurate data. Additionally, we demonstrated that SIC-IR has excellent sensitivity of identifying patients with ventilator associated pneumonia and improves documentation of a patients severity of illness. Our long term goal is to develop a highly discriminative predictive model to improve the diagnosis of infections in critically ill patients. 3

4 Table of Contents: pages List of Tables 5 List of Figures 8 Section 1. - Introduction Chapter 1: Background/Significance/Acknowledgement..10 Section 2 Retrospective analysis of our culture practice in determining urinary tract, bacteremia, and respiratory infections in critically ill trauma patients Chapter 2: Urinary Tract Infections 15 Chapter 3: Bacteremia Chapter 4: Respiratory Tract Infections.63 Chapter 5: Changing the Fever Workup Utilizing a Multi-Technique Modeling Approach to More Accurately Diagnose Infections.. 90 Section 3 Creation and Validation of the Surgical Intensive Care Infection Registry (SIC-IR) Chapter 6: Creation of SIC-IR.126 Chapter 7: Validation of SIC-IR..145 Section 4 Utility of SIC-IR Chapter 8. Monitoring Infections in the ICU..178 Chapter 9: SIC-IR Documents Sicker Patients 201 4

5 Table List Table 2.1: Study population demographics and outcomes..29 Table 2.2: Number of urine cultures and urinary tract infections...30 Table 2.3: STICU daily urinary culture practice. 31 Table 2.4: STICU urinary culture results 32 Table 3.1: Demographics 54 Table 3.2: The number of blood cultures and bacteremias Table 3.3: Bivariate analysis of our blood culture practice...56 Table 3.4: The relative risks of fever and leukocytosis triggering blood cultures 57 Table 3.5: The relative risk of fever and leukocytosis associated with bacteremia Table 3.6: Outcome comparison of patients with and without bacteremia 59 Table 4.1: Study Population Demographics and Outcomes 80 Table 4.2: Practice of Obtaining Respiratory Cultures 81 Table 4.3: Efficacy of Identifying Positive Respiratory Cultures. 82 Table 5.1: Variables used in mathematical modeling 108 Table 5.2: Description of mathematical models used in this evaluation Table 5.3: Criteria used to compare the mathematical models.109 Table 5.4: Clinical and demographic data of the study cohort..110 Table 5.5: Logistic regression analysis identifying variables associated with our STICU culture practice Table 5.6: Comparison of training and testing set of the ten modeling variables Table 5.7: Analysis of mathematical modeling techniques

6 Table 5.8: Comparison of training and testing set of the ten modeling variables in patient-days with at least one culture obtained..114 Table 5.9: Analysis of mathematical modeling techniques in patient days where cultures were obtained 115 Table 7.1: Outcomes and methods used to validate SIC-IR utilizing the DeLone and McLean Model as a framework 166 Table 7.2: Resident users of SIC-IR Table 7.3: Comparison of progress note accuracy between the SIC-IR unit and the non SIC-IR unit.168 Table 8.1: Summary of SIC-IR versus IC reporting of NHSN data. 195 Table 8.2: Summary of SIC-IR versus IC reporting of NHSN data normalized to administrative data Table 8.3: Two by two table for SIC-IR reported cases of ventilator associated pneumonia used to calculate sensitivity and specificity 197 Table 8.4: Two by two table for IC reported cases of ventilator associated pneumonia used to calculate sensitivity and specificity.198 Table 9.1: List of diagnoses and procedures taught to the SIC-IR B/DM 220 Table 9.2: List of the commonly missed diagnoses which have a significant impact on MS-DRG assignment.221 Table 9.3: List of STICU documentation procedures occurring in phase 1 and phase Table 9.4: Demographics of study patients Table 9.5: Clinical variables compared between phase 1 and

7 Table 9.6: SIC-IR B/DM outputs during phase Table 9.7: ICD-9 and DRG outcomes between phase 1 and phase

8 Figure List Figure 2.1: The percent of patients with fever, leukocytosis, or both. 33 Figure 2.2: The percent of UTIs by temperature range 34 Figure 2.3: The percent of UTIs by leukocyte count range.35 Figure 3.1. Blood culture episodes by day..60 Figure 3.2: Percent of positive blood and central venous catheter tip cultures compared to leukocyte count 61 Figure 3.3: Percent of positive blood and central venous catheter tip cultures compared to body temperature.62 Figure 4.1: Analysis of patients with fever, leukocytosis, and fever with leukocytosis..83 Figure 4.2: Percent of patients remaining in STICU with fever, leukocytosis, and fever with leukocytosis by week Figure 4.3: Percent of positive respiratory cultures by temperature range. 85 Figure 4.4: Percent of positive respiratory cultures by WBC range 86 Figure 4.5: Distribution of Respiratory Cultures and Treated Respiratory Infections per Patient..87 Figure 5.1: Flow diagram of patient selection to evaluate our STICU culture practice and the impact the fever workup has on this practice Figure 5.2: 2 x 2 table evaluating our STICU culture practice..123 Figure 5.3: Flow diagram of patient selection to evaluate mathematical models. 124 Figure 5.4: Flow diagram of patient selection to evaluate mathematical models in patient-days when cultures were obtained

9 Figure 6.1: Clinical data movement from the actual patient to administrative databases and reasons for potential loss of data integrity 141 Figure 6.2: Example of a SIC-IR STICU patient s Daily Rounding Sheet. 142 Figure 6.3: The architecture of SIC-IR and the movement of data and information to and from the clinician user 143 Figure 6.4: Schematic representation of data entry from the true clinical state of the patient into SIC-IR..144 Figure 7.1: Clinical data SIC-IR dashboard Figure 7.2: Structured SIC-IR data entry form..173 Figure 7.3: Free text SIC-IR data entry form 174 Figure 7.4: Schematic representation of data entry from the true clinical state of the patient into SIC-IR..175 Figure 7.5: The DeLone and McLean Model of Information System Success. 176 Figure 7.6: SIC-IR resident use survey study results. 177 Figure 8.1: SIC-IR daily information and pharmacy. 199 Figure 8.2: SIC-IR daily information and pharmacy.200 Figure 9.1: Endpoints of hospital administrative data.227 Figure 9.2: The hospital reimbursement equation 228 Figure 9.3: Handwritten billing documentation template Figure 9.4: SIC-IR B/DM graphical user interface. 230 Figure 9.5: SIC-IR B/DM data dashboard..231 Figure 9.6: Graphical user interface for selecting additional ICD-9 codes

10 Chapter 1. Introduction 10

11 Background/Significance The work encompassed in this thesis involves four main sections. The first one is the introduction and overview which I am going over now. There are three remaining sections. Section two describes a retrospective analysis of our (Surgical and Critical Care units) culture practice for determining urinary tract infections, blood stream infections, and respiratory infections in critically ill trauma patients. Specifically, we were interested in looking at our culture practice in the surgical intensive care for trauma patients during their first fourteen days of admission. We were especially interested in evaluating the role of fever and leukocytosis in determining our culture practice and then evaluating to see whether or not our culture practice, and specifically fever and leukocytosis was associated with actually developing a positive infection or positive culture. This work is summarized in Chapters 2 through 5. The one thing that Chapters 2 through 5 consistently illustrate is the fact that we do culture based on fever and leukocytotis, however fever and leukocytotis are not clearly associated with positive cultures (urinary, blood or respiratory). Chapter 5 summarizes all three infections looking at the impact of fever and/or leukocytotis in obtaining cultures (aka the fever workup ). In chapter 5 we calculate the ability of various mathematical modeling techniques to assist beyond our current practice in the identification of infectious complications (urinary tract infections [UTI], bacteremia, or respiratory tract infections [RTI]) in critically ill trauma patients. The goal of that pilot study was not to create an infection workup, but rather determine the feasibility of using mathematical modeling techniques, and identify a technique which has the greatest potential to assist in the development of future clinical decision support. In this section we make the argument 11

12 that our current workup algorithm using clinical information at the bedside is not very accurate. This leaves us to question our efficacy in diagnosing infections in the intensive care unit when our strongest predictive factor for getting cultures is fever and/or leukocytotis which downstream does not show to be associated with infections or at least positive cultures. Section 3 deals with trying to find a solution to the problems of retrospective data. Our main problem with our retrospective data was incomplete data and. the fact that we did not have all of the variables that we thought were important. We also felt it was important to get data prospectively at the point of patient care. Thus, we created the Surgical Intensive Care - Infection Registry referred to as SIC-IR to document and track all patients from admission to discharge from the ICU. We have two chapters in this thesis that are dedicated to development and validation of SIC-IR. The first chapter is Chapter 6 which deals with the creation of SIC-IR. This manuscript/chapter describes how SIC-IR was initially created and the fundamentals behind its design. It also describes the rationale for its data capture. Since this manuscript we are continuing to work on a more recent prototype which will increase some of the automated downloads as well as improving some of the graphic user interface. However, we then did an evaluation study to validate SIC-IR as an accurate reliable database that is also easy to use. This is a manuscript (Chapter 7) that has been accepted for publication in the Journal of American College of Surgeons and won the best paper for the resident paper competition. I was the mentor and primary investigator for the project. The resident paper competition is the longest standing competition recognized by the American College of Surgeons and is now extending beyond the US and all over the Americas as 12

13 well as Australia and Europe. Again, this work done in developing SIC-IR and then showing the validation study is recognized by the College and received the first place award out of over 500 papers that were initially submitted. Section 4 of this thesis has two chapters. Chapter 8 deals with monitoring infections in the ICU and demonstrates that we can use SIC-IR at the point of care to document infections. We were interested in comparing the sensitivity and specificity of diagnosing ventilator associated pneumonia using SIC-IR and comparing this to the surveillance technique that is employed by our hospital infection control partners. Chapter 9 describes how SIC-IR documents patients more accurately and thoroughly. The hypothesis was that utilizing SIC-IR attending billing and documentation module would increase the documented severity of illness, risk of mortality, number of diagnosis, and the reimbursement of surgical trauma intensive care unit patients. We successfully demonstrated these things when comparing it to a historic cohort which is three months prior to the utilization of the SIC-IR billing and documentation module Responsibilities/Acknowledgement The work encompassed in the following pages consists of work that I have done since entering Case Western Reserve University Master s program of clinical investigation. This work was done since my arrival and starting the program in July 1, 2005 and has been enhanced by the educational courses and opportunities provided through the CRSP program. I was the primary investigator for the manuscripts/chapters included in the thesis but not necessarily the first author on all the submitted manuscripts, as I feel that this is important to have collaboration and part of my experience at 13

14 MetroHealth has been mentoring. Joseph F. Golob is a resident who I have mentored for the past twoyears. A great deal of this work was done in collaboration with him and others. We have divided up some of the authorship based on some opportunities for awards as well as sharing credit. I can honestly say though that I am responsible for all of the work and have been involved in every step of the way in developing this body of work. I have also had substantial collaboration with Adam Fadlalla. whom I would like to give recognition to. Dr. Fadlalla is relatively a newcomer to the scene of medical informatics; however he has had a long history of doing modeling for businesses. He is a computer scientist and has a PhD. in computer science and technology from Cleveland State University and also has a Master s degree in business as well as a Master s in statistics. He has been a great colleague to assist with some of these advanced modeling techniques as well as assisting us with data base design. The following the body of work encompasses my thesis for masters credit through the CRSP program. This is also part of my work done as the original K12 program. I was the initial cohort chosen in the K12 which is now morphed into the CTSA program. I would also like to give credit and thanks to the following people: My SIC-IR Partners: Adam Fadlalla, PhD and Joseph F. Golob, MD My Boss, Chairman, and Surgical Research Mentor: Mark A. Malangoni, MD My Multidisciplinary Research Mentors: Randall D. Cebul, MD and Alfred F. Connors My wife: Rebecca L. Y. Claridge 14

15 Chapter 2. FEVER AND LEUKOCYTOSIS IN CRITICALLY ILL TRAUMA PATIENTS: IT S NOT THE URINE 15

16 FEVER AND LEUKOCYTOSIS IN CRITICALLY ILL TRAUMA PATIENTS: IT S NOT THE URINE Joseph F. Golob Jr., M.D. *, Mark J. Sando, BS *, William R. Phipps, M.D, Charles J. Yowler, M.D. *, Mark A. Malangoni, M.D. *, Adam M.A. Fadlalla Ph.D**, and Jeffrey A. Claridge, M.D. * * MetroHealth Medical Center Department of Surgery. Case Western Reserve University School of Medicine, Cleveland, OH ** Cleveland State University Department of Information Science Trumbull Memorial Hospital Department of Surgery and Critical Care, Cortland, OH Presented: The 27 th annual meeting of the Surgical Infection Society in Toronto, Canada. April 20, :30-5:00 PM. Acknowledgement JA Claridge is supported by the National Institutes of Health, National Institute of Child Health and Human Development, Multidisciplinary Clinical Research Career Development Programs Grant K12 RR Corresponding Author: Jeffrey A. Claridge, M.D. MetroHealth Medical Center Hamann Building Room H939 Cleveland, OH Phone: Fax: jclaridge@metrohealth.org 16

17 ABSTRACT: Background: Infectious complications are a major cause of morbidity and mortality in critically ill trauma patients. Fever and leukocytosis in these patients often trigger an extensive laboratory workup which includes a urine culture (UCx). The purpose of this study was to: 1) define the current practice for obtaining UCxs in trauma patients admitted to the surgical and trauma intensive care unit (STICU), and 2) determine if there is an association between fever or leukocytosis and urinary tract infections (UTIs) during the initial 14 hospital days. Methods: An 18-month retrospective cohort analysis was performed on consecutive trauma patients admitted for 2 days to the STICU at a level I trauma center. Data collected included: demographics, injuries, and the first 14 days of daily maximal temperature (TMax), leukocyte count, and UCx results. Fever and leukocytosis were defined as Tmax 38.5ºC and leukocyte count 12,000 / mm 3, respectively. UTIs were diagnosed with a positive UCx ( 10 5 organisms). Results: Five hundred and ten patients were evaluated for a total of 3839 patient-days. The mean patient age and injury severity score was 49 ± 1 years and 19 ± 1, respectively. Seventy two percent were males and 91% had blunt injuries. Four hundred and seven UCxs were obtained and 42 patients (8%) had 60 UTIs. The cohort had an indwelling urinary catheter in place for 97% of the patient-days yielding an infection density of 16 UTIs / 1000 urinary catheter-days. There was a significant association of obtaining a UCx with fever and fever with leukocytosis (p <0.001), but no association of UTI with fever, leukocytosis, or the combination of fever and leukocytosis. Analysis using temperature and leukocyte count as continuous variables showed no temperature or leukocyte range 17

18 associated with UTIs. Independent risk factors for UTI calculated by logistic regression were female gender, older age, low ISS, and no antibiotics 24 hours before obtaining the UCx. Conclusions: The practice for obtaining a UCx in the STICU trauma patient was related to fever and fever with leukocytosis. However fever, leukocytosis, or both were not associated with UTIs. These data suggest that there is an unnecessary emphasis on UTI as a source of fever and leukocytosis in injured patients during their first 14 STICU days. Our study suggests that the paradigm for evaluating UTI as a cause for fever needs to be reevaluated in critically ill trauma patients. 18

19 BACKGROUND: Infections are extremely common in the intensive care unit with approximately two million nosocomial infections occurring annually in the United States(1). Previous work has shown that infection rates are higher in surgical and trauma intensive care units (STICU) compared to medical intensive care units (2). Trauma patients in the STICU have a higher risk of nosocomial infections compared to general surgical patients (3) with 37%-45% of trauma patients experiencing an infectious complication (4, 5). The most commonly reported nosocomial infections include the lower respiratory tract, urinary tract, surgical site, sinusitis, and catheter-related blood stream infections. Although infections are common in the critically injured, they are often difficult to diagnose and their evaluation and treatment expose patients to unnecessary tests, invasive procedures, higher medical costs, and inappropriate use of antibiotics (1, 6). The surgical intensivist must often differentiate between infectious and non-infectious causes of fever and the systemic inflammatory response syndrome (SIRS). This differentiation is more difficult in trauma patients as studies have demonstrated that 92% of critically injured patients have a SIRS response during their first week in the STICU and 69% demonstrate SIRS during the second week as a normal response to their injuries (4). Fever and leukocytosis are two components of the SIRS response that often trigger a broad infection workup including cultures of blood, urine, respiratory secretions, and a chest radiograph. This common clinical practice for evaluating fever and leukocytosis is often referred to as the fever workup and its clinical usefulness and cost effectiveness has been questioned previously in the surgical patient (7-9). The concern is that this shotgun approach to the evaluation of fever and leukocytosis may potentially be 19

20 unnecessary in the critically ill trauma patient during their first two weeks in the STICU as a result of the SIRS response to injury and inflammation. To begin dissection of this complex problem, we investigated the diagnosis of urinary tract infections (UTIs) as part of the fever workup in critically ill trauma patients. Urinary tract infections are a common nosocomial infection in critically injured patients. It has been estimated that UTIs will increase the duration of hospitalization by one to four days with an average excess cost of approximately $600 per UTI (1). This results in health care costs between $424 - $451 million dollars yearly attributable to the diagnosis and treatment of nosocomial UTIs (10). There were two main objectives for this study. The first objective was to define our STICU fever workup practice for diagnosing UTIs based on fever and leukocytosis in critically injured patients over the first 14 days of admission. The second objective was to evaluate the efficacy of this practice by determining if fever and leukocytosis were associated with the development a UTI. DESIGN AND METHODS: A retrospective review was conducted on all injured trauma patients admitted to the STICU at a level I urban trauma center over 18 months (January 2004 through June 2005). Patients were excluded if they were admitted to an outside hospital for greater than 24 hours prior to transfer, were admitted to a non-icu floor before STICU admission, or had an initial STICU length of stay less than 48 hours. Data were collected from the patient s chart, electronic laboratory information system (Epic Systems Corporation: Madison, WI) and our trauma registry (dbase: Vestal, NY). Data collected included patient demographics and injuries as well as daily 20

21 maximal temperature, leukocyte count, and antibiotic therapy. In addition, the presence of an indwelling urinary catheter and urine culture (UCx) results were recorded for the first 14 hospital days or until the patient was discharged from the STICU. Fever was defined as a maximum temperature of 38.5ºC and leukocytosis as a leukocyte count 12,000 / mm 3 during a 24 hour period. A UTI was defined as a positive UCx ( 10 5 organisms/ml) (11). Data was analyzed using SPSS (SPSS Inc: Chicago, IL) software. A p-value of 0.05 was considered statistically significant. Numeric data is expressed as mean ± standard error of the mean and compared using the student t-test. Categorical data is described as a percentage and compared using either Chi-square or Fisher s exact test as appropriate. Logistic regression analysis was done to evaluate independent risk factors for the development of UTIs. Clinically relevant factors that were potentially (p<0.20) associated with UTI were used for logistic regression analysis. In addition to the factors determined by bivariate analysis, fever and leukocytosis were also evaluated by backward stepwise logistic regression analysis. Using this data set, sensitivities and specificities were estimated for various levels of probability for the development of UTI. Receiver operator characteristics were generated on the basis of various probabilities for conversion. This study was reviewed and approved by our institution s Institutional Review Board. RESULTS: Study Population Five hundred ten critically ill trauma patients with a mean injury severity score (ISS) of 19.4 ± 0.5 met the study criteria for inclusion in our analysis. The mean age was 49 ± 1 years-old and 72% were males. The population had a total of 3,839 patient-days 21

22 of data collected. Additional patient demographics and outcomes are summarized in Table 1. A total of 407 UCxs were obtained. Overall, a UCx was obtained on 11% of the patient-days. An indwelling urinary catheter was in place during 97% of the patientdays. Forty-two patients (8%) had 60 UTIs. This yielded an infection density of 16 UTIs per 1000 urinary catheter-days (Table 2). All patients with a UTI had an indwelling urinary catheter in place at the time of diagnosis. A comparison of patients diagnosed with a UTI compared to those patients without a UTI demonstrated that patients with UTIs had statistically longer STICU length of stay (15.8 ± 1.2 vs.9.4 ± 0.5; p=0.003), hospital length of stay (17.2 ± 1.2 vs.12.8 ± 0.6; p=0.027) and higher hospital mortality (21% vs. 6%; p<0.001). To further describe the patient population, the frequencies of fever, leukocytosis, and the combination of fever and leukocytosis were evaluated daily. The percent of patients remaining in the STICU by day who had fever, leukocytosis, and fever with leukocytosis is illustrated in Figure 1. Fever and leukocytosis were common in our patient population. Twenty nine percent of the patient-days the study population was febrile and 41% of the patient-days they had a leukocytosis. The combination of fever and leukocytosis was observed on 12% of the patient-days. Defining the Practice of Obtaining Urine Cultures The relationship between obtaining UCxs and fever, leukocytosis, and the combination of fever and leukocytosis is shown in Table 3. Fever was observed in 70% of the patients who had a UCx obtained vs. 24% in those who were not cultured (p < 0.001). Similarly, fever with leukocytosis was observed in 27% of the patients who were 22

23 cultured vs. 10% of those not cultured (p< 0.001). Evaluation of temperature and leukocyte count as continuous variables was also performed. The mean temperature for patients who were cultured was significantly higher (p<0.001) than for patients who were not cultured (38.6 ± 0.03ºC vs ± 0.1ºC, respectively). The mean leukocyte count of patients who were cultured vs. those not cultured was not significantly different (12.0 ± 0.3 and 11.6 ± 0.1, respectively; p=0.288). For comparison purposes, table 3 also contains data regarding UCxs which were obtained in patients with NO fever and NO leukocytosis. Fourteen percent of the urine cultures obtained were in patients without fever and leukocytosis. The lack of fever and leukocytosis was associated with not obtaining a UCx (RR = 0.22; 95%CI = ). These data demonstrates that obtaining a UCx in the STICU was associated with fever and the combination of fever with leukocytosis, but not leukocytosis alone. Defining the Efficacy of Obtaining Urine Cultures Four hundred seven UCxs were obtained and 14.7% were positive. The relationship between the results of UCx and fever, leukocytosis, and the combination of fever with leukocytosis is shown in Table 4. In addition, this table also contains the results of urine cultures obtained in patients with NO fever and NO leukocytosis. There was no association between UTI and the presence of fever, leukocytosis, or fever with leukocytosis. The lack of fever and leukocytosis together was not associated with having a UTI. Mean body temperature was similar for patients with a negative UCx and patients with a UTI (38.7 ± 0.03 ºC vs ± 0.09 ºC, respectively; p = 0.495). The mean leukocyte count in those with a negative UCx was higher at 12.3 ± 0.3 compared to 10.7 ± 0.5 in those with UTI (p = 0.005). 23

24 The practice for obtaining UCxs in critically ill trauma patients during their initial hospitalization was associated with fever and fever with leukocytosis, but neither of these categorical clinical parameters were associated with UTI. Thus, we investigated temperature and leukocyte count as continuous variables to determine if there was range at which UTI were more common. Figure 2 demonstrates the percentage of UTIs related to body temperature and Figure 3 depicts UTIs related to leukocyte count. There was no temperature or leukocyte range associated with significantly more UTIs. Examination of Confounding Variables by Logistic Regression Analysis Logistic regression analysis was used to determine which variables were independent risk factors of UTIs. The significant risk factors were increasing age (OR = 1.05; 95% CI = ), decreasing ISS (OR=1.05; 95% CI = ), female gender (OR = 3.03; CI = ), and no antibiotics the day prior to obtaining the UCx (OR = 2.25; CI = ). Logistic regression did not demonstrate an association of fever and leukocytosis with UTI. Receiver operator analysis demonstrated the area under the curve = DISCUSSION: The importance of UTIs and other nosocomial infections in the ICU population have been shown previously (5, 12-14). The total cost of workup and treatment for nosocomial infections has been estimated to be approximately 4.5 billion dollars annually(13). Some of this money used for the diagnostic studies in the evaluation of fever in the surgical population have been determined to be inefficient (7, 9). Despite these previous studies, no specific changes have been made to the common clinical practice for diagnostic workup of fever and/or leukocytosis in the ICU. To begin to 24

25 understand this complex problem, we felt it necessary to dissect the fever workup to a specific infection (UTI) in a very specific population (critically ill trauma patients). The results of this study show that our institution s STICU workup for UTIs in critically ill trauma patients followed the common practice of habitually obtaining urinary cultures as part of the fever workup. The actual definition of a fever and its workup is difficult to elucidate. Multiple studies and reviews have been contradictory with regard to the exact temperature of fever as well as which tests should be done in search of an infectious etiology of this fever (6, 9, 15). Our analysis was unable define a temperature range where UTIs were more common. Similarly, we were not able to define a leukocyte range associated with UTIs. The STICU practice of obtaining a UCx was associated with fever and the combination of fever and leukocytosis even though these clinical parameters were not associated with the development of UTIs during the initial 14 days of hospitalization. This suggests inefficiencies and an area of potential improvement in the diagnosis of UTIs in the critically ill trauma patient. The incidence density of UTIs in this cohort was 16 / 1000 indwelling urinary catheter-days, which is consistent with other reports for patients in medical and surgical ICUs (16-20). The results from our study are similar to that reported by Bochicchio et al of 18 UTIs / 1000 catheter-days in 1,172 trauma patients admitted to the ICU (21). Bochiccio s results in addition to the data reported in our study supports that trauma patients have a higher incidence for UTIs compared to the general surgery patient or the medical ICU patient (2, 3). Despite the higher incidence of UTIs in the trauma population, the risk factors to trigger a diagnostic workup of UTI remain illusive. 25

26 The independent risk factors for UTI in our study included age, female gender, lower ISS, and no antibiotic use the day prior to obtaining the UCx. Age and female gender have been confirmed in previous studies (18, 21) to be risk factors for UTIs. The literature contains contradicting data regarding the association of ISS and nosocomial infections including UTIs. A study by Jamulitrat et al. of 222 nosocomial infections in trauma patients treated in the STICU showed ISS as a good predictor of infection (incidence rate ratio = 1.65; 95% CI = ) (22). Another study by Claridge et al demonstrated that ISS was an independent risk factor for the development of infection in critically injured patients(23). However these studies did not evaluate UTIs independently and included all nosocomial infections. Our present study demonstrated increasing ISS was protective for the development of UTIs. A retrospective study by Hurr et al on 113 trauma patients concluded that ISS and APACHE II scores were not good predictors of nosocomial infections in trauma patients (13). Therefore the data presented in our study indicating the inverse association between ISS and UTI should be interpreted with caution. One interpretation may be that patients with higher ISS developed other infections besides urinary tract infections. Current antibiotic use and the diagnosis of UTIs has not been previously investigated. Our data showed that antibiotics the day prior to obtaining a UCx was protective for the diagnosis of UTI. This is likely as a result of the antibiotics initializing a sterilization of the urine. The limitations of this study include its retrospective design within a single institution. This retrospective design made it difficult to evaluate the exact rationale for obtaining UCxs. In patients without fever or leukocytosis there was likely a high clinical suspicion of infection that could have been based on clinical deterioration, the appearance 26

27 of the urine, or the presence of a fever the day prior to obtaining the UCx. Also, the low number of defined UTIs decreases the power of our logistic regression analysis. The diagnosis of infections in the ICU is a very complex problem and evaluating for UTIs is only one facet in this workup. However, our study had specific objectives to first examine our practice of obtaining a UCx in critically ill trauma patients over their initial two weeks of admission and then to determine if the criteria which we obtained the UCx is associated with UTIs. The strength of these specific objectives is their ability to dissect a complex problem to a specific patient population, time frame, and nosocomial infection. This study also alerted our group to several quality improvement initiatives which have begun in our STICU. Our results revealed that a urinalysis (UA) was not routinely obtained prior to each UCx. A total of 168 UCxs (41%) had a combined UA obtained within 24 hours of the UCx yielding a negative predictive value of 90% and a positive predictive value of 35%. Our institution has since initiated a UA with reflex UCx standing order whenever a UCx alone has been ordered. In addition, our STICU does not have a standard operating procedure for obtaining patient body temperatures and utilizes the oral, axillary, and rectal techniques. We have realized the limitations of these methods and have begun investigating other modalities such as temperature probes within indwelling urinary catheters. We conclude that our practice for obtaining UCxs in STICU trauma patients during their initial 14 hospital-days was associated with fever and the combination of fever and leukocytosis; but neither of these factors were associated with UTIs. The conclusions presented within this study should not be used when trying to extrapolate 27

28 these data results to the critically ill surgical or medical patients, or trauma patients after 14 days of admission. The common practice of performing a fever workup in trauma patients which often includes a UCx should be modified to an infectious workup and employ careful consideration to the laboratory tests ordered in search of the infectious etiologies. Factors such as female gender, advanced age, and antibiotics may have more importance in predicting UTIs than does isolated fever and leukocytosis. Further research will be required to form a model to improve the efficiency of the trauma patient infectious workup. However, at this point we do not recommend obtaining a UCx on traumatically injured patients during their initial first two weeks of hospitalization based on fever and/or leukocytosis alone. 28

29 Table 1: Study population demographics and outcomes Patients (n) 510 Patient-days 3839 Age (years) 49 ± 1 Males (%) 72% Caucasian (%) 73% African-American (%) 20% Injury Severity Score (ISS) 19.4 ± 0.5 Blunt injury (%) 91% ICU length of stay (days) 10 ± 0.5 Length of stay (days) 13 ± 1 Mortality (%) 7% 29

30 Table 2: Number of urine cultures and urinary tract infections Urine cultures (n) 407 Percent of patient-days with a urine culture 11% Urinary tract infections (UTIs) (n) 60 Patients with UTIs (n) 42 Incidence of UTIs (%) 8% Incidence density of UTIs 16 / 1000 indwelling urinary catheter days 30

31 Table 3: STICU daily urinary culture practice based on fever, leukocytosis, and fever + leukocytosis. Also reported is the relative risk of obtaining cultures in patients without fever and leukocytosis Cultured (N=407) Fever ( 38.5º C) 285 / 407 (70%) Leukocytosis ( 12,000 / mm 3 ) 167 / 395* (42%) Fever + Leukocytosis 108 / 395* (27%) NO Fever + NO Leukocytosis 56 / 395* (14%) Not Cultured (N=3432) 832 / 3432 (24%) 1246 / 3209* (39%) 332 / 3209 * (10%) 1482 / 3209* (46%) RR of Obtaining Cultures 95% Confidence Interval p-value < < <0.001 * The number of number of patient-days in the leukocytosis groups is lower because a leukocyte count was not obtained every day. RR = Relative Risk

32 Table 4: STICU urinary culture results based on fever, leukocytosis, and fever + leukocytosis. Also reported is Positive Cultures (N=60) Fever ( 38.5º C) 41 / 60 (68%) Leukocytosis ( 12,000 / mm 3 ) 19 / 57* (33%) Fever + Leukocytosis 12 / 57* (21%) NO Fever + NO Leukocytosis 11 / 57* (19%) Negative Cultures (N=347) 244 / 347 (70%) 148 / 338* (44%) 96 / 338* (28%) 45 / 338* (13%) RR of UTI 95% Confidence Interval p-value * The total number of positive and negative cultures in the leukocytosis groups are lower since a leukocyte count was not obtained on every patient each day. RR = Relative Risk the relative risk of having a positive urine culture without the presence of fever and leukocytosis. 32

33 Figure 1: The percent of patients with fever, leukocytosis and the combination of fever and leukocytosis by day in the STICU. Patients with FEVER, LEUKOCYTOSIS and FEVER+LEUKOCYTOSIS by Day 100% 90% Percent of patients remaining in STICU 80% 70% 60% 50% 40% 30% 20% 10% LEUKOCYTOSIS ( 12,000 /mm 3 ) FEVER ( 38.5ºC) FEVER+LEUKOCYTOSIS 0% STICU Day Remaining patients

34 Figure 2: The percent of UTIs by temperature range. Percentage of Positive Urine Cultures by Temperature Range with Regression Line 100 Percent of positive urine cultures R = Temperature range (C) Positive cultures Total cultures < >

35 Figure 3: The percent of UTIs by leukocyte count range. Percent of Positive Urine Cultures by WBC Range with Regression Line 100 Percent of positive urine cultures R = WBC Range ( * 10 3 /mm 3 ) Positive cultures Total cultures < >

36 Bibliography: 1. Jarvis WR. Selected aspects of the socioeconomic impact of nosocomial infections: morbidity, mortality, cost, and prevention.[see comment]. Infection Control & Hospital Epidemiology. 1996;17(8): Craven DE, Kunches LM, Lichtenberg DA, Kollisch NR, Barry MA, Heeren TC, et al. Nosocomial infection and fatality in medical and surgical intensive care unit patients. Archives of Internal Medicine. 1988;148(5): Wallace WC, Cinat M, Gornick WB, Lekawa ME, Wilson SE. Nosocomial infections in the surgical intensive care unit: a difference between trauma and surgical patients. The American surgeon Oct;65(10): Hoover L, Bochicchio GV, Napolitano LM, Joshi M, Bochicchio K, Meyer W, et al. Systemic inflammatory response syndrome and nosocomial infection in trauma. Journal of Trauma-Injury Infection & Critical Care. 2006;61(2):310-6; discussion Papia G, McLellan BA, El-Helou P, Louie M, Rachlis A, Szalai JP, et al. Infection in hospitalized trauma patients: incidence, risk factors, and complications. The Journal of trauma Nov;47(5): O'Grady NP, Barie PS, Bartlett J, Bleck T, Garvey G, Jacobi J, et al. Practice parameters for evaluating new fever in critically ill adult patients. Task Force of the American College of Critical Care Medicine of the Society of Critical Care Medicine in collaboration with the Infectious Disease Society of America. Critical care medicine Feb;26(2): Freischlag J, Busuttil RW. The value of postoperative fever evaluation. Surgery Aug;94(2): Marino PL. The ICU Book. Second ed: Lippicott Williama & Wilkins; Schey D, Salom EM, Papadia A, Penalver M. Extensive fever workup produces low yield in determining infectious etiology. American Journal of Obstetrics & Gynecology. 2005;192(5): Sedor J, Mulholland SG. Hospital-acquired urinary tract infections associated with the indwelling catheter. The Urologic clinics of North America Nov;26(4): CDC. CDC Definitions of Nosocomial infections. [cited; Available from: Bochicchio GV, Joshi M, Knorr KM, Scalea TM. Impact of nosocomial infections in trauma: does age make a difference? The Journal of trauma Apr;50(4):612-7; discussion Hurr H, Hawley HB, Czachor JS, Markert RJ, McCarthy MC. APACHE II and ISS scores as predictors of nosocomial infections in trauma patients. American Journal of Infection Control. 1999;27(2): Montalvo JA, Acosta JA, Rodriguez P, Hatzigeorgiou C, Gonzalez B, Calderin AR. Factors associated with mortality in critically injured trauma patients who require simultaneous cultures. Surgical Infections. 2006;7(2):

37 15. Marik PE. Fever in the ICU.[see comment]. Chest. 2000;117(3): Laupland KB, Bagshaw SM, Gregson DB, Kirkpatrick AW, Ross T, Church DL. Intensive care unit-acquired urinary tract infections in a regional critical care system. Critical care (London, England) Apr;9(2):R Misset B, Timsit JF, Dumay MF, Garrouste M, Chalfine A, Flouriot I, et al. A continuous quality-improvement program reduces nosocomial infection rates in the ICU. Intensive care medicine Mar;30(3): van der Kooi TI, de Boer AS, Mannien J, Wille JC, Beaumont MT, Mooi BW, et al. Incidence and risk factors of device-associated infections and associated mortality at the intensive care in the Dutch surveillance system. Intensive care medicine Feb;33(2): Wagenlehner FM, Loibl E, Vogel H, Naber KG. Incidence of nosocomial urinary tract infections on a surgical intensive care unit and implications for management. International journal of antimicrobial agents Aug;28 Suppl 1:S Zolldann D, Spitzer C, Hafner H, Waitschies B, Klein W, Sohr D, et al. Surveillance of nosocomial infections in a neurologic intensive care unit. Infect Control Hosp Epidemiol Aug;26(8): Bochicchio GV, Joshi M, Shih D, Bochicchio K, Tracy K, Scalea TM. Reclassification of urinary tract infections in critically ill trauma patients: a timedependent analysis. Surgical Infections. 2003;4(4): Jamulitrat S, Narong MN, Thongpiyapoom S. Trauma severity scoring systems as predictors of nosocomial infection. Infection Control & Hospital Epidemiology. 2002;23(5): Claridge JA, Crabtree TD, Pelletier SJ, Butler K, Sawyer RG, Young JS. Persistent occult hypoperfusion is associated with a significant increase in infection rate and mortality in major trauma patients. Journal of Trauma-Injury Infection & Critical Care. 2000;48(1):8-14; discussion

38 Chapter 3. FEVER AND LEUKOCYTOSIS IN CRITICALLY ILL TRAUMA PATIENTS: IT S NOT THE BLOOD 38

39 FEVER AND LEUKOCYTOSIS IN CRITICALLY ILL TRAUMA PATIENTS: IT S NOT THE BLOOD Running Title: Bacteremia Workup in Trauma Jeffrey A. Claridge, M.D.* (jclaridge@metrohealth.org) Joseph F. Golob Jr., M.D.* (jgolob@metroheatlh.org) Adam M.A. Fadlalla, PhD (a.fadlalla@csuohio.edu) Mark A. Malangoni, M.D.* (mmalangoni@metrohealth.org) Jeffrey Blatnik, M.D.* (jeff.blatnik@yahoo.com) Charles J. Yowler, M.D.* (cyowler@metrohealth.org) * Case Western Reserve University School of Medicine, MetroHealth Medical Center Cleveland, OH Cleveland State University Cleveland, OH Corresponding Author and Reprints Jeffrey A. Claridge, M.D. MetroHealth Medical Center Room H939, Hamann Bldg 2500 MetroHealth Drive Cleveland, OH Phone: Fax: jclaridge@metrohealth.org JA Claridge is supported by the National Institutes of Health, National Institute of Child Health and Human Development, Multidisciplinary Clinical Research Career Development Programs Grant 1KL2RR

40 ABSTRACT BACKGROUND: The diagnosis of bacteremia in critically injured patients is difficult and is often based on fever and/or leukocytosis. This inefficient practice is termed the fever workup. The objectives of this study were to determine 1) if our intensive care unit obtains blood cultures based on fever and/or leukocytosis over the initial 14 days of hospitalization after trauma and 2) the efficacy of this diagnostic workup for diagnosing bacteremia. METHODS: An 18-month retrospective cohort analysis was performed on consecutive trauma patients admitted for 2 days to a level I trauma center. Data collected included demographics, injuries, and the first 14 days maximal daily temperature and leukocyte count. In addition, the results of all blood culture episodes (defined as obtaining central venous or peripheral blood cultures) and central venous catheter tip semi-quantitative cultures were recorded. Fever was defined as a maximum daily temperature of 38.5ºC and leukocytosis as a leukocyte count 12,000/mm 3 of blood. All positive blood cultures were individually evaluated for the source of bacteremia and classified as either true bacteremias or contaminants. RESULTS: Five hundred and ten patients were evaluated for a total of 3839 patientdays. The mean age and injury severity score (ISS) were 49 ± 1 years and 19 ± 1, respectively. Seventy-two percent were males and 91% had blunt injuries. Four hundred twenty-five blood culture episodes were obtained and 25 (6%) bacteremias were identified in 23 patients (5%). A significant association was found between obtaining blood cultures in patients with fever (RR=7.7; 95%CI ), leukocytosis (RR=1.3; 95%CI= ) and fever+leukocytosis (RR=3.2; 95%CI= ). However, no significant association was found between these clinical signs and the diagnosis of 40

41 bacteremia. Logistic regression analysis using gender, age, mechanism, ISS, hospital day, and antibiotic use failed to identify independent risk factors that predicted bacteremia on patient days that had blood cultures. In fact fever alone was inversely associated with bacteremia. CONCLUSION: Our intensive care unit follows the common fever workup practice and obtains blood cultures based on the presence of fever and leukocytosis. However, fever and leukocytosis were not significantly associated with bacteremia, suggesting inefficiency in this practice. Further prospective study will be required to identify whether other parameters are more predictive as to when blood cultures in critically ill trauma patients should be obtained. 41

42 INTRODUCTION Infections in the intensive care unit are extremely common and result in significant patient morbidity and mortality. Data have demonstrated that surgical intensive care units have more infectious complications than medical intensive care units [1, 2]. Within the surgical intensive care unit, trauma patients have more infections than general surgical patients [1-3]. Despite the increased incidence of infections in trauma patients, these complications are often difficult to identify and clinicians frequently rely on fever and leukocytosis as workup triggers. In 1998, the Society of Critical Care Medicine in collaboration with the Infectious Disease Society of America concluded that because fever can have many infectious and non-infectious etiologies, a new fever in a patient in the intensive care unit should trigger a careful clinical assessment rather than automatic orders for laboratory and radiographic tests [4]. Differentiating between infectious and non-infectious causes of fever and leukocytosis can be especially difficult in trauma patients as these signs are often present as part of the systemic inflammatory response syndrome (SIRS) due to injury [5]. Unfortunately, the appropriate clinical parameters that should initiate an efficient infection workup compared to the current fever workup in trauma patients remain elusive. Aerobic and anaerobic blood cultures are common microbiologic tests obtained as part of the fever workup. The overall diagnostic yield of blood cultures has been shown to be poor, and studies have demonstrated various conflicting risk factors associated with bacteremia [6-8]. Previous research has questioned using temperature and leukocyte count as triggers for obtaining blood cultures in the surgical and burn 42

43 patient populations [9, 10]. The concern of missing a bacteremia further complicates matters and likely influences the workup at any sign including fever. The purpose of this study was to evaluate our blood culture practice as it relates to fever and leukocytosis, and to evaluate if fever and/or leukocytosis are predictors of positive blood cultures in a critically ill trauma population during the first two weeks in the surgical and trauma intensive care unit (STICU). We hypothesized that our STICU practice is to obtain blood cultures in the face of fever and leukocytosis, but neither of these clinical signs would be associated with identifying a bacteremia in critically ill trauma patients. METHODS We performed a retrospective review of critically ill trauma patients admitted to a level I trauma center STICU over an 18 month period. Patients were included if they were older than 18 years, spent at least two days in the STICU, and were not admitted to the regular nursing floor or another hospital for more than 24 hours before being admitted to our STICU. Data collected included demographics, injuries, daily maximal temperature, daily maximal leukocyte count, and blood culture episodes when obtained. A blood culture episode included blood culture bottles from a sterile peripheral site or from a central venous catheter utilizing BD BACTEC Standard Anaerobic/F and BD BACTEC Plus Aerobic/F bottles (Becton, Dickenson, and Company: Sparks, MD). Blood cultures were obtained by critical care trained registered nurses who used aseptic technique with the protocol to instill 5-7 ml in the anaerobic bottle and 8-10 ml in the aerobic bottle. In addition, central venous catheter tip quantitative culture results were collected if they were obtained.. 43

44 Fever was defined as a daily maximal temperature 38.5ºC and leukocytosis as a leukocyte count 12,000/mm 3. Bacteremia was identified by a positive blood culture with an organism determined not to be a contaminant. Contaminating organisms included diptheroid species, Propionibacterium acnes, micrococci, or coagulase negative staphylococcus in a single bottle or multiple bottles from the same site without evidence of a source such as a positive central venous catheter tip. Catheter tips were considered positive if a semi-quantitative rolling culture technique yielded 15 colony forming units (CFUs) of a single bacterium. Each patient with a bacteremia was then individually investigated for a source of the infecting organism. SPSS version 15 (SPSS Inc; Chicago, IL) was used for all statistical analysis. All categorical data is expressed as percentages and compared with Pearson Chi-square or Fisher s Exact Test where appropriate. Non-categorical data is expressed as mean ± standard error of the mean and compared with Student s t-test. Statistical significance was set at a p-value This study was reviewed and approved by the MetroHealth Institutional Review Board. RESULTS This 18 month review identified 510 critically ill trauma patients who met inclusion criteria. The mean age of this trauma patient population was 49 ± 1 years and 72% were males. The mean injury severity score (ISS) was 19.4 ± 0.5. A total of 3839 patient-days of data were evaluated. A central venous catheter was in place for 47% of the patient-days. Table 1 summarizes patient demographics. The average STICU length of stay was 10 ± 0.5 days and a mean hospital length of stay was 13 ± 1 days. Thirty-six patients (7%) died while in the STICU. 44

45 There were 425 blood culture episodes in 202 (40%) patients. Figure 1 demonstrates the distribution of blood cultures episodes by day. These episodes yielded 1253 blood culture bottles: 1185 were from a peripheral site and 68 from a central venous catheter. Fifty-three central venous catheter tips were cultured during this study period. A total of 25 bacteremias were identified in 23 patients. The most common etiology of the bacteremia was found to originate from the respiratory system. Two patients had a positive central venous catheter tip in addition to positive blood cultures with the same organism and were diagnosed with a catheter related blood stream infection (CR-BSI). This yielded a CR-BSI rate of 1.1 per 1000 central-line days. The most common bacterium isolated in patients with a bacteremia was methicillin-sensitive Staphylococcus aureus. Table 2 summarizes the blood culture results. Defining our STICU practice for obtaining blood cultures To initiate our hypothesis testing, we first investigated our STICU practice for obtaining blood cultures. The bivariate analysis of patient days that were cultured versus days that were not cultured is shown on table 3. Specifically, we evaluated if there was an association between fever and/or leukocytosis and obtaining blood cultures in critically injured patients. The relative risk of fever triggering a blood culture was statistically significant at 7.2 (95% CI = ). Likewise, both leukocytosis (RR=1.3; 95%CI= ) and the combination of fever with leukocytosis (RR=3.2; 95%CI= ) were associated with obtaining a blood culture (table 4). The absence of fever and leukocytosis was associated with a significantly lower risk of acquiring a blood culture in critically ill trauma patients. When temperature and leukocyte count were investigated as continuous variables, there was a similar association. The mean maximal body 45

46 temperature of patients who were cultured was significantly higher at 38.8 ± 0.03ºC vs ± 0.1ºC in patients who were not cultured (p=0.006). Likewise the daily leukocyte count in patients who were cultured was significantly higher compared to patients not cultured (cultured=12.2 x 10 3 ± 0.25 vs. not cultured=11.6 x 10 3 ± 0.09; p=0.017). The efficacy of our blood culture practice Next we determined if fever and leukocytosis was associated with bacteremia (table 5). Fever, leukocytosis, and their combination were not associated with bacteremia on patient days when blood cultures were obtained. In fact, an evaluation of patient days without leukocytosis demonstrated the fever alone was inversely associated with bacteremia. On patient days with no leukocytosis and bacteremia, 45% did not have a fever compared to only 23% that were febrile (p = 0.04). Thus, the relative risk of bacteremia in patients fever only (no leukocytosis) was 0.39 (CI = ). The mean temperature on the day of obtaining the blood culture in patients with bacteremia was 38.7 ± 0.4ºC compared to 38.8 ± 0.03ºC in those with a negative blood culture, no significant difference. Likewise, the mean leukocyte count in patients who were diagnosed with bacteremia was 13.0 x 10 3 ± 0.5 compared to 12.3 x 10 3 ± 1.0 in patients with a negative blood culture, no significant difference. Figures 2 and 3 demonstrate that there were no temperature or leukocyte count ranges associated with positive blood cultures and/or positive central venous catheter tip cultures. Logistic regression analysis using gender, age, ISS, hospital day, and antibiotic use failed to identify independent risk factors that predicted bacteremia on patient days that blood cultures were obtained. Bacteremic patients had longer STICU length of stay and hospital length of stay as well as more ventilator-days, central-venous catheter days, and antibiotic-days (table 46

47 6). Twenty-two percent of the critically ill trauma patients who had bacteremia expired while in the STICU compared to a six percent mortality among patients without a bacteremia (p=0.016). DISCUSSION The results of this study demonstrate that our STICU practice for obtaining blood cultures in critically ill trauma patients is associated with fever and leukocytosis, but neither of these two clinical signs were associated with patients having bacteremia on days when cultures were obtained. Also, we were unable to identify a range of increased temperature or leukocyte counts associated with bacteremias in this critically ill trauma patient population. Fever and leukocytosis are part of the SIRS response, common in trauma patients with or without an infectious process. Previous data has shown that 92% of seriously injured patients will exhibit a SIRS response during their first hospitalized week and 69% during their second week [1]. Differentiating between infectious and non-infectious causes of this SIRS response is difficult, and often involves obtaining blood cultures as part of the workup. In a large prospective study of more than 11,000 critically ill patients, 9% were suspected of having clinical sepsis. A microbiological infection was documented in 71% of these patients, and bacteremia was identified in 53% [11]. Such a low overall incidence of bacteremia in a group of patients with a high clinical suspicion for infection suggests that an even lower incidence of bacteremia may be present in patients with only fever and/or leukocytosis. The overall diagnostic yield of blood cultures has previously been shown to be poor [6-8]. In our study, the 1253 blood culture bottles only identified 23 patients with 47

48 bacteremia. Blood cultures are considered the gold-standard for diagnosing bacteremia, therefore calculating a sensitivity of this test is not feasible. However, many have demonstrated that blood cultures are often falsely positive secondary to contamination and have a positive predictive value of approximately 50% [6, 8]. Conflicting data exist regarding concurrent antibiotic use affecting blood culture results and the utility of repeat blood cultures [6, 12]. Our data demonstrated that there was no association between positive blood cultures and antibiotics on the day of, or day prior to obtaining a culture. Several researchers have questioned the use of anaerobic blood cultures in routine screening of febrile patients, but these blood culture bottles are still widely used [6, 13, 14]. The single best method to improve the diagnostic utility of blood cultures is not related to individual patient characteristics at all, but rather increasing the blood volume in each blood culture bottle acquired [8, 13, 15]. Because of the risk of bacteremia in trauma patients, and the high overall mortality associated with bacteremia progressing to sepsis, there is generally a very low threshold to send blood cultures. In our study, bacteremia was associated with a significantly higher mortality rate of 22% further demonstrating the concern of this clinical condition. Others have attempted to identify predictive factors associated with positive blood cultures without much success. In 1991, Theuer et al concluded that neither the magnitude of the absolute leukocyte count nor the maximum temperature at the time of phlebotomy predicted a positive blood culture in perioperative patients [15]. Swenson et al. concluded that among surgical patients with bloodstream infections, fever is sensitive for bacteremia, but it is not specific and the common temperature threshold of 38.5ºC was questioned [10]. Murray et al. reporting on critically ill burn patients 48

49 concluded that neither fever, leukocytosis, nor the neutrophil percentage is associated with positive blood cultures [9]. Our data suggests that critically ill trauma patients are another discrete group that does not demonstrate a correlation between fever and leukocytosis with bacteremia. There are several limitations to our study. The database does not contain all risk factors previously shown to be associated with bacteremia (e.g. hypotension and need for hemodynamic support), which limited our ability to suggest when blood cultures should be obtained. In order to best identify risk factors for bacteremia, blood cultures may need to be taken more often, and even in patients not suspected to have an infection. In addition to the problems associated with retrospective studies, our STICU does not have a standard operating procedure for obtaining patient body temperatures, and utilizes oral, axillary, and rectal techniques. Furthermore, there was no quantification of the amount of blood obtained per culture bottle although nursing is encouraged to follow protocol. This study only identified 25 bacteremias in 23 patients. Such a low number of infections introduces type-ii error and limited our ability to perform analyses such as logistic regression to determine potential risk factors for bacteremia in trauma patients. It also must be noted that we only analyzed the risk of bacteremia on patient days that had a culture obtained. This limits the study population and the results would change if you made the assumption that patient days that were not cultured were non-bacteremia days. The possibility of a type II error in this study does exist, especially with leukocytosis. However, this data suggests that fever and leukocytosis are not strong predictors of bacteremia. 49

50 This study further demonstrates that the blood culture practice as part of fever workup needs to be reevaluated for critically ill trauma patients, especially in the first two weeks after injury. Neither fever nor leukocytosis appears to be associated with bacteremia, but intensivists still use these criteria to trigger blood cultures. Prospective data from an accurate research database will be required to answer when blood cultures should be obtained in critically ill trauma patients. 50

51 Bibliography [1] Hoover L, Bochicchio GV, Napolitano LM, Joshi M, Bochicchio K, Meyer W, et al. Systemic inflammatory response syndrome and nosocomial infection in trauma. Journal of Trauma-Injury Infection & Critical Care. 2006;61(2):310-6; discussion 6-7. [2] Papia G, McLellan BA, El-Helou P, Louie M, Rachlis A, Szalai JP, et al. Infection in hospitalized trauma patients: incidence, risk factors, and complications. The Journal of trauma Nov;47(5): [3] National Nosocomial Infections Surveillance (NNIS) System Report, data summary from January 1992 through June 2004, issued October Am J Infect Control Dec;32(8): [4] O'Grady NP, Barie PS, Bartlett J, Bleck T, Garvey G, Jacobi J, et al. Practice parameters for evaluating new fever in critically ill adult patients. Task Force of the American College of Critical Care Medicine of the Society of Critical Care Medicine in collaboration with the Infectious Disease Society of America. Critical care medicine Feb;26(2): [5] Miller PR, Munn DD, Meredith JW, Chang MC. Systemic inflammatory response syndrome in the trauma intensive care unit: who is infected? Journal of Trauma-Injury Infection & Critical Care. 1999;47(6): [6] Darby JM, Linden P, Pasculle W, Saul M. Utilization and diagnostic yield of blood cultures in a surgical intensive care unit. Critical Care Medicine. 1997;25(6):

52 [7] Henke PK, Polk HC, Jr. Efficacy of blood cultures in the critically ill surgical patient. Surgery. 1996;120(4):752-8; discussion 8-9. [8] Shafazand S, Weinacker AB. Blood cultures in the critical care unit: improving utilization and yield.[see comment]. Chest. 2002;122(5): [9] Murray CK, Hoffmaster RM, Schmit DR, Hospenthal DR, Ward JA, Cancio LC, et al. Evaluation of White Blood Cell Count, Neutrophil Percentage, Elevated Temperature as Predictors of Bloodstream Infection in Burn Patients. Archives of Surgery. 2007;142(7): [10] Swenson BR, Hedrick TL, Popovsky K, Pruett TL, Sawyer RG. Is fever protective in surgical patients with bloodstream infection? Journal of the American College of Surgeons. 2007;204(5):815-21; discussion [11] Brun-Buisson C, Doyon F, Carlet J, Dellamonica P, Gouin F, Lepoutre A, et al. Incidence, risk factors, and outcome of severe sepsis and septic shock in adults. A multicenter prospective study in intensive care units. French ICU Group for Severe Sepsis. JAMA. 1995;274(12): [12] Schermer CR, Sanchez DP, Qualls CR, Demarest GB, Albrecht RM, Fry DE. Blood culturing practices in a trauma intensive care unit: does concurrent antibiotic use make a difference? Journal of Trauma-Injury Infection & Critical Care. 2002;52(3): [13] James PA, al-shafi KM. Clinical value of anaerobic blood culture: a retrospective analysis of positive patient episodes. Journal of Clinical Pathology. 2000;53(3): [14] Ortiz E, Sande MA. Routine use of anaerobic blood cultures: are they still indicated?[see comment]. American Journal of Medicine. 2000;108(6):

53 [15] Theuer CP, Bongard FS, Klein SR. Are blood cultures effective in the evaluation of fever in perioperative patients? American Journal of Surgery. 1991;162(6):

54 Number of Patients 510 Patient-Days 3839 Age 49 ± 1 years Males (%) 367 (72%) Caucasian (%) 372 (73%) African-American (%) 102 (20%) ISS 19 ± 0.5 Blunt Mechanism (%) 464 (91%) Table 1: Demographics ISS = Injury Severity Score 54

55 Blood Cultures Obtained and True Bacteremias Identified Number of Blood Culture Episodes 425 Number of Cultured Patients 202 (40%) Total Blood Culture Bottles 1253 From Peripheral Site 1185 From Central Venous Catheter 68 Central Venous Catheter Tip Cultures 53 Bacteremias Identified 25 Patients with a Bacteremia 23 Etiology of True Bacteremias Respiratory System 7 CR-BSI 2 Urinary System 1 Central Nervous System 1 No Identified Source 14 Bacteria Isolated MSSA 9 MRSA 3 Polymicrobial 3 Table 2: The number of blood cultures and bacteremias identified along with the etiology and species of bacteria isolated. Others included Streptococcus pneumoniae, Bacteroides fragilis, Klebsiella pneumoniae, Enterococcus fecalis, Bacteroides oris, Pseudomonas aeuroginosa, alpha strep, Aeromonas hydrophila, Enterobacter cloacae, and Acinetobacter species CR-BSI: Catheter related blood stream infection MSSA: Methicillin sensitive Staphylococcus aureus MRSA: Methicillin resistant Staphylococcus aureus 55

56 Not Cultured Cultured p value Age (years) <0.001 ISS <0.001 Male (%) <0.001 Bunt mechanism (%) On Antibiotics day of culture (%) <0.001 Antibiotics the day before (%) Presence of Central Line (%) <0.001 Fever (%) <0.001 Leukocytosis (%) Fever and Leukocytosis (%) <0.001 Fever or Leukocytosis (%) <0.001 Leukocyte count (x 10 3 cells/cc) Temperature (º Celsius) Fever only (no leukocytosis) (%) <0.001 Leukocytosis only (no fever) (%) <0.001 Table 3. Bivariate analysis of our blood culture practice based on 3839 days with 425 culture episodes. Numerical data expressed as means. 56

57 Relative Risk of Obtaining a Blood Culture 95% Confidence Interval p-value Fever <0.001 Leukocytosis Fever and <0.001 Leukocytosis Fever or Leukocytosis <0.001 No Fever and No Leukocytosis <0.001 Table 4: The relative risks (unadjusted) and 95% confidence intervals of fever and leukocytosis triggering blood cultures in our STICU trauma patients 57

58 Relative Risk of 95% Confidence Bacteremia Interval p-value Fever Leukocytosis Fever and Leukocytosis Fever or Leukocytosis No Fever and No Leukocytosis Table 5: The relative risk (unadjusted) and 95% confidence intervals of fever and leukocytosis being associated with bacteremia on patient days that cultures were obtained. 58

59 Bacteremic Patients Non-Bacteremic p-value (n=23) Patients (n=487) STICU length of 21.6 ± ± 0.5 <0.001 stay (days) Hospital length of 26.1 ± ± 0.5 <0.001 stay (days) Ventilator-days 9.0 ± ± 0.2 <0.001 Central line-days 5.9 ± ± 0.2 <0.001 Antibiotic-days 6.9 ± ± 0.2 <0.001 Mortality 22% 6% Table 6: Outcome comparison of patients with and without bacteremia. Values expressed as means ± standard error of the mean 59

60 Figure 1. Blood culture episodes by day Number of culture episodes Percent of total culture episodes STICU DAY 60

61 Figure 2: Percent of positive blood and central venous catheter tip cultures compared to leukocyte count Percent Positive Blood Cultures 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% r = Leukocyte Count Range (10^3) Positive Cultures Negative Cultures

62 100% Figure 3: Percent of positive blood and central venous catheter tip cultures compared to body temperature 90% Percent Positive Blood Cultures 80% 70% 60% 50% 40% 30% 20% 10% r = % Temperature Range (C) Positve Cultures 37º 37.1º-37.5º 37.6º-38.0º 38.1º-38.5º 38.6º-39.0º 39.1º-39.5º 39.6º All Cultures

63 Chapter 4. THE FEVER WORKUP AND RESPIRATORY CULTURE PRACTICE IN CRITICALLY ILL TRAUMA PATIENTS 63

64 THE FEVER WORKUP AND RESPIRATORY CULTURE PRACTICE IN CRITICALLY ILL TRAUMA PATIENTS Jeffrey A. Claridge, MD*, Mark J. Sando BS*, Joseph F. Golob MD*, Adam M.A. Fadlalla, Ph. D**, Joel R. Peerless, MD*, and Charles J. Yowler MD* *MetroHealth Medical Center, Department of Surgery: Division of Trauma, Critical Care, Burns, and Life Flight. Case Western Reserve University School of Medicine, Cleveland, OH ** Cleveland State University Department of Information Science Acknowledgement JA Claridge is supported by the National Institutes of Health, National Institute of Child Health and Human Development, Multidisciplinary Clinical Research Career Development Programs Grant 1 KL2RR Corresponding Author: Jeffrey A. Claridge, M.D. MetroHealth Medical Center Hamann Building Room H939 Cleveland, OH Phone: Fax: jclaridge@metrohealth.org 64

65 ABSTRACT INTRODUCTION: Due to the morbidity of infections, fever and leukocytosis(f&l) in critically ill patients often trigger a "workup" including a respiratory secretion culture(rcx). The rationale of this practice has not been thoroughly evaluated after trauma. HYPOTHESIS: F&L would be associated with obtaining a RCx, but would not be associated with a positive culture or treating a respiratory infection in critically injured patients during the first 14 intensive care unit (ICU) days. METHODS: An 18-month retrospective analysis was performed on consecutive ICU trauma patients admitted for 2 days to a level I trauma center. Data collected included demographics, injuries, RCxs(bronchoalveolar lavage or tracheal aspirate), and the presence of fever ( 38.5 C) and leukocytosis( 12,000/mm 3 ) over the first 14 ICU days. A diagnosis of respiratory infection was defined by the clinicians' decision for a 7 day course of antibiotics. RESULTS: A total of 510 patients with a mean age of 49 and injury severity score of 19 were evaluated for a total of 3839 patient-days. 211 patients had 489 RCxs obtained (2.4RCxs/patient); 94 (19%) were obtained on consecutive days. Obtaining a Rcx was associated with fever (RR=4.8 95CI= ) and the combination of F&L (RR=2.6 95%CI= ), but not leukocytosis alone. Neither fever, leukocytosis, nor F&L predicted a positive RCx. 128 patients were treated for a respiratory infection. Treatment 65

66 of respiratory infections was contrary to the RCx results 24% of the time. The sensitivity and specificity of a positive RCx being associated with respiratory infection was 97% and 46%, respectively. CONCLUSIONS: F&L were associated with the decision to obtain RCxs, but neither were associated with positive RCxs. RCx results had a low specificity and did not consistently impact treatment decisions. Factors other than F&L alone should influence the decision to obtain RCxs during the first 14 days in the ICU after trauma. 66

67 INTRODUCTION Diagnosing and treating infections in critically injured patients is complex. The total cost of workup and treatment for nosocomial infections alone has been estimated to be approximately 4.5 billion dollars annually [1]. Previous studies have shown that infection rates are higher in surgical and trauma intensive care units (STICU) compared to medical intensive care units with 37%-45% of trauma patients experiencing an infectious complication [2, 3]. Infections in the critically injured patient are often difficult to diagnose and their evaluation and treatment may lead to unnecessary tests, procedures, and increased medical costs [4]. Evaluation for infections in these patients is often triggered by the presence of fever and leukocytosis leading to a broad infection workup including urine, blood, and respiratory secretion cultures, as well as chest radiographs. The rationale for such workup has not been fully evaluated in trauma patients. Fever and leukocytosis are part of the systemic inflammatory response syndrome (SIRS) and can be elicited by both infectious and non-infectious causes. The differentiation between infectious vs. noninfectious causes of SIRS is difficult as 92% of critically injured patients have a SIRS response during their first week in the STICU and 69% demonstrate SIRS during the second week [2]. Previous studies of critically ill ICU patients have shown that the diagnostic studies obtained in the evaluation of fever are often inefficient [5, 6]. While classically associated with nosocomial infections, fever and leukocytosis lack diagnostic usefulness in predicting these ICU infections [7]. Current recommendations, specifically for critically ill ICU patients, state that careful clinical assessment should be employed rather 67

68 than automatic laboratory orders [8-10]. Despite this evidence, it appears that fever and leukocytosis often triggers an extensive workup for infection in the STICU. Our previous work has shown that fever, leukocytosis, and the combination of fever with leukocytosis are not associated with the development of urinary tract infections or bacteremias in critically ill trauma patients during their initial 14 days of hospitalization [11, 12]. To further dissect the fever workup we felt it necessary to examine our respiratory culture practice, as well as the association of fever and leukocytosis with the actual development of a respiratory infection. Lower respiratory tract infections are the most common nosocomial infection in critically injured patients accounting for 22% to 47% of all infections [2, 13]. Even 10 years ago it was estimated that respiratory tract infections will increase the duration of hospitalization by 6.8 to 30 days with an average excess cost of approximately $5,000 per infection [4]. The diagnostic yield of respiratory cultures in trauma patients has also been questioned [14-16] leading to the need for further assessment into understanding when to order these cultures as well as how to interpret their results. The first aim of this project was to define our current practice of obtaining respiratory cultures based on fever and leukocytosis in STICU patients over the initial 14 days of admission. The second objective was to evaluate the results of our respiratory culture practice by assessing the correlation between fever and leukocytosis and the development of a positive respiratory culture. It is our hypothesis that the fever and leukocytosis will be associated with obtaining respiratory cultures however neither of these clinical parameters will be associated with positive RCx results. 68

69 DESIGN AND METHODS A retrospective review was conducted on all injured trauma patients admitted to the STICU at a level I urban trauma center over 18 months (January 2004 through June 2005). Patients were excluded if they were admitted to an outside hospital for greater than 24 hours prior to transfer, were admitted to a non-icu floor before STICU admission, or had an initial STICU length of stay less than 48 hours. Data were collected from the patient s chart, electronic laboratory information system (Epic Systems Corporation: Madison, WI) and our trauma registry (dbase: Vestal, NY). Data collected included patient demographics and injuries as well as daily maximal temperature, leukocyte count, and antibiotic therapy for the initial 14 hospital days or until the patient was discharged from the STICU. Fever was defined as a maximum temperature of 38.5ºC and leukocytosis as a leukocyte count 12,000 / mm 3 during a 24 hour period. A positive RCx was defined as a qualitative tracheal aspirate culture producing any pathologic isolate which is not normal oral flora, or a bronchoalveolar lavage (BAL) culture producing 10 4 cfu/ml. A respiratory infection was defined as a clinicians decision to treat with at least a seven day course of appropriate antibiotic therapy following treatment guidelines from the American Thoracic Society and Infectious Diseases Society of America. These guidelines recommend a seven to eight day course of antibiotics for patients having a respiratory infection who received initially appropriate therapy and had a good clinical response [17]. 69

70 Data was analyzed using SPSS (SPSS Inc: Chicago, IL) software. A p-value of 0.05 was considered statistically significant. Numeric data is expressed as mean ± standard error of the mean and compared using the student t-test. Categorical data is described as a percentage and compared using either Chi-square or Fisher s exact test as appropriate. Using this data set, sensitivities and specificities were estimated for the association of a positive RCx with being treated for a respiratory infection. This study was reviewed and approved by our institution s Institutional Review Board. RESULTS Study Population Five hundred ten critically ill trauma patients with a mean injury severity score (ISS) of 19.4 ± 0.5 met the study criteria for inclusion in our analysis. The mean age was 49 ± 1 years-old and 72% were males. The population had a total of 3,839 patient-days of data collected. Additional patient demographics and outcomes are summarized in Table 1. A total of 489 RCxs were obtained in 211 patients for an average of 2.4 RCx/patient. Of these cultures, 466 were tracheal aspirates of which 316 (68%) were positive, and 23 were BALs of which 18 (78%) were positive. Ninety-four RCx (19%) were obtained on consecutive days. One hundred twenty-eight patients (25%) were treated for a respiratory infection. To further describe the patient population, the frequencies of fever, leukocytosis, and the combination of fever and leukocytosis were evaluated daily. The percent of patients remaining in the STICU by day who had fever, leukocytosis, and fever with 70

71 leukocytosis is illustrated in Figure 1. Fever and leukocytosis were common in our patient population. Figure 2 shows that 79% of patients staying in the STICU for at least 7 days were febrile and that 80% of them had leukocytosis at some point during the first week. In patients staying 2 weeks in the STICU, 73% were febrile and 85% had leukocytosis at some point during the second week. A combination of fever and leukocytosis was present in 42% of patients during the first week, and 55% of patients remaining during the second week. Defining the Practice of Obtaining Respiratory Cultures Table 2 illustrates the relationship between obtaining RCxs and fever, leukocytosis, and fever with leukocytosis. Fever was associated with obtaining a respiratory culture in our patient population (RR = 4.84). Likewise, fever with leukocytosis was associated with obtaining a respiratory culture (RR = 2.60). No association was observed with leukocytosis alone and obtaining a respiratory culture. These data show that our STICU respiratory culture practice is associated with fever and fever with leukocytosis, but not leukocytosis alone. Also, not having a fever and not having leukocytosis was protective against receiving a respiratory culture (RR = 0.28). The mean temperature for patients who were cultured was significantly higher (p<0.001) than for those who were not cultured (38.6 ± 0.03 vs ± 0.1, respectively). The mean leukocyte count for patients who were cultured vs. those who were not cultured was not significantly different (12.0 ± 0.2 vs ± 0.1, respectively; p=0.085). Defining the Efficacy of Identifying Positive Respiratory Cultures Four hundred eighty-nine respiratory cultures were obtained and 68% were positive. The relationship between the results of RCx and fever, leukocytosis, and the 71

72 combination of fever with leukocytosis is shown in Table 3. There was no association between patients with fever or leukocytosis and having a positive RCx. Therefore fever, leukocytosis, and fever with leukocytosis are not associated with positive RCxs in critically ill trauma patients during their initial 14 days of hospitalization. The mean temperature for patients with a positive culture was not significantly different from patients with a negative RCx (38.7 ± 0.04 vs ± 0.05, respectively; p=0.325). The mean leukocyte count in those with a positive RCx vs. those with a negative RCx was also not significantly different (12.0 ± 0.3 vs ± 0.47, respectively; p=0.704). The practice for obtaining RCxs in critically ill trauma patients was associated with fever and fever with leukocytosis, but neither of these parameters were associated with positive RCx. Thus, we investigated temperature and leukocyte count as continuous variables to determine if there was a range at which positive RCx were more common. There appeared to be no temperature or leukocyte range associated with significantly more positive RCxs as illustrated in Figures 3 and 4, respectively. Our final goal was to evaluate the association of respiratory infections with a positive or negative RCx. We determined that 128 patients were treated for a respiratory infection. Figure 5 illustrates that, 46 patients with a positive RCx were not treated for an infection, and 4 patients with a negative RCx were treated for an infection. In other words, 50 out of 211 patients cultured (24%) had treatment decisions that were contrary to the result of their RCx. Additionally, two patients (0.6%) who did not receive a respiratory culture were treated for a respiratory infection. The sensitivity and specificity of a positive RCx being associated with treating a respiratory infection were 97% and 46%, respectively. 72

73 DISCUSSION This study indicates that our STICU practices the common fever workup and obtains RCxs in response to fever and leukocytosis. The actual definition of a fever and its workup is unclear as many reviews have been contradictory with regard to the exact temperature of fever as well as which tests should be performed in search of an infectious etiology of this fever [6, 9, 10]. The Society of Critical Care Medicine defines fever as > 38.3 C and recommends that in the absence of obvious infectious process only temperatures > 38.3 C warrant further investigation [9, 10]. Previous work has suggested that temperatures of 38.5ºC and 39.0ºC are not predictive of infection, with positive predictive values of 5% and 0%, respectively, for determining infection in postoperative patients [18]. A study by Frank, et al. reported that up to 50% of surgical patients had a maximum body temperature 38.0ºC and 25% had a maximum temperature of 38.5ºC in the initial 24 hour postoperative period [19]. These temperature elevations were similarly associated with extent and duration of surgery as well as increased IL-6 response, suggesting that early postoperative fever is a manifestation of perioperative stress, [19]. Due to the high incidence of SIRS in the trauma population [2, 20]causing fevers of non-infectious origin, we chose to use the traditional postoperative fever definition of 38.5ºC. It has been recognized that fever in the intensive care unit frequently triggers a battery of diagnostic tests that are costly, expose the patient to unnecessary risks, and often produce misleading or inconclusive results [9]. Focused reviews on the subject in critically ill ICU patients have led to recommendations that the presence of fever in the absence of clinically obvious sources of infection warrants blood cultures and 73

74 observation before performing further tests or beginning empiric antibiotics [9, 10]. In regards to assessing for respiratory infections, a consensus report from the Infectious Diseases Soceity of America and the American Thoracic Society recommends a combination of clinical criteria be utilized to evaluate the need for further diagnostic workup. This workup should include lower respiratory tract specimen culture and empiric antibiotic therapy until the results of the cultures are received at which time therapy should be adjusted accordingly [17]. In our analysis, fever and the combination of fever and leukocytosis were associated with obtaining respiratory cultures however neither of these clinical parameters were associated with a positive respiratory culture during the initial 14 days of hospitalization. A meta-analysis of 14 studies of ventilator-associated pneumonia (VAP) found that clinical features such as fever, leukocytosis, purulent pulmonary secretions, and radiographic infiltrates individually are not associated with an increased likelihood of VAP [7]. Similarly, Wunderink suggests that such clinical parameters are appropriate for initial suspicion of infection, but that the lack of specificity suggests the need for additional testing or information [21]. In both studies, combination of these clinical parameters was associated with an increased likelihood of infection, but did not establish the diagnosis of VAP. In our study, fever and leukocytosis were not associated with positive RCx suggesting the absence of respiratory infection. Studies aimed at understanding the diagnostic workup for respiratory infection in critically ill patients have led to varying results. The use of fever and leukocytosis as clinical criteria to trigger further workup may not be beneficial in critically ill trauma patients since non-infectious SIRS is common [2]. Critically ill STICU patients also 74

75 frequently have persistent purulent respiratory secretions and abnormal chest radiographs which are not always representative of respiratory infection [13]. It has been estimated that up to 50% of trauma patients with clinical evidence of pneumonia are undergoing a SIRS response [15]. This makes the diagnosis of respiratory infection in these patients all the more challenging. Use of the Clinical Pulmonary Infection Score (CPIS) has received credit for diagnosing and treating patients with pneumonia, however, in a study by Croce, et al. the CPIS was not able to differentiate ventilator-associated pneumonia from SIRS in critically injured patients [22]. It seems that commonly used clinical indicators for determining evaluation and treatment of respiratory infection are less informative in trauma patients indicating the need for a greater emphasis on independent risk factors and/or diagnostic procedures. The study presented here showed that RCx results had a high sensitivity (97%), but low specificity (46%) for predicting respiratory infection indicating that respiratory infection is less likely with a negative culture however a positive culture is not necessarily predictive of respiratory infection (PPV=72%). This finding is consistent with previous reports which show the sensitivity and specificity of endotracheal aspirate culture to be 90% and 68%, respectively [16]. Positive RCxs were not associated with fever or leukocytosis and the result of the RCx did not consistently impact decisions to treat patients for respiratory infections. In this study 24% of patients who received RCxs had treatment decisions that were contrary to the result of said culture. A study of 116 patients with community acquired pneumonia found that tracheal aspirates in particular did not contribute significantly to patient management with antimicrobial treatment directed to diagnostic results in only one patient [23]. RCxs may be ordered reflexively 75

76 based on fever and fever with leukocytosis however their results do not always impact treatment decisions. This may suggest that not only are RCxs being ordered based on the wrong clinical criteria in trauma patients, but also that they are not being properly utilized or interpreted in this population. The method of obtaining respiratory secretions for culture that is most useful in identifying infection in critically ill patients has been questioned. A multi-center study by the Canadian Critical Care Trials Group concluded that cultures of endotracheal aspirates (ETA) as well as BAL samples are associated with similar mortality rates and antibiotic usage in mechanically ventilated patients [24]. Wood, et al. found moderate culture result agreement between ETA and BAL and suggests that ETA can be used as an equivalent diagnostic technique in VAP as a less invasive, and less costly approach [25]. Current recommendations similarly suggest using tracheal aspirates as a sufficient starting point to prompt further investigation if results are positive [10, 17]. Bonten et al., demonstrated only 58% concordance of tracheal aspirate with BAL cultures and found more isolates in 26% of tracheal aspirate cultures suggesting growth of normal oral flora as contaminant by this method [14]. Other studies indicate that tracheal aspirates may over diagnose respiratory infections leading to over-use of antibiotics and that bronchoscopic techniques may be more useful in trauma patients [14, 16]. Mondi et al., found that quantitative deep endotracheal aspirate (QDEA) incorrectly diagnosed 31% of intubated patients with VAP using a cutoff of 10 5 cfu/ml and 42% with a cutoff of 10 4 cfu/ml when compared to BAL cultures < 10 5 cfu/ml [16]. In a study of 43 mechanically ventilated trauma patients, BAL was also able to effectively differentiate pneumonia from SIRS suggesting that invasive diagnosis may be more important than clinical indicators in 76

77 diagnosing respiratory infection in critically injured patients [13, 15]. While these studies indicate a possible advantage for invasive diagnostic techniques in trauma patients, further clinical predictors of respiratory infection and when to order respiratory secretion cultures are lacking. The limitations of this study include its retrospective design within a single institution making it difficult to completely ascertain the rationale behind obtaining RCx and treating respiratory infections. Defining respiratory infection was also difficult retrospectively, however, we chose to define infection as a clinicians decision for a 7 day course of antibiotics with the intention of treating a respiratory infection based on guidelines from the ATS and IDSA [17]. This definition was based on our objective to assess our culture practice, assuming that we are obtaining RCxs in order to influence treatment decisions. Analysis of patients treated for respiratory infections showed that RCx results did not consistently impact treatment decisions. Respiratory infections that were treated contrary to culture results may have been due to the presence of other clinical criteria such as worsening chest radiographs, the appearance of more purulent secretions, or patient deterioration. We did not analyze chest radiographs as part of this study, and this is a limitation of our ability to understand culture and treatment rationale. New infiltrates on chest radiographs marginally increase the likelihood of pneumonia in ventilated patients and the absence of infiltrate makes pneumonia less likely [7]. Also, our patient population received a significantly higher number of qualitative tracheal aspirates than quantitative BAL cultures. Our findings that positive RCx results had a low specificity for predicting treatment of respiratory infection may indeed have been different if more invasive diagnostic techniques had been utilized. 77

78 Diagnosing infections in the STICU setting is a difficult problem with numerous factors predisposing to infection as well as influencing clinical decisions to evaluate and treat patients. The specific objectives of our study were to define our practice of obtaining RCxs in critically ill trauma patients over their initial two weeks of admission and determine if the criteria for which we obtained the culture is associated with a positive culture. The focus of these objectives on a specific population, time frame, and nosocomial infection helps to provide one piece of this complex puzzle. The results of this study also indicate some inappropriate use of the respiratory culture in our STICU. 211 patients in our cohort received 489 respiratory cultures for an average of 2.4 RCx per patient. Nineteen percent of all respiratory cultures were collected on consecutive days. We feel that the increased cost and manpower resources utilized to gather and work-up these additional cultures does not justify the limited information which they provide as in most cases the results of back to back cultures will probably be equivalent. With the cost of each tracheal aspirate culture with sensitivities at our institution being $116, we estimate a savings of $21,808 per year with elimination of these consecutive cultures alone. Further savings would most likely be realized with utilization of clinical criteria other than fever and leukocytosis prompting fewer diagnostic evaluations for respiratory infection. By decreasing the number of cultures obtained, we would similarly decrease empiric antibiotic use leading to even greater cost reduction. Currently, our STICU has begun utilizing a novel electronic medical record system which provides physicians with daily reminders of the previous day s culture orders. We hope this will help limit the number of back to back respiratory cultures as 78

79 well as trigger thorough evaluation of clinical parameters rather than reflex pan-culture orders when assessing for origin of fever and/or leukocytosis. CONCLUSION In conclusion, this study suggests that the fever workup paradigm for evaluating respiratory infections in critically injured trauma patients needs to be reevaluated. The current triggers of fever and leukocytosis for obtaining RCxs are not associated with having a positive culture result. Our practice of obtaining RCxs also involves gathering cultures on consecutive days, and our treatment of respiratory infection did not correspond with culture results nearly one-fourth of the time. We were unable to identify a temperature or leukocyte range associated with significantly more positive RCxs. These findings lead us to conclude that factors other than fever and leukocytosis need to be considered when determining whether or not to obtain a RCx. Careful consideration of clinical factors and patient condition should be employed when deciding to order RCxs, as well as in making treatment decisions with RCx results. This will aid in eliminating unnecessary tests and in decreasing hospital costs. Further research analyzing independent risk factors for various infections will be required to create a new infectious workup model for STICU trauma patients. At this point however, we do not recommend obtaining RCx in traumatically injured patients during their initial 14 days of hospitalization based on fever and/or leukocytosis alone. 79

80 STUDY POPULATION DEMOGRAPHICS Patients 510 Patient - Days 3839 Age 49 ± 1 years Males 367 (72%) Caucasian 372 (73%) African American 102 (20%) Injury Severity Score 19 ± 0.5 Blunt Mechanism 464 (91%) STUDY POPULATION OUTCOMES STICU Length of Stay 10 ± 0.5 days Overall Length of Stay 13 ± 1 days Mortality 36 (7%) Table 1: Study Population Demographics and Outcomes 80

81 Relative Risk of Obtaining a Respiratory Culture Based on FEVER, LEUKOCYTOSIS, and FEVER+LEUKOCYTOSIS RR of Being Cultured 95% CI P-Value Fever <0.001 Leukocytosis Fever +Leukocytosis <0.001 NO Fever + NOLeukocytosis <0.001 Table 2: Practice of Obtaining Respiratory Cultures Data analysis indicates that fever and the combination of fever and leukocytosis are associated with obtaining a RCx. 81

82 Relative Risk of Positive Respiratory Culture Based on FEVER, LEUKOCYTOSIS, and FEVER+LEUKOCYTOSIS RR of Positive Culture 95% CI P-Value Fever Leukocytosis Fever + Leukocytosis NO Fever + NO Leukocytosis Table 3: Efficacy of Identifying Positive Respiratory Cultures Data indicates that fever, leukocytosis, and the combination of fever and leukocytosis are not associated with positive RCx. 82

83 Patients with FEVER, LEUKOCYTOSIS and FEVER+LEUKOCYTOSIS by Day 100% Percent of patients remaining in STICU 90% 80% 70% 60% 50% 40% 30% 20% 10% LEUKOCYTOSIS ( 12,000 /mm 3 ) FEVER ( 38.5ºC) FEVER+LEUKOCYTOSIS 0% STICU Day Remaining patients Figure 1: Analysis of patients with fever, leukocytosis, and fever with leukocytosis, measured as percent of patients remaining in the STICU each day. 83

84 Percent of patients with fever, leukocytosis, and fever + leukocytosis by week 90.00% 80.00% 70.00% % of patients remaining in STICU 60.00% 50.00% 40.00% 30.00% Week 1 Week % 10.00% 0.00% % of patients with fever % of patients with leukocytosis % of patients with fever + leukocytosis Figure 2: Percent of patients remaining in STICU with fever, leukocytosis, and fever with leukocytosis by week. Patients included in this figure were those who stayed the entire 7 days for week one, and the entire 14 days for week 2. 84

85 Percentage of positive respiratory cultures by temperature range 90% 80% 70% 60% R 2 = % positive cultures 50% 40% 30% 20% 10% 0% Temperature Range ( C) Positive cultures Total cultures < > Figure 3: Percent of positive respiratory cultures by temperature range 85

86 Percentage of positive respiratory cultures by WBC range 90% 80% 70% 60% % positive cultures 50% 40% 30% R 2 = % 10% 0% WBC Range Positive cultures Total cultures < > Figure 4: Percent of positive respiratory cultures by WBC range 86

87 510 STICU Patients Respiratory Culture Obtained 211 patients (41%) Respiratory Culture Not Obtained 299 patients (59%) Positive Culture 168 patients (80%) Negative Culture 43 patients (20%) Resp Infection 2 patients (0.6%) No Resp Infection 297 patients (99.4%) Resp Infection 122 patients (73%) No Resp Infection 46 patients (27%) Resp Infection 4 patients (9%) No Resp Infection 39 patients (91%) Figure 5: Distribution of Respiratory Cultures and Treated Respiratory Infections per Patient This indicates that 46 patients with a positive RCx were not treated for an infection, and that 4 patients with a negative RCx were treated for an infection. In other words, 50 out of 211 patients cultured (24%) were treated contrary to the result of their RCx. 87

88 Bibliography [1] Hurr H, Hawley HB, Czachor JS, Markert RJ, McCarthy MC. APACHE II and ISS scores as predictors of nosocomial infections in trauma patients. Am J of Infect Control. 1999;27(2): [2] Hoover L, Bochicchio GV, Napolitano LM, Joshi M, Bochicchio K, Meyer W, et al. Systemic inflammatory response syndrome and nosocomial infection in trauma. J Trauma 2006;61(2):310-6; discussion 6-7. [3] Papia G, McLellan BA, El-Helou P, Louie M, Rachlis A, Szalai JP, et al. Infection in hospitalized trauma patients: incidence, risk factors, and complications. J Trauma. 1999;47(5): [4] Jarvis WR. Selected aspects of the socioeconomic impact of nosocomial infections: morbidity, mortality, cost, and prevention.[see comment]. Infect Control Hosp Epidemiol. 1996;17(8): [5] Freischlag J, Busuttil RW. The value of postoperative fever evaluation. Surgery. 1983;94(2): [6] Schey D, Salom EM, Papadia A, Penalver M. Extensive fever workup produces low yield in determining infectious etiology. Am Journal Obstet Gynecol. 2005;192(5): [7] Klompas M. Does this patient have ventilator-associated pneumonia? JAMA. 2007;297(14): [8] Cunha BA. Fever in the intensive care unit.[comment]. Intensive Care Med. 1999;25(7): [9] Marik PE. Fever in the ICU.[see comment]. Chest. 2000;117(3): [10] O'Grady NP, Barie PS, Bartlett JG, Bleck T, Garvey G, Jacobi J, et al. Practice guidelines for evaluating new fever in critically ill adult patients. Task Force of the Society of Critical Care Medicine and the Infectious Diseases Society of America.[see comment]. Clin Infect Dis. 1998;26(5): [11] Golob JF, Fadlalla AMA, Blatnik J, Malangoni MA, Yowler CJ, Claridge JA. Fever and Leukocytosis in critically ill trauma patients: It's not the blood.. J Trauma: Abstract IN PRESS [12] Golob JF, Sando MJ, Phipps WR, Yowler CJ, Malangoni MA, Claridge JA. Fever and Leukocytosis in critically ill trauma patients: It's not the urine. Surgical Infections: IN PRESS [13] Coopersmith C, Kollef M. Dale D, Federman D, eds: Critical Care, Postoperative and Ventilator-Associated Pneumonia In: ACS Surgery Online. WebMD Inc [14] Bonten MJ, Bergmans DC, Stobberingh EE, van der Geest S, De Leeuw PW, van Tiel FH, et al. Implementation of bronchoscopic techniques in the diagnosis of ventilatorassociated pneumonia to reduce antibiotic use. Am J Respir Crit Care Med. 1997;156(6): [15] Croce MA, Fabian TC, Schurr MJ, Boscarino R, Pritchard FE, Minard G, et al. Using bronchoalveolar lavage to distinguish nosocomial pneumonia from systemic 88

89 inflammatory response syndrome: a prospective analysis. J Trauma. 1995;39(6):1134-9; discussion [16] Mondi MM, Chang MC, Bowton DL, Kilgo PD, Meredith JW, Miller PR. Prospective comparison of bronchoalveolar lavage and quantitative deep tracheal aspirate in the diagnosis of ventilator associated pneumonia. J Trauma. 2005;59(4):891-5; discussion 5-6. [17] American Thoracic Society; Infectious Diseases Society of America. Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia Am J Crit Care Med. 2005;171: [18] Vermeulen H, Storm-Versloot MN, Goossens A, Speelman P, Legemate DA. Diagnostic accuracy of routine postoperative body temperature measurements.[see comment]. Clin Infect Dis. 2005;40(10): [19] Frank SM, Kluger MJ, Kunkel SL. Elevated thermostatic setpoint in postoperative patients. Anesthesiology. 2000;93(6): [20] Circiumaru B, Baldock G, Cohen J. A prospective study of fever in the intensive care unit.[see comment]. Intensive Care Med. 1999;25(7): [21] Wunderink RG. Clinical criteria in the diagnosis of ventilator-associated pneumonia. Chest. 2000;117(4 Suppl 2):191S-4S. [22] Croce MA, Swanson JM, Magnotti LJ, Claridge JA, Weinberg JA, Wood GC, et al. The futility of the clinical pulmonary infection score in trauma patients. J Trauma. 2006;60(3):523-7; discussion 7-8. [23] Ewig S, Schlochtermeier M, Goke N, Niederman MS. Applying sputum as a diagnostic tool in pneumonia: limited yield, minimal impact on treatment decisions. Chest. 2002;121(5): [24] Canadian Critical Care Trials G, Heyland D, Dodek P, Muscedere J, Day A. A randomized trial of diagnostic techniques for ventilator-associated pneumonia.[see comment]. N Engl J Med. 2006;355(25): [25] Wood AY, Davit AJ, 2nd, Ciraulo DL, Arp NW, Richart CM, Maxwell RA, et al. A prospective assessment of diagnostic efficacy of blind protective bronchial brushings compared to bronchoscope-assisted lavage, bronchoscope-directed brushings, and blind endotracheal aspirates in ventilator-associated pneumonia. J Trauma. 2003;55(5):

90 Chapter 5. ENHANCING THE FEVER WORKUP UTILIZING A MULTI-TECHNIQUE MODELING APPROACH TO MORE ACCURATELY DIAGNOSE INFECTIONS 90

91 ENHANCING THE FEVER WORKUP UTILIZING A MULTI-TECHNIQUE MODELING APPROACH TO MORE ACCURATELY DIAGNOSE INFECTIONS Adam MA Fadlalla PhD, Joseph F. Golob Jr., MD*, and Jeffrey A. Claridge, MD* Cleveland State University: Department of Computer and Information Science Cleveland, OH * MetroHealth Medical Center: Department of Surgery Cleveland, OH Running Title: PREDICTIVE MODELING OF ICU INFECTIONS This work and JA Claridge was supported in part by Grant Number 1KL2RR from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. This work was also partially funded by a grant awarded to JA Claridge from the Surgical Infection Society. Presented at the 28 th annual meeting of the Surgical Infection Society in Hilton Head, SC May 7 th May 9 th, Corresponding author and reprints: Jeffrey A. Claridge MD MetroHealth Medical Center 2500 MetroHealth Drive Cleveland, OH jclaridge@metrohealth.org 91

92 ABSTRACT Background: Accurate differentiation between infectious and non-infectious etiologies of the systemic inflammatory response syndrome (SIRS) within trauma patients remains illusive. Mathematical modeling techniques in combination with computerized clinical decision support may assist with this differentiation process. The purpose of this study was to determine the capability of various mathematical modeling techniques to predict infectious complications (urinary tract infections [UTI], bacteremia, and respiratory tract infections [RTI]) in critically ill trauma patients, and compare the performance of these models to a standard "fever workup" practice. Methods: An 18-month retrospective database was created using data collected daily from critically ill trauma patients admitted for at least two days to a surgical and trauma intensive care unit. Utilizing ten variables previously shown to be associated with infectious complications; decision trees, neural networks, and logistic regression analysis models were created to predict the presence of UTI, bacteremia, or RTI. These models were created after the data sample was split into a 70% training set and a 30% testing set. Models were compared to each other by calculating sensitivity, specificity, positive predictive value, negative predictive value, overall accuracy, discrimination, and calibration. Models were also compared to the fever workup defined as obtaining blood, urine, or respiratory secretion cultures based on fever and/or leukocytosis. Results: The study population consisted of 243 infected patient-days defined as the day when an infectious complication was identified with a confirmatory positive microbiologic culture, and 2243 non-infected patient-days which included all days from 92

93 non-infected patients without a culture or with a negative culture. Two hundred fortythree non-infected patient-days were randomly chosen to combine with the 243 infecteddays which created a modeling sample of 486 patient-days. Utilizing the testing set (n=141), the fever workup had a sensitivity of 78%, an accuracy of 66%, and a discrimination of 0.67 for identifying infections. Decision trees had the best modeling performance with a sensitivity of 83%, an accuracy of 82%, and a discrimination of 0.91 for identifying infections. Both neural networks and decision trees outperformed logistic regression analysis. A second analysis was performed utilizing the same 243 infected days and only those non-infected patient-days associated with a negative culture workup (n=236). Utilizing this testing set (n=139), the fever workup had a sensitivity of 72%, an accuracy of 47%, and a discrimination of 0.50 for identifying infections. Decision trees again had the best modeling performance for infection identification with a sensitivity of 79%, an accuracy of 83%, and a discrimination of Conclusion: Compared to a standard fever workup, our data demonstrates that the use of mathematical modeling techniques has the ability to improve the robustness and accuracy of predicting UTI, bacteremia, and RTI in critically ill trauma patients. Decision tree analysis appears to have the best potential to use within our computerized clinical decision support to assist in the differentiation of infectious and non-infectious etiologies of the SIRS response. 93

94 BACKGROUND Infections are common complications treated in the surgical and trauma intensive care unit (STICU) [1]. Studies have demonstrated that approximately two million nosocomial infections occur annually, and the cost of diagnosing and treating these infections exceeds 4.5 billion dollars each year [2, 3]. Despite being common in the STICU, infections can be difficult to diagnose. Common clinical signs associated with infectious complications such as fever and leukocytosis become less reliable in a surgical and trauma patient population due to their association with inflammation rather than infection [4-9]. We have previously demonstrated that obtaining blood, urine, and respiratory secretion cultures in critically ill trauma patients was associated with fever ( 38.5ºC) and/or leukocytosis ( 12.0 x 10 3 / mm 3 blood). However, neither fever nor leukocytosis was associated with trauma patients having a positive blood, urine, or respiratory secretion culture [10-12]. Similar results have been described by others for a variety of populations including general surgery and burn patients [4, 8, 13-15]. In the future, we anticipate that computers with clinical decision support will assist physicians and improve the diagnosis of infections in the STICU. To do this, one needs an accurate medical informatics system and validated prediction rules of infectious complications. We have developed and validated the Surgical Intensive Care Infection Registry (SIC-IR) as our informatics tool [16, 17]. This present study was performed to evaluate the potential utility of mathematical models in predicting infections. SIC-IR is an electronic medical record system designed specifically for research as well as physician and administrative documentation in our two STICUs. SIC-IR 94

95 prospectively collects more than 100 clinical variables daily on each STICU patient at the point of patient care. These variables include all the National Healthcare Safety Network data as well as all the Joint Commission intensive care unit core measures. SIC-IR was specifically created with the capability to incorporate advanced clinical decision support generated from robust mathematical models to assist clinicians with the diagnosis of infectious complications. We hypothesize that the addition of multiple clinical and demographic variables into computerized mathematical models will improve the accuracy and efficiency of identifying infections in critically ill trauma patients. Which of these models offer the best predictability of infection remains unclear? Possible modeling techniques which can be utilized within SIC-IR include decision trees, neural networks, and logistic regression analysis. The purpose of this pilot study was twofold. First, to define our STICU culture practice and the impact the fever workup (obtaining blood, urine, or respiratory secretion cultures based on fever and/or leukocytosis) has on this practice. Second, calculate the ability of various mathematical modeling techniques to assist beyond our current practice in the identification of infectious complications (urinary tract infections [UTI], bacteremia, or respiratory tract infections [RTI]) in critically ill trauma patients. The goal of this study is not to create an infection workup, but rather determine the feasibility of using mathematical modeling techniques, and identify a technique which has the greatest potential to assist in the development of clinical decision support within the SIC-IR system. METHODS Database creation and study sample selection 95

96 An 18-month (January-04 through June-05) retrospective database was created with consecutive critically ill trauma patients who were admitted to our institution s 27 bed STICU. To be included in the database, patients had to be at least 18 years-old, could not be admitted to the regular nursing ward or another hospital for more than 24 hours prior to STICU admission, and were required to stay at least two days in the STICU. Data collected included patient demographics, injury mechanism (blunt vs. penetrating), and injury severity score (ISS). In addition, daily clinical data was collected during the patient s initial 14 hospital days and included maximal body temperature, maximal leukocyte count, antibiotic treatments, need for mechanical ventilation, presence of a central venous catheter, microbiologic culture results, and confirmed infectious complications. The population used for this analysis was created by identifying all infected patient-days within the database. An infected patient-day was defined as the day when an infectious complication was identified by a confirmatory positive microbiologic culture (blood, urine, or respiratory secretion). To identify non-infected patient-days, the remaining patient-days were further refined by identifying and removing any day from patients with an identified infection. For example, if a patient was identified as having UTI on STICU day three, all days before and after STICU day three were removed from the analysis. This resulted in a group of non-infected patient-days which were defined as any day with negative microbiologic cultures or days when the clinical suspicion of infection was so low that no cultures were obtained. This patient-day removal was done because we were unable to determine when an infected patient should be considered 96

97 disease free (i.e.: At what day after treatment initiation is an infected patient considered non-infected?). Evaluation of our STICU culture practice and the influence of the fever workup These infected and non-infected patient-days were then analyzed to determine our STICU practice of obtaining blood, urine, and/or respiratory secretion cultures (tracheal aspirate or bronchoalveolar lavage), and to determine the influence the fever workup has on this practice. This was accomplished by calculating the relative risk of fever or leukocytosis triggering a microbiologic culture. Then, a bivariate analysis was performed to identify those variables which had an association with obtaining a culture. Variables evaluated in this analysis are listed in Table 1 and include variables which have been previously shown to be associated with infectious complications or have a direct impact on microbiologic culture results [7, 18-25]. Any variable with a p-value < 0.1 was placed into a backward stepwise logistic regression analysis to determine those variables which were independently associated with obtaining a culture. The odds ratios were then compared to determine the impact fever or leukocytosis has on our STICU practice of triggering a culture workup. Description of mathematical models and their statistical evaluation A decision tree, an artificial neural network, and a logistic regression analysis model were created to predict infectious complications in critically ill trauma patients during their initial 14 hospital-days. Decision trees are tools for recursively partitioning data to assign cases to specific target classes such as infected or non-infected. Artificial neural networks were developed from engineering research and attempt to mimic the action of human neurons but within computer software. These networks model inputs 97

98 and outputs to identify patterns leading the neurons to a specific outcome. Finally, logistic regression analysis is a more familiar modeling tool used in the medical literature. This tool determines if a group of clinical variables including covariates has a unique predictive relationship with a specific outcome. See table 2 for a more detailed description of these mathematical models. For unbiased results, models should be developed on a data set that is different from the set they are tested [26-28]. For developing the models in this analysis, a random sample of non-infected patient-days was selected to match the number of infected-days in a 1:1 ratio. This sample was then split 70% for a training set and 30% for a testing set. Decision trees, neural networks, and logistic regression models were trained using the training set to predict the presence of infection. These models were created utilizing the clinical and demographic variables listed in Table 1. The models were then tested on the testing set and were compared to each other and to the fever workup by calculating sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), overall accuracy, and discrimination. Hosmer-Lemeshow (H-L) chi-square statistic was also computed for the three models to compare their calibration. All models were created using SPSS (SPSS Inc: Chicago, IL). The neural network was created using a multi-perceptron implementation from SPSS and the rules of the decision tree using the SPSS implementation of the C5.0 algorithm. The definitions of sensitivity, specificity, PPV, NPV, overall accuracy, discrimination, and H-L calibration statistic are listed in Table 3. All categorical variables are expressed as percentages and compared using Chi square or Fisher s Exact Test when appropriate. Continuous variables are expressed as means ± standard error of the mean and compared 98

99 using Student s T-tests after evaluation with Levene s Test to determine if equal or unequal variances should be assumed. A p-value of 0.05 was used for statistical significance. This study was reviewed and approved by the MetroHealth institutional review board. RESULTS: The retrospective database used for this analysis consisted of 510 patients with 3839 patient-days of data collected. There were a total of 243 infected patient-days which left 3596 patient-days remaining. From these remaining patient days, all days were excluded from patients who had an identified infection. This left 2243 non-infected patient-days to combine with the 243 infected-days which created the 2486 patient-day study sample used in this analysis (Figure 1). This study sample had a 10% prevalence of infection (bacteremia, UTI, RTI) and 2% missing data. Table 4 lists the clinical and demographic data of the study cohort. The 243 infected-days (identified in 126 patients) consisted of 60 UTIs, 27 bacteremias, and 178 RTIs. Twenty-two infected patient-days had multiple identified infections. Evaluation of our STICU culture practice and the influence of the fever workup Four hundred seventy nine patient-days (19%) were associated with obtaining any one of the three microbiologic cultures of interest: blood, urine, or respiratory secretion. One thousand two hundred fifty one patient-days (50%) were associated with either fever or leukocytosis. The relative risk was 3.4 (95% CI = ) for fever or leukocytosis triggering a microbiologic culture (Figure 2). Seventy nine percent of the blood, urine, and respiratory secretion cultures acquired were obtained on patient-days with fever or 99

100 leukocytosis. These data support that our STICU culture practice is strongly influenced by the fever workup. Bivariate analysis using variables listed in Table 1 identified ICU day (p<0.001), age (p=0.002), injury severity score (p<0.001), male gender (p=0.009), presence of a central venous catheter (p<0.001), mechanical ventilation (p<0.001), and presence of fever or leukocytosis (p<0.001) as being associated with obtaining a culture. These variables were placed in a backward stepwise logistic regression analysis which identified ICU day, presence of a central venous catheter, mechanical ventilation, and presence of fever or leukocytosis as independent predictors of obtaining a blood, urine, or respiratory secretion culture (Table 5). The presence of fever or leukocytosis had the highest odds ratio of the variables remaining in the logistic regression analysis suggesting the major impact of the fever workup on our STICU culture practice. Therefore, in the remainder of this analysis, the fever workup will be used as a surrogate marker of our STICU culture practice when comparing the generated mathematical models. Evaluation of mathematical models for predicating infections A random 243 non-infected patient-days were chosen to combine with the 243 infected-days in a 1:1 ratio. These 486 patient-days were then spilt into a 70% model training set (n=345) and 30% testing set (n=141) (Figure 3). Each of the three models were first trained to identify infectious complications by analyzing the variables in Table 1 within the training set. The models were then exposed to the testing set, and the accuracy of identify infections was analyzed. Table 6 shows a comparison of the clinical and demographic variables between the training and testing sets. There were no 100

101 significant difference between any of the training and testing set variables. The results of testing the models are in Table 7. Fever or leukocytosis alone had an accuracy of 66% with a sensitivity of 78% for identifying infectious complications. The neural network and decision tree modeling techniques both increased the accuracy of identifying infections compared to the fever workup ; however the neural network demonstrated a slight decrease in sensitivity whereas the decision tree increased the sensitivity. Compared to the fever workup, logistic regression analysis had a decrease in both sensitivity and accuracy for identifying infections. All models showed an increase in discrimination with areas under their respective receiver operating characteristic curve between 0.81 and H-L calibration statistic was 58.28, , and for the logistic regression, the decision tree, and the neural network respectively. Each of the model s results listed in Table 7 were created using the definition of non-infected patient-day as any day with a negative microbiologic culture or a day when the clinical suspicion of infection was so low that no cultures were obtained. Realizing the potential of missing infected patient-days with this definition, we performed an identical analysis using the same 243 infected patient-days and only the non-infected patient-days when negative microbiologic cultures was obtained (Figure 4). This second analysis was done to remove some of the selection bias and potential false negatives which may be found in our initial models. For this second study sample, a comparison of the training and testing set variables are listed in Table 8, and the model results are presented in Table 9. There was no significant difference between the training and testing set for any of the variables 101

102 used. Fever or leukocytosis was 47% accurate for identifying infections in patients who were cultured. Mathematical modeling increased this accuracy to 83% in the decision tree model. Both the neural network and decision tree analysis increased the sensitivity of identifying infections to 79% and 86% respectively, but logistic regression lost sensitivity to 66%. Decision tree analysis again had the best discrimination of all the models with an area under the receiver operating characteristic curve of 0.87 DISCUSSION: Our analysis demonstrated that the presence of fever or leukocytosis dominates our STICU s trauma patient culture practice. In a population thought to be infected as demonstrated by the physician obtaining a microbiologic culture, fever or leukocytosis alone identified 72% (sensitivity) of those with an infection. However, on patient-days that had cultures obtained and the presence of fever or leukocytosis, only 50% (PPV) of cultures were positive. Obtaining these multiple cultures often results in an inefficient diagnostic practice. Mathematical modeling utilizing ten variables associated with infectious complications can improve the sensitivity, specificity, PPV, NPV, accuracy, and discrimination of diagnosing infections. Statistical and mathematical models in the trauma and critical care population have been used in the past. As performance comparisons between hospitals and physicians increase, models have been used to predict outcomes for both trauma (adult and pediatric) and critical care patients. [29-34]. A very common modeling technique used in the trauma and critical care literature continues to be logistic regression analysis. This technique was used to create the Trauma and Injury Severity Score (TRISS) which gives a probability of death based on anatomical and physiological severity indicators 102

103 [35]. More recently, other mathematical models have been gaining momentum in the medical community. For example, artificial neural networks have been used in mammography algorithms, predicting prognosis of patients experiencing a myocardial infarction, predicting outcomes after liver transplantation, and assisting in the diagnosis of pulmonary embolisms [36-39]. However, models other than logistic regression analysis have been slow to adopt in the medical community due to their complexity and conflicting data when comparing performance directly to logistic regression analysis [29-31, 34] The two other modeling techniques we evaluated included artificial neural networks and decision tree analysis. Neural networks are best suited in situations that are unstructured, data-intensive, with high uncertainty, hidden relationships, and noise. We believe they are potentially a good modeling technique for predicting infections in the critically ill trauma patients. Predicting infections in such patients is data-intensive, highly uncertain, and the data could be quite noisy. However, developing a good neural network model requires a significant amount of time for trial and refinement, and interpreting the resulting model is a serious challenge. Decision trees have key merits including their ability to handle missing data, and the ease of explaining and deploying their resulting models. However, they are based on step (not smooth) functions, and hence may provide only coarse estimators for relationships that do not fit a step function characterization. Statistical modeling of infectious complications has been performed by Leibovici and colleagues in their development of TREAT, a clinical decision support system for antibiotic selection in inpatients with common bacterial infections [40-43]. This group 103

104 employed a causal probabilistic network modeling technique which was populated with common clinical signs and symptoms of infection identified in the literature. This allowed the TREAT system to calculate a probability of identifying an infectious complication and offered an empiric antibiotic selection. This system was evaluated in a prospective study in three countries and concluded that appropriate empirical antibiotic treatment increased in the intervention group while length of hospital stay, costs related to future resistance, and total antibiotic costs were all decreased [42]. Our models differ from TREAT in that we are using variables identified in the literature to be associated with infections and then our own data to generate the models. We are not relying on probabilities identified from other authors to assist in our model generation. This will allow our system to continue to learn and adapt as new prospective data is gathered within the SIC-IR system. There are several limitations to this study which must be addressed in context with the overall objective of this analysis. Our goal was not to describe when an infectious workup should be initiated, but rather determine if mathematical modeling using multiple variables associated with infections can improve the predictive power over the fever workup. The database used for this analysis was collected in a retrospective fashion and has the inherit weakness of this type of data. The database also lacked specific important variables which may help to predict infections such as vital signs and hemodynamic support requirements. However, for our analysis we feel the ten variables used were very relevant to the goal of proving mathematical modeling may help identify infections in trauma patients. The current SIC-IR system offers prospective data collected at the point of care and includes multiple other variables associated with 104

105 infection which will improve the robustness of future models. The assumption that patients with no cultures were not infected was addressed by performing our second modeling analysis looking at only patients who were cultured. This analysis also proved that modeling can increase the accuracy and discrimination without sacrificing sensitivity in a population believed to be clinically infected. Finally, all analyses were limited to only three infections most commonly evaluated in the fever workup: UTIs, respiratory tract infections, and bacteremias. This may introduce a low signal to noise ratio in the models which results in variables which are unimportant becoming more significant. However, the recommended number of infections per variable in the model should be at least ten creating the requirement of 100 infected-days. Our data had 243 infected-days which yielded just over 20 infected-days per variable entered into the model. In conclusion our STICU practice of obtaining microbiologic cultures in critically ill trauma patients is dominated by fever and/or leukocytosis. This practice results in sensitivities between 72-80% and accuracies of 47-66% in identifying infectious complications. Mathematical models beyond logistic regression analysis improved sensitivity, specificity, PPV, NPV, accuracy, and discrimination of infection identification. Decision tree analysis performed the best with sensitivities between 83-86% and accuracies of approximately 82% for identifying infections in critically ill trauma patients. Thus, to create an "infection workup" to assist with the differentiation between infectious and non-infectious etiologies of the SIRS response, decision trees should be considered for designing computerized clinical decision support within our Surgical Intensive Care Infection Registry (SIC-IR). 105

106 FIGURE LEGEND Figure 1: Flow diagram of patient selection to evaluate our STICU culture practice and the impact the fever workup has on this practice Infected patient-day: The day when an infectious complication was identified by a confirmatory positive microbiologic culture (blood, urine, or respiratory secretion) Non-infected patient-day: Any day with negative microbiologic cultures or days when the clinical suspicion of infection was so low that no cultures were obtained STICU: Surgical and trauma intensive care unit Fever workup : Obtaining blood, urine, or respiratory secretion cultures based on fever and/or leukocytosis Figure 2: 2 x 2 table evaluating our STICU culture practice This 2 x 2 table demonstrated that our STICU practices the fever workup and obtains microbiologic cultures based on fever ( 38.5ºC) or leukocytosis (>12.0 x 10 3 / mm 3 blood). Values are expressed in patient-days. STICU: Surgical and trauma intensive care unit Fever workup : obtaining blood, urine, or respiratory secretion cultures based on fever and/or leukocytosis Figure 3: Flow diagram of patient selection to evaluate mathematical models The sample used for mathematical modeling was created by randomly choosing 243 noninfected patient-days to combine with the 243 infected patient-days. This sample was then split into a 70% model training set and a 30% testing set. Figure 4: Flow diagram of patient selection to evaluate mathematical models in patient-days when cultures were obtained 106

107 The sample used for mathematical modeling was created by identifying the 236 noninfected patient-days with a negative microbiologic culture and combining them with the 243 infected patient-days. This sample was then split into a 70% model training set and a 30% testing set. 107

108 Variable Name Variable Type ICU Day (days) Continuous Age (years) Continuous Max body temperature (ºC) Continuous Max leukocyte count (10 3 / ml 3 blood) Continuous Injury Severity Score (ISS) Continuous Gender Dichotomous Mechanism of injury Dichotomous Presence of a central-line Dichotomous Presence of mechanical ventilation Dichotomous Presence of antibiotics the day prior to Dichotomous obtaining cultures Fever or leukocytosis Dichotomous Table 1: Variables used in mathematical modeling and determination of our STICU microbiologic practice These variables were collected daily on each critically ill trauma patient and then placed into various mathematical models to predict infections (urinary tract infection, bacteremia, respiratory tract infection). All mathematical models used temperature and leukocyte count as continuous variables. Fever or leukocytosis replaced max body temperature and max leukocyte count when evaluating our culture practice. STICU: Surgical and trauma intensive care unit 108

109 Model Comparison Criteria Sensitivity Specificity Positive predictive value (PPV) Negative predictive value (NPV) Accuracy Discrimination Calibration Equation / Method of Calculation True Positive / (True Positive + False Negative) True Negative / (False Positive + True Negative) True Positive / (True Positive + False Positive) True Negative / (False Negative + True Negative) (True Positive + True Negative) / (True Positive + False Positive + True Negative + False Negative) Measured by the area under the receiver operating characteristic curve Measured by the Hosmer-Lemeshow (H-L) chi-square statistic Table 3: Criteria used to compare the mathematical models to each other and to the fever workup Fever workup : Obtaining blood, urine, and respiratory secretion cultures based on fever and/or leukocytosis 109

110 Age (years) 48.8 ± 0.4 Daily max temperature (ºC) 37.9 ± 0.02 Daily max leukocyte count (x 10 3 / ml 3 of blood ) 11.2 ± 0.1 Injury severity score 20.2 ± 0.2 Male 74% Blunt Trauma 91% Patient-days with central venous catheter 40% Patient-days with ventilator requirement 55% Table 4: Clinical and demographic data of the study cohort (n=2486 patient-days) Continuous variables are expressed as means ± standard error of the mean per patientday. Dichotomous variables are expressed as percents per patient-day. 110

111 Variable associated with obtaining a culture (blood, urine, or respiratory Odds Ratio 95% Confidence Interval secretion culture) ICU Day Presence of central venous catheter Mechanical ventilation Presence of fever or leukocytosis Table 5: Logistic regression analysis identifying variables associated with our STICU culture practice Presence of fever or leukocytosis had the highest odds ratio suggesting the importance of the fever workup in our STICU culture practice. The area under the receiver operating curve was 0.80 which suggests not all variables associated with our culture practice were recorded and analyzed. Fever workup : Obtaining blood, urine, and respiratory secretion cultures based on fever and/or leukocytosis 111

112 Training Set Testing Set N=345 N=141 p-value Age (years) 47.5 ± ± Max Daily Temperature (ºC) 38.3 ± ± Max Daily Leukocyte Count (x 10 3 / ml 3 of blood ) 11.2 ± ± Injury Severity Score (ISS) 22.7 ± ± ICU Day 6.2 ± ± Males 71% 69% Blunt Injury 95% 91% Patient-days with central venous catheter 50% 50% Patient-days with ventilator requirement 69% 75% Presence of antibiotics day before 65% 65% Table 6: Comparison of training and testing set of the ten modeling variables Values are reported as mean ± standard error of the mean per patient day or percent per patient-day 112

113 Sensitivity Specificity PPV NPV Accuracy Discrimination Fever Workup 78% 55% 63% 72% 66% 0.67 Neural Network 76% 78% 79% 76% 77% 0.84 Decision Tree 83% 80% 79% 85% 82% 0.91 Logistic Regression 62% 65% 69% 58% 63% 0.81 Table 7: Analysis of mathematical modeling techniques compared to the fever workup All values are reported for the model testing set (n=141) Fever Workup : obtaining blood, urine, or respiratory secretion cultures based on fever and/or leukocytosis PPV: Positive predictive value NPV: Negative predictive value Discrimination: Area under the receiver operating characteristic curve 113

114 Training Set Testing Set N=340 N=139 p-value Age (years) 46.4 ± ± Max Daily Temperature (ºC) 38.6 ± ± Max Daily Leukocyte Count (x 10 3 / ml 3 of blood ) 12.3 ± ± Injury Severity Score (ISS) 23.6 ± ± ICU Day 6.7 ± ± Males 79% 78% Blunt Injury 91% 94% Patient-days with central venous catheter 54% 56% Patient-days with ventilator requirement 83% 82% Presence of antibiotics day before 80% 77% Table 8: Comparison of training and testing set of the ten modeling variables in patient-days with at least one culture obtained Values are reported as mean ± standard error of the mean per patient day or percent per patient-day 114

115 Sensitivity Specificity PPV NPV Accuracy Discrimination Fever Workup 72% 18% 50% 38% 47% 0.50 Neural Network 79% 67% 65% 80% 72% 0.76 Decision Tree 86% 80% 81% 85% 83% 0.87 Logistic Regression 66% 58% 58% 66% 62% 0.74 Table 9: Analysis of mathematical modeling techniques in patient days where cultures were obtained All values are reported for the testing set (n=139) Fever Workup : obtaining blood, urine, or respiratory secretion cultures based on fever and/or leukocytosis PPV: Positive predictive value NPV: Negative predictive value Discrimination: Area under the receiver operating characteristic curve 115

116 Goal of Model Types of Model Method of Prediction Decision Tree Provides algorithms for generating rules which partitions data leading to the prediction of a target outcome variable Classification trees: decision trees which predict categorical variable outcomes Regression trees: decision trees which predict continuous variable outcomes Variety of different algorithms which generate prediction rules (rules are stored in leaf nodes) o CHAID (Chisquared Automa tic Interact ion Detecti on) [44] Artificial Neural Network Computerized intelligent systems which simulate the inductive power of the human brain to abstract essential data needed to predict specific outcomes Recurrent or non-recurrent: how the layer of data processing elements (neurons) are organized and connected Breaks independent variables into an interconnected network of simple units called neurons (connectioni sm) Neurons lead data through a simple input layer, any Logistic Regression Analysis Attempts to predict probabilities of a binary outcome as a function of one or more independent variables Direct: All independent variables enter the model at the same time Sequential: using a block-chi square method (likelihood ratio test), the full model with independents and covariates is evaluated. Then the model is run again with the block of independents which were dropped to determine the importance of the covariants Stepwise: the model determines automatically which independent variables to add/drop from the model Evaluates a group of independent variables collectively termed x and their probability (p) of an outcome p ' log it( p) = log = α + x b 1 p α is the intercept parameter and b is a vector of slope parameters 116

117 o CART (Classif ication and Regress ion Trees) [45] o QUEST [46] o C5.0 [47] Iterative splitting of independent variables into discrete groups which are as different as possible Ideally a split leads to a pure leaf node which contains only one of the outcomes being predicted number of hidden layers, and an output layer Similar to a human brain, neural networks can learn from mistakes by modifying the connections between neurons (delta learning rule) Table 2: Description of mathematical models used in this evaluation. 117

118 Bibliography [1] Edwards JR, Peterson KD, Andrus ML, Tolson JS, Goulding JS, Dudeck MA, et al. National Healthcare Safety Network (NHSN) Report, data summary for 2006, issued June American Journal of Infection Control. 2007;35(5): [2] Hurr H, Hawley HB, Czachor JS, Markert RJ, McCarthy MC. APACHE II and ISS scores as predictors of nosocomial infections in trauma patients. American Journal of Infection Control. 1999;27(2): [3] Jarvis WR. Selected aspects of the socioeconomic impact of nosocomial infections: morbidity, mortality, cost, and prevention.[see comment]. Infection Control & Hospital Epidemiology. 1996;17(8): [4] Freischlag J, Busuttil RW. The value of postoperative fever evaluation. Surgery Aug;94(2): [5] Hoover L, Bochicchio GV, Napolitano LM, Joshi M, Bochicchio K, Meyer W, et al. Systemic inflammatory response syndrome and nosocomial infection in trauma. Journal of Trauma-Injury Infection & Critical Care. 2006;61(2):310-6; discussion 6-7. [6] Miller PR, Munn DD, Meredith JW, Chang MC. Systemic inflammatory response syndrome in the trauma intensive care unit: who is infected? Journal of Trauma-Injury Infection & Critical Care. 1999;47(6): [7] Papia G, McLellan BA, El-Helou P, Louie M, Rachlis A, Szalai JP, et al. Infection in hospitalized trauma patients: incidence, risk factors, and complications. The Journal of trauma Nov;47(5): [8] Schey D, Salom EM, Papadia A, Penalver M. Extensive fever workup produces low yield in determining infectious etiology. American Journal of Obstetrics & Gynecology. 2005;192(5): [9] Wallace WC, Cinat M, Gornick WB, Lekawa ME, Wilson SE. Nosocomial infections in the surgical intensive care unit: a difference between trauma and surgical patients. The American surgeon Oct;65(10): [10] Claridge JA, Sando MJ, Golob JF, Fadlalla AM, Peerless JP, CJ Y. The "Fever Workup" and Rrespiratory Culture Practice in Critically Ill Trauma Patients. Being Reviewed [11] Golob JF, Claridge JA, Fadlalla A, Malangoni M, Blatnik J, Yowler CJ. Fever and Leukocytosis in Critically Ill Trauma Patients: It's not the blood. Being reviewed [12] Golob JF, Claridge JA, Sando MJ, Phipps WR, Yowler CJ, Fadlalla A, et al. Fever and Leukocytosis in Critically Ill Trauma Patients: It's Not the Urine. Surgical Infections. 2008;9(1): [13] Wagenlehner FM, Loibl E, Vogel H, Naber KG. Incidence of nosocomial urinary tract infections on a surgical intensive care unit and implications for management. International Journal of Antimicrobial Agents. 2006;28 Suppl 1:S [14] Murray CK, Hoffmaster RM, Schmit DR, Hospenthal DR, Ward JA, Cancio LC, et al. Evaluation of White Blood Cell Count, Neutrophil Percentage, Elevated Temperature as Predictors of Bloodstream Infection in Burn Patients. Archives of Surgery. 2007;142(7):

119 [15] Theuer CP, Bongard FS, Klein SR. Are blood cultures effective in the evaluation of fever in perioperative patients? American Journal of Surgery. 1991;162(6):615-8; discussion 8-9. [16] Fadlalla AMA, Golob JF, Claridge JA. The Surgical Intensive Care - Infecton Registry (SIC-IR): A research registry with daily clinical support capabilities.. The American Journal of Medical Quality. In press. [17] Golob JF, Fadlalla AMA, Kan JA, Patel NP, Yowler CJ, Claridge JA. Validation of SIC-IR : A medical informatics system for intensive care unit research, qualtiy of care improvement, and daily patient care. Journal of the American College of Surgeons. In Press. [18] Charalambous C, Swoboda SM, Dick J, Perl T, Lipsett PA. Risk factors and clinical impact of central line infections in the surgical intensive care unit. Archives of Surgery. 1998;133(11): [19] Pawar M MY, Kapoor P, Sharma J, Gupta A, Trehan N. Central venous catheter-related blood stream infections: incidence, risk factors, outcome, and associated pathogens. Journal of Cardiothoracic & Vascular Anesthesia. 2004;18(3): [20] Angstwurm MW, Gaertner R, Schopohl J. Outcome in elderly patients with severe infection is influenced by sex hormones but not gender. Critical care medicine. 2005;33(12): [21] Bochicchio GV, Joshi M, Bochicchio K, Shih D, Meyer W, Scalea TM. Incidence and impact of risk factors in critically ill trauma patients. World Journal of Surgery. 2006;30(1): [22] Bochicchio GV, Joshi M, Knorr KM, Scalea TM. Impact of nosocomial infections in trauma: does age make a difference? The Journal of trauma Apr;50(4):612-7; discussion 7-9. [23] Crabtree TD, Pelletier SJ, Gleason TG, Pruett TL, Sawyer RG. Gender-dependent differences in outcome after the treatment of infection in hospitalized patients. JAMA. 1999;282(22): [24] Jamulitrat S, Narong MN, Thongpiyapoom S. Trauma severity scoring systems as predictors of nosocomial infection. Infection Control & Hospital Epidemiology. 2002;23(5): [25] Coopersmith C, Kollef M. ACS Surgery Online. In: Dale D, Federman D, eds. Critical Care, Postoperative and Ventilator-Associated Pneumonia: WebMD Inc [26] Bleeker SE, Moll HA, Steyerberg EW, Donders A, Derksen-Libsen G, Grobbee DE, et al. External validation is necessary in prediction research: A clinical example. Journal of Clinical Epidemiology. 2003;56: [27] Terrin N, Schmid CH, Griffith JL, D'Agostino RB, Selker HP. External validity of predictive models: A comparison of logistic regression, classification trees, and neural networks. Journal of Clinical Epidemiology. 2003;56: [28] Vergouwe Y, Steyerberg EW, Eijkemans JC, Habbema D. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. Journal of Clinical Epidemiology. 2005;58: [29] Becalick DC, Coats TJ. Comparison of artificial intelligence techniques with UKTRISS for estimating probability of survival after trauma. UK Trauma and Injury Severity Score. Journal of Trauma-Injury Infection & Critical Care. 2001;51(1):

120 [30] Clermont G, Angus DC, DiRusso SM, Griffin M, Linde-Zwirble WT. Predicting hospital mortality for patients in the intensive care unit: a comparison of artificial neural networks with logistic regression models.[see comment]. Critical Care Medicine. 2001;29(2): [31] DiRusso SM, Chahine AA, Sullivan T, Risucci D, Nealon P, Cuff S, et al. Development of a model for prediction of survival in pediatric trauma patients: comparison of artificial neural networks and logistic regression. Journal of Pediatric Surgery. 2002;37(7): ; discussion [32] DiRusso SM, Sullivan T, Holly C, Cuff SN, Savino J. An artificial neural network as a model for prediction of survival in trauma patients: validation for a regional trauma area.[see comment]. Journal of Trauma-Injury Infection & Critical Care. 2000;49(2):212-20; discussion [33] Gabbe BJ, Cameron PA, Wolfe R, Simpson P, Smith KL, McNeil JJ. Prehospital prediction of intensive care unit stay and mortality in blunt trauma patients. Journal of Trauma-Injury Infection & Critical Care. 2005;59(2): [34] Wolfe R, McKenzie DP, Black J, Simpson P, Gabbe BJ, Cameron PA. Models developed by three techniques did not achieve acceptable prediction of binary trauma outcomes. Journal of Clinical Epidemiology. 2006;59(1): [35] Boyd CR, Tolson MA, Copes WS. Evaluating Trauma Care: The TRISS Method. The Journal of trauma. 1988;27: [36] Baxt WG. Use of an artificial neural network for the diagnosis of myocardial infarction.[see comment][erratum appears in Ann Intern Med 1992 Jan 1;116(1):94]. Annals of Internal Medicine. 1991;115(11): [37] Doyle HR, Dvorchik I, Mitchell S, Marino IR, Ebert FH, McMichael J, et al. Predicting outcomes after liver transplantation. A connectionist approach. Annals of Surgery. 1994;219(4): [38] Tourassi GD, Floyd CE, Sostman HD, Coleman RE. Artificial neural network for diagnosis of acute pulmonary embolism: effect of case and observer selection. Radiology. 1995;194(3): [39] Vyborny CJ, Giger ML. Computer vision and artificial intelligence in mammography. AJR American Journal of Roentgenology. 1994;162(3): [40] Leibovici L, Gitelman V, Yehezkelli Y, Poznanski O, Milo G, Paul M, et al. Improving empirical antibiotic treatment: prospective, nonintervention testing of a decision support system. Journal of Internal Medicine. 1997;242(5): [41] Leibovici L, Paul M, Nielsen AD, Tacconelli E, Andreassen S. The TREAT project: decision support and prediction using causal probabilistic networks. International Journal of Antimicrobial Agents. 2007;30 Suppl 1:S [42] Paul M, Andreassen S, Tacconelli E, Nielsen AD, Almanasreh N, Frank U, et al. Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster randomized trial.[see comment]. Journal of Antimicrobial Chemotherapy. 2006;58(6): [43] Paul M, Nielsen AD, Goldberg E, Andreassen S, Tacconelli E, Almanasreh N, et al. Prediction of specific pathogens in patients with sepsis: evaluation of TREAT, a computerized decision support system. Journal of Antimicrobial Chemotherapy. 2007;59(6):

121 [44] Kass G. An exploratory technique for investigating large quantities of categorical data. Applied Statistics. 1980;29(2): [45] Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and Regression Trees. New York: Chapman and Hall [46] Loh WY, Shih YS. Split selection methods for classification trees. Statistica Sinica. 1997;7: [47] Quinlan JR. C4.5: Programs for Machine Learning: Morgan Kaufmann Publishers

122 Retrospective Database 3839 Patient-days 243 Infected Patient-days 3596 Patient-days Remove all days from patients with an identified infected-day (n=1353 patient-days) 2486 Patient-days used for analysis of our STICU culture practice 2243 Non-infected Patient-days Figure 1 122

123 Any Culture? Yes No Fever And/Or Leukocytosis? Yes No Figure 2 Relative risk of fever OR leukocytosis triggering a culture = 3.4 (95% CI = ) 123

124 Retrospective Database 3839 Patient-days 243 Infected Patient-days 3596 Patient-days Remove all days from patients with an identified infected-day (n=1353 patient-days) 486 Patient-days used for modeling Random 243 patientdays 2243 Non-infected Patient-days 70% Training set (N=345) 30% Testing set (N=141) Figure 3 124

125 Retrospective Database 3839 Patient-days 243 Infected Patient-days 479 Patient-days used for modeling 3596 Patient-days 2243 Non-infected Patient-days Remove all days from patients with an identified infected-day (n=1353 patient-days) 70% Training set (N=340) 30% Testing set (N=139) Negative culturedays (N=236) No culture-days (N=2007) Figure 4 125

126 Chapter 6 Creation of SIC-IR (The Surgical Intensive Care Infection Resgistry) 126

127 THE SURGICAL INTENSIVE CARE-INFECTION REGISTRY (SIC-IR): A RESEARCH REGISTRY WITH DAILY CLINICAL SUPPORT CAPABILITIES Running Title: DEVELOPMENT OF SIC-IR Adam M.A. Fadlalla, Ph.D, Joseph F. Golob Jr, M.D. *, Jeffrey A. Claridge, M.D. * Cleveland State University Department of Computer and Information Science: Cleveland, OH * Case Western Reserve University School of Medicine, MetroHealth Medical Center, Department of Surgery: Cleveland, OH Reprint Requests and Corresponding Author: Jeffrey A. Claridge, M.D. MetroHealth Medical Center Room H939, Hamann Bldg 2500 MetroHealth Drive Cleveland, OH Phone: Fax: jclaridge@metrohealth.org JA Claridge is supported by the National Institutes of Health, National Institute of Child Health and Human Development, Multidisciplinary Clinical Research Career Development Programs Grant K12 RR Presented at the 2007 American Medical Informatics Association Annual Symposium 127

128 ABSTRACT: Infections in the surgical and trauma intensive care unit (STICU) are responsible for significant patient morbidity and mortality. Research into these infectious complications often utilizes administrative databases or clinical information systems designed for documentation and billing of daily patient care. Neither of these sources is intended for research, and many researchers have questioned their accuracy. We developed the Surgical Intensive Care Infection Registry (SIC-IR) specifically as a research data repository used to monitor STICU infections. SIC-IR is a relational database application designed to collect quality data and integrate with daily patient care. SIC-IR prospectively collects and archives over 100 clinical variables daily on each STICU patient to ensure completeness and correctness of the registry. Furthermore, SIC-IR aids in clinical activities by providing patient summaries and medical record documentation. SIC-IR provides accurate data for STICU infection research and enables the users to easily undertake quality of care improvement initiatives. KEY WORDS: 1) Information systems, inpatient 2) Intensive care units 3) Infections 128

129 INTORDUCTION: Clinical research to improve patient outcomes requires accurate prospective databases. This paper describes the design, implementation, and validation of SIC-IR (Surgical Intensive Care Infection Registry); a system developed by a multidisciplinary team to create an accurate database that is used for infection research in the surgical and trauma intensive care unit (STICU). SIC-IR is also used as a daily clinical care support tool that creates all patient care summaries and documentation. BACKGROUND: Nosocomial infections and quality of care improvement programs There is increasing awareness about nosocomial infections and their devastating consequences. Major daily newspapers such as the New York Times and the Wall Street Journal have printed articles relating to hospital infections including Clostridium difficile colitis and methicillin-resistent Staphylococcus aureus [1-3]. Signs in hospital rooms and elevators remind patients to refuse care until their physician or nurse wash their hands to stop the spread of these dreadful contagions. These infections are becoming an epidemic in our hospitals with approximately two million nosocomial infections occurring in the United States[4] with a total cost of approximately 4.5 billion dollars annually for their diagnosis and treatment[5]. Publicity of nosocomial infections has transformed patients from passive into active participants in their medical care, and also mandates the necessity that we as health care providers work to improve the system. The nosocomial infection epidemic has reached critical importance in the STICU. Previous research has shown that the STICU has significantly more infections (including nosocomial infections) compared to the medical intensive care unit[6]. Within the 129

130 STICU, trauma patients particularly have higher rates of infectious complications compared to general surgery patients[7]. Estimates have shown 37%-45% of all trauma patients will experience some type of infection with 75%-85% of these infections being classified as nosocomial[8, 9]. Within the last several years, multiple STICU quality of care initiatives have been started by numerous organizations in an attempt to improve the incidence rate and morbidity of infectious complications. The Joint Commission, the nation s leading accrediting body in health care, recognizes the magnitude of nosocomial infections within hospitals and publishes the National Patient Safety Goals. Goal 7 of the 2008 hospital program is: Reduce the risk of heath care-associated infections. To reach this goal, the Joint Commission requires hospitals to comply with the current World Health Organization Hand Hygiene Guidelines and mange as sentinel events all identified cases of unanticipated death or permanent loss of function associated with health careassociated infections[10]. Nosocomial infections and quality of care are key factors which must be thoroughly and appropriately researched in order to significantly improve STICU patient outcomes. In 2006, The Society of Critical Care Medicine Outcomes Task Force released a how-to guide for ICU quality of care improvement programs[11]. Two of their seven recommendations were related to information technology (IT) and included creation of an accurate data collection system as well as formation of a data reporting system. These systems will allow clinicians and other stakeholders to identify current problems as well as monitor quality improvements[11]. Our group found it necessary to employ modern medical informatics through the development of SIC-IR (Surgical Intensive Care 130

131 Infection Registry). SIC-IR was designed to collect accurate data on infections in critically ill STICU patients, and to provide a reporting system that meets the IT recommendations set forth by the Society of Critical Care Medicine. SIC-IR has several similarities and differences to currently available commercial infection control surveillance software such as TheraDoc Infection Control Assistant and Premier s SafetySurveillor -Infection Control. Like SIC-IR, these tools collect real-time data to help identify infectious complications quickly and accurately. However, TheraDoc Infection Control Assistant and SafetySurveillor -Infection Control were designed for trained infection control team utilization while SIC-IR is used by the physicians caring for the patient at the time of patient care while the patient is in the STICU. In addition, SIC-IR is currently collecting organized data on issues other than those which are infection related to assist with other quality of care projects and to be a valuable tool for clinical research studies. Limitation of administrative databases for research A key weakness in many research studies is that researchers often rely on administrative or clinical databases which were not designed specifically for research. Administrative databases are used because they are easy to access, available in electronic format, and are cost effective means of studying large populations[12]. In addition, researchers use commercially available clinical information systems designed for documenting clinical care to support their research endeavors. Both administrative and clinical systems were not designed for clinical research support and their accuracy in this regard has been questioned[12-18]. Figure 1 is a schematic representation of data flow 131

132 from patients to administrative databases and potential sources of errors of omission and commission as described in part by Roberts, et al.[19]. The Department of Veterans Affairs (VA) recognized the shortcomings of using administrative and clinical systems for research and consequently developed the VA National Surgical Quality Improvement Program (NSQIP) in The NSQIP was developed to accurately report morbidity and mortality for each surgical procedure performed at VA medical centers and has accumulated approximately 1.3 million records from 123 institutions[20, 21]. The database was specifically designed for research purposes, and has the potential to accurately report risk-adjusted mortality rates for major noncardiac surgical procedures. The American College of Surgeons has adopted the NSQIP technology and encourages non-va hospitals to contribute data in order to continually improve surgical outcomes through research based on an accurate database. NSQIP is an example of a successful research database which has been shown to be very accurate and has gained wide acceptance across the nation [20]. METHODS SIC-IR Design Objectives: Recognizing the research limitations of administrative and clinical databases, we designed SIC-IR as a research specific prospective registry to allow evaluation of patient care. We are initially focusing on the problems associated with infectious complications in the STICU. To gain user support, several clinical care components were added to SIC- IR to help with daily patient care activities in return for clinician manual data entry. System Description 132

133 We developed and copyrighted a STICU prototype infection registry and clinical information system called SIC-IR (Surgical Intensive Care Infection Registry). SIC-IR is a unique application specifically designed for researching infectious complications in the STICU. However, it was designed to easily allow addition of new modules to study other critical care issues and permit portability to other institutions with only minor modifications. In addition to its registry qualities, SIC-IR serves as a modified electronic medical record (EMR) for all STICU patients. The application is not yet a complete EMR, but performs key EMR aspects such as data display organization, data integration, and clinical decision support. SIC-IR provides functionality for creating initial patient histories and physicals, daily progress notes, and transfer of patient care documentation. In addition, SIC-IR automatically generates a daily rounding sheet which gives a clinical problem list as well as calculations of central line days, indwelling urinary catheter days, and ventilator days for each patient. The daily rounding sheet also includes five day trends for hematocrit, leukocyte counts, creatinine, and microbiology culture results (Figure 2). To the best of our knowledge, there is no existing STICU computerized application designed specifically for research use which also offers comprehensive daily clinical support. SIC-IR was created through collaboration of physicians and an information and computer scientist to provide an innovative research infrastructure and to integrate well into the process of daily patient care. It is carefully designed with a deliberate separation of its relational back-end and its friendly graphical user interface (GUI) front-end (Figure 3). Such a separation was intended to provide easy portability of SIC-IR by not 133

134 constraining it to a specific database management system. For example, its data repository can reside in any relational database management system such as Microsoft Access, Microsoft SQL Server, Oracle, or IBM DB2. The SIC-IR database is built using the relational model design principles which are the most proven and accepted data modeling techniques. The SIC-IR data model has normalized tables, well-defined data relationships, and built-in rules to automatically enforce data integrity constraints to ensure the accuracy of the data repository. The GUI front-end is developed using Microsoft Access forms, queries, reports, macros, and a significant amount of VBA (Visual Basic for Applications ) code to achieve all the functionality provided by SIC- IR. SIC-IR integrates data from multiple sources and uses three different methods, of varying reliability, to acquire data. The first, and most reliable method, is electronic entry via an automatic download from the hospital laboratory information system. Automatic data loading ensures data consistency and accuracy. SIC-IR is collecting over 100 clinical patient variables on a daily basis with 25% of these variables having an automatic electronic interface requiring no manual data entry. Collected variables include daily information regarding laboratories, microbiology results, vital signs, antibiotics, and the Centers for Disease Control defined infections. SIC-IR also accumulates information regarding indwelling urinary catheters, central lines, ventilators, use of steroids, blood product transfusions, and all Joint Commission core measures (e.g. venous thromboembolism prophylaxis, gastrointestinal bleeding prophylaxis, and head of bed elevation to 30º)[22]. The second method is manual entry with daily review by a member of the STICU team for completeness and correctness. For example, antibiotics 134

135 are manually entered by the STICU resident physicians from the medication administration record via drop down menus and then later reviewed by the STICU pharmacist. The third, and least reliable method, is manual data entry without subsequent review. For example, vital signs are entered by the resident physicians from the nurse s flow sheets, but are not subsequently verified for accuracy. Figure 4 is a schematic representation of these data acquisition methods along with their respective gold standards to where the data is located within our institution s paper chart. SIC-IR is HIPAA (Health Insurance Portability and Accountability Act) compliant in that it is multi-user with secure log-in and audit trail capabilities. The audit trail enables the tracing of any changes on any data element to the user who added, deleted, or modified the data element to a data field level. SIC-IR also has a regular backup and archiving schedule, and a restore and recovery procedure. RESULTS SIC-IR was validated in a three month prospective study which evaluated the accuracy of the data repository for future research studies and the improvements in clinical efficiency. Data which was electronically captured in the repository had accuracy rates of 100% (accuracy was defined as being both complete and correct). Data which was manually entered and then subsequently reviewed by a member of the STICU team had accuracy rates of 95%. However, data which was manually entered and not subsequently reviewed had accuracy rates as low as 70%, which is not different from the accuracy for the same data in the hand written paper chart that is currently used for research and for entering information into administrative databases. 135

136 A survey study of resident physician users demonstrated that SIC-IR integrated well into the daily patient care and was thought to improve overall quality of care. A time motion study of data entry established that utilizing SIC-IR saved four minutes of resident prerounding time per patient compared to the current standard operating procedure of looking up data on the hospital clinical information system and hand writing daily notes. When utilizing SIC-IR, prerounding time included the time it took the resident to review all data, evaluate the patient, enter data into the research repository, and complete patient care documentation. These four minutes of time improvement yielded over an hour and half time savings with a full STICU. SIC-IR proved to be significantly more accurate for data which was electronically captured or reviewed after entry, and as accurate as the paper chart for clinical data which was entered and not subsequently reviewed. With its quality data, which we expect will continue to improve with more electronic data capture; SIC-IR will serve as a useful prospective research registry. As significantly, SIC-IR was well accepted by the STICU team and improved resident efficiency - a key benefit in light of the Accreditation Council for Graduate Medical Education 80-hour work week mandate. CONCLUSIONS AND FUTURE DIRECTIONS OF SIC-IR Due to the research limitations of administrative and clinical systems, accurate research registries are urgently needed to support both research and evidence-based medicine practices. We developed SIC-IR first as a research specific registry to study infectious complications in critically ill patients and then as a comprehensive tool for clinical care support. SIC-IR was validated and proved to be an accurate registry which integrated well with daily STICU care. Consequently, multiple research projects are 136

137 underway using SIC-IR. Our research goals are to use the accurate data stored in SIC- IR s registry to create mathematical models which may help identify infectious causes of the systemic inflammatory response syndrome in STICU patients. A more efficient system for identifying infections may yield earlier appropriate treatment and decrease the morbidity and mortality attributed to infectious complications. SIC-IR is currently being used in our 27 bed Level 1 trauma STICU on a daily basis. During its first six months of use, SIC-IR has collected over 4000 patient-days of data on more than 700 patients. We are currently in the process of adding more electronic data acquisition components to SIC-IR to decrease manual data entry, increase accuracy, and continue the data integration process from the multiple clinical information systems in our institution. We are adding the capability to automate tracking patient vitals, ventilator settings, medications as well as adding a billing and documentation module for attending physicians use. We continue to make improvements to SIC-IR s GUI to maintain physician acceptance. With pay for performance and medical center outcome comparisons gaining momentum, research specific registries such as NSQIP and SIC-IR will continue to be developed to ensure accurate research and high quality outcome analysis. 137

138 Bibliography [1] Goldstein J. Evolution in Real Time: How Bacteria Beat Antibiotics. The Wall Street Journal June 8, [2] Krauss C. Bacteria that Strike Elderly Spead in Canadian Hopsitals. The New York Times August 9, [3] Press A. Deadly Germ is Becoming Wilder Threat. The New York Times December 2, [4] Jarvis WR. Selected aspects of the socioeconomic impact of nosocomial infections: morbidity, mortality, cost, and prevention.[see comment]. Infection Control & Hospital Epidemiology. 1996;17(8): [5] Hurr H, Hawley HB, Czachor JS, Markert RJ, McCarthy MC. APACHE II and ISS scores as predictors of nosocomial infections in trauma patients. American Journal of Infection Control. 1999;27(2): [6] Craven DE, Kunches LM, Lichtenberg DA, Kollisch NR, Barry MA, Heeren TC, et al. Nosocomial infection and fatality in medical and surgical intensive care unit patients. Archives of Internal Medicine. 1988;148(5): [7] Wallace WC, Cinat M, Gornick WB, Lekawa ME, Wilson SE. Nosocomial infections in the surgical intensive care unit: a difference between trauma and surgical patients. The American surgeon Oct;65(10): [8] Hoover L, Bochicchio GV, Napolitano LM, Joshi M, Bochicchio K, Meyer W, et al. Systemic inflammatory response syndrome and nosocomial infection in trauma. Journal of Trauma-Injury Infection & Critical Care. 2006;61(2):310-6; discussion

139 [9] Papia G, McLellan BA, El-Helou P, Louie M, Rachlis A, Szalai JP, et al. Infection in hospitalized trauma patients: incidence, risk factors, and complications. The Journal of trauma Nov;47(5): [10] 2008 National Patient Safety Goals: Hospital Program [cited; Available from: s.htm [11] Curtis JR, Cook DJ, Wall RJ, Angus DC, Bion J, Kacmarek R, et al. Intensive care unit quality improvement: a "how-to" guide for the interdisciplinary team.[see comment]. Critical Care Medicine. 2006;34(1): [12] Aronsky D, Haug PJ, Lagor C, Dean NC. Accuracy of administrative data for identifying patients with pneumonia. American Journal of Medical Quality. 2005;20(6): [13] Halpern N, Pastores S, Thaler H, Greestein R. Critical care medicine use and cost among Medicare beneficiaries : Major discrepancies between two United States federal Medicare databases. Critical Care Medicine. 2007;35(3): [14] Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc Sep-Oct;4(5): [15] Iezzoni LI. Assessing quality using administrative data. Ann Intern Med Oct 15;127(8 Pt 2): [16] Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information 139

140 systems. Implications for outcomes research.[see comment]. Annals of Internal Medicine. 1993;119(8): [17] Ward NS. The accuracy of clinical information systems. Journal of Critical Care. 2004;19(4): [18] Ward NS, Snyder JE, Ross S, Haze D, Levy MM. Comparison of a commercially available clinical information system with other methods of measuring critical care outcomes data. Journal of Critical Care. 2004;19(1):10-5. [19] Roberts RJ, Stockwell DC, Slonim AD. Discrepancies in administrative databases: implications for practice and research.[comment]. Critical Care Medicine. 2007;35(3): [20] Davis CL, Pierce JR, Henderson W, Spencer CD, Tyler C, Langberg R, et al. Assessment of the reliability of data collected for the department of veterans affairs national surgical quality improvement program. J Am Coll Surg Apr;204(4): [21] Khuri SF, Daley J, Henderson W, Barbour G, Lowry P, Irvin G, et al. The National Veterans Administration Surgical Risk Study: risk adjustment for the comparative assessment of the quality of surgical care.[see comment]. Journal of the American College of Surgeons. 1995;180(5): [22] JCAHO. Attributes of core measures performance measures and associated evaluation criteria. [cited; Available from: 140

141 Actual Patient Data What actually happened to the patient Patient Medical Record What is documented about the patient Coding and Billing Process What is captured about the patient Administrative Database What is archived about the patient Discordance between actual care and documented care Coding errors Coding bias Data reliability Figure 1: Clinical data movement from the actual patient to administrative databases and reasons for potential loss of data integrity. 141

142 Figure 2: Example of a SIC-IR STICU patient s Daily Rounding Sheet. 142

143 Clinician user PRESENTATION LAYER Microsoft Access forms and reports APPLICATION LAYER VBA Code Hospital Information Systems DATA LAYER Any relational database management system Figure 3: The architecture of SIC-IR and the movement of data and information to and from the clinician user 143

144 True clinical state of all STICU patients Data Type Daily labs and microbiology Medications Clinical data Data Gold Standard Lab information system Medication administration record (MAR) Nurse and respiratory flow sheets Method of Data Capture into SIC-IR ELECTRONIC MANUAL MANUAL SIC-IR Method of SIC-IR Data Validation Built-in data integrity rules Reviewed daily by the pharmacy staff Not reviewed Figure 4: Schematic representation of data entry from the true clinical state of the patient into SIC-IR. Data arrives into the registry from the patient record gold standards via three methods: 1) Electronic data transfer 2) Manual data entry with a daily review and correction by the pharmacy staff and 3) Manual data entry without a second review. STICU: Surgical and trauma intensive care unit SIC-IR: Surgical Intensive Care Infection Registry 144

145 Chapter 7 Validation of SIC-IR 145

146 VALIDATION OF SIC-IR : A MEDICAL INFORMATICS SYSTEM FOR INTENSIVE CARE UNIT RESEARCH, QUALITY OF CARE IMPROVEMENT, AND DAILY PATIENT CARE Joseph F. Golob Jr, M.D.*, Adam M.A. Fadlalla, Ph.D, Justin A. Kan BA*, Nilam P. Patel, Pharm.D, Charles J. Yowler, M.D., FACS*, Jeffrey A. Claridge, M.D. FACS* * Department of Surgery, MetroHealth Medical Center, Case Western Reserve University School of Medicine, Cleveland, OH Cleveland State University Department of Computer and Information Science, Cleveland, OH MetroHealth Medical Center, Department of Pharmacy, Cleveland, OH Article Type: Original Scientific Article Reprint Requests and Corresponding Author: Jeffrey A. Claridge, M.D. MetroHealth Medical Center Room H939, Hamann Bldg 2500 MetroHealth Drive Cleveland, OH Phone: Fax: jclaridge@metrohealth.org JA Claridge and the project described was supported by Grant Number 1KL2RR from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH) and NIH Roadmap for Medical Research, and its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. JA Claridge and the project described was also supported by the young investigators award given by the Surgical Infection Society. JF Golob is the 2008 American College of Surgeons Committee on Trauma Resident Paper Competition Clinical Winner. 146

147 ABSTRACT: BACKGROUND: We developed a prototype electronic clinical information system called the Surgical Intensive Care Infection Registry (SIC-IR), to prospectively study infectious complications and monitor quality of care improvement programs in the surgical and trauma intensive care unit (STICU). The objective of this study was to validate SIC-IR as a successful health information technology with an accurate clinical data repository. STUDY DESIGN: Utilizing the DeLone and McLean Model of Information Systems Success as a framework, we evaluated SIC-IR in a three month prospective crossover study of physician use in one of our two STICUs (SIC-IR unit vs. non SIC-IR unit). Three simultaneous research methodologies were employed: a user survey study, a pair of time-motion studies, and an accuracy study of SIC-IR s clinical data repository. RESULTS: The SIC-IR user survey results were positive for system reliability, graphical user interface, efficiency, and overall benefit to patient care. There was a significant decrease in pre-rounding time of nearly four minutes per patient on the SIC-IR unit compared to the non SIC-IR unit. The SIC-IR documentation and data archiving was accurate % of the time depending on the data entry method utilized. This accuracy was significantly improved compared to normal hand-written documentation on the non SIC-IR unit. CONCLUSIONS: SIC-IR proved to be a useful application both at an individual user and organizational level, and will serve as an accurate tool to conduct prospective research and monitor quality of care improvement programs. 147

148 ABBREVIATIONS AND ACRONYMS SIC-IR: Surgical Intensive Care Infection Registry STICU: surgical and trauma intensive care unit GUI: graphical user interface HIT: health information technology IS: information systems IRB: institutional review board 148

149 INTRODUCTION: The published limitations of clinical information systems and administrative databases[1-7] for conducting accurate outcomes research led our group to develop the Surgical Intensive Care Infection Registry (SIC-IR) to study infectious complications in the surgical and trauma intensive care unit (STICU). SIC-IR is a relational database application developed specifically for research by employing a graphical user interface (GUI) that utilizes a combination of structured data entry fields as well as free text fields which are completed by the physician treating the patient (see figures 1, 2, and 3 for GUI examples). This combination facilitate both ast data entry at the point of patient care as well as uncomplicated data validation and evaluation. SIC-IR prospectively collects over 100 clinical variables on all STICU patients with 25% of these variables imported electronically requiring no manual data entry. Figure 4 is a schematic representation of the three data entry methods utilized by SIC-IR s clinical data repository. The three methods include an electronic download from current hospital information systems, manual entry with subsequent daily data review, and manual entry without further data review. In addition to SIC-IR s research registry qualities, the application provides daily clinical care functionality for creating an initial patient history and physical, daily progress notes, procedure notes as well as transfer of patient care documentation. SIC-IR generates a daily rounding sheet which gives a clinical problem list as well as calculations of STICU-days, central venous catheter-days, indwelling urinary catheterdays, and ventilator-days for each patient. The daily rounding sheet also includes five day trends for hematocrit, leukocyte count, creatinine, and microbiology culture results. 149

150 After more than one year of research and development, SIC-IR was rolled-out into one of the two STICUs in our institution (SIC-IR unit). Our initial goal was to validate SIC-IR s registry as an accurate clinical data repository for future research and monitoring of quality of care initiatives. In addition, we wanted to verify that SIC-IR is a reliable, well integrated application to be used for daily patient care and welcomed by the STICU multidisciplinary team. Methods of health information technology (HIT) validation are currently an area of significant research. Both quantitative and qualitative methods are employed when studying HIT, and the exact variables to be researched for new systems are constantly evolving. Adhikari et al. published a review of intensive care unit HIT assessment which incorporated three phases: 1-safety and technical performance, 2-clinical effectiveness (for patients and/or physicians) and 3-outcomes in a heterogeneous population [8]. A review by Van Der Meijden et al. suggested using the well validated DeLone and McLean Model for Information System Success as a guide when studying HIT [9]. Our group decided to employ the DeLone and McLean Model since this model incorporates all the recommendations set forth by Adhikari, with the potential to yield a thorough quantitative and qualitative assessment of SIC-IR. The DeLone and McLean Model was first published in 1992 and was based on the theoretical and empirical information systems (IS) research conducted in the 1970 s and 1980 s [10]. Since then, more than 100 articles have used and validated this model to help define the dependent variables of IS success [11]. The original model evaluates five different IS dimensions which are schematically represented in figure 5. These dimensions include system quality, information quality, use/user satisfaction, individual 150

151 impact, and organizational impact. System quality is intended to evaluate the reliability, accessibility, and ease of use of the system. Information quality evaluates the accuracy (completeness and correctness) of the data stored within the application. Use and user satisfaction investigates the user approval and/or disapproval of the system. Finally, outcomes are measured through individual impact which appraises the effects of the system on its users, and organizational impact which evaluates influences the system has on the organization where it is being used [9-11]. Table 1 depicts the outcome measures evaluated during the validation of SIC-IR using the DeLone and McLean Model as a framework. Studying these qualitative and quantitative outcomes we hypothesized that SIC-IR would be a reliable, efficient, accepted, and accurate clinical registry for research and quality improvement initiatives, as well as a valuable addition for daily patient care. METHODS: The validation of SIC-IR occurred in a 748 bed tertiary academic medical center with two STICUs (27 total beds). Our institution is the only American College of Surgeons accredited Level I adult and pediatric trauma center in Cleveland, OH. In 2006, the STICUs treated over 1700 critically ill patients with 50%-65% being trauma patients. Our institution currently utilizes an outpatient electronic medical record, but lacks an inpatient component relying on the paper based chart for daily documentation of patient care. A three month (March 2007 May 2007) prospective validation trial was conducted using SIC-IR in one of two adjacent STICUs within our institution (SIC-IR unit-15 beds vs. Non SIC-IR unit-12 beds). Patients in both units are clinically similar and are cared for by a single STICU multidisciplinary team consisting of one surgical intensivist, a surgical fellow, one critical care pharmacist, and multiple resident 151

152 physicians. In order to research all the outcomes listed in table 1, three simultaneous studies were conducted: a resident use survey study, a pair of time-motion studies, and a data accuracy study. Resident Use Survey Study: Crossover Design On the first day of their STICU rotation, all residents were thoroughly trained in the use of SIC-IR and were given detailed instructions on documentation requirements for all critically ill patients. Residents were then selected at random to care for patients on either the SIC-IR unit or the non SIC-IR unit. Residents would spend the first half of their rotation (two weeks) on the randomly assigned unit. After two weeks, the residents would cross over to care for patients on the other unit. While on the SIC-IR unit, SIC-IR was used for all aspects of daily patient care including initial patient history and physical documentation as well as daily progress notes. The daily rounding sheet was printed and placed at the bed side to be accessible during both resident pre-rounds and formal attending rounds. SIC-IR was also used to generate a resident sign-out list to be used for transfer of STICU patient care between resident care givers. Residents on the non SIC- IR unit would perform daily patient tasks as they would normally do on other rotations including handwriting all documentation and utilizing the hospital s clinical information system to follow all laboratory and culture results. The non SIC-IR unit managed transfer of patient care with a Microsoft Excel spreadsheet which required daily updating of any new information. At the end of each two week block, residents were given a survey based on previously published work in the field [12]. The survey used a five point Likert scale with 1 signifying strong disagreement and 5 suggesting strong agreement with the survey 152

153 statement. The survey covered the resident assessment of SIC-IR s reliability, efficiency, user interface, amount of noneducational busy work, ease of documentation, sign-out preparation, sign-out quality, value of the daily rounding sheet, and overall SIC-IR value to patient care. The residents would complete the survey comparing their SIC-IR experience to previous rotations and/or care given on the non SIC-IR unit. Time-Motion Study: Pre-rounds and Attending Rounds Two different time-motion studies were conducted simultaneously during the three month study. Three days per week, a single resident was selected at random to be followed during their daily evaluation and documentation of STICU patients (prerounds). This randomization of residents alternated between the SIC-IR unit and non SIC-IR unit. Residents were timed during each patient they evaluated. During this time, residents would review all patient clinical data, examine the patient, and document their findings and treatment plan in a daily progress note. The time keeping was started when the resident initiated evaluation of the patient and concluded when daily documentation was complete. On the SIC-IR unit, this time included the time required for daily data entry for both research and patient care. On the non SIC-IR unit this included the time to complete all daily hand-written documentation. The second time-motion study occurred two days per week when a single research team member (JAK) attended formal STICU patient rounds with the entire multidisciplinary team. During this time, each patient was presented to the attending surgical intensivist by the resident, all clinical data was reviewed, and a formal treatment plan was created. Total time was recorded for all patients evaluated on the SIC-IR and non SIC-IR units. In addition, unanswered clinical questions were recorded. An 153

154 unanswered question was defined as those clinical questions asked on rounds regarding specific clinical aspects of patient care which was unknown to the residents, requiring them to search for the answer. These unanswered questions were classified into clinical categories including: general labs, microbiology, past medical history, current diagnosis / injuries, or procedures. Accuracy Study: The final research methodology involved evaluation of data accuracy (completeness and correctness) within the SIC-IR clinical data repository as well as an accuracy comparison of daily documentation between the SIC-IR unit and the non SIC- IR unit. Since all documentation is generated by SIC-IR for patients on the SIC-IR unit, evaluation of the data contained in the daily progress notes served as a surrogate for evaluating the accuracy of the data stored within the database itself. Three days per week, ten random patient charts were reviewed; five on the SIC-IR unit and five on the non SIC-IR unit. Daily progress notes within these charts were prospectively reviewed for four different clinical attributes dealing with infectious complications. All residents were blinded to these attributes which included microbiology culture results, daily maximum body temperature, antibiotics (indication, day of treatment, and stop date), and ventilator settings. The information within the daily progress notes were compared to the data gold standards listed in figure 4 and a percent agreement was calculated (percent agreement = number accurate / total reviewed). To be considered accurate, the data within the daily note must have been both complete (not missing) and correct. The percent agreement between the daily progress notes in the SIC-IR unit was then compared to the percent agreement of the progress notes from the non SIC-IR unit. 154

155 SPSS statistical software (SPSS Inc, Chicago, IL) was used for data analysis in all portions of this study. Categorical variables are expressed as percents and compared using Chi square or Fisher s exact test where appropriate. Continuous variables are expressed as means ± standard error of the mean and compared using Student s T-tests after evaluation with Levene s Test to determine if equal variance can be assumed. A p- value of 0.05 was chosen for statistical significance. A 3 month evaluation was chosen to allow enough data collection to conform to an 80% power analysis calculation with an alpha-error of 0.05 to access accuracy differences of 5% between the SIC-IR and the non SIC-IR unit. All aspects of this validation study were reviewed and approved by our institutional review board (IRB) and informed consent was obtained from all resident users of SIC-IR. RESULTS: Over 2300 patient-days of data were evaluated between the SIC-IR and non SIC- IR units during this validation study. A total of twenty-four residents used SIC-IR during the three month evaluation. Fifty-four percent were males and 58% were surgical residents. Nineteen of the twenty-four residents were part of the STICU team. The additional five residents were cross cover residents who had limited use of SIC-IR and therefore were not included in the survey study. Table 2 lists additional demographics of all the resident users. The remaining results within this section will be described in relation to the DeLone and McLean Model components. System Quality Over the three month study, SIC-IR had 6.5 hours (0.3%) of down time used for normal database maintenance and repair of system malfunctions. The electronic data 155

156 download from the institution s clinical laboratory system failed three times (3%). Although this download failure did affect the accuracy of the SIC-IR unit daily progress notes, the error was identified and corrected in each instance to keep the overall data stored within the database tables complete and correct. Figure 6 displays the resident survey results. Question 1 assesses SIC-IR reliability which was defined as the availability and functionality of the system. Reliability had a mean Likert score of 3.5 with a mode of 4. 56% of the resident users agreed that SIC-IR was a reliable system. The resident s mean Likert score of the SIC-IR user interface was 4.2 with a mode of 4 and is displayed in the response to Question 2. No residents thought the user interface was difficult to use. Information Quality A total of 191 daily patient notes were reviewed in the SIC-IR unit during the data accuracy portion of this study. Culture results, which were electronically captured, showed 100% accuracy. Antibiotics, which were manually entered and then checked by the pharmacy staff, were 95% accurate. Data which was manually entered and not reviewed resulted in a significant decrease in accuracy. Maximum body temperature was accurate 82% of the time (99% complete and 82% correct). In other words, residents manually entered the patient s maximal body temperature incorrectly or not at all 18% of the time compared to the value recorded on the nursing flow sheet. Finally, the five required manually entered fields for ventilator settings were 74% accurate (95% complete and 77% correct). Twenty six percent of patients on the ventilator had either incomplete or incorrect ventilator settings documented in SIC-IR compared to the respiratory therapy flow sheets. Table 3 summarizes all of the accuracy results. 156

157 User Satisfaction User satisfaction was qualitatively studied with several survey questions. Figure 6-questions 3, 4, and 5 summarize resident user satisfaction. Question 3 addresses busy work when using SIC-IR compared to previous experiences. Busy work was defined as noneducational work such as looking up patient lab results and recopying them to patient documentation. Two thirds of the residents felt SIC-IR decreased the busy work in everyday patient care. Question 4 attempted to evaluate the ease of documentation (daily notes and history and physicals) utilizing the SIC-IR. The mean and mode Likert score were both 3 with one-third of the residents being indifferent to using SIC-IR for documentation compared to handwriting notes. Finally, question 5 addressed the overall patient care benefits offered by SIC-IR. Seventy eight percent of the residents felt SIC-IR improved the patient care they offered to critically ill patients (mean Likert score = 3.8, mode = 4). Individual Impacts Individual impacts were evaluated using the resident survey data as well as the time-motion study of pre-rounding efficiency. Questions 6 and 7 (Figure 6) deal with SIC-IR s impact on resident transfer of patient care via generation of sign-out lists. Seventy eight percent of the residents felt the sign-out generation was easier using SIC- IR and 55% felt the quality of the sign-out was improved compared to previous experiences. The pre-rounding time-motion study evaluated 66 resident pre-rounding experiences on the SIC-IR unit and 61 on the non SIC-IR unit. The SIC-IR unit had a statistically significant decrease in mean pre-rounding time per patient compared to the 157

158 non SIC-IR unit (14:43 ± 0:32 vs. 18:27 ± 0:54; p<0.001). This resulted in a 3 minute and 44 second time saving per patient during pre-rounds in the SIC-IR unit. Organizational Impact Organizational impact was evaluated by team rounding efficiency using a timemotion study and survey results. In addition, a comparison of resident documentation between the SIC-IR unit and the non SIC-IR unit was evaluated using the accuracy study results. Using the time motion study, no significant differences were noted in unanswered questions or mean rounding time per patient. The mean rounding time in the SIC-IR unit was 8 minutes and 28 seconds per patient compared to 8 minutes and 9 seconds per patient on the non SIC-IR unit (p=0.704). However, qualitative evaluation of efficiency yielded 65% of the residents feeling that rounds on the SIC-IR unit were more efficient (figure 6-question 8). Question 9 assessed the impact of the daily rounding sheets on patient care and was viewed as very useful by 88% of the residents (mean Likert = 4.3, mode = 4). Using data collected during the accuracy study, SIC-IR s impact on documentation was evaluated. The results displayed in table 3 demonstrate that the two units had well matched patient characteristics. There was no difference between the SIC- IR unit and the non SIC-IR unit for patients who had cultures over the previous five days (p=0.964), patients on antibiotics (p=0.819), and patients requiring mechanical ventilation (p=0.579). Overall, the number of patients who had completed notes on the chart was not statistically different between the two units, but the SIC-IR unit did show a 50% reduction in the number of missing notes (18 on the non SIC-IR unit vs. 9 on the SIC-IR unit). 158

159 Evaluating documented culture results, 100% of the notes on the SIC-IR unit had a complete listing of recent cultures compared to only 29% on the non SIC-IR unit (p<0.001). Of the results which were documented, 100% of the SIC-IR notes were correct and 87% of the notes in the non SIC-IR unit were correct (p=0.05). Combining the completeness and correctness for culture results, the SIC-IR note was 100% accurate compared to the non SIC-IR note s 20% accuracy (p<0.001). Daily progress notes on the SIC-IR unit listed patient maximal daily body temperature 99% of the time compared to 93% of the time on the non SIC-IR unit (p=0.001). However no statistically significant difference was noted in accuracy as temperature was 82% accurate on the SIC-IR unit and 76% accurate on the non SIC-IR unit compared to the documented values on the nursing flow sheet (p=0.178). A list of the current antibiotics on daily documentation was not statistically different between the two units (SIC-IR unit = 95%, non SIC-IR unit = 93%; p=0.464). However, differences were noted in documentation of antibiotic treatment components such as antibiotic indication, antibiotic day of treatment, as well as treatment stop date. Antibiotic indications were listed two times more frequently on the SIC-IR unit compared to the non SIC-IR unit. Likewise, the day of treatment was found five times more often, and the antibiotic stop date was identified eight times more often on the SIC-IR unit. A complete list of ventilator settings was documented on 95% of the SIC-IR unit notes but only 80% in the non SIC-IR unit (p<0.001). However, similar to body temperature results, manual entry without further review resulted in a significant decrease in accuracy. These settings were 74% accurate on the SIC-IR unit documentation compared to 70% on the non SIC-IR unit. 159

160 DISCUSSION Nosocomial infections result in significant morbidity and mortality to STICU patients. Improvements in diagnosis, treatment, and prevention of these infectious complications will require accurate clinical research and subsequent quality of care improvement programs. Current research in this area is often retrospective or uses hospital administrative databases. These administrative databases were not intended for research purposes, and their completeness and correctness have been characterized as poor[4, 13, 14]. Even databases developed specifically for clinical use are not perfect, and their accuracy depends on the type of information contained in the database, the person entering the data, and the timing of data entry[3, 15]. We developed SIC-IR first as a research tool and then as a clinical care adjunct. Using the DeLone and McLean Model of Information System Success as a guide, the goals of this study were to show SIC-IR is an accurate research registry which offers daily patient care improvements and integrates well into the STICU environment. Survey results from resident users showed SIC-IR to be a reliable and robust clinical application. SIC-IR had minimal downtime and isolated data download malfunctions. These data download malfunctions were easily identified and corrected to insure database integrity and accuracy of 100%. The information quality of antibiotics (data which was manually entered and subsequently reviewed for completeness and correctness) had an accuracy of 95%. However, data which was manually entered and not reviewed lost significant accuracy (daily maximal temperature = 82% and ventilator settings = 74%). Interestingly, there was no difference in progress note accuracy in manually entered data between the SIC-IR 160

161 and non SIC-IR units. This suggests that manual transfer of data from the clinical gold standards via keyboard entry or handwriting have similar error rates. Inaccuracies in handwritten resident-generated daily progress notes has previously been shown to be >60% for some specific entries such as vascular lines, medications, and patient weights[16]. An attempt to increase accuracy of this data with the use a computerized entry system (personalized digital assistant) was undertaken, but with only modest improvements, none of which were statistically significant [16]. Therefore, our results of approximately 20-25% inaccuracies in documentation were not unexpected with either keyboard entry or hand written data capture. Currently, to improve data accuracy for research and documentation, we are adding more electronic data acquisition to SIC-IR including data for body temperature and ventilator settings. SIC-IR s graphical user interface was deemed by the residents as easy to use and navigate. The residents felt that there was less noneducational busy work when using SIC-IR, but were indifferent between handwriting documentation vs. inputting the information into SIC-IR. Physician use has been shown to be one of the biggest obstacles to HIT implementation, so the positive user satisfaction we observed was a pleasing outcome [17-19]. The positive user satisfaction also stemmed from many of the individual impacts SIC-IR offered. Residents felt that the SIC-IR generated resident sign-out for transfer of patient care was easy to create and was of higher quality than current methods. Also, and more importantly to the busy resident in an 80 hour / work week mandate, SIC-IR saved almost four minutes per patient for daily documentation. This resulted in a 56 minute time savings when the SIC-IR unit had a full capacity of patients. Time efficiency has 161

162 rarely been shown to be possible with physician data entry into electronic health records, and Poissant et al. felt time efficiency should not be a dependent variable for assessment of HIT applications [20]. Knowing that user time is one of the most important aspects to the success of HIT [21], the significant time savings in our study will increase physician buy-in as well as improve user compliance. SIC-IR also offered organizational impacts to the multidisciplinary STICU team. Although there was no statistically significant quantitative improvement in time efficiency or unanswered questions of formal attending rounds using SIC-IR, the residents felt qualitatively that rounding efficiency was improved in the SIC-IR unit (figure 6-question 8). This may be a result of the residents feeling more prepared with a better understanding of the patients they were treating. In addition to improved legibility, there was tremendous improvements in daily documentation with computer generated SIC-IR notes. SIC-IR documentation demonstrated significant enhancement in accuracy of culture results, as well as improvements in antibiotic documentation. CONCLUSION: Improvements in STICU care for nosocomial infections will require prospective research from an accurate registry. Using the DeLone and McLean model as a guide we were able to show that SIC-IR was a successful clinical application in system quality, information quality, user satisfaction, individual impacts and organizational impacts. SIC-IR has the capability to achieve database accuracies of % compared to gold standards when data is electronically downloaded or reviewed daily. Manually entered data which was not reviewed after entry loses substantial accuracy, but this accuracy can be improved with a movement to more electronically downloaded data. SIC-IR 162

163 integrated well with the STICU team and offered many advantages to the residents and the team. Overall, SIC-IR proved to be reliable, efficient, and accepted, and will be an accurate clinical registry from which to conduct multiple prospective research studies and monitor quality improvement programs. 163

164 FIGURE LEGENDS: Figure 1: Clinical data SIC-IR dashboard. This is an example of one of SIC-IR s daily data displays used to assist physicians with clinical care. The dashboard lists the patient s medications as listed in the pharmacy information system, most recent laboratory studies with trends (electronically downloaded from the hospitals clinical laboratory system), as well as recent microbiology results. Figure 2: Structured SIC-IR data entry form. This is an example of SIC-IR s structured data entry fields used to obtain daily information such as the Joint Commission intensive care unit core measures (e.g. deep venous thromboembolism prophylaxis) and the National Healthcare Safety Network infection related research data (e.g. presence and location of a central venous access catheter). Figure 3: Free text SIC-IR data entry form. This is an example of the unstructured / free text fields used by physicians to capture initial history and physical information. Figure 4: Schematic representation of data entry from the true clinical state of the patient into SIC-IR. Data arrives into the registry from the patient record gold standards via three methods: 1) Electronic data transfer 2) Manual data entry with a daily review and 3) Manual data entry without a second review. This figure has been modified from a previous publication regarding the development of SIC-IR (The American Journal of Medical Quality, in press) STICU: Surgical and trauma intensive care unit 164

165 Figure 5: The DeLone and McLean Model of Information System Success. Adopted from: DeLone W, McLean, ER. Information system success: The quest for the dependent variable. Information Systems Research. 1992;3(1):60-95 Figure 6: SIC-IR resident use survey study results. * Bimodal lowest mode displayed. Reliable always available when needed and functioned well; busy work - sign-out list updating, patient data gathering, ext; documentation daily notes, initial history and physicals, and procedure notes; sign-out quality assessment of continuity of care and resident to resident communication 165

166 DeLone and McLean Model Domain System Quality Outcome Measured Reliability of SIC-IR Validation Method Used Resident use survey study and system downtime analysis Resident use survey study Ease of use of SIC-IR Information Quality Data completion and correctness within SIC-IR Data accuracy study Noneducational busy work when using SIC-IR User Satisfaction Ease of daily documentation when using SIC-IR Resident use survey study Impression of overall patient care when using SIC-IR Prerounding efficiency Time-motion study Individual Impact Ease of sign-out generation Quality of sign-out Resident use survey study Rounding efficiency Time-motion study Impression of SIC-IR rounds Organizational Impact Impression of the SIC-IR Resident use survey study daily rounding sheet Daily documentation improvements Data accuracy study Table 1: Outcomes and methods used to validate SIC-IR utilizing the DeLone and McLean Model as a framework. Thirteen different outcomes were measured using three different research methodologies following the domains of the DeLone and McLean Model as a guide. 166

167 Total number of resident users 24 Number of males (% of total resident users) 13 (54%) Surgery residents (% of total resident users) 14 (58%) Emergency department residents (% of total resident users) 5 (21%) Anesthesia residents (% of total resident users) 3 (13%) Other residents (% of total resident users) 2 (8%) Number of STICU team residents (% of total resident users) 19 (79%) Number of cross cover residents-not surveyed (% total resident users) 5 (21%) Number of surveys returned from STICU team residents 18/19 (95%) Table 2: Resident users of SIC-IR. 95% of the STICU team residents who participated in the evaluation of SIC-IR use returned surveys. 167

168 SIC-IR Unit Non SIC-IR P- value (N=191) Unit (N=189) Completed notes on the chart (%) 182 (95%) 171 (91%) CULTURES RESULTS ON PROGRESS NOTES % of patients with cultures 54% 54% % of patients with complete listing 100% 29% <0.001 % of patients with correct results 100% 87% % of patients with accurate results 100% 20% <0.001 TEMPERATURE MAX (TMax) ON PROGRESS NOTES % of patients with TMax completed 99% 93% % of patients with TMax correct 82% 81% % of patients with TMax accurate 82% 76% ANTIBIOTICS (ABX) ON PROGRESS NOTES % of patients on antibiotics 59% 58% % of patients with accurate Abx 95% 93% % of patients with indication listed 95% 57% <0.001 % of patients with day of Rx listed 95% 19% <0.001 % of patients with stop date listed 95% 12% <0.001 VENTILATOR SETTINGS ON PROGRESS NOTES % of patients on the ventilator 70% 67% % of patients with settings completed 95% 80% <0.001 % of patients with setting correct 77% 87% % of patients with setting accurate 74% 70% Table 3: Comparison of progress note accuracy between the SIC-IR unit and the non SIC-IR unit. Rx = treatment 168

169 BIBLIOGRAPHY: [1] Aronsky D, Haug PJ, Lagor C, Dean NC. Accuracy of administrative data for identifying patients with pneumonia. American Journal of Medical Quality. 2005;20(6): [2] Halpern N, Pastores S, Thaler H, Greestein R. Critical care medicine use and cost among Medicare beneficiaries : Major discrepancies between two United States federal Medicare databases. Critical Care Medicine. 2007;35(3): [3] Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc Sep-Oct;4(5): [4] Iezzoni LI. Assessing quality using administrative data. Ann Intern Med Oct 15;127(8 Pt 2): [5] Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research.[see comment]. Annals of Internal Medicine. 1993;119(8): [6] Ward NS. The accuracy of clinical information systems. Journal of Critical Care. 2004;19(4): [7] Ward NS, Snyder JE, Ross S, Haze D, Levy MM. Comparison of a commercially available clinical information system with other methods of measuring critical care outcomes data. Journal of Critical Care. 2004;19(1):10-5. [8] Adhikari N, Lapinsky SE. Medical informatics in the intensive care unit: overview of technology assessment. Journal of Critical Care. 2003;18(1):

170 [9] Van Der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc May-Jun;10(3): [10] DeLone W, McLean, ER. Information system success: The quest for the dependent variable. Information Systems Research. 1992;3(1): [11] DeLone W, McLean, ER. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. Journal of Management Information Systems. 2003;19(4):9-30. [12] Van Eaton EG, Horvath KD, Lober WB, Rossini AJ, Pellegrini CA. A randomized, controlled trial evaluating the impact of a computerized rounding and signout system on continuity of care and resident work hours. Journal of the American College of Surgeons. 2005;200(4): [13] Black N, Payne M. Directory of clinical databases: improving and promoting their use.[see comment]. Quality & Safety in Health Care. 2003;12(5): [14] Linder JA, Bates DW, Williams DH, Connolly MA, Middleton B. Acute infections in primary care: accuracy of electronic diagnoses and electronic antibiotic prescribing. J Am Med Inform Assoc Jan-Feb;13(1):61-6. [15] Stein HD, Nadkarni P, Erdos J, Miller PL. Exploring the degree of concordance of coded and textual data in answering clinical queries from a clinical data repository. J Am Med Inform Assoc Jan-Feb;7(1): [16] Carroll AE, Tarczy-Hornoch P, O'Reilly E, Christakis DA. The effect of point-ofcare personal digital assistant use on resident documentation discrepancies. Pediatrics Mar;113(3 Pt 1):

171 [17] Dansky KH, Gamm LD, Vasey JJ, Barsukiewicz CK. Electronic medical records: are physicians ready? Journal of healthcare management / American College of Healthcare Executives Nov-Dec;44(6):440-54; discussion [18] Hier DB, Rothschild A, LeMaistre A, Keeler J. Differing faculty and housestaff acceptance of an electronic health record. International journal of medical informatics Aug;74(7-8): [19] van Ginneken AM. The computerized patient record: balancing effort and benefit. International journal of medical informatics Jun;65(2): [20] Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc Sep-Oct;12(5): [21] Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidencebased medicine a reality. J Am Med Inform Assoc Nov-Dec;10(6):

172 Figure 1: Clinical data SIC-IR dashboard 172

173 Figure 2: Structured SIC-IR data entry form 173

174 Figure 3: Free text SIC-IR data entry form 174

175 True clinical state of all STICU patients Data Type Daily labs and microbiology Medications Clinical data Data Gold Standard Clinical lab system Medication administration record (MAR) Nurse and respiratory flow sheets Method of SIC-IR Entry ELECTRONIC MANUAL (reviewed by pharmacy) MANUAL (not reviewed) SIC-IR Figure 4: Schematic representation of data entry from the true clinical state of the patient into SIC-IR 175

176 System Quality Use Individual Impact Organizational Impact Information Quality User Satisfaction Figure 5: The DeLone and McLean Model of Information System Success 176

177 Resident Likert Survey Results Following the DeLone and McLean Model (N=18) Organizational Impact Individual Impact User Satisfaction System Quality Question 9 (Mean=4.3; Mode=4) The Daily Rounding Sheet was useful Question 8 (Mean=3.7; Mode=4) Rounding efficiency was improved with SIC-IR Question 7 (Mean=3.7; Mode=3*) Sign-out quality was better with SIC-IR Question 6 (Mean=4.1; Mode=4) Sign-out generation was easier with SIC-IR Question 5 (Mean=3.8; Mode=4) Overall patient care was better with SIC-IR Question 4 (Mean=3.4; Mode=3) Documentation was easier with SIC-IR Question 3 (Mean=3.7; Mode=4) There is less busy work with SIC-IR Question 2 (Mean=4.2; Mode=4) SIC-IR user interface was easy to use Question 1 (Mean=3.5; Mode=4) SIC-IR is a reliable system * Bimodal - lowest displayed 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Agree (4,5) Neutral (3) Disagree (1,2) Percent Responding Figure 6: SIC-IR resident use survey study results. 177

178 Chapter 8 Monitoring infections in the ICU 178

179 WHO IS MONITORING YOUR INFECTIONS: SHOULDN T YOU BE? Jeffrey A. Claridge, M.D.*, Joseph F. Golob, Jr., M.D.*, Adam M.A. Fadlalla, Ph.D., Beth M. D Amico, BS*, Joel R. Peerless, M.D.*, Charles J. Yowler, M.D.*, and Mark A. Malangoni, M.D.* *Department of Surgery, MetroHealth Medical Center, Case Western Reserve University School of Medicine Cleveland, OH Department of Computer and Information Sciences, Cleveland State University: Cleveland OH Corresponding Author and Reprints Jeffrey A. Claridge, M.D. MetroHealth Medical Center Room H939, Hamann Bldg 2500 MetroHealth Drive Cleveland, OH Phone: Fax: jclaridge@metrohealth.org This work and J. A. Claridge was supported in part by Grant Number 1KL2RR from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. This work was also supported by the Surgical Infection Society through the young investigators grant which was awarded to J. A. Claridge in

180 Background: In the era of pay for performance and outcome comparisons among institutions, it is imperative to have reliable and accurate surveillance methodology for monitoring infectious complications. The current monitoring standard often involves a combination of prospective and retrospective analysis by trained infection control (IC) teams. We have developed a medical informatics application, the Surgical Intensive Care - Infection Registry (SIC-IR), to assist with infection surveillance. The objectives of this study were to 1) evaluate for differences in data gathered between the current IC practices and SIC-IR and 2) determine which method has the best sensitivity and specificity for identifying ventilator associated pneumonia (VAP). Methods: A prospective analysis was conducted in two surgical and trauma intensive care units (STICU) at a level I trauma center (Unit 1-8 months, Unit 2-4 months). Data were simultaneously collected by the SIC-IR system at the point of patient care and by IC utilizing multiple different administrative and clinical modalities. Data collected by both systems included patient-days, ventilator-days, central line-days, number of VAPs, and number of catheter related blood steam infections (CR-BSIs). VAPs and CR-BSIs were classified using the Centers for Disease Control definitions. VAPs were individually analyzed and true infections were defined by a physician panel blinded to methodology of surveillance. Using these true infections as a reference standard, sensitivity and specificity for both SIC-IR and IC were determined. Results: A total of 769 patients were evaluated by both surveillance systems. There were statistical differences between the median number of patient days/month and ventilatordays/month when IC was compared to SIC-IR. There was no difference in the rates of CR-BSI/1000 central line-days per month, however VAP rates were significantly 180

181 different between the two surveillance methodologies (SIC-IR: 14.8/1000 ventilator-days, IC: 8.4/1000 ventilator-days; p=0.008). The physician panel identified 40 patients (5%) who had 43 VAPs. SIC-IR identified 39 and IC documented 22 of the 40 patients with VAP. SIC-IR had a sensitivity and specificity of 97% and 100%, respectively for identifying VAP. This compared to an IC sensitivity of 56% and a specificity of 99%. Conclusions: Utilizing SIC-IR at the point of patient care by a multidisciplinary STICU team offers more accurate infection surveillance with a high sensitivity and specificity. This can be accomplished without additional resources and engages the physicians treating the patient. 181

182 BACKGROUND Surveillance of nosocomial infections has traditionally been carried out by trained infection control (IC) professionals in order to supply timely and accurate data about infectious complications, as well as to provide feedback about the efficacy of infectionrelated patient care practices [1]. The rationale for tracking hospital acquired infections (HAI) as a means to reduce overall infection rates can be traced to the Center for Disease Control (CDC) landmark trial, the Study on the Efficacy of Nosocomial Infection Control (SENIC). The SENIC study concluded that nearly one-third of nosocomial infections could be prevented by infection surveillance and control programs [2, 3]. In the current environment of patient outcome comparisons among institutions, pay for performance calculations, and national efforts to establish quality improvement practices, surveillance of HAI remains particularly relevant. Currently, IC teams use various combinations of prospective and retrospective methods to monitor HAI. The methodology that is chosen determines efficacy, cost, and resource expenditure. The accuracy of IC reporting has been shown to depend upon both the timing and frequency of monitoring [4]. Prospective daily review of all patient records, while considered to be the gold standard, is both time-consuming and costly [2]. Other documented methods of surveillance include retrospective and/or prospective review of targeted patients records, such as those with positive cultures or those with an intensive care unit stay greater than five days [2]. The success of all of these surveillance methodologies rely on thorough health care documentation [3, 5, 6]. We have designed and implemented the Surgical Intensive Care-Infection Registry (SIC-IR), which has been validated to be an accurate and reliable medical 182

183 informatics program [7, 8]. SIC-IR is a real-time data acquisition research and clinical tool that is utilized at the point of care to assist in caring for critically ill patients. In addition to its research registry qualities, SIC-IR serves as an electronic medical record for all STICU patients. The purpose of this study was to identify differences in infection-related data collected by SIC-IR and IC, as well as determine which of these methods had the best sensitivity and specificity for identifying ventilator associated pneumonia (VAP). We hypothesized that the SIC-IR system would be more accurate in data collection and infection identification when compared to IC due. METHODS A twelve month prospective analysis was conducted in two surgical and trauma intensive care units (STICU) at a level I trauma center (Unit 1 8 months of data collection; Unit 2 4 months of data collection). Data simultaneously collected by both the SIC-IR system and IC included patient-days, ventilator-days, central line-days, number of VAPs, and number of catheter-related bloodstream infections (CR-BSI). SIC-IR is a research specific prospective registry, designed with input from a multidisciplinary team of surgical intensivists, surgical residents, pharmacists, and computer scientists, that facilitates the evaluation of infectious complications in the surgical and trauma intensive care unit (STICU). SIC-IR collects over 100 clinical variables daily on each STICU patient, including components implicated in infectious complications, such as patient demographics (age, sex, race), vital statistics, laboratory values, current antibiotic treatment, prior and current infectious complications, comorbidities, and impact of time and interventions (e.g. length of time in the STICU, 183

184 time to tracheostomy, and surgical interventions). SIC-IR is integrated with the hospital s laboratory information system as well as the medication administration record for automatic data loading to ensure registry consistency and accuracy. SIC-IR also accumulates information regarding indwelling urinary catheters, central venous access devices, ventilator requirements, use of steroids, blood product transfusions, and all Joint Commission intensive care unit core measures (e.g. venous thromboembolism prophylaxis, gastrointestinal bleeding prophylaxis, and head of bed elevation to 30º) [9]. The SIC-IR system provides functionality for creating patient admission histories and physicals, daily progress notes, procedure notes, and transfer of patient care documentation. The SIC-IR system utilized a graphical user interface to obtain the infection information daily at the point of patient care, with reporting completed by physicians treating the patient (See Figures 1 & 2 for examples of data entry forms). Infectionrelated data was reviewed daily by the STICU pharmacist, who was present during daily rounds. In this study, a SIC-IR patient-day was recorded when either an admission history and physical or a daily progress note was completed for a particular patient. If the treating physician responded yes to daily mandatory questions regarding requirement for mechanical ventilation or the presence of a central venous catheter, a SIC-IR ventilator-day or central line-day was recorded, respectively. In addition, ventilator and central line data were updated throughout the day as physicians completed mandatory SIC-IR procedure notes when intubating or placing central venous catheters. The IC team employed multiple different clinical and administrative modalities to obtain data. Patient-days were derived from the hospital s administrative midnight 184

185 census. Ventilator-days were based on respiratory therapy billing records defined as having a ventilator in the patient s room. Central line-days were calculated from once a day rounds by a designated STICU nurse recording presence or absence of a central line. IC surveillance practices involve one of two IC trained nurses prospectively monitoring all patients charts with positive microbiology culture results, as well as daily data interrogation for any patients staying in the STICU for five or more days. VAPs and CR-BSI were defined using the CDC definitions [10]. Bronchoalveolar lavage (BAL) specimens were considered positive with growth at a level of 10 4 organisms per ml. For this study, BAL was performed based on review of chest radiograph, assessment of respiratory secretions, fever, and leukocytosis. A semiquantitative rolling technique was used for central venous catheter tip cultures, and was considered positive with the growth of 15 colony forming units of a single type of bacteria. Central line and ventilator use, as well as VAP and CR-BSI rates were calculated using the National Healthcare Safety Network (NHSN) surveillance methodology [1]. Rate calculations are performed by normalizing the number of VAP and CR-BSI to 1000 ventilator and central line-days, respectively. Our data is compared to the NHSN data using the surgical intensive care units as a reference because our units contain less than 80% trauma patients. Data was analyzed using SPSS 15.0 (SPSS Inc., Chicago IL). Numbers are expressed as percentiles (median=50 th percentile) and compared between SIC-IR and IC using Wilcoxon Signed Ranks Test for two related samples. Discrepancies in reported cases of VAP between IC and SIC-IR were individually analyzed and true infections were identified by a physician panel that was blinded to surveillance methodology. The 185

186 physician panel consisted of five practicing surgical intensivists at our institution. Using these true infections as a reference standard, sensitivity and specificity for both SIC-IR and IC were determined. RESULTS This analysis evaluated 769 consecutive patients admitted to the STICU over the 12 month period. Table 1 summarizes SIC-IR versus IC NHSN data. SIC-IR documented significantly more patient-days per month compared to IC (p=0.003). However, IC recorded significantly more ventilator-days per month (p=0.009). There was no significant difference in the number of central line-days between the two systems. SIC-IR data calculated significantly less ventilator and central line utilization per month. At 56%, the SIC-IR report of ventilator use was 11% less than IC, and placed SIC-IR results between the 75 th and 90 th percentile on the NHSN scale for surgical ICUs. In contrast, the IC report of 67% ventilator use per month would be greater than the 90 th percentile nationally. SIC-IR calculated 52% central line utilization while IC calculated 57%, placing both SIC-IR and IC results between the 25 th and 50 th percentile for surgical ICUs [1]. SIC-IR recorded significantly more episodes of VAP per month compared to IC (p=0.009), and when converted to a VAP rate per 1000 ventilator-days per month, SIC-IR had nearly double the VAP rate of IC (p=0.008). Compared to surgical ICUs nationally, SIC-IR reported a VAP rate greater than the 90 th percentile on the NHSN scale, while IC reported a rate between the 75 th and 90 th percentile [1]. There were no significant differences between SIC-IR and IC with respect to number of CR-BSI per month, nor CR-BSI rate. The rate of CR-BSI calculated from both SIC-IR and IC data is between 186

187 the 0 th and 10 th percentile on the NHSN scale [1]. Further comparisons of CR-BSI were not made due to the low incidence. A second analysis of central line utilization, ventilator utilization and VAP rates were performed to eliminate the differences in the methodology of the way IC and SIC- IR determines ventilator days and patient days. Central line use and ventilator use between SIC-IR and IC were normalized to the same administrative data denominator (midnight census). In addition, VAP rates were also normalized to administrative data (ventilator billed-days). These results are shown in Table 2. This normalization analysis removed the statistical difference between central venous catheter utilization; however ventilator utilization and VAP rates remain statistically different between the two surveillance methodologies. The two reporting systems agreed on the diagnosis of VAP in 21 patients. SIC-IR detected 18 patients with VAP that infection control failed to report. In contrast, infection control discovered only 1 patient with VAP that SIC-IR overlooked. After investigating all episodes of VAP identified by IC and SIC-IR, a physician panel identified 40 patients who accounted for 43 episodes of VAP. SIC-IR had a sensitivity of 97% and specificity of 100% for identifying patients with VAP (Table 3). IC sensitivity and specificity were 56% and 99%, respectively (Table 4). Of the 18 patients that only SIC-IR reported as having VAP, 14 had positive BAL cultures at the level of 10 4 organisms. The remaining four patients that IC did not record were those who had had a positive respiratory secretion cultures and had been treated with an antibiotic regimen for VAP with a documented response to therapy. DISCUSSION 187

188 The results of this study demonstrate that our current practice of monitoring hospital acquired infections can be improved by utilizing a computerized registry in real time to prospectively track all patients in the STICU. The SIC-IR system documented significantly more patient-days per month, but at the same time reported fewer ventilatordays and less ventilator and central line utilization when matched against infection control. Of 40 patients in this analysis diagnosed with ventilator associated pneumonia, SIC-IR detected 39 while infection control recognized only 22, giving SIC-IR a higher sensitivity and specificity. When converted to a VAP rate to allow for both intra- and inter- facility comparisons, the VAP rate obtained from SIC-IR was nearly double that of IC (14.8/1000 ventilator-days vs. 8.4/1000 ventilator days). Previous investigations of infection surveillance have highlighted key issues underscored by the results of our analysis. Retrospective examination of all patient records tends to be of low yield, with prior studies reporting sensitivities of 74 and 88%, while requiring significant time expenditure [5, 11, 12]. However, concentrating efforts by examining only discharge summaries or singling out records containing positive microbiology cultures does not improve accuracy of retrospective review [5, 13]. Prospective examination of patients with positive cultures is generally accepted as a better case-identification method, with investigators reporting sensitivities ranging from 75 to 91% [4, 14, 15]. Emori et al. specifically reviewed the accuracy of National Nosocomial Infection Surveillance (NNIS) personnel in finding infections based on site, and found the sensitivities to vary widely, from 68% for pneumonia to 85% for bloodstream infections [6]. Reviewing patient data on a more frequent basis was 188

189 associated with increased sensitivities in prior reviews [4], but evaluating only those patients with positive cultures did not improve results [16]. Consistently, models with the greatest sensitivity and specificity for identifying hospital acquired infections have incorporated computer assistance in the surveillance system. Evans et al. reported a computer monitoring method was able to identify 90% of HAI, while infection control teams found only 76% [16]. Similarly, Bouam et al. reported the use of an automated system missed only 9% of infections, compared to the 41% missed by infection control [13]. This study confirmed that monitoring practices by infection control personnel reliably have high specificity, but lower sensitivity for detecting hospital acquired infection. That is, it is rare for infection control teams to falsely identify a patient as having disease; however, it is common for teams to fail to detect patients who are truly infected [6]. Nationally, only 90% of hospital acquired infections actually have an organism cultured [16]. The ten percent not defined by positive cultures are based solely upon physician clinical judgment, and may be one important group of patients ultimately missed by infection control surveys [16]. Several factors may account for discrepancies noted between SIC-IR and IC reporting of VAP. Regardless of whether a detection cutoff of 10 4 or 10 5 organisms is used to define a positive BAL culture, IC missed 14 VAPs based on positive BAL results. Even at the more stringent cutoff of 10 5, SIC-IR would have documented VAP in 8 patients that IC failed to identify. Furthermore, there may have been some misinterpretation on the part of IC personnel regarding BAL lab results that reported normal oral flora, as oral flora are not normal organisms to culture from the lungs. 189

190 Identification of patients with infectious complications requires more than simply looking for positive microbiology cultures; indeed, four patients in this analysis were diagnosed and treated for VAP based solely on physician judgment and respiratory secretion cultures. Thus it seems a system that requires input from physicians, who are ultimately responsible for integrating information and formulating clinical judgments, is a better surveillance model for identifying patients with infection. Infection control reports rely on passive surveillance, for example, by respiratory therapists and nursing staff whose primary role is not infection surveillance [2, 17]. Heavily lab-based and record-based IC practices may not utilize ward rounds and routine discussion with caregivers to monitor HAI [2]. A limitation of this study surrounds our assumption that all patients in the population not identified by SIC-IR or IC as having an infection were actually disease free. Five percent of patients in this analysis were diagnosed with ventilator associated pneumonia by either SIC-IR or IC, or both. We feel that the combined efforts of infection control and SIC-IR captured virtually every case of diagnosed VAP. Another limitation of this study is that resource utilization was not specifically analyzed. This could be evaluated by looking at the number of FTE s utilized and time spent collecting data and then evaluating it. Although we have no quantitative data, we are confident that SIC-IR utilizes fewer additional resources. Multiple people are utilized to capture IC data and retrospective review requires time. IC data was only available two to three months after the fact. In contrast, SIC-IR captures data in real time as part of the routine work flow to assist with the delivery of patient care. 190

191 In summary, the results of this analysis highlight the potential for improved infection surveillance in the STICU through the use of a prospective registry to track all patients at the point of care. SIC-IR proved to have a high sensitivity for detecting hospital acquired infection, while at the same time being readily integrated into physician practice. As the CDC has stated, data collection should never be an endpoint in itself; rather, it should be used to achieve the goal of decreasing HAI [17]. However, to truly decrease HAI and make comparison among hospitals the true infection rate must be documented. The utilization of SIC-IR allows for true standardization of surveillance methodology using real time data evaluation 24 hours a day, seven days a week. SIC-IR uses fewer resources and provides more accurate infection surveillance than standard IC practices. 191

192 Bibliography [1] Edwards JR, Peterson KD, Andrus ML, Tolson JS, Goulding JS, Dudeck MA, et al. National Healthcare Safety Network (NHSN) Report, data summary for 2006, issued June American Journal of Infection Control. 2007;35(5): [2] Pottinger JM, Herwaldt LA, Peri TM. Basics of surveillance--an overview. Infection Control & Hospital Epidemiology. 1997;18(7): [3] Scheckler WE, Brimhall D, Buck AS, Farr BM, Friedman C, Garibaldi RA, et al. Requirements for infrastructure and essential activities of infection control and epidemiology in hospitals: a consensus panel report. Society for Healthcare Epidemiology of America.[see comment]. Infection Control & Hospital Epidemiology. 1998;19(2): [4] Delgado-Rodriguez M, Gomez-Ortega A, Sierra A, Dierssen T, Llorca J, Sillero- Arenas M. The effect of frequency of chart review on the sensitivity of nosocomial infection surveillance in general surgery. Infection Control & Hospital Epidemiology. 1999;20(3): [5] Belio-Blasco C, Torres-Fernandez-Gil MA, Echeverria-Echarri JL, Gomez-Lopez LI. Evaluation of two retrospective active surveillance methods for the detection of nosocomial infection in surgical patients. Infection Control & Hospital Epidemiology. 2000;21(1):24-7. [6] Emori TG, Edwards JR, Culver DH, Sartor C, Stroud LA, Gaunt EE, et al. Accuracy of reporting nosocomial infections in intensive-care-unit patients to the National Nosocomial Infections Surveillance System: a pilot study.[erratum appears in 192

193 Infect Control Hosp Epidemiol 1998 Jul;19(7):479]. Infection Control & Hospital Epidemiology. 1998;19(5): [7] Fadlalla AMA, Golob JF, Claridge JA. The Surgical Intensive Care - Infecton Registry (SIC-IR): A research registry with daily clinical support capabilities.. The American Journal of Medical Quality. In press. [8] Golob JF, Fadlalla AMA, Kan JA, Patel NP, Yowler CJ, Claridge JA. Validation of SIC-IR : A medical informatics system for intensive care unit research, qualtiy of care improvement, and daily patient care. Journal of the American College of Surgeons. In Press. [9] JCAHO. Attributes of core measures performance measures and associated evaluation criteria. [cited; Available from: [10] CDC. CDC Definitions of Nosocomial infections. [cited; Available from: [11] Haley RW, Culver DH, White JW, Morgan WM, Emori TG, Munn VP, et al. The efficacy of infection surveillance and control programs in preventing nosocomial infections in US hospitals. American Journal of Epidemiology. 1985;121(2): [12] Haley RW, Schaberg DR, McClish DK, Quade D, Crossley KB, Culver DH, et al. The accuracy of retrospective chart review in measuring nosocomial infection rates. Results of validation studies in pilot hospitals. American Journal of Epidemiology. 1980;111(5): [13] Bouam S, Girou E, Brun-Buisson C, Karadimas H, Lepage E. An intranet-based automated system for the surveillance of nosocomial infections: prospective validation 193

194 compared with physicians' self-reports.[see comment]. Infection Control & Hospital Epidemiology. 2003;24(1):51-5. [14] Bouletreau A, Dettenkofer M, Forster DH, Babikir R, Hauer T, Schulgen G, et al. Comparison of effectiveness and required time of two surveillance methods in intensive care patients. Journal of Hospital Infection. 1999;41(4): [15] Delgado-Rodriguez M, Gomez-Ortega A, Llorca J, Lecuona M, Dierssen T, Sillero-Arenas M, et al. Nosocomial infection, indices of intrinsic infection risk, and inhospital mortality in general surgery. Journal of Hospital Infection. 1999;41(3): [16] Evans RS, Larsen RA, Burke JP, Gardner RM, Meier FA, Jacobson JA, et al. Computer surveillance of hospital-acquired infections and antibiotic use. JAMA. 1986;256(8): [17] CDC. Outline for Healthcare-Associated Infections Surveillance; 2006 April

195 SIC-IR INFECTION CONTROL Percentiles Percentiles median 25 th -75 th median 25 th -75 th p- value Patient-days Central line-days Ventilator-days Central line utilization 52% 57% (NHSN percentile) (25 th -50 th 45%-53% ) (25 th -50 th ) 52%-59% Ventilator utilization 56% 67% (NHSN percentile) (75 th -90 th 54%-59% ) (>90 th ) 61%-71% Number of VAPs Number of CR-BSI VAP rate (NHSN percentile) (>90 th ) (75 th -90 th ) CR-BSI rate 0 0 (NHSN percentile) (0-10 th 0-0 ) (0-10 th ) Table 1: Summary of SIC-IR versus IC reporting of NHSN data. All values are reported per month NHSN= National Healthcare Safety Network Utilization and rates are calculated utilizing the NHSN equations: Utilization = (number of device days / total patient days) * 100% Rate = (number of infections / number of device days) *

196 SIC-IR (normalized to admininistrative data) INFECTION CONTROL Percentiles Percentiles median 25 th -75 th median 25 th -75 th p- value Central line utilization 58% 57% (NHSN percentile) (25 th -50 th 51%-60% ) (25 th -50 th ) 52%-59% Ventilator utilization 63% 67% (NHSN percentile) (>90 th 61%-65% ) (>90 th ) 61%-71% VAP Rate (NHSN percentile) (>90 th ) (75 th -90 th ) Table 2: Summary of SIC-IR versus IC reporting of NHSN data normalized to administrative data. All values are reported per month NHSN= National Healthcare Safety Network Utilization and rates are calculated utilizing the NHSN equations normalized to administrative data: Utilization = number of device days / midnight census patient-days * 100% Rate = (number of VAPs / number of ventilator days per administrative data) *

197 SIC-IR reported cases VAP Actual cases of ventilator associated pneumonia (VAP) VAP no VAP VAP reported no VAP reported cases 729 Total pts: 769 VAP Table 3: Two by two table for SIC-IR reported cases of ventilator associated pneumonia used to calculate sensitivity and specificity 197

198 IC reported cases of VAP Actual cases of ventilator associated pneumonia (VAP) VAP no VAP VAP reported no VAP reported cases 729 Total pts: 769 VAP Table 4: Two by two table for IC reported cases of ventilator associated pneumonia used to calculate sensitivity and specificity 198

199 Figure 1: SIC-IR daily information and pharmacy. Note mandatory prompts regarding presence of a central line and mechanical ventilation. 199

200 Figure 2: SIC-IR daily information and pharmacy. Note infection, date of confirmation, organism cultured, and antibiotic treatment record. 200

201 Chapter 9 SIC-IR documents sicker patients 201

202 SIC-IR (SURGICAL INTENSIVE CARE INFECTION REGISTRY) DOCUMENTS SICKER PATIENTS Joseph F. Golob Jr MD *, Adam M.A. Fadlalla PhD, Joel R. Peerless MD *, Charles J. Yowler MD *, Jeffrey A. Claridge MD * MetroHealth Medical Center: Department of Surgery Case Western Reserve University School of Medicine Cleveland, OH Cleveland State University Cleveland, OH JA Claridge is supported by the National Institutes of Health, National Institute of Child Health and Human Development, Multidisciplinary Clinical Research Career Development Programs Grant 1KL2RR This work was also partially funded by a grant awarded to JA Claridge from the Surgical Infection Society. Corresponding author and reprints: Jeffrey A. Claridge MD MetroHealth Medical Center 2500 MetroHealth Drive Cleveland, OH jclaridge@metrohealth.org 202

203 INTRODUCTION Hospital administrative databases are often criticized for inaccuracy when utilized for research and quality improvement projects [1-7]. As data moves from the patient to the medical record, through the coding/billing department, and into the administrative database, critical information is often overlooked or absent. A significant loss of data occurs from the discordance between valid patient diagnoses and what is actually documented in the medical record [8]. This discordance leads to missing information which creates erroneousness administrative databases. Weak administrative systems can have significant impact on two major endpoints of administrative data: hospital reimbursement and public reporting (Figure 1). Administrative data is used by the Centers for Medicare and Medicaid Service (CMS) as well private insurance companies to drive hospital reimbursement for inpatient care. More recently, administrative data is being collected by the private sector and used for hospital and physician comparisons. Raw administrative data, as well as data evaluated by CMS and the public, is then utilized for local and national quality of care initiatives, safety improvement projects, pay for performance calculations, and health care reform. The seven key potions of administrative data which CMS utilizes for hospital reimbursement include: primary diagnosis, secondary diagnoses, procedures, complications, comorbidities, age (< 17 or 17 years-old), and discharge status (alive or dead). This data is then used to identify one of 745 Medicare severity-diagnosis related groups (MS-DRG) [9]. Each MS-DRG is associated with a single reimbursement amount and a single DRG-relative weight (DRG-RW). The DRG-RW is a geographic insurance factor based solely on the cost of resources required to care for patients within the 203

204 specific MS-DRG. The DRG-RW does not evaluate individual patient characteristics or severity of illness in the calculation of resource cost. However, in general, sicker patients have a higher DRG-RW. For example, resource utilization is much higher for an end stage renal patient on hemodialysis who undergoes splenectomy after a motor vehicle crash compared to a patient with normal kidney function undergoing the same procedure for the same indication. The end stage renal patient would be assigned a different MS- DRG with a higher DRG-RW. This DRG-RW then enters into the hospital reimbursement equation which is DRG-RW multiplied by the hospital base rate equals the amount compensated to the hospital (Figure 2). The hospital base rate is also known as the case-mix index. This index is derived from the sum of all the DRG-RWs obtained in the entire hospital divided by the total number of Medicare and Medicaid patients treated. Since DRG-RW is linked to severity of illness through resource utilization, this case-mix index represents the mean severity of illness of all Medicare and Medicaid patients treated by the hospital. Thus, four ways to improve hospital reimbursement include: decreasing resource utilization, decreasing hospital length of stay, increasing the hospital base rate (case-mix index), and increasing the individual patient s DRG-RW [10]. An increase in the DRG-RW and the case-mix index can be achieved by improving the documentation within the medical record to accurately capture all patient diagnoses and procedures rendered. The second major end-point of administrative data is public reporting where companies are using data to compare hospitals and physicians. A computer and an internet service provider is all that is required to see how a hospital performs on a local 204

205 and national level. Multiple companies on the World Wide Web offer various hospital and physician comparisons utilizing diverse profiling methodologies. The transparency of these profiling methodologies is not always clear. Devastating effects may result from an inaccurate hospital or physician profile. No matter how well designed the comparative methodology utilized; the outcomes of these comparisons can only reflect the severity of illness and level of care provided based on medical record documentation [11]. Currently in our hospital, all inpatient documentation is performed using handwritten notes in a paper-based medical record. To improve documentation in our two surgical and trauma intensive care units (STICUs), we developed and are currently using the SIC-IR system (Surgical Intensive Care Infection Registry). SIC-IR is an information technology application designed specifically for researching infectious complications as well as assisting in daily patient care by offering physicians clinical decision support tools. SIC-IR is used for all physician STICU documentation including daily progress notes, initial history and physicals, and procedure notes. SIC-IR prospectively collects over 100 clinical variables daily on each STICU patient. This data is collected at the point of patient care with one-third of these variables being directly imported from various hospital information systems (e.g. the pharmacy and laboratory clinical information systems) requiring no human data entry [12, 13]. Resident physicians utilize SIC-IR s graphical user interface (GUI) to enter the remaining patient care data using a mixture of structured and free-text entry methods. The residents patient care plans are then reviewed by the attending intensivist who makes addendums and cosigns the documentation before the typed note is placed in the medical record. 205

206 An attending billing and documentation module (B/DM) was developed and integrated into the SIC-IR GUI and data repository. Using data captured in SIC-IR s clinical warehouse, the B/DM application was trained to suggest and capture 70 different patient diagnoses (ICD-9 codes) and six STICU procedures (CPT codes) to ensure complete and accurate documentation of the patient s condition (Table 1). After being accepted by the attending intensivist, these ICD-9 and CPT codes are then placed in the patient s daily medical record documentation as well as the STICU intensivist s professional bill to assist the coding/billing department. Within the 70 diagnoses taught to SIC-IR are nine of the nineteen most commonly missed comorbid conditions and complications which CMS recognizes as having the biggest potential to change MS- DRG status (Table 2) [14]. The objective of this study was to determine if the SIC-IR B/DM can improve STICU documentation by increasing the number of ICD-9 and CPT codes captured by our hospital s administrative databases. Our hypothesis was that using the SIC-IR B/DM to accurately capture the patient s condition would increase the documented severity of illness, documented risk of mortality, DRG-RW, and thus the DRG estimated hospital reimbursement. METHODS A six-month (July 2007 December 2007), prospective, two phase study of the SIC-IR B/DM was conducted in two STICUs located within a regional American College of Surgeons Level I trauma center. The two STICUs consisted of 27 total beds, and cared for a mixture of critically ill trauma and surgical patients (~150 patients / month). All patients are cared for by a multidisciplinary team consisting of a single attending 206

207 intensivist, a critical care pharmacist, a surgical critical care fellow, multiple resident physicians, and multiple nurses. The SIC-IR B/DM was not used during the first three months of the study (phase 1), which required the attending intensivists to daily identify and manually document all ICD-9 diagnoses and procedures for each STICU patient. This process consisted of handwritten professional billing and medical record documentation. The professional billing documentation was performed utilizing a pre-printed template which required the attending physician to select the daily evaluation and management (E/M) code with or without a modifier, followed by manual identification of patient diagnoses using a checkbox list of common STICU patient ICD-9 codes (Figure 3). In addition, any STICU procedures were listed with appropriate CPT codes. These handwritten bills were then turned into the inpatient billing department where data was entered into an administrative database. The phase 1 attending intensivist medical record documentation was performed using handwritten patient notes (with or without a note template) in addition to cosigning the required daily resident documentation. During phase 1, all resident chart documentation was performed using the normal SIC-IR features which had been used and validated in our two STICUs [13]. The phase 1 SIC-IR system did not have the ability to archive STICU procedures or suggest and document patient diagnoses. After patient discharge, the entire medical record was sent to the coding department who reviewed the chart and identified all ICD-9 diagnoses and other necessary information to select the appropriate MS-DRG which was archived in an administrative database. 207

208 The SIC-IR B/DM was initiated on October 1, 2007 and signified the start of phase 2. During phase 2, the SIC-IR B/DM created all daily professional billing documentation for each STICU patient as well as offered the attending electronic medical record documentation capabilities. The SIC-IR B/DM would electronically process all patient data in the repository to determine if any of the 70 diagnoses taught to SIC-IR fit the patient s clinical condition. The ICD-9 codes would be suggested to attending intensivist in an easy to use GUI. The GUI included patient demographics, referring surgical attending, list of patient diagnoses and injuries, suggested SIC-IR diagnoses, and any STICU procedures performed by the attending or the housestaff (Figure 4). In addition, a button could be used to display laboratory results with trends, microbiologic culture results, current medications, and confirmed infectious complications (Figure 5). The intensivists were not required to use SIC-IR for medical record documentation during phase 2, but they were required to use the professional billing capabilities. This required the attending to choose an E/M code with or without a modifier, review and accept the SIC-IR diagnoses, and review and accept any STICU procedures archived by the system. The bill was then printed and forward to the billing department as was done in phase 1. If the attending did not bill the patient, or removed a diagnosis or procedure, a reason why was required using a drop down menu of options presented in the GUI. If the attending wanted to add a diagnosis or procedure not captured by SIC-IR, this could be done by manually entering the ICD-9 or CPT code or selecting it from a list within the GUI (Figure 6). If the attending elected to use the medical record documentation, they would manually enter their physical exam, assessment, and plan on the GUI. The printed attending note would include their typed 208

209 information along with a list of accepted ICD-9 codes for that day and a summary of all diagnoses during the patient s STICU stay. The resident documentation experienced minimal changes during phase 2. The only additions were several procedure note templates used to document STICU procedures, and a list of daily identified SIC-IR diagnoses which were displayed on the GUI and printed on the daily note under a Current Problem List. The residents were instructed to review this list and make necessary modifications and comments in their patient assessment and plans. Table 3 is a comparison summary of attending and resident daily documentation procedures during phase 1 and 2. Data was obtained from SIC-IR and our hospitals administrative databases during phase 1 and 2. Two different groups of data were collected during this evaluation. One group of data consisted of variables used to compare the clinical characteristics of the phase 1 and phase 2 patient populations. The second group of data was comprised of outcome variables used to test our hypothesis. Clinical characteristics used to compare the two phases included: age, gender, admitting service, STICU patient-days, hospital length of stay, indwelling urinary catheter-days, central venous catheter-days, antibioticdays, ventilator-days, infectious complications, and mortality. Outcome variables included: number of CPT and ICD-9 codes per patient at discharge, DRG-RW, DRG geographic mean length of stay, DRG estimated reimbursement, all patient refineddiagnosis related group relative weight (APR-DRG-RW), APR-DRG severity of illness, and APR-DRG risk of mortality. The APR data was obtained from proprietary benchmarking software leased by our institution from the 3M Company. The software is designed to help healthcare 209

210 organizations meet their quality improvement and reporting needs [15]. The APR-DRG system modifies the traditional MS-DRG system by adding risk-adjustment methodologies. Unlike the MS-DRG system, the APR-DRG system evaluates all patients (not Medicare / Medicaid patients only), treats age as a linear variable, and includes four classes of severity of illness and risk of mortality. Each patient is given a severity of illness and risk of mortality score from 1 to 4 (1=minor; 2=moderate; 3=major; 4=severe). Similar to the MS-DRG methodology, the APR-DRG system relies solely on medical record documentation. SPSS version 15 (SPSS Inc; Chicago, IL) was used for all statistical analysis. All categorical data is expressed as percentages and compared with Pearson Chi-square. Non-categorical data is expressed as mean ± standard error of the mean and compared with Student s t-test after evaluation with Levene s Test to determine if equal or unequal variance should be utilized. The APR severity of illness and risk of mortality are ordinal variables and were compared using non-parametric tests (Mann-Whitney U). Statistical significance was set at a p-value This study was reviewed and approved by the MetroHealth Institutional Review Board. This trial was registered with ClinicalTrials.gov (NCT ). RESULTS There were 436 patients evaluated in phase 1 with 2202 STICU patient-days of data collected. Phase 2 had 378 patients with 2308 STICU patient-days. Phase 2 patients were slightly older. There were no differences in gender or admitting service (Table 4). Phase 2 had a one day longer mean STICU length of stay, but mean hospital length of stay was not different. There was no difference in indwelling urinary catheter-days, 210

211 central line-days, and antibiotic-days. Mean ventilator-days were statistically one day longer in phase 2 compared to phase 1. There was no difference in the mean number of infections per patient or mortality between the two groups (Table 5). A total of 2308 potential patient bills (one bill per STICU patient-day) should have been created by the SIC-IR B/DM. SIC-IR created 2206 or 96% of these potential patient bills. The 4% lost was a result of an application programming error which removed several patients admission bills. Of the bills generated by SIC-IR, 99% were reconciled; defined as being reviewed by the attending intensivist and either accepted or rejected. Of the 2206 generated SIC-IR bills, 63% were forwarded to the billing department leaving 37% not billed (Table 6). The most common E/M code of those bills forwarded to the billing department was a Level 3 subsequent visit (43%). A critical care code (99219) was used 30% of the time. However, the number of critical care codes did significantly increased per administrative bill between phase 1 and phase 2 (0.99 ± 0.8 vs. 1.5 ± 0.2; p=0.005). Most common reasons for not forwarding a bill included lack of critical care attending documentation or evaluated by the fellow only (46%), patient to be discharged from the STICU (25%), or the STICU attending did not bill for the patient since they were admitted at night and evaluated by the on call trauma attending (15%). The SIC-IR B/DM suggested 6657 ICD-9 diagnoses during phase 2. 99% of these suggested diagnoses were accepted by the attending intensivist and were placed on daily resident documentation and the attending professional billing sheet. Thirty-six of the 128 rejected ICD-9 codes were because the patient did not have the suggested diagnosis. During phase 2 the SIC-IR B/DM captured 207 STICU procedures through the use of the incorporated procedure note templates. 145 of these procedures were 211

212 actually billed. The main reason for not billing the other 62 was because an attending was not present for the procedure. Investigating the ICD-9 and DRG outcomes within the administrative database demonstrated the SIC-IR B/DM had significant improvements in all areas which are displayed in Table 7. The number of ICD-9 codes increased by 3 per patient. The DRG- RW statistically increased by 26% above phase 1 from 3.8 to 4.8. The DRG geographic mean LOS was increased from 8 days to nearly 10 days. The DRG reimbursement was increased by 21% between phase 1 and phase 2 which equated into a $6,665 increase per patient bill. This resulted in a nearly 2.4 million dollar increase in estimated DRG reimbursement during phase 2 despite fewer patients evaluated and treated. The all patient refined data obtained from the 3M software also increased significantly during phase 2. The phase 2 APR-DRG-RW increased 20% over phase 1. The APR-severity of illness and risk of mortality significantly increased by 7% and 14% respectively. DISCUSSION Weak administrative databases can have a significant impact beyond hospital reimbursement and physician comparisons. Administrative databases are often used to obtain research populations by querying for specific ICD-9 or DRG codes. Administrative databases created from poor medical record documentation can result inaccurate patient populations and possibly inappropriate research conclusions [1, 16]. In addition, with pay for performance and a cost-based DRG system on the horizon, documentation and coding will become even more paramount. Our data demonstrates that improving the discordance between a critically ill patient s actual condition and what is documented in the medical record and on 212

213 professional billing can significantly improve data archived in administrative databases. The implementation of a home grown electronic billing and documentation module with the capabilities to suggest patient diagnoses significantly improved our two STICUs diagnostic code capture. The SIC-IR B/DM resulted in approximately three more captured ICD-9 codes at discharge, a 26% increase in DRG-RW, a two day increase in DRG estimated hospital length of stay, and nearly a $7,000 per patient increase in hospital reimbursement. The system also increased our documented severity of illness by 7% and risk of mortality by 14%, results which will allow for better risk adjusted hospital and physician comparisons. Grogan et al in a 2004 article published in the Journal of the American College of Surgeons demonstrated that a handwritten daily progress note template including common post surgical diagnoses significantly improved documentation. The 12 month study demonstrated significant improvements in number of captured ICD-9 codes, DRG- RW and risk of mortality when using the note template. The template was not mandatory and these improvements occurred with only a 57% template use [17]. Another study by Heistein et al at the Ohio State University burn unit combined informatics and note templates in this before-and-after study. They investigated a computerized history and physical as well as a progress note template created by Clinicomp, a medical informatics company in San Diego, CA. The notes were completed with check boxes and drop down menus as well as free text typing and contained appropriate data to allow the highest level of billing. The group observed significant billing improvements with the use of the Clinicomp electronic notes and concluded that the note templates allowed for a more accurate record of clinical services [18]. The development of the SIC-IR B/DM 213

214 combined both note templates and informatics with the addition of clinical decision support to suggest patient diagnoses based on accurate prospective data obtained at the patient s bedside on a daily basis. Our data confirms that documentation and coding can have significant impact on hospital reimbursement. Many surgeons question these improvements because they do not see any direct compensation as a result of their increased work for documentation. The Balance Budget Act of 1997 required CMS to link physician relative value units to DRG-RW. Thus, the higher the DRG-RW of the patient population treated, the higher the individual physician s RVUs [10]. In addition, academic physicians are becoming increasingly dependent on hospital support due to declining professional reimbursement. Increasing hospital revenue has the potential to directly impact the individual physician s salary [10]. Also computerized billing has been shown to improve payments for professional trauma care as well as decreasing insurance denials, both of which have direct impact on individual physicians [19]. Limitations of this study include the differences noted in the clinical data comparisons of the phase 1 and phase 2 populations. Phase 2 patients were slightly older, stayed one extra day in the STICU and remained on the ventilator one extra day. However hospital length of stay and mortality was not different. We do not feel these differences can account for such significant changes in our outcome variables. Also, hospital length of stay is not used in any of the MS-DRG or APR-DRG calculations. This study also lacked a washout period after the initiation of the SIC-IR B/DM. We used an intention to treat analysis meaning patients discharged early in the initiation of the B/DM had both handwritten bills and SIC-IR bills. These patients were analyzed in 214

215 the phase 2 population. We were unable to allow a washout period because nurses trained in 3M documentation improvement practices were starting in our institution and would cause two documentation changes occurring simultaneously. In conclusion, the SIC-IR B/DM introduced significant improvements in medical record documentation resulting in improved administrative databases. Utilizing medical informatics techniques to accurately assist physician documentation and billing can significantly increase the documented illness burden and hospital reimbursement of a large STICU population. This will permit more accurate risk-adjustment of patient populations to allow fair comparisons between individual physicians and hospital systems. 215

216 FIGURE LEGEND Figure 1: Endpoints of hospital administrative data. Two major endpoints of administrative data include the Centers for Medicare and Medicaid Services and public reporting. Figure 2: The hospital reimbursement equation. DRG-RW is multiplied by the hospital base rate to give the amount compensated to the hospital. The hospital base rate is also known as the case-mix index which is the mean DRG-RW of all Medicare and Medicaid patients treated in the hospital. DRG-RW = diagnosis related group relative weight Figure 3: Handwritten billing documentation template. The attending intensivist would manually complete this form daily for each patient they wished to professionally bill. The attending was required to select the appropriate E/M code as well as choose diagnoses and enter procedures. This form was then forwarded to the inpatient billing department. E/M = evaluation and management Figure 4: SIC-IR B/DM graphical user interface. This is a screen shot from the SIC- IR B/DM main data entry page used by the attending physicians. Here is where the intensivist would review suggested diagnoses and procedures completed, choose an E/M code, and enter medical record documentation. SIC-IR = Surgical Intensive Care Infection Registry B/DM = Billing and Documentation Module E/M = evaluation and management 216

217 Figure 5: SIC-IR B/DM data dashboard. This dashboard is accessible from the B/DM and lists patients laboratory studies with trends, microbiology results, infections which have been confirmed on daily rounds, and current medications. SIC-IR = Surgical Intensive Care Infection Registry B/DM = Billing and Documentation Module Figure 6: Graphical user interface for selecting additional ICD-9 codes. This form lists common intensive care unit diagnoses to assist attendings with choosing the appropriate ICD-9 codes. 217

218 Bibliography [1] Aronsky D, Haug PJ, Lagor C, Dean NC. Accuracy of administrative data for identifying patients with pneumonia. American Journal of Medical Quality. 2005;20(6): [2] Halpern N, Pastores S, Thaler H, Greestein R. Critical care medicine use and cost among Medicare beneficiaries : Major discrepancies between two United States federal Medicare databases. Critical Care Medicine. 2007;35(3): [3] Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc Sep-Oct;4(5): [4] Iezzoni LI. Assessing quality using administrative data. Ann Intern Med Oct 15;127(8 Pt 2): [5] Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research.[see comment]. Annals of Internal Medicine. 1993;119(8): [6] Ward NS. The accuracy of clinical information systems. Journal of Critical Care. 2004;19(4): [7] Ward NS, Snyder JE, Ross S, Haze D, Levy MM. Comparison of a commercially available clinical information system with other methods of measuring critical care outcomes data. Journal of Critical Care. 2004;19(1):10-5. [8] Roberts RJ, Stockwell DC, Slonim AD. Discrepancies in administrative databases: implications for practice and research.[comment]. Critical Care Medicine. 2007;35(3): [9] CMS. IMPROVING MEDICARE'S HOSPITAL INPATIENT PROSPECTIVE PAYMENT SYSTEM 2007 [cited 2/23/2008]; Available from: [10] Tarantino D. Making the most of DRGs. Physician Executive. 2002;28(6):50-2. [11] Weygandt P, Gaspar K, Gerhart S. Fitting PQRI into the Emergin CMS Quality Puzzle. Focus: Newsletter of the American College of Medical Quality. Winter [12] Fadlalla AMA, Golob JF, Claridge JA. The Surgical Intensive Care - Infecton Registry (SIC-IR): A research registry with daily clinical support capabilities.. The American Journal of Medical Quality. In press. [13] Golob JF, Fadlalla AMA, Kan JA, Patel NP, Yowler CJ, Claridge JA. Validation of SIC-IR : A medical informatics system for intensive care unit research, qualtiy of care improvement, and daily patient care. Journal of the American College of Surgeons. In Press. [14] Schmidt K, Stegman M, Magani R. DRG Desk Reference: The ultimate resource for improving the new MS-DRG assignment practices. USA [15] 3M. 3M Health information Systems [cited 2/25/2008]; Available from: [16] Howard AE, Courtney-Shapiro C, Kelso LA, Goltz M, Morris PE. Comparison of 3 methods of detecting acute respiratory distress syndrome: clinical screening, chart review, and diagnostic coding. American Journal of Critical Care. 2004;13(1):

219 [17] Grogan EL, Speroff T, Deppen SA, Roumie CL, Elasy TA, Dittus RS, et al. Improving documentation of patient acuity level using a progress note template. J Am Coll Surg Sep;199(3): [18] Heistein JB, Coffey RA, Buchele BA, Gordillo GM. Development and initiation of computer generated documentation for burn patient care. Journal of Burn Care & Rehabilitation. 2002;23(4): [19] Reed RL, 2nd, Davis KA, Silver GM, Esposito TJ, Tsitlik V, O'Hern T, et al. Reducing trauma payment denials with computerized collaborative billing. The Journal of trauma Oct;55(4):

220 DIAGNOSIS ICD-9 CODE WHEN SIC-IR B/DM SUGGESTS DIAGNOSIS Infection Related Diagnosis BACTEREMIA-CATHETER RELATED When placed on confirmed diagnosis form in the SIC-IR GUI BACTEREMIA-NONCATHETER RELATED When placed on confirmed diagnosis form in the SIC-IR GUI C. DIFF INFECTION When placed on confirmed diagnosis form in the SIC-IR GUI EAR / MASTOIDITIS When placed on confirmed diagnosis form in the SIC-IR GUI EMPYEMA When placed on confirmed diagnosis form in the SIC-IR GUI ENDOCARDITIS When placed on confirmed diagnosis form in the SIC-IR GUI ESOPHAGITIS-CANDIDAL When placed on confirmed diagnosis form in the SIC-IR GUI EYE-CONJUNCTIVITIS When placed on confirmed diagnosis form in the SIC-IR GUI FUNGEMIA When placed on confirmed diagnosis form in the SIC-IR GUI FUNGEMIA-CANDIDEMIA When placed on confirmed diagnosis form in the SIC-IR GUI INTRAABDOMINAL INFECTION-CHOLANGITIS When placed on confirmed diagnosis form in the SIC-IR GUI INTRAABDOMINAL INFECTION-CHOLECYSTITIS When placed on confirmed diagnosis form in the SIC-IR GUI INTRAABDOMINAL INFECTION-DIVERTICULITIS When placed on confirmed diagnosis form in the SIC-IR GUI INTRAABDOMINAL INFECTION-HEPATITIS When placed on confirmed diagnosis form in the SIC-IR GUI INTRAABDOMINAL INFECTION-NOS When placed on confirmed diagnosis form in the SIC-IR GUI INTRAABDOMINAL INFECTION-PANCREATITIS When placed on confirmed diagnosis form in the SIC-IR GUI INTRAABDOMINAL INFECTION-SPLEEN When placed on confirmed diagnosis form in the SIC-IR GUI MEDIASTINITIS When placed on confirmed diagnosis form in the SIC-IR GUI MENINGITIS When placed on confirmed diagnosis form in the SIC-IR GUI NECROTIZING SOFT TISSUE INFECTION When placed on confirmed diagnosis form in the SIC-IR GUI ORAL / PHARYNX INFECTION When placed on confirmed diagnosis form in the SIC-IR GUI OSTEOMYELITIS When placed on confirmed diagnosis form in the SIC-IR GUI PNEUMONIA-COMMUNITY AQUIRED When placed on confirmed diagnosis form in the SIC-IR GUI PNEUMONIA-NOSOCOMIAL When placed on confirmed diagnosis form in the SIC-IR GUI PNEUMONIA-VENTILATOR ASSOCIATED When placed on confirmed diagnosis form in the SIC-IR GUI SINUSITIS When placed on confirmed diagnosis form in the SIC-IR GUI SKIN-ABSCESS When placed on confirmed diagnosis form in the SIC-IR GUI SKIN-BREAST/MASTITIS When placed on confirmed diagnosis form in the SIC-IR GUI SKIN-CELLULITIS When placed on confirmed diagnosis form in the SIC-IR GUI SKIN-NOS When placed on confirmed diagnosis form in the SIC-IR GUI SKIN-PRESSURE ULCER When placed on confirmed diagnosis form in the SIC-IR GUI SKIN-TRAUMATIC WOUND INFECTION When placed on confirmed diagnosis form in the SIC-IR GUI SURGICAL SITE INFECTION-DEEP When placed on confirmed diagnosis form in the SIC-IR GUI SURGICAL SITE INFECTION-ORGAN SPACE When placed on confirmed diagnosis form in the SIC-IR GUI SURGICAL SITE INFECTION-SUPERFICIAL When placed on confirmed diagnosis form in the SIC-IR GUI TRACHEOBRONCHITIS When placed on confirmed diagnosis form in the SIC-IR GUI UTI-COMMUNITY AQUIRED When placed on confirmed diagnosis form in the SIC-IR GUI UTI-NOSOCOMIAL When placed on confirmed diagnosis form in the SIC-IR GUI Other Laboratory Abnormality Diagnosis LEUKOCYTOSIS Suggested if leukocyte count > 12.0 HYPERNATREMIA Suggested if sodium > 148 HYPONATREMIA Suggested if sodium < 135 HYPERKALEMIA Suggested if potassium > 5.3 HYPOKALEMIA Suggested if potassium < 3.3 HYPERCHLOREMIA Suggested if chloride > 111 HYPOCHLOREMIA Suggested if chloride < 97 UREMIA 586 Suggested if blood urea nitrogen > 35 HYPERGLYCEMIA Suggested if glucose > 150 HYPOGLYCEMIA Suggested if glucose < 70 HYPERCALCEMIA Suggested if calcium > 10.4 HYPOCALCEMIA Suggested if calcium < 8.0 HYPERMAGNESEMIA Suggested if magnesium > 2.8 HYPOMAGNESEMIA Suggested if magnesium < 1.6 HYPERPHOSPHATEMIA Suggested if phosphorus > 4.8 HYPOPHOSPHATMEIA Suggested if phosphorus < 2.1 ACIDEMIA Suggested if ph < ALKALOSIS Suggested if ph > HYPOXEMIA Suggested if PO2 < 70 HYPERCAPNIA Suggested if PCO2 > 45 Common ICU Diagnosis FEVER Suggested if max daily temp > 38.5ºC SEVERE PROTEIN CALORIE MALNUTRITION 262 Suggested if patient is on total parentral nutrition PLEURAL EFFUSION Suggested if daily chest x-ray read indicates presense of pleural effusion PULMONARY COLLAPSE / ATTELECTASIS Suggested if daily chest x-ray read indicates presense of collapse or atelectasis PULMONARY CONTUSION Suggested if daily chest x-ray read indicated presence of pulmonary contusions ACUTE RESPIRATORY FAILURE (TRAUMA / POST-OP) If patient is on the ventilator BLOOD LOSS ANEMIA (ACUTE) Suggested if patient received at least 1 unit of packed red blood cells DEFECTIVE COAGULATION NOS Suggested if patient received at least 1 unit of fresh frozen plasma THROMBOCYTOPENIA Suggested if patient received at least 1 pack of platelets HYPERGLYCEMIA REQUIRING INSULIN DRIP Suggested if patient is on an insulin drip PROCEDURE CPT CODE WHEN SIC-IR B/DM SUGGESTS PROCEDURE INTUBATION When a SIC-IR procedure note is completed by resident of attending INSERTION OF NEW CENTRAL VENOUS CATHETER When a SIC-IR procedure note is completed by resident of attending REPLACEMENT OF NEW CENTRAL VENOUS CATHETER When a SIC-IR procedure note is completed by resident of attending INSERTION OF PULMONARY ARTERY CATHETER When a SIC-IR procedure note is completed by resident of attending INSERTION OF ARTERIAL CATHETER When a SIC-IR procedure note is completed by resident of attending INSERTION OF A THOACOSTOMY TUBE When a SIC-IR procedure note is completed by resident of attending Table 1: List of diagnoses and procedures taught to the SIC-IR B/DM. ICD-9=International Classification of Disease Version 9 CPT= Current Procedural Terminology GUI=Graphical User Interface 220

221 Anemia due to blood loss, acute * Hematemesis Atrial flutter Hyponatremia* Atelectasis * Malnurtiriton* Cachexia Melena Cardiogenic shock Pleural effusion* Cardiomyopathy Pnumothorax Cellulitis* Renal failure (end stage renal disease) Congestive heart failure (specific forms) Respiratory failure* Decubitus ulcer * Urinary tract infection* Diabetes mellitus with ketoacidosis Table 2: List of the commonly missed diagnoses (complications and comorbidities) which have a significant impact on MS-DRG assignment. Adopted from INGENIX DRG Desk Reference: The ultimate resource for improving the new MS-DRG assignment practices * denotes diagnoses which can be recognized by the SIC-IR B/DM 221

222 Attending medical record documentation Attending professional billing documentation Resident physician documentation STICU procedure documentation Phase 1 (No SIC-IR B/DM) Handwritten with or without a progress note template Handwritten using a billing template. All patient diagnoses and procedures individually selected Electronically: using SIC- IR without automatic diagnosis identification or procedure archiving Handwritten procedure notes Phase 2 (SIC-IR B/DM) Handwritten with or without a progress note template or typed using SIC-IR B/DM Electronically: SIC-IR B/DM automatically suggests 70 ICD-9 codes and 6 CPT codes Electronically using SIC-IR B/DM to capture 70 ICD-9 codes automatically Electronic procedure notes with archiving and ability to remind attending physician on billing documentation Table 3: List of STICU documentation procedures occurring in phase 1 and phase 2 STICU=Surgical and trauma intensive care unit ICD-9=International Classification of Disease Version 9 CPT= Current Procedural Terminology 222

223 Phase 1 N=436 pts 2202 pt-days Phase 2 N=378 pts 2308 pt-days P-value Mean age (years) 50.1 ± ± Males 68% 64% Trauma patients 60% 57% General surgery 11% 9% Cardiac surgery 6% 10% Neurosurgery 12% 11% Table 4: Demographics of study patients Age is expressed as mean per patient ± standard error of the mean 223

224 Phase 1 N=436 pts 2202 STICU patient-days Phase 2 N=378 pts 2308 STICU patient-days P-value Mean STICU LOS 5.0 ± ± Mean Hospital LOS 10.2 ± ± Mean Indwelling Urinary Catheter-Days Mean Central Venous Catheter-Days 4.8 ± ± ± ± Mean Antibiotic-Days 2.4 ± ± Mean Ventilator-Days 2.7 ± ± Mean Number of Infections 0.22 ± ± Mortality 5% 6% Table 5: Clinical variables compared between phase 1 and 2 All values are expressed as means per patient ± standard error of the mean STICU = Surgical and trauma length of stay LOS = Length of stay 224

225 Number of potential professional bills during phase 2 Number of professional bills generated by SIC-IR (96% of potential bills) Number of reconciled professional bills 2188 (99% of generated bills) Number of professional bills turned into billing department 1381 (63% of generated bills) Number of professional bills not turned into billing department 825 (37% of generated bills) Table 6: SIC-IR B/DM outputs during phase 2 225

226 Number of ICD-9 codes Phase 1 N=444 administrative bills Phase 2 N=354 administrative bills p-value Relative % change from phase ± ± 0.3 < % DRG-RW 3.8 ± ± % DRG geographic mean LOS DRG Estimated Reimbursement 8.0 ± ± % $32, ± $ $38, ± $ % APR DRG-RW 4.1 ± ± % APR DRG Severity of Illness 2.7 ± ± 0.05 <0.001* +7% APR DRG Risk of Mortality 2.1 ± ± 0.06 <0.001* +14% Table 7: ICD-9 and DRG outcomes between phase 1 and phase 2. All results are expressed as means per patient administrative bill ± standard error of the mean * Calculated utilizing Mann-Whitney U non-parametric statistic ICD-9=International Classification of Disease Version 9 DRG = Diagnosis related group DRG-RW = Diagnosis related group relative weight LOS = Length of stay APR = All patients refined (A proprietary 3M technology) 226

227 CMS Private Insurances Hospital Reimbursement Hospital Administrative Data Quality Initiatives Safety Initiatives Pay-for-Performance Health Care Reform Public Reporting Hospital and Physician Comparisons Figure 1 227

228 DRG-RW X Hospital Base Rate = Hospital Reimbursement Case-Mix Index Sum of all DRG-RW Total number of Medicare and Medicaid patients treated Figure 2 228

229 Figure 3 229

230 Figure 4 230

231 Figure 5 231

232 Figure 6 232