Computerized Provider Order Entry: Initial Analysis of Current and Predicted Provider Ordering Workflow

Transcription

1 Computerized Provider Order Entry: Initial Analysis of Current and Predicted Provider Ordering Workflow by Niiokai Alcide A thesis submitted in conformity with the requirements for the degree of Master of Health Science in Clinical Biomedical Engineering, Institute of Biomaterials and Biomedical Engineering, University of Toronto Supervised by Dr. Edward Etchells Department of Medicine, Centre for Health Services Sciences, and Information Services, Sunnybrook Health Sciences Center Copyright by Niiokai Alcide 2009

2 Niiokai Alcide Computerized Provider Order Entry: Initial Analysis of Current and Predicted Provider Ordering Workflow November 2009 Master of Health Science in Clinical Biomedical Engineering Institute of Biomaterials and Biomedical Engineering University of Toronto Abstract Background: Computerized Provider Order Entry (CPOE) allows providers to enter medication and service orders electronically. Workflow analysis is a critical component of CPOE implementation. Objectives 1. To develop a nosology for provider ordering workflow. 2. To describe actual provider ordering workflow focusing on chart and computer usage 3. To model the impact of computerized ordering on provider workflow in three future state scenarios Method: 20 hours of participant observation was performed for nosology development, time motion studies totaling 47 hours and predictive modeling to project effects of possible implementation scenarios Results/Conclusions: Unique nosology was developed. Clinicians spent 27% of their time with the patient, 2.2% writing and 1.1% locating patient charts. Our study predicted that the E-All scenario (computerization of all orders) would be the best implementation choice. Limitations: Small sample size (25 clinicians), participant frame of reference and other assumptions may have affected the results of this study. ii

3 Acknowledgements I would like to express my most profound gratitude to the entire CPOE team at the Sunnybrook Health Science Center, my dynamic supervisor Dr Edward Etchells, my thesis supervisory committee: Dr. Kaveh Shojania and Paul Milgram, my colleague Julie Chan for her help with my observational studies. My mother Catherine, father Anthony, my fiancée Tiffany and my sister Kimmel who all pushed me to make the necessary sacrifices to achieve my goals. Without your support my efforts would have been futile. Thank you iii

4 Table of Contents Chapter 1: Background Benefits of CPOE Importance of pre-implementation workflow analysis Fragmentation of clinician workflow Current state ordering process at Sunnybrook Methods for evaluating clinician workflow Predictive modeling Chapter 2: Objectives Chapter 3: Methods Study environment Participant observation Time motion study Measures Other data sources Data analysis Predictive modeling Chapter 4: Results Results: Objective 1: Nosology development Results: Objective 2: Time motion study Predictive modeling decision tree Base case analysis One way sensitivity analysis Two way sensitivity analysis Chapter 5: Discussion Development of a Nosology Description of provider workflow Effects of order entry on provider workflow Chapter 6: Limitations Chapter 7: Recommendations Chapter 8: Conclusion. 63 References Appendix iv

5 LIST OF FIGURES Figure 1: Data collection tool Figure 2: Screen shots from time-motion study software Figure 3: Decision Tree Figure 4: One way sensitivity graph for time to find a free PC workstation Figure 5: One way sensitivity graph for time to log on to EPR system Figure 6: One way sensitivity graph for time to enter order into EPR system Figure 7: One way sensitivity graph for time to write order in chart Figure 8: Two way sensitivity graphs for time to find free pc and logon to EPR system Figure 9: Two way sensitivity graphs for time to find free pc and enter order into EPR system. 46 Figure 10: Two way sensitivity graphs for time logon to EPR system and enter order v

6 LIST OF TABLES Table 1: Summary of reviewed literature on impact of CPOE on physician workflow... 6 Table 2: Types of order clusters, and clinician workflow for each type of order cluster in various CPOE implementation scenarios Table 3: Avantages and Disadvantages of workflow evaluation methods Table 4: Ordering activity nosology Table 5: Characteristics of study and participants Table 6: (a)time captured by activity (b) Usage of patient chart and EPR system Table 7: Charateristics of order clusters for time-motion study Table 8: Interobserver reliability results Table 9: Predictive modeling variables Table 10: Probabilities used for predictive modeling Table 11: Averaged variables used in decision tree analysis Table 12: One way senstivity analysis of probability variables Table 13: One way sensitivity analysis of averaged variables Table 14: Two way sensitivity analysis results Table 15: Comparison of time motion study results with prior studies vi

7 LIST OF ABBREVIATIO S ADE: Adverse Drug Event CDS: Clinician Decision Support CPOE: Computerized Provider Order Entry EPR: Electronic Patient Record E-MED: electronic ordering of medication orders only E-MOST: electronic ordering of medication, radiology, laboratory and echo-cardiology orders only E-ALL: electronic ordering of all orders OE: Order Entry PC: Personal Computer PDA: Personal Data Assistant PO: Participant Observation POE: Provider Order Entry SPSS: Statistical Package for the Social Sciences TM: Time Motion WS: Workstation vii

8 Chapter 1: BACKGROU D Computerized Provider Order Entry (CPOE) is a computer application that accepts the provider s (physician, pharmacist or nurse) orders for medications, laboratory tests, diagnostic radiology, and other diagnostic tests. CPOE replaces the manual process whereby the providers record orders using paper based methods. According to Drazen (2000), the immediate interest in CPOE is focused on medication order entry and its potential to reduce medication prescribing errors. The application can compare the orders against standards for dosing, check for allergies or interactions with other medications and warn the physician about potential problems. CPOE systems may also reduce costs through avoided adverse events, reduced utilization and shorter lengths of stay, and reduce unnecessary variations in care by encouraging recommended care practices (Drazen 2000). However, unintended consequences and CPOE induced errors can stem from improper design and implementation. There are many reports of failed CPOE implementation, and a low rate of the adoption of this technology. Goddard et al in 2000 reported the reasons for the failure of an implementation undertaken in three acute care hospitals. They highlighted financial pressures, personal unwillingness to change and integrator inexperience with health care. In addition to those reasons, other studies have reported lack of medical professional (clinician) involvement, inadequate capture and analysis of pre and post implementation workflows, and lack of post implementation clinical and technical support as other frequently reported reasons for failure. Keel et al (2005) 1.1 The benefits of CPOE A major potential benefit of CPOE is the reduction of medication errors. Bates et al (1999) evaluated the impact of computerized physician order entry on medication errors. 1 Non-

9 2 intercepted medication errors, defined as medication errors that reached the patient, were reduced by 86%. Non-missed dose medication errors (i.e. medication errors resulting from route errors, frequency errors, substitutions, drug-drug interactions, inappropriate drugs, illegible orders, known allergies to drugs, drugs not being available, avoidable delays in treatment, and preparation errors) fell by 81%. CPOE may also reduce adverse drug events. The effect of CPOE with clinical decision support (CDS) on potential adverse drug events (ADEs) was explored by Wolfstadt et al in The authors reviewed 10 CPOE studies and found that 5 reported statistically significant reductions in ADEs. One of the studies reported a significant reduction in the rate of total ADEs per 100 drug orders from 1.0 in a paper-based unit to 0.15 in a computer-based unit consisting of CPOE with CDS (P<.01) (Colpaert et al 2006). The second benefit of CPOE is that it serves as a powerful tool for the reduction of unnecessary variation in care by encouraging recommended practices and increasing responsiveness to new information (Drazen et al, 2000). CPOE facilitates the standardization of care through decision support and standardized order sets. In one study there was a 94% compliance with the new recommendations after 4 weeks of using computerized decision support, compared to 16% before the implementation of the system. (Teich et al 1996). A final potential benefit of CPOE is cost reduction. Cost reductions may be realized by avoiding errors, improving efficiency, optimizing drug utilization, and avoiding the costs of adverse drug events. Shojania et al (1998) demonstrated reduction of use of an expensive antibiotic with a CPOE platform and clinical decision support. Clinicians wrote 32% fewer orders for the drug and the overall duration of therapy with the drug was 36% lower than that of the study s control

10 3 group. Another study showed that CPOE reduced utilization of services, wastage of medications, decreased length of stay and overall costs (Classen et al 1997). CPOE implementation and maintenance also carries costs however, CPOE requires an analysis and redesign of important clinical processes, as well as an implementation of technology. According to Kuperman (2003), technical costs, costs of process redesign, and cost of implementation and support often deter the adoption of this technology. However, cost savings from high costs of ADEs both medical and legal are solid incentives to put CPOE into widespread use. A study at Brigham Women s Hospital revealed an implementation cost of $1.9 million, with a yearly maintenance of $500,000. The institution now reports a return on initial investment of $5 to $10 million in annual savings (Leapfrog Group 2008). 1.2 Importance of Pre implementation Workflow Analysis The effect of CPOE on provider workflow is one of the major pitfalls in the adoption of the technology. Successful implementation requires adequate knowledge of local workflow patterns. Niazkhani et al. (2008), emphasize the important differences between conceptual (what is supposed to happen) and actual workflow patterns. Successful CPOE projects require an understanding of these differences, so that implementation can maximize efficiency to ensure that the clinical information system is in tune with the healthcare professionals existing processes and workflows. (Shabot 2004) Workflow routines vary tremendously amongst various groups of healthcare providers. Callen et al (2008) described the dissimilar ways doctors and other health professionals carry out

11 4 their routines at the point of care. Physicians and residents review test results, provide or seek consults, and order drugs, tests and other services. Nursing staff on the other hand execute orders and are involved in direct care for patients. Nurses would therefore spend less time interacting with a computer while carrying out their daily routine. In contrast, physicians are involved in indirect care, as they spend more time with the paper charts and workstations. The article also reveals that there is tremendous diversity in the work patterns of clinicians and that careful analysis must be in place before they can adapt to any workflow changes. For example, Niazkhani et al. (2009) reported that providers from one institution s CPOE implementation experienced greater workflow support after the implementation and felt that the system was in sync with their habits. The changes in provider workflow brought about by CPOE may increase the amount of time spent on the medication ordering process and for clinical documentation. As a result, many physicians express concern that ordering with CPOE takes longer than ordering with paper. Some studies have confirmed this aspect of CPOE. Possiant et al undertook a review of 23 studies centered on electronic health records and clinical documentation times, from the perspective of physicians and nurses. The general trend of the studies was an increase in the time spent documenting on paper versus the computer. Only 10 of the studies focused on physicians and 3 of the 10 studies included order entry (CPOE) systems. The three studies (Bates 1994, Shu 2001 and Tierney 1993) reported similar results. Bates et al. revealed that in a post CPOE implementation inpatient setting, 15 clinicians spent on average 44 more minutes daily (5.3% to 10.3%) writing orders. On a pre and post implementation study conducted on 43 clinicians for 1554 hours, Shu et al (2001) found an increase from 2.1 % to 9.0% (approximately 35 minutes

12 5 more per clinician per day) for time spent order writing. Similarly in 1993, Tierney et al observed 24 clinicians in a pre and post implementation setting over 957 hours and reported an average increase of 33 minutes for writing orders (2.5% to 9.3% minutes) after CPOE implementation during day shifts ( 10am to 8pm). Other current studies have shown that electronic ordering can be time-neutral compared with a paper process. A study conducted by Overhage et al 2001 in an inpatient setting on pre and post implementation ordering times revealed that physicians and residents spent 2.12 minutes longer per patient (6.2% to 6.9% of total time over a typical 30 minute clinic visit) writing orders using CPOE than control physicians using paper-based methods with some computer usage for displaying lab results. Similarly, Lo et al revealed an increase in physician order writing time by 2 minutes (11% to 15% of total time) per patient, for a 30 minute average patient care period using an integrated EPR (Electronic Patient Record) and CPOE system. They also reported that clinicians spent on average 5.3 minutes for ordering medication and services and 3.5 minutes reading various lab results on the EPR-CPOE system per every 30 minutes of patient care. In contrast, physicians using paper-based ordering spent on average 3.3 minutes per 30 minutes of patient care writing orders and 1.7 minutes using the computer to review lab results. Given the above mentioned time changes, the researchers concluded that clinicians do not spend more time overall on patient care after the implementation of CPOE. The results of these studies may however only pertain to the particular study environment, as all orders and charting activities were hand written and lab results were reviewed electronically. The following table summarizes the results of the reviewed literature pertaining to time spent ordering.

13 6 Study Study Setting and Design Participants Sample Size Workflow Classification Methods Main measures Results/ comments Bates et al Inpatient Pre and post Medical 22 Time Motion Time spent on order entry. Time Interventional participants 44 minutes 1994 implementation Interns for other activities after order longer per day than control (5.3% to 10.5% entry implementation of total time spent on order writing. Lo et al Inpatient Pre and post Physicians 17 Time Motion Measure of clinician time spent on Interventional participants 2 minutes longer 2007 implementation 85 activities per patient than control on time spent on order writing. Overhage et Internal medicine practice Physicians 34 Time Motion Time spent on various activities Interventional participants 2.12 minutes al 2000 Inpatient Pre and post per patient longer per patient than control (6.2% to 6.9) implementation % of total time spent on order writing. Teirney et al Inpatient Pre and post Interns 24 Time motion Time motion study of selected Interventional participants 33 minutes 1996 implementation Medical interns on system s time longer per day than control (2.5% to 9.3% students, consumption of total time spent on order writing. Shu et al Inpatient Pre and post Interns 43 Random sampling Impact of CPOE system on Interventional participants 35 minutes 2001 implementation Medical clinician time longer per day than control (2.1% to 9% of students, total time spent on order writing. Table 1: Summary of reviewed literature on impact of CPOE on physician workflow 1.3 Fragmentation of clinician workflow Over the past 15 years, there has been gradual computerization of various aspects of the ordering process. For example at Sunnybrook today, physicians must write, then electronically enter, radiology and echocardiography orders. By contrast, medication orders and all other orders (such as nursing orders) are written on paper order sheets. Clinical documentation is done almost entirely on the paper chart, but discharge summaries and certain clinical notes are available on the computer. Clinical data review is done both on the computer, for radiology and laboratory results, and the paper chart (such as pulmonary function tests, electroencephalography reports).this process fragmentation can create problems for clinicians workflow, because many

14 7 tasks now require both the paper chart and the computer workstation for successful completion. For example, a physician may need to view laboratory results on the computer while writing progress notes in the paper chart. Clinicians may also be performing multiple tasks simultaneously, or sequentially, with the paper chart or computer. For example, a clinician could be using the computer to review results so that orders can be written in the paper chart. The critical workflow consideration is that the clinician is performing the tasks with both the paper chart and the computer. Process fragmentation, and the related concept of task sequencing, may also represent an opportunity, because clinicians may now be more willing to increase the amount of time spent at the computer in order to reduce process fragmentation. Previous clinician workflow studies have not thoroughly explored the amount of time clinicians spend multitasking while performing tasks centered on ordering medication and services in a pre CPOE implementation setting. Studies by Overage and Lo, where the setting involved the use of paper charts and computer for results display, revealed a less drastic impact of CPOE on physician time and workflow as clinicians were already spending time on the computer. Other workflow studies have not thoroughly considered the concepts of process fragmentation or task sequencing. Studies by Bates (1994), Shu (2001), and Overhage et al. (2001) group clinician tasks as isolated activities. For example they generated categories by media (computer or chart) and included write, read and other activities as sub categories. During the observation periods, observers used their judgment to determine what the major activity would be logged as, and possibly disregarding the fact that the clinician may have been

15 8 multitasking at the time. This may have impacted the results of the studies as one of the activities would be chosen more frequently than another, without accounting for performing simultaneous tasks. The end result would therefore be less time being recorded for particular tasks. A study by Asaro took a different approach and captured to some degree the multitasking nature of clinician workflow. They captured the various overlapping (simultaneously performed) tasks of clinicians in an emergency department. For example tasks were grouped as using EPR system while on the phone, writing orders while talking to nurse, and using the EPR system while working with the patient chart. The latter category thus included all the possible uses of the chart, namely writing order, reviewing or documentation. The study was not aimed at documenting the particular activities being performed or whether they included the EPR system and chart, but focused on how much time was spent doing any type of multitasking. The present study will explore the current fragmented nature of physicians workflow, focusing on whether the clinician is using a computer workstation, a paper chart, or both, during ordering, documentation, or clinical data review. These data will allow a more accurate prediction of the impact of changes to the ordering process on clinician workflow. 1.4 Current State Medication Ordering Process at Sunnybrook Currently at Sunnybrook clinicians are required to write all medical orders in the patient s chart. In addition echocardiography and diagnostic imaging are required to be entered directly into the EPR system by the ordering provider. Laboratory orders are then entered into the electronic system by nurses or ward clerks. For a typical medication order, a clinician would

16 9 write the order on an order sheet found in the patient s chart. The chart is then flagged (red knob) to alert nurses that an order has been written. After acknowledging the order, the nurse removes the yellow carbon copy of the order sheet and places it in the pharmacy tray for pickup by pharmacy technicians. Pharmacy technicians then collect order sheets on a unit at 30 to 45 minute intervals. The forms are taken to the pharmacy department for pharmacist review. The pharmacist will enter the order into the computerized dispensing system. The first doses are dispensed in the pharmacy department, then brought to the unit by pharmacy technicians and placed in the patient s tray in the medication cart for administration by nurses. The Sunnybrook CPOE project is currently choosing between three future state implementation scenarios (i) Electronic Med (E-Med): Medication orders are only entered electronically by the ordering provider. There are no other changes to current state. All other orders are written on paper chart. The physician must then also electronically enter radiology and echocardiography orders. (ii) Electronic Med-Lab-Radiology-Echo (E-Most): Medication, laboratory, radiology and echocardiography orders entered only electronically by the ordering provider. There are no paper chart orders for medication, laboratory, radiology and echocardiography. All remaining orders are written on paper chart. (iii) All Orders Electronic (E-All): All orders only entered electronically by the ordering provider; no orders written on paper chart.

17 10 Clinicians do not necessarily write orders solely for medications at a single time. A clinician may also write a lab order and a general order (for example, activity as tolerated) at the same time. We define the term order cluster as a group of orders written by a single clinician at a single time for a single patient. Prior studies have examined order sets, which are predefined standard groups of orders, but there has been little consideration of the concept of an order cluster in studies of clinician workflow. Clusters could include any combination of various types of orders. For example, a doctor may write an order for Heparin (medication), a Complete blood count (Lab), and nothing by mouth for 24 hours (general order). The analysis of a cluster of orders is highly relevant when process fragmentation occurs. Depending on the characteristic of the orders in the order cluster, and the workflow for each type of order in the order cluster, the clinician will have a very different workflow. For example, if an order cluster has three orders (see example in bottom row of table 2, below), the clinician may need to interact with a paper chart only, both a chart and computer, or a computer only, depending on the implementation plan chosen. No prior time motion studies or predictive models of CPOE implementation have explicitly accounted for the concept of order clusters and process fragmentation when evaluating the potential effects of CPOE on clinician workflow. Our predictive model will explore three implementation scenarios: E-Meds: all computerized entry of medication orders, no chart for medications; E-Most: computerized entry of medications, laboratory, and echocardiography orders (no paper use); and E-All: all orders are entered electronically, no use of paper chart. The three proposed implementation scenarios will thus incorporate combinations of the order clusters highlighted in table 2 below.

18 11 Media Required to create Order Order Type Frequency* Typical Example Current E-Meds E-Most E-All % State Medication Only 38% Lasix 40 mg po once daily Paper chart Computer Computer Computer Laboratory/radiology or 8% Electrolytes tomorrow Paper Paper Computer Computer echocardiography only morning chart** chart** General order only 13% Daily weights Paper Paper Paper Computer chart chart chart Medication plus 13% 1) Lasix 40 mg po once daily Paper Both Computer Computer laboratory/radiology or echocardiography 2) Electrolytes tomorrow morning chart Medication plus general order 11% 1) Lasix 40 mg po once daily 2) Daily weights Paper chart Both Both Computer Laboratory/radiology or 4% 1) Electrolytes tomorrow AM Paper Paper Both Computer echocardiography and General Order 2) Daily weights chart Chart All types (medication, 13% 1) Lasix 40 mg po once daily Paper Both Both Computer laboratory/radiology/echocardio graphy, and general order) 2) Electrolytes tomorrow AM chart* 3) Daily weights *Data obtained from CPOE project team based on 300 order clusters written on D2 at Sunnybrook, October **In some cases the clinician would use BOTH the chart and the computer, but not in the example on this table E-Meds : computerized entry of medication orders no chart for medications E-Most: computerized entry of medications, laboratory, and echocardiography orders (no paper use)e-all: all orders are entered electronically no paper use Table 2: Types of order clusters, and clinician workflow for each type of order cluster in various CPOE implementation scenarios

19 Methods for Evaluating Clinician Workflow There are numerous methods for the evaluation of clinician workflow in a hospital setting. Some of the more commonly used methodologies are highlighted in the table below. Method of Workflow Evaluation Participant observation: whereby an observer is immersed in daily activities of the observed and makes note of activities of interest Work sampling: this technique is used to measure work activity at set intervals of time using an observer or a self logged method Self Reported Survey- Participants respond directed questions. Time Motion Studies: Method which allows time stamps to be attached to performing various activities. Random sampling: Participants are given pagers which alarm to remind them to record activities Strength Observer sees all activities and can concentrate only on the relevant ones General understanding of activities observed as observer observes participants in groups Easier for observer to blend in, as many are being observed simultaneously. Directed to large number of participants and they can remain anonymous Gives more detailed information Observer is in close proximity to participant and can log exact times for activities. Less disruptive from the observer s perspective and can be used on a 24 hour basis. No human observer necessary Weakness The observer effect- Participants may change behavior due to the presence of the observer. Difficulty in noting activities. The observer effect- Participants may change behavior due to the presence of the observer. One observer to multiple participants thus some aspects of work habits may be overlooked. Responses are not always accurate and or honest The incidence of the observer effect is greater. Labor intensive as it requires one on one observations Individuals may still forget to record activities and they may become bothered by the alarms Table 3: Advantages and disadvantages of workflow evaluation methods Participant Observation and Time-motion studies were chosen for this study as they are complementary methods for workflow evaluation. The participant observation formulates the categories from the observed activities. The time motion studies then allow the association of a time factor to the generated categories. Participant observation is a qualitative research method involving direct participation of the researcher in the events being studied. In participant

20 13 observation the researcher is immersed in the day-to-day activities of the population being studied. The objectives are to record conduct and reveal patterns for various tasks under the widest range of possible settings. Data obtained through participant observation serve as a check against participants subjective reporting of what they believe should be done and what they do. Through this method, researchers can also uncover factors important for a thorough understanding of the research problem but that were unknown when the study was designed. (Pina 2006) Participant observation studies are an important tool in understanding how to redesign CPOE systems to avoid harm and achieve the full potential of benefit for improved patient safety. For example a participant observation study by Jamie Pina in 2006 on an EPR system revealed that issues such as duplicate patient records arose due to understaffing in the clinic s data entry department and variations in the current staff s levels of training on the system. The information can be used to suggest and project work patterns when the change is implemented. Participant observations can be passive, whereby the observer does not take part in the activities of the observed, or active where the observer participates and is involved in the daily activities of the observed. Active participant observation is known to foster increased observer acceptance as he/she appears to have a more significant role. Passive participant observation however, is a less intrusive method than active participant observation as there is no observer involvement in the observed activities. Thus active participant observation may in some ways increase the occurrence of the observer effect which is behavioral change in the observed brought about by the observer s presence..

21 14 From reviewed literature, time-motion studies have been noted to be the most accurate method for capturing task duration in the evaluation of workflow compared to simple participant observation random sampling and self reported surveys (Lo et al. 2007). Time motion studies are used to measure time required for the completion of a particular task. The fragmented nature of clinicians work and the relatively short duration of specific tasks makes a time-motion study a critical research method choice in workflow analysis (Overhage et al. 2001) The time motion study can highlight the time spent during each area of a clinician s daily patient care workflow, despite the direct (patient interaction) or indirect (no patient interaction) nature of the activities. The time motion studies and participant observation can be performed simultaneously. Time motion study results may identify aspects of order entry which are more time consuming and allow CPOE project teams to predict the various ways in which a fully integrated EPR and CPOE system may impact on task times. The development of time-motion software that runs on Personal Data Assistants (PDAs) has greatly increased the resolution and speed of data recording, which was previously done with a stopwatch and paper, limiting the scope of such studies. (Khan 2006). Using stop watches to record durations cannot provide information about the interactions of activities across care providers. Capturing starting and ending times of activities along with a description of the activity is difficult, particularly in an emergency department or a general medicine unit, due to frequently changing and overlapping tasks. In this setting handheld computer applications have been found useful in capturing activities at high granularity. The applications are capable of highlighting the most general to the more specific activities performed by clinicians.(asaro 2004) Previous studies dating as far back as 1993 have utilized time motion studies for time

22 15 capture using laptop computers or stopwatches. More recent studies, e.g. VanDenKerkhof et al in 2003, have incorporated handheld devices (PDAs) for increased accuracy and mobility. 1.6 Predictive Modeling Predictive modeling involves the creation of a statistical model of future behavior. A predictive model is composed of predictor variables that are likely to influence future behavior or results. A good understanding of current state and future state workflow are also essential to building a meaningful predictive model. To create a predictive model, data are collected from workflow observation and time motion studies and then a predictive model is formulated. Predictions are made and the model is validated (or revised) as additional data become available. Predictive modeling will aid in the projection of the changes to clinician ordering workflows which may be experienced after CPOE implementation. The predicted model may serve as a guide to decisions about hardware and software revisions which are essential for the success of POE systems. Predictive modeling techniques include neural networks, Markov analysis and decision trees. Before any analysis can take place, the modeling techniques must match the requirements of the clinical problem, but should not be too complex for the available data (Chapman et al 2003). In addition, the issues of accuracy, simplicity, timeframe and availability of data are key determinants of which decision model is best suited. A neural network model is generated through a set of computer algorithms which mimic human brain function. The purpose of a neural network is to learn to recognize patterns in data

23 16 by making various associations. The neural network model can be trained from data samples, allowing it to make predictions by detecting similar patterns in future data. The major advantage of neural networks lies in their ability to represent both linear and non-linear relationships and in their ability to learn these relationships directly from the data being modeled. (Traditional linear models are simply inadequate when it comes to modeling data that contains non-linear characteristics.) The disadvantage of neural networks is that there is tremendous difficulty in interpreting results, as the model structure does not relate directly to the features of the underlying problem. Neural networks were deemed too complex for this preliminary analysis of ordering workflow, given the simple nature of the data captured. A Markov analysis uses a sequence of events, and analyzes the tendency of one event to be followed by another. Using this analysis, it is possible to generate a new sequence of random but related events, which will look similar to the original. A Markov process is useful for analyzing dependent random events, i.e. events whose likelihood depends on what happened last. Markov analyses allow modeling of complex systems over time and they rely on sequenced events as in the case of the intention to order medication. The major limitation of this decision analysis methodology is that in some cases Markov analyses are too complex for the situation being analyzed and, similar to the neural networks, results are not easily interpreted. A decision tree is a branching structure in which various node symbols are used to represent different kinds of events, including decisions, probabilities and uncertainties, and a node s branches represent the outcomes or alternatives associated with that event. Every series of actions and outcomes is clearly represented with a distinct path. This decision analysis model offers easy construction and interpretable results, making it highly amenable to explanation and

24 17 human inspection. However, a decision tree represents a situation in its simplest form and as a result assumptions must be clearly defined and accounted for. Given the preliminary nature of our study and the nature of the captured data, decision tree analysis was chosen to predict values for average ordering times given the various CPOE implementation scenarios. We reviewed sample studies reported by Detsky et al (1997) to ensure that this method would be sufficient to capture the key aspects of our study given the many assumptions generated. Our relatively small data set and small number of probability variables made using a decision tree a better choice.

25 Chapter 2: Objectives 1. To develop a nosology for provider workflow 2. To describe actual provider workflow, with a focus on (i) time spent with the paper chart (ii) time spent with the computer workstation, and (iii) time spent with both chart and computer workstation simultaneously 3. To model the predicted impact of computerized ordering on time to complete an order in three future state CPOE implementation scenarios (i) Electronic Med (E-Med): Medication orders are only entered electronically by the ordering provider. There are no other changes to current state. All other orders are written on paper chart. The physician must then also electronically enter radiology and echocardiography orders. (ii) Electronic Med-Lab-Radiology-Echo (E-Most): Medication, laboratory, radiology and echocardiography orders entered only electronically by the ordering provider. There are no paper chart orders for medication, laboratory, radiology and echocardiography. All remaining orders are written on paper chart. (iii) All Orders Electronic (E-All): All orders only entered electronically by the ordering provider; no orders written on paper chart. 18

26 Chapter 3: Materials and Methods 3.1 Study Environment We enrolled a convenience sample of staff physicians and junior and senior residents working on D2, a general medicine and nephrology ward at Sunnybrook Health Science Center. We chose D2 because it is the ward where the CPOE pilot implementation at Sunnybrook is being conducted. Participants were recruited on a voluntary basis via direct approach on the study unit. Before each observation period, the observer informed participants about the purpose of the study, after which the individual gave verbal consent to be observed. Each participant was given a unique identifier to enable the observer to keep track of how many times the participant was observed. 3.2 Participant Observation Observations took place only on the patient care unit in the vicinity of the nursing station and the hallways. One observer shadowed 5 clinicians on the study unit for 20 hours over 10 days (2 hours/day). The observer noted all activities and processes carried out daily for 20 pilot observation periods highlighting usage of the patient charts and the EPR system. After the 13 th observation period it was noted that no new categories emerged, as the observer was able to group the activities using previous information. 19

27 Time Motion Study To capture the time spent on various clinical activities we utilized time motion study software installed on Personal data assistants. (Figure 1 below) Figure 1: HP IPAQ Personal Data Assistant. Device: Hewlett Packard IPAQ, Personal Data Assistant Model: Rx3115 (above) Operating system: Windows Mobile 2003 Time Motion study Software: UMTPlus by Laubrass Inc installed on the 2 PDAs The nosology developed during the participant observation phase of the study was programmed into the time motion study software (UMTPlus). The configuration was then uploaded into the PDA. The device utilized color touch screen technology which made it easy for choosing the correct color coded keys and entering auxiliary study notes/ occurrences.

28 21 (a) (b) (c) (d) Figure 2 PDA screen for time motion study.(a) Main Study screen,(b) Sub screen for activities such as Reviewing Chart, Writing order and progress notes, phone orders and verbal orders (c) Sub screen for Walking, (d) Sub screen for Other/ Miscellaneous activities. Time-Motion Observations One of the main focuses of the study was to determine the amount of time a clinician spends interacting with the patient s chart, computer or both. As a result we used the clinician, rather than the chart or the patient, as the unit of analysis. Each observation period lasted for minutes. First, the level of training of participant and the time of day were recorded. Using the time motion study software (UMTPlus)

29 22 installed on the PDA, the observer keyed in the activity and location of the participant and focused on medication and services ordering tasks and usage of workstations (which are located in the nursing stations and in a conference room) and 2 workstations on wheels (WOWs) located in the hallway near the nursing station. The observer remained 8 feet from the participant for the duration of observation, but did not enter patient rooms. The observer did not witness any instances where the clinician took a chart into a patient s room, and there are no workstations in patient rooms. After each session, participants were asked whether the presence of the observer made them uneasy and ultimately affected their behavior. Finally, the observer retrieved the patient s paper chart(s) used by the clinician and recorded the characteristics of the newly written orders. The observation data were uploaded to a desktop computer via a cradle connection and imported into Microsoft Excel. This procedure was repeated for 105 observation periods. Our target was 50 hours of observation, but the observations totaled 47 hours, as a few sessions were cut short because participants were leaving the hospital or called upon for an emergency. To evaluate inter-observer reliability of time motion study software, a second observer was trained during a 30 minute session. The use of the tool was demonstrated and the structure of the task categorization scheme was reviewed. The second observer then practiced on the handhelds for 30 minutes for one observation period. The second observer had already conducted workflow observations on D2 for other aspects of the CPOE project, and was very familiar with the electronic ordering process at Sunnybrook. When the second observer was comfortable with

30 23 the software and categorization of activities, both observers simultaneously but independently coded 5 observation periods (150 minutes of data). 3.4 Measures Major categories of activity were: Reviewing chart, writing notes in chart, writing order, walking, talking, in patient room and miscellaneous tasks. Sub-categories were based on ordering media type (chart, computer or both) (See Table 2). 3.5 Other Data Sources To supplement our analysis of the types of orders written, we obtained data from the Sunnybrook CPOE team. These data are from 300 order clusters written for D2 patients in October 2008 and sent to the D2 satellite pharmacy for verification. In general, orders that contain at least one medication are sent to the pharmacy, so these data tend to over-represent medication orders. Orders that contain no medication orders will tend not to be sent to the pharmacy. Also, the CPOE project team had done studies measuring the time required to enter 10 medication orders into the test electronic system. 3.6 Data Analysis UMTPlus included a statistical suite which was used to tabulate and perform the descriptive statistics on the data captured during each observation period. The data was exported to a MS Excel format. Microsoft Excel 2007 statistical analysis suite was then used to perform more indepth statistics and generate graphical results.

31 24 Interobserver reliability was evaluated using intraclass correlation coefficients. The data for the two observers was entered into SPSS v15 (statistical program) and the intraclass coefficients were generated along with the 95% confidence interval. 3.7 Predictive Modeling The main output variable of the model was the average time required to write/enter a typical order. The other aspects of CPOE, such as order acknowledgement by nurses, verification of orders by pharmacists, and medication administration by nurses were not explored in this predictive model. We modeled current state ordering workflow, as well as future workflow changes after CPOE implementation, based on several potential future state scenarios Current state: Medication and general orders written on chart, Echo and diagnostic imaging written on both chart and then entered into EPR/CPOE system Future state scenarios: (i) Electronic Med (E-Med): Medication orders are only entered electronically by the ordering provider. There are no other changes to current state. All other orders are written on paper chart. The physician must then also electronically enter radiology and echocardiography orders. (ii) Electronic Med-Lab-Radiology-Echo (E-Most): Medication, laboratory, radiology and echocardiography orders entered only electronically by the ordering provider. There are no paper chart orders for medication, laboratory, radiology and echocardiography. All remaining orders are written on paper chart.

32 25 (iii) All Orders Electronic (E-All): All orders only entered electronically by the ordering provider; no orders written on paper chart. The participant observation and time motion data are a central part of the predictive model. The following variables were used in the predictive modeling/decision tree analysis. We used data from our time motion study plus the data obtained from the Sunnybrook CPOE project team to estimate the following variables: Proportion of Orders entered electronically without any written order Proportion of Orders written on paper chart AND entered electronically Proportion of orders written on the paper chart only. We used data from the time motion study to estimate: The proportion of clinician time spent at a computer (either logged into EPR, or using the computer for other reasons) The proportion of time spent with the paper chart The proportion of time with the paper chart AND at a computer. We used data from the time motion study and as well as the data obtained from the CPOE project team to estimate the following time variables: Time to Locate chart ( T_ locate_chart ) Time to Locate free workstation (T_ findpc ) Time to Log into EPR (T logon.epr )

33 26 Time to Log out of EPR (T logoff.epr ) Time to Enter order into EPR (T enter.epr ) Time to Write order on chart (T write-order ) TreeAge Pro R 2009 was used to model the decision trees for the above scenarios and to perform decision analyses. The output of the decision tree was the average time to enter a typical order in each of the four scenarios. We performed one-way and two-way sensitivity analyses on the decision trees using various combinations of the study variables. 3.8 Sample Size A sample of 25 clinicians participated in this study. During the observation period there were approximately 12 staff physicians or clinical associates and 20 residents on the five general medicine teams. This represents the teams on service on the D2 general medicine unit for one month. Each participant was observed between 1 to 5 periods for minutes each. The time period was derived during participant observation/nosology development periods, when it was noted that 30 minutes was sufficient to capture the nature of clinician activities without encountering unnecessary repetition. In addition we chose the time interval based on the fact that participants may feel burdened by our presence if they were being observed for a prolonged period of time.

34 Ethics Prior to any observations and data collection, we obtained ethics approval from the Sunnybrook Health Sciences Center s and University of Toronto s research ethics review board

35 Chapter 4: Results 4.1 Results: Objective 1: osology Development (Table 4) We successfully grouped the clinical activities centered on patient care and medication ordering using paper charts and the available EPR system. From the table, activities were categorized into major groups, e.g. review chart, writing order, talking, walking and miscellaneous. Further groupings were constructed for activities which included the use of computers (for both EPR and non EPR usage) and the patient charts simultaneously. Table 4: Ordering activity osology. Generated from participant observation periods and review of prior literature Major Activity Categories At WorkStation Review Patient Chart Writing Med/Service Order Writing Progress Notes in Chart Specific Activity PC Use and EPR Use Pc Use and no EPR Use No PC Use Not at Workstation At Workstation no PC use At Workstation PC use for Non EPR Activity At Workstation PC use for EPR Not at Workstation At Workstation no PC use At Workstation PC use for Non EPR Activity At Workstation PC use for EPR Not at Workstation At Workstation no PC use At Workstation PC use for Non EPR Activity At Workstation PC use for EPR Telephone Med/Service Order Not at Workstation At Workstation no PC use At Workstation PC use for Non EPR Activity At Workstation PC use for EPR 28

36 29 Verbal Med/Service Order Walking Talking Miscellaneous Activities Not at Workstation At Workstation no PC use At Workstation PC use for Non EPR Activity At Workstation PC use for EPR On Unit To Nurse Station To other Unit Look for Nurse To Patient Room To Clinician To Patient To Family Telephone Non- Ordering Answering Page Reading Reference Text Locate Progress notes sheet Locating Order sheet Locate Auxiliary Forms In Patient's Room Checking MAR Locating Patient's Chart Waiting for Chart Wait for Free workstation Logging into EPR Logging out of EPR Reviewing EPR only Ordering Services in EPR only