Mental models as a practical tool in the engineer s toolbox

Transcription

1 International Journal of Production Research, Vol. 43, No. 14, 15 July 2005, Mental models as a practical tool in the engineer s toolbox DAVID SINREICH*, DANIEL GOPHER, SHAY BEN-BARAK, YARIV MARMOR and RAKEFET LAHAT Davidson Faculty of Industrial Engineering and Management, Technion Israel Institute of Technology, Haifa 32000, Israel (Revision received July 2004) Industrial engineering methods are very successful in coping with well-structured systems and processes. However, when it comes to analysing, planning and controlling systems which contain unstructured processes, managers and engineers are faced with a much more difficult task. This is especially true in systems where teams and individuals have a significant role in the daily operation, monitoring and decision-making. In these situations, the processes may be performed differently by different individuals depending on their perceptions, concepts, ideas and perceived system status, all of which are denoted as the operators Mental Model (MM) of the system. This study develops a similarity measure to quantify the differences between MMs. This is done by eliciting the operators subjective perceptions of the system and their role within it (Mental Model), and comparing them to a standard description reference model which represents management s policy of how the system should be operated. Analysing the differences between these models may facilitate intervention approaches in closing these gaps and may help in creating better-synchronized and synergistic teamwork. Use of this similarity measure is demonstrated in a hospital ED environment. Keywords: Mental models; Quantitative processes comparison; Emergency departments operation; Emergency department patient types; System operations 1. Introduction Traditional industrial engineering methods for coping with the management of processes in production and service systems address processes systematically from a quantitative point of view. In most cases, these methods provide sufficient information to facilitate design, planning and control. This is especially true when dealing with structured processes in which the people who operate and monitor the process have very little control over the way the process is performed or the pace at which it is performed. In those cases, traditional methods can forecast, design, optimize and tune variables such as capacity, quality and resource consumption in an adequate way. *Corresponding author. sinreich@ie.technion.ac.il International Journal of Production Research ISSN print/issn X online # 2005 Taylor & Francis Group Ltd DOI: /

2 2978 D. Sinreich et al. In reality, alongside the structured processes one can find an abundance of unstructured processes processes in which operators and managers have a wide freedom of choice regarding their work style, process flow, order of precedence, etc. The thing that most distinguishes unstructured processes from structured ones is the high dependency on the individual in charge. This means that the process may be performed differently by different individuals depending on their perceptions, concepts, ideas and perceived system status (Mental Models). Chiasson (1997) states that MMs present additional essential information and insight into the system s potential performance beyond the information gained through traditional industrial engineering direct field observations. An example of an unstructured process is the Research and Development phase (R&D). In such a setting, the staff has almost total freedom to choose the way to conduct the process and there is room for considerable creativity. Nevertheless, it was found that most of the R&D projects in an organization require similar procedures, and consume similar resources (Adler et al. 1995). Thus, such processes can also be optimized to some extent. Another example is the medical processes performed in an Emergency Department (ED) by physicians and nurses. Managing those processes requires considerable flexibility in the decision-making by the personnel. On the other hand, there is a set of very rigid procedures and regulations that have to be followed. Since the criterion measures in unstructured processes are not entirely clear or well defined, no optimal or objective model can be developed for such processes; the best we can hope for is a model which is consistent and in correspondence with the decisions management makes in order to run the system. Sanders and Moray (1991) state that errors can be defined only in relation to correct and desired system behaviour. This means that the knowledge gained by a model which describes the correct system performance is essential for the detection of errors and faulty system performance. In light of these issues the question raised is whether quantitative methods can be used to analyse the performance of unstructured processes. Is it possible to predict the behaviour of systems which are driven by such processes? This study will try to show that there may be ways of dealing with these questions formally and effectively. The present study employs as a reference a model which describes the way management perceives and wishes these processes to be performed. This can be denoted as the Management s Operation Policy (MOP) model which constitutes management s agreed views of the resource usage in the system, i.e. the role workers and operators have in the system including the relationships between all team members. This can also be denoted the mental model of the system. This study tries to quantify the differences between the MOP model and the individual s (workers and operators) perceptions of these processes. Such a quantitative measure can facilitate a comparison between individuals operating the system and assess the degree of matching between their perceptions and the defined MOP reference. Even though each of these individuals has a unique background, knowledge and expertise, they have to agree on who is doing what and when, as stated by Cooke et al. (2000): Team members need to coordinate their activities with others who are working toward the same goal. This knowledge will determine their ability to operate as a cohesive group, where each member knows his responsibility, what role is expected of him and the overall timing, schedule and precedence of the different process stages. Based on the literature (Cannon-Bowers et al. 1993, 1995, Kraiger and

3 Mental models as a practical tool 2979 Wenzel 1997, Stout et al. 1999, Webber et al. 2000) common or shared task perceptions (situational maps) between team members, including workers and management, will lead to better operations, reduce the potential for system failures and achieve overall effective team functioning. Since these unstructured processes are most likely performed differently by different individuals, some may deviate from the defined standard procedure, norm or reference and result in reduced synchronization and coordination between team members. Such problems can lead to faulty performance. The fact is that faulty performance which stems from different or inappropriate perceptions or situational maps may cause system malfunction or even accumulate into a major system failure or breakdown. For example, the crash of Eastern Airline flight 401 in Miami was caused due to falling through the cracks of the flight control task while a minor troubleshooting task dominated the crew s attention (Rouse et al. 1992). This is a classical case where the main task is abandoned by all team members since the situational map of each of the team members states that the main task is being taken care of by someone else. Another example is the accident at the Three Mile Island facility, which was caused due to an incomplete, or irrelevant perceived system status (Rouse et al. 1992). In this case, concentrating on a local state variable while neglecting the complete system status resulted in a loss of the entire facility. 1.1 An overview of mental models When people interact with systems, simple as well as complex, they develop a subjective internal representation of the system and of their role in the environment in which they operate. Such a representation has been labeled in the literature as the Mental Model (MM) of the system. A MM is a combination of the individual s subjective perceptions, concepts, ideas and perceived system status. This term is common among researchers who study the interaction between humans and complex engineering systems (Moray 1999). The MM is usually a simplification of reality, which is achieved by decomposing the system in terms of an abstraction of hierarchies that facilitate different levels of reasoning (Rasmussen 1986, Moray 1990). However, one thing should be clear The term Mental Model is widely used and its meanings vary just as widely (Moray 1999). In contrast to objective models of systems and processes, MMs are prone to be biased in their estimates and assessments of the likelihood of cause and effect events (Tversky and Kahneman 1974) as well as misunderstand system operations (Evans 1995). MMs are frequently less consistent, and their structure is highly sensitive and depends on their usage rate. They evolve with experience and lack defined boundaries. Moray in Mental Model in Theory and Practice (1999) chapter 8 page 203 states the following: Nonetheless, MMs provide an individual with performance tools to describe and explain facts regarding the system, and the ability to predict future system behavior based on the current system state. The closer the match is between the MM of the system and the actual system, the higher are the performance levels that can be achieved (Rouse et al. 1992). The literature provides us with several classifications for MMs. This overview will briefly describe two of them. MMs of complex systems and processes and MMs of teamwork.

4 2980 D. Sinreich et al MMs of complex systems and processes. In the early 1970s it became evident that very few answers could be given within the traditional Human Factors research to problems which emerged when studying tasks of monitoring automated manufacturing processes or complex systems. Such tasks do not demand psychomotor interaction, however they challenge the mental abilities of the operator (Bainbridge 1974). Most studies of such situations resorted to simulations as opposed to real life systems due to methodological difficulties related to the ability of controlling the studied variables. Some of the studies were conducted in laboratories while others were conducted in the field using some sort of as-in-a-real-case script. The focus of these studies was cognitive. Bainbridge (1974) tried to understand the cognitive processes that made up the MM. Researchers found that individual s MMs differ. The difference may stem from the individual s expertise level (Kaufman and Patel 1998, Patel and Arocha 1998) or differences in the reasoning style of professionals in the same system MMs of teamwork. In the case of teamwork, the team shares resources and information and there should be an accepted distribution of responsibility. In addition, teams have to maintain adequate communication which will result in proper coordination and synchronization. Therefore, the existence of a common MM among all team members is especially important. Cannon-Bowers et al. (1993), Kraiger and Wenzel (1997), Stout et al. (1999) and Webber et al. (2000) and many others argue that the greater the similarity between team member s MMs, the better they perform as a team and the better the safety level is. Rouse et al. (1992) discuss the relationship between the lack of a common MM in teamwork as the main cause of system failure (e.g. the Three Mile Island accident and the crash of Eastern Airlines flight 401 in Miami). However, in other cases, the team requires versatility, original thought and unique ideas rather than wide acceptance. Such is the case in an R&D team, which is responsible for the conceptual design phase of a product. Different MMs of the team members may lead to better, more efficient design results as larger parts of the solution space are covered. Since MMs cannot be observed directly, indirect methods have been developed to elicit and capture them. Langan-Fox et al. (2000) list the following eight common elicitation techniques: Cognitive Interviewing Techniques, Verbal Protocol Analysis, Content Analysis, Visual Card Sorting Techniques, Repertory Grid Technique, Causal Mapping, Pairwise Rating Methods and Ordered Tree Technique. Sasse (1991) outlines several methodological problems related to the capturing of MMs. The first is over-reliance on performance data. This is related to the fact that people may do the right thing for the wrong reasons. Users performing well on a benchmarking task may hold unfounded misconceptions regarding the system, which may eventually lead, when certain circumstances arise, to faulty performance. Therefore, exclusive reliance on performance data in order to derive MMs is not recommended. Nevertheless, a lot can be learned from human errors while interacting with the system. A second drawback is the lack of ecological validity. This is related to the fact that most experiments in eliciting MMs use artificial situations where the individual s interaction with the system is short-term. A third problem raised is related to the paradigm of thinking aloud. The claim is that individuals are only partially cognizant of their acts, thus they cannot express aloud all the cognitive processes underlining task performance.

5 Mental models as a practical tool 2981 As a result, the description of MM based on those protocols may not be complete. Moreover, the fact that thinking aloud forces individuals to convert subconscious processes into conscious ones may bias the processes and alter them. For these models to be useful, the MMs need to be represented systematically and analysed using well-documented techniques. Langan-Fox et al. (2000) list the following three common analysis and representation techniques: Multidimensional Scaling, Distance Ratio Formula and Pathfinder. As argued earlier, processes may be performed differently by different individuals depending on their MMs. In a complex system these uncoordinated nonstandard processes may cause information loss and conflicts in the system. Cook and Woods (1994) and Cook et al. (2000) refer to these events as gaps in the system s MM. A single or even a few gaps may not have any harmful effects, however the mere existence of even one gap increases the likelihood of such an occurrence. These gaps may hinder the performance of the system since some procedures may not be performed in a desired well-optimized manner, as a result more personnel may be required for a longer duration than necessary. The more gaps there are, the greater the chances are for faulty performance. Eventually, conditions and system status may cause these gaps to align themselves such that a major system breakdown will occur. The present study attempts to develop a quantitative measure that can be used to assess the differences between the MMs of the different individuals operating the system as well as the differences of their MMs and the MOP model of the same system. Such a measure can be used to bridge the gap between the different MMs in order to enhance teamwork, coordination and facilitate better and more efficient overall system performance. In summary, the paper is written from an industrial engineering vantage point, and the main focus is on production processes in which the process does not have a formal, objective, or optimal structure, for determining the type of resources and the order in which they should be used in production. These situations are hence not amenable to traditional analysis and optimization methods of Industrial Engineering and are highly susceptible to individual variation in applied work sequences. The tenets of the present work are threefold: 1. Proposing the management operation model as a reference model; 2. Proposing, similar to other production processes, to conceptualize resources and transition between resources as the major building blocks of the management and staff members work process model; and 3. Emphasizing the importance of the similarity between models for the coordination and synchronization of the overall work in the unit. The major effort is to map and restate the properties of the working environment in terms of resources and transitions. We accept the fact that this is a simpleminded first-step approximation of the MM notion. However, the weakness of this approach, namely its simplicity, is also its strength. It is easy to examine its applicability, advantages, disadvantages and possible benefits. The paper is organized as follows: section 2 discusses ways to elicit and describe MMs and develops a method for quantifying the differences between these models. Section 3 describes the environment in which the developed model was tested. The comparison results are summarized in section 4. Finally, a discussion and final remarks appear in section 5.

6 2982 D. Sinreich et al. 2. Describing and quantifying mental models In order to assess and quantify the difference between MMs of the individuals in a work unit and the MOP model of the work processes in this unit, a common method to describe the MMs has to be used. Such a description should represent resource consumption, causality and probabilistic transitions of stages within the processes. These are all key factors in describing the process control mechanism. The literature provides several methods designed to describe processes, such as queuing models PERT/CPM charts. These models are quite complex and complicated to solve, while at the same time are based on simplified assumptions, which are not always realistic. Other researchers chose to use simulation to model processes and systems. Simulation enables accurate and precise modeling of processes and systems, however at the cost of having to model every minute detail of the system. The process description method chosen in this study is the common Process Chart (PC). This simple method meets all requirements it enables simple tracking of resource use and consumption, it represents very clearly the precedence dimension of the process while at the same time has high face validity and is simple to learn and understand even by non-professionals. Finally, PCs are generic enough to represent a wide array of processes ranging from healthcare to industrial systems. A few simple process charts are illustrated in figure 1. For example, all three process charts contain three resources,,,. In PC1, event D, performed by resource, follows event C, performed by resource. In PC2, event D by resource again follows event C performed by resource, and so forth. 2.1 Eliciting mental models using process charts An open interview (cognitive interview technique) will be used to elicit the MM. In these interviews the subjects will be asked to map the processes they are familiar with in the system. In the next phase, the simple methodology of describing a process using a process chart will be explained. Finally, due to problems related to eliciting MMs as reported in the literature and discussed earlier, it was decided that the subjects will be asked to verify and modify the charts themselves until they accept them as an accurate representation of the system s operation a b g a b g a b g A A A B B B C C D D D E Figure 1. Example of three different process charts.

7 Mental models as a practical tool 2983 Temporal data of process duration and of activity duration can also be collected at this stage. 2.2 Comparing process charts As indicated earlier, key factors in representing processes are resource activities and resource transitions (flows). Let us consider the process chart as a graph which comprises nodes (activities) and arcs (resource transitions). A similar approach in which the elicited data is viewed as a graph was reported by Langfield-Smith and Wirth (1992). In this study the comparison is done by calculating the differences between the adjacency matrices which represent the graphs (cognitive maps). In the current study, any similarity/dissimilarity measure between two process charts is based on the two components, denoted hereafter as activities (a) and relationships (r), which make up the graphs (process charts). The first component represents the activities (nodes) of the different resources in the charts, e.g. A, B, C, D and E in figure 1. The activity similarity measure a ij can be obtained using a ij e ij ¼ e ij þ b ij þ b ji, ð1þ where e ij denotes the number of identical activities in process charts i and j (in the coding phase identical activities that carry different names should be recognized and handled as such) and b ij denotes the number of activities that exist in PC i which do not exist in PC j (see that b ij 6¼ b ji ). It is clear that 0 a ij 1. In the case that both PCs are identical in terms of their activities (not necessarily their relationships), then we have a ij ¼ 1 while a ij ¼ 0 if no common activities exist, by definition a ij ¼ a ji. The second component represents the relationships between activities (arcs) in the chart. A relationship is defined by the activities it connects (there may be more than one connection between activities) and by the direction of the connecting arc, e.g. the arcs that connect activities AB, BC and CD in PC1 in figure 1. The first step in calculating the relationship similarity measure is to calculate the relationship intensity matrices H i between each pair of resources k and l for each of the process charts. The relationship intensity is a function of the number of arcs f i kl between each pair of resources and the weights y associated with each resource! k,! l. This value can be calculated as follows: h i kl ¼ f i kl! k! l : ð2þ In the case of probabilistic relationships the value of h i kl may be a non-integer number. Based on the relationship intensity, the sum of all the common arcs c ij and the sum of all exclusive arcs d ij between any two process charts i and j can be calculated, y Setting the resource weights is not a trivial issue. The weights can be set based on their relative cost or on how critical the relationships between the resources are to the safe completion of the entire process or even based on management wishes. Nonetheless this issue is beyond the scope of this paper.

8 2984 D. Sinreich et al. as shown in equations (3) and (4), respectively. Finally, the relationship similarity measure r ij can be obtained using equation (5). c ij ¼ X X minfh i kl,h j kl g: k l ð3þ d ij ¼ X X h i kl h i kl : k l ð4þ r ij ¼ cij c ij þ d ij : ð5þ It is clear that 0 r ij 1. In the case where both PCs are identical in terms of their relationships (arcs) r ij ¼ 1, while r ij ¼ 0 if no common relationships exist between the two process charts. By definition r ij ¼ r ji. Finally, based on the activity and relationship similarity measures, a combined measure s ij, which defines the overall similarity level between any two process plans i and j, is calculated as the average of the individuals similarity measures: s ij ¼ aij þ r ij : ð6þ 2 In this model the absence of any resource in one of the process charts has the same effect on the a ij measure regardless of which resource it is. In the case where not all resources are of the same importance, there are those which are more critical than others, indicating that not all dissimilarities are of equal importance. To incorporate this into the model a weight has to be associated with each resource to reflect its relative importance. The comparison method presented in Langfield-Smith and Wirth (1992) does not make a distinction between activities and relationships and, as a result, this information is lost in the comparison. In contrast, the similarity calculation method suggested in the current study distinguishes between the two. In the case where low similarity values are obtained, the current method enables management to identify the cause of the discrepancies. This distinction is important in our opinion, especially since management resource allocation decisions are based on which resource is performing each of the activities. In this way management can determine if each resource knows which activities fall under his or her responsibility and whether they know with whom they need to interact in each case or event. In addition, since the adjacency matrices which represent the cognitive maps are sparse, similarity results obtained by the current similarity calculation method better represent the actual similarity levels between the subjects perceptions. In contrast, the difference measures obtained by Langfield-Smith and Wirth (1992) are all concentrated near the 0 extreme point, which makes it much harder to differentiate between the individuals tested. 2.3 Comparing two process charts an example In order to illustrate the calculation procedure of the similarity measure, three process charts, PC1 3 presented in figure 1, are used. Choosing PC1 and PC2, the following values are obtained: e 12 ¼ 3, b 12 ¼ 1, b 21 ¼ 1. Using these values in equation (1) results in an activity similarity measure

9 Mental models as a practical tool 2985 of a 12 ¼ 0.6. It should be noted that, in the case where activity D is performed by different resources, it should be considered as two different activities. Choosing PC1 and PC3: e 13 ¼ 3, b 13 ¼ 1, b 31 ¼ 1anda 13 ¼ 0:6: Choosing PC2 and PC3: e 23 ¼ 2, b 23 ¼ 2, b 32 ¼ 2anda 23 ¼ 0:333: Using the vector weights (2, 1, 1) for each of the resources illustrated in figure 1 (,, ), respectively, the relationship intensity matrices H i for each of the three process charts PC1 3 can be calculated using equation (2) as follows: H 1 ¼ B A ¼ B 0 0 A, H 3 ¼ 2 0 0C A, H2 ¼ B 0 0 A : Using these values in conjunction with equations (3) and (4) the common and exclusive relationship values can be calculated as follows: c 12 ¼ 0 þ 2 þ 0 þ 0 þ 0 þ 1 þ 0 þ 0 þ 0 ¼ 3, c 13 ¼ 2, c 23 ¼ 2, d 12 ¼ 0 þ 0 þ 0 þ 0 þ 0 þ 0 þ 2 þ 1 þ 0 ¼ 3, d 13 ¼ 7, d 23 ¼ 6: Based on these values and equation (5) the relationship similarity measure can be calculated as follows: r 12 ¼ 3=ð3 þ 3Þ ¼0:5, r 13 ¼ 0:222, r 23 ¼ 0:25: Finally, the overall similarity measure between the three process charts illustrated in figure 1 can be calculated using equation (6) as follows: s 12 ¼ð0:6þ0:5Þ=2 ¼ 0:55, s 13 ¼ 0:41, s 23 ¼ 0:29: These calculations show that the greatest similarity is between PC1 and PC2 while the lowest similarity is obtained when comparing PC2 and PC3. 3. Environment case description The setting that was chosen to test the proposed tool is an Emergency Department (ED) in a large rural hospital in Israel, which handles about patients a year. The ED is divided into three major wards: the surgical-orthopedic ward, the internal ward and the internal fast-track. Internist physicians serve in both the internal and fast-track wards alternately. Surgeons and orthopedists serve in the surgicalorthopedic ward. Nurses in the ED serve alternately in all wards. 3.1 Reference model The first step in the project was to determine the MOP model of all processes within the investigated system which will serve as a reference model for comparison with all other elicited staff MMs.

10 2986 D. Sinreich et al. Table 1. Patient type and mix in the ED. Ward Patient (process) type In the ward (%) In the ED (%) Combined (%) Surgical-orthopedic Minor trauma Minor surgical Severe surgical Orthopedic Internal Fast track Severe internals one nurse Severe internals two nurses The process of developing the MOP reference model started by interviewing the EDs managerial staff. Through these interviews the procedure of how reality can be translated into a process chart was explained. Next, using the gathered input the different process charts were developed. The EDs director physician, the head nurse and the director of the surgical-orthopedic ward were then asked to verify and modify the charts until they accepted them as an accurate representation of the system s operation. Through these iterative sessions, seven main processes were identified (listed in table 1). For each patient type a process chart was extracted representing the treatment a typical patient type goes through. In addition, management was asked to estimate the relative patient mix. Figure 2 illustrates the process that a fast-track patient goes through as management views it. The second stage was to verify these processes through direct observations of the ED staff in operation. During a period of nine days, data were collected in the ED. The collected data recorded 1528 patients, which represent about 90% y of all the administered patients in the ED in that period. The recorded mix of the seven patient types identified is listed in table 1. Due to the nature of the chosen system, it is obvious that in this case in order to achieve all system goals (patients care) the team members have to share a common view and agree how and when to perform each stage of the process. 3.2 Eliciting the process chart from the participating medical staff Five physicians and the four nurses who form part of the permanent ED medical staff agreed to participate in the study; the information related to these staff members is summarized in table 2. Nurses usually work shifts in both wards, so they are familiar with all types of activities in the ED performed around the clock. Physicians who are part of the permanent ED personnel usually work in their specific ward. The elicitation process started with interviews that were conducted with the chosen medical staff members. Through these interviews the procedure of how reality can be translated into a process chart was explained. Next, using the gathered input the different process charts were developed. The physicians and nurses were then y The 10% who were not included contain mostly contaminated data and a few rare cases, which, in most cases, are not handled by the main investigated ED.

11 Mental models as a practical tool 2987 Physician Nurse Stretcher bearer Imaging Laboratory Other Reception Vital signs Blood ECG - 20% I.V ECG gauge Blood pressure gauge 50% Examination Blood sample 50% X-ray Blood sample Collecting replies Decision Recording Discharge Administrative discharge Notation: Parallel events by definition 10% Relative frequency of events. Figure 2. Internal fast-track patient type one of the seven processes in the reference model. Table 2. Medical staff participating in the study. Participant Period of service as a professional (years) Period of service in the ED (years) Education Typical shifts Main expertise MD Specialistþ Morning, evening Internist sub-specialty MD Specialist Morning Internist MD Specialist Morning, evening Family MD MD Morning, evening, night Orthopedist MD Specialist Morning, evening, night Surgeon Nurse BA Morning, evening, night Nurse Registered Morning, evening nurse Nurse Registered Morning, night nurse Nurse Registered nurse Morning, evening

12 2988 D. Sinreich et al. asked to verify and modify the charts until they accepted them as their accurate representation of the system s operation. 4. Results Each member of the ED medical staff was asked to identify the major patient types handled by the ED and chart the process each patient has to go through. Figures 3 and 4 illustrate two elicited process charts, one of an orthopedic patient as obtained from MD 4 and one of a fast-track patient as obtained from Nurse 4. If a specific Physician Nurse Stretcher bearer Imaging Laboratory Other Reception Vital signs blood Examination Samples Decision Cast Treatment discharge/ Admission Figure 3. The process chart of an orthopedic patient as captured from MD 4. Physician Nurse Stretcher bearer Imaging Laboratory Other Reception 60% Reception treatment 40% Examination 30% X-Ray Samples Consultation Blood Samples Decision Discharge Figure 4. The process chart of a fast-track patient as captured from Nurse 4.

13 Mental models as a practical tool 2989 patient type was not identified by one of the participating staff the matching with the reference model was made based on the closest patient type model which was identified. No participant identified more patient types than the seven types of the reference model. Since MD 1 3 are internists who operate in the internal ward, only three patient types which pertain to this ward were considered (fast-track, severe internal one nurse and severe internal two nurses). Also, as MD 4 5 operate in the surgical-orthopedic ward, only four patient types which pertain to this ward were considered (minor trauma, minor surgical, severe surgical and orthopedic). Table 3 summarizes the process identified by each of the staff members participating in the study. 4.1 The similarity between the elicited MM and the reference model Once all participants process charts were derived, the similarity level s ij between each of these models and the appropriate MOP reference model was evaluated using the measure described in section 2.2. When calculating the intensity relationship matrices, the ED physicians and nurses were considered to be scarce resources, therefore their associated weight was set to 2 while the weight associated with the rest of the ED resources was set to 1. The similarity measures values when comparing the elicited models illustrated in figures 3 and 4 with their corresponding reference model are 0.35 and 0.38, respectively. These are below average similarity values and represent a different view some ED staff members have regarding the processes they are in charge of. For example, the process chart illustrated in figure 4 shows that Nurse 4 believes that, in 40% of the fast-track patients she sees, she can make the decision whether to send them to X-ray, laboratories or even summon a specialist, without consulting the fast-track physician. Figure 5 displays the similarity results for each of the participants with the MOP model and the average for each patient type (process chart). It is important to note that the graphic proximity of two points in two separate graphs does not necessarily indicate a close match between these two elicited process charts. However, it does indicate that these two process charts are within a similar distance from the MOP process chart. Thus, the question is what was the gap between the MOP chart and the different process charts obtained from the medical staff participating in the study? The comparison results illustrated in figure 5 show that the average similarity measure was 0.47 (0.56 for minor trauma patients and 0.41 for minor surgical patients). These values show relatively sizable differences between staff members perceptions of the processes and management s operation policies. 4.2 Cross-comparison between elicited process charts The next step in the study included a cross-comparison for each of the elicited patient types. These comparisons assess the similarity or agreement level between the different staff members regarding a specific patient type. Two extremes among the staff members were identified. The lowest agreement measure (0.4) was obtained in the comparison of the orthopedic patient type, presented in table 4 (all similarity values exceeding 0.5 are highlighted). This patient type also has the lowest similarity results in the two professional sub-groups, nurses and physicians, of 0.42 and 0.38, respectively. It is also interesting to note that only two of the six relevant

14 Model Table 3. The process identified by participating staff in comparison with the reference model. Process type Reference model Minor trauma Minor surgical Severe surgical Orthopedic Fast track Severe internals one nurse Severe internals two nurses MD 1 Fast track Intermed. internals Severe internals MD 2 Internals Internals Internals MD 3 Fast track Intermed. internals Severe internals MD 4 Minor orthop. Intermed. orthop. Urgent orthopedic Orthopedic illnesses MD 5 Trauma Surgical Surgical Trauma Nurse 1 Trauma Trauma Trauma Trauma Fast track Regular internal Complicated internal Nurse 2 Trauma Trauma Trauma Trauma Fast track Internals Internals Nurse 3 Minor trauma Minor trauma Minor trauma Orthopedic Fast track Immediate internal Immediate internal Nurse 4 Walking trauma Walking trauma Lying trauma Lying trauma Fast track Minor internals Intermed severe internals 2990 D. Sinreich et al.

15 Mental models as a practical tool 2991 Figure 5. The similarity value between participants MM and the reference model. Table 4. Comparison between elicited process charts for orthopedic patients. Nurse 1 Nurse 2 Nurse 3 Nurse 4 MD 4 MD 5 Nurse 1 Nurse Nurse Nurse MD MD Table 5. Comparison between elicited process charts for fast-track patients. Nurse 1 Nurse 2 Nurse 3 Nurse 4 MD 1 MD 2 MD 3 Nurse 1 Nurse Nurse Nurse MD MD MD staff members identified this patient type (see table 3). Nonetheless, table 1 shows that 28% of the patients attending the ED were classified in this category. The highest similarity measure (0.61) was obtained in the comparison of the staff s scores for the fast-track patient type, presented in table 5. In this case the majority of the similarity scores were higher than 0.5. The scores obtained for this patient type reveal that the agreement level between nurses as a sub-group was considerably larger than the agreement level between physicians as a sub-group or even the agreement level between the two different sub-groups (physicians and nurses). In this case, six of the seven relevant staff members included a fast-track category in their patient type classification (see table 3). While MD 2, who did not identify this category, claimed that all internal patients go through the same process.

16 2992 D. Sinreich et al. Table 6. The average similarity values achieved within and between each professional group for each patient type. Minor trauma Minor surgical Severe surgical Orthopedic Fast track Severe internal one nurse Severe internal Two nurses Avg n Avg N Avg n Avg n Avg n Avg n Avg n Overall MDs Nurses MDs nurses He ignored the fact that this process takes place at a different location, and chose to describe it within all other internal patient procedures. The comparison of the average similarity measures obtained for each professional group in each of the seven patient types is presented in table 6. The highest and lowest average score obtained for each group is highlighted, while the overall high and low average scores are underlined. Since the fast track is operated by one physician and one nurse, the information presented in table 5 indicates that when MD 1 works at the fast track with one of the nurses it is likely that fewer disagreements will arise compared with the case were MD 2 or MD 3 work there with Nurse 4. This may eventually lead to better coordination and fewer mishaps. The elicited charts, which represent the processes these two patient types undergo, were also compared using the method developed by Langfield-Smith and Wirth (1992), as shown in tables A1 and A2 of Appendix A. The results obtained by the two methods reveal that comparisons which achieved high similarity values using one comparison method also achieved high similarity values using the other method and again the average similarity between the process charts which describe the fasttrack patient process was greater than the average similarity between the process charts which describe the orthopedic patient process. 5. Discussion The results obtained in this field study of the ED unit support the suggestion that the similarity measure presented in this paper can provide meaningful insight into the way the system operates. The results can lead management to the weak links in the operation chain and assist professionals in locating gaps in the staff s perceptions of the system and its processes by comparing them to an MOP reference model, a model which represents the system as management conceptualized it and wishes it to operate. Once gaps are detected and analysed, reinforcement and external intervention measures have to be introduced such as staff instruction and training. At this stage the similarity measure can be re-employed to assess the success of the intervention program and determine if further action is needed. Another aspect of this study is teamwork-related MMs. When a team is required to work synergistically in order to achieve its objectives, the main issue is the level

17 Mental models as a practical tool 2993 of agreement among team members on how best to perform the required tasks. What is the role of each team member, and how should responsibility be divided? Rouse et al. (1992) claim that the more similar the team s MMs are, the more efficient and more synergistic its performance. The similarity measure developed in this study is a possible tool to quantify the level of mutual agreement among the individuals which constitute a team. 5.1 Evaluating the results obtained for the ED Although this study was not designed as a comprehensive analysis of the presented ED, it will not be complete without briefly discussing the results obtained, and demonstrating the use of the suggested similarity measure. The average agreement level between the staff s elicited MMs and the management reference model is This is a relatively low score, which may indicate that the MOP model and those of the medical staff at the ED unit vary considerably. This is not to say that management s MOP model of the system is the optimal way to operate the system. Nevertheless, this is the model which guides management in all the decisions taken regarding the system, e.g. resource allocation, priority setting and operation. If this is not a true representation of the way the system operates, these may be the incorrect decisions to make. On the other hand, if this is the correct way to run the ED, the results indicate that the staff is not adhering to management s policy guidelines and instructions, which may eventually increase the probability of conflict, confusion and errors. On the other hand, it is conceivable to have some differences between these models. Previous studies have shown that staff members view the system in a much simpler and narrower manner than management s view. The present study reinforces these findings, as shown in table 3. The nurses identified only four patient types compared with management s seven types. At the same time the appropriate physician groups identified three internal patient types and 2.33 trauma patient types compared with management s four and three patient types, respectively. The elicited process plans were less detailed in both professional sub-groups. Nurses and physicians described, on average, fewer activities (16% and 18%, respectively) and fewer activity relationships (7% and 13%, respectively) than management s operational policy model. The staff s MM cross-comparison revealed differences in their description of the processes patient types go through. As shown in table 6, the highest and lowest agreement levels were obtained for the fast-track and orthopedic patients, respectively. This issue requires further elaboration. The causes of the low agreement levels regarding the orthopedic process may stem from the following reasons: orthopedic patients were treated in the same physical space with other trauma and general surgical patients. This mix may cause confusion and as result lower levels of organization in the process. In addition, the physician in charge of the orthopedic process was not a licensed specialist in orthopedics. Finally, in recent years management has not invested any special effort towards improving this situation. The result of this study may suggest that such an action is warranted. In comparison, the process which obtained the highest agreement scores can be argued to have benefited from management s increased focus. In the last few years management has invested considerable time and effort into setting guidelines and improving this process in order to improve its quality and reduce

18 2994 D. Sinreich et al. patients duration-of-stay. As a result, fast-track patients are treated in an area separate from all other ED processes. The physicians in charge (alternately) are licensed internists. These two contrasting cases exhibit two opposites of management s operational attention. The similarity measure suggested in this study was able to detect these differences. High levels of agreement were obtained between staff and management, and between and within professional sub-groups in the case where management played an active role. On the other hand, disagreement and lack of coordination were revealed in the process which management neglected. Teamwork in the ED refers to the ability of the two professional sub-groups, nurses and physicians, to interact efficiently in order to handle the different patient types. The similarity measure was able to reveal the average agreement levels between these sub-groups. In principal, cooperation levels should be as high as possible. However, further research is required in order to determine acceptable agreement levels between professional sub-groups that will ensure efficient interaction and cooperation. In conclusion, the similarity measure suggested in this study was able to capture fundamental issues related to operation of the analysed unit. It shows the possible contribution of an industrial engineering-based modeling approach to the study of less structured working environments. In particular, it is a formal approach to the study of management and workers mental models which provides useful information to guide management decisions and operations. Appendix A Tables A1 and A2 illustrate the difference measure between the elicited process charts as calculated using the method suggested by Langfield-Smith and Wirth (1992). In this method a 0 value indicates both graphs are identical, while a difference value of 1 indicates both graphs do not have anything in common. Comparing our results with the results above reveals that the best match between staff members for the fast-track and orthopedic patients was achieved in both case between Nurses 1 and 3 (0.82 and 0.51). As expected the DR value between these two nurses was the lowest at 0.03 and 0.1, respectively, which also indicates the best match. Moreover, a statistical paired t-test revealed that the hypothesis that the Table A1. Comparison between elicited process charts for orthopedic patients. Orthopedic Nurse 1 Nurse 2 Nurse 3 Nurse 4 MD 1 MD 2 MD 3 MD 4 MD 5 DR Nurse Nurse Nurse Nurse MD 1 MD 2 MD 3 MD MD

19 Mental models as a practical tool 2995 Table A2. Comparison between elicited process charts for fast-track patients. Fast track Nurse 1 Nurse 2 Nurse 3 Nurse 4 MD 1 MD 2 MD 3 MD 4 MD 5 DR Nurse Nurse Nurse Nurse MD MD MD MD 4 MD 5 average results each of the tested individuals achieved using both methods are similar cannot be rejected (P ¼ orthopedic patients, P ¼ fast-track patients). References Adler, P.S., Mandelbaum, A., Nguyen, V. and Schwerer, E., From project to process management: an empirically-based framework for analyzing product development time. Manage. Sci., 1995, 41, Bainbridge, L., Analysis of verbal protocols from a process control task. In The Human Operator in Process Control, edited by E. Edwards and F. Lees, Chapter 10, 1974 (Taylor & Francis: Oxford). Cannon-Bowers, J.A., Salas, E. and Converse, S., Shared mental models in expert team decision making. In Individual and Group Decision Making: Current Issues, edited by N. Castellan and J. John, pp , 1993 (Lawrence Erlbaum: Hillsdale, NJ). Cannon-Bowers, J.A., Tannenbaum, S.I., Salas, E. and Volpe, C.E., Defining team competencies and establishing training requirements. In Team Effectiveness and Decision Making in Organizations, edited by R. Guzzo and E. Salas, pp , 1995 (Jossey-Bass: San-Francisco, CA). Chiasson, D., Mental models in industrial jobs. In Engineering Psychology and Cognitive Ergonomics: Job Design and Product Design, edited by D. Harris, Vol. 2, pp , 1997 (Ashgate Publishing: Aldershot, UK). Cook, R.I. and Woods, D.D., Operating at the sharp end: the complexity of human error. In Human Error in Medicine, edited by M.S. Bogner, pp , 1994 (Lawrence Erlbaum: Hillsdale, NJ). Cook, R.I., Render, M. and Woods, D.D., Gaps in the continuity of care and progress on patient safety. British Med. J., 2000, 320, Cooke, N.J., Salas, E., Cannon-Bowers, J.A. and Stout, R., Measuring team knowledge. Human Factors, 2000, 42, Evans, J.B.T., Thinking and reasoning. In Longman Essential Psychology Cognitive Psychology, pp , 1995 (Longman: New York). Kaufman, D.R. and Patel, V.L., Progressions of mental models in understanding circulatory physiology. In Human Cognition A Multidisciplinary Perspective, edited by I. Singh and R. Parasuraman, pp , 1998 (Sage Publications: New Delhi, India). Kraiger, K. and Wenzel, L.H., Conceptual development and empirical evaluation of measures of shared mental models as indicators of team effectiveness. In Team Performance Assessment and Measurement: Theory, Methods and Application, edited by M.T. Brannick and E. Salas, pp , 1997 (Lawrence Erlbaum: Hillsdale, NJ). Langan-Fox, J., Code, S. and Langfield-Smith, K., Team mental model techniques, methods, and analytic approaches. Human Factors, 2000, 42, Langfield-Smith, K. and Wirth, A., Measuring differences between cognitive maps. J. Op. Res. Soc., 1992, 43,