Evaluation of Medical Command and Control Using Performance Indicators in a Full-Scale, Major Aircraft Accident Exercise

ORIGINAL RESEARCH Evaluation of Medical Command and Control Using Performance Indicators in a Full-Scale, Major Aircraft Accident Exercise Dan Gryth, MD, PhD; 1 Monica Rådestad, RN; 1 Heléne Nilsson, RN; 2 Ola Nerf, RN; 1 Leif Svensson, MD, PhD; 1 Maaret Castrén, MD, PhD; 1 Anders Rüter, MD, PhD 2 1. Department of Clinical Science and Education, Södersjukhuset, Karolinska Institutet, Stockholm Prehospital Centre, Stockholm, Sweden 2. Centre for Teaching and Research in Disaster Medicine and Traumatology, Linköping University, Linköping, Sweden Correspondence: Dan Gryth Department of Clinical Science and Education, Södersjukhuset Karolinska Institutet Stockholm Prehospital Centre S-118 83 Stockholm Sweden E-mail: dan.gryth@sodersjukhuset.se This work was supported by the Department for Disaster Medical Planning (REK), Stockholm, Sweden. This work was performed by the Stockholm Prehospital Centre (SPC) in cooperation with the Centre for Teaching and Research in Disaster Medicine and Traumatology (KMC), Linköping University, Linköping, Sweden. Keywords: aircraft accident exercise; airport drill; disaster evaluation; disaster medicine; disaster preparedness; lessons learned; performance indicators; prehospital management; safety; security; strategic management Abbreviations: CPR = cardiopulmonary resuscitation EMS = emergency medical services Received: 26 May 2009 Accepted: 15 June 2009 Revised: 13 July 2009 Web publication: 29 March 2010 Abstract Introduction: Large, functional, disaster exercises are expensive to plan and execute, and often are difficult to evaluate objectively. Command and control in disaster medicine organizations can benefit from objective results from disaster exercises to identify areas that must be improved. Objective: The objective of this pilot study was to examine if it is possible to use performance indicators for documentation and evaluation of medical command and control in a full-scale major incident exercise at two levels: (1) local level (scene of the incident and hospital); and (2) strategic level of command and control. Staff procedure skills also were evaluated. Methods: Trained observers were placed in each of the three command and control locations. These observers recorded and scored the performance of command and control using templates of performance indicators. The observers scored the level of performance by awarding 2, 1, or 0 points according to the template and evaluated content and timing of decisions. Results from 11 performance indicators were recorded at each template and scores >11 were considered as acceptable. Results: Prehospital command and control had the lowest score. This also was expressed by problems at the scene of the incident. The scores in management and staff skills were at the strategic level 15 and 17, respectively; and at the hospital level, 17 and 21, respectively. Conclusions: It is possible to use performance indicators in a full-scale, major incident exercise for evaluation of medical command and control. The results could be used to compare similar exercises and evaluate real incidents in the future. Gryth D, Rådestad M, Nilsson H, Nerf O, Svensson L, Castrén M, Rüter A: Evaluation of medical command and control using performance indicators in a full-scale, major aircraft accident exercise. Prehosp Disaster Med 2010;25(2):118 124. Introduction Disaster medicine can be difficult to evaluate scientifically. Rüter et al studied 13 major incident reports in Sweden. Based on these reports and results from a modeling process, standards for major incident medical management were developed. 1,2 These performance indicators enable minor parts of different components of disaster management to be evaluated. 3 Using these indicators, it is possible to identify areas in which improvements can and should be made, and less attention can be paid to what already functions adequately. Also, if standards are expressed numerically, statistical methods can be applied and results can be compared. 4 Before measuring results from performance indicators in real incidents, it is advisable to first develop a system for education and training in which these indicators are taught and used. If the staff involved in incident management are not informed of the criteria the evaluation is based on, the evaluation results probably will not lead to improvement. One advantage of the use of performance indicators is that they can be used through the whole chain, from education and training (and functional Prehospital and Disaster Medicine http://pdm.medicine.wisc.edu Vol. 25, No. 2

Gryth, Rådestad, Nilsson, et al 119 Performance Indicators: Prehospital Management Measurable indicator Within.min from alert Score Putting on tabard Directly 0 First report to dispatch Content of first report Formulate guidelines for response Establish contact with strategic level of command Liaison with fire and police Second report from scene Content of second report Indicating first patient transport Establish level of medical ambition First patient evacuated Information to media on scene 2 min 0 METHANE 0 3 min 0 5 min 1 5 min 0 10 min 0 Verifying first report Indicating first patient transport 0 10 min 1 15 min 1 30 min 0 TOTAL 3 Approval level >11 Table 1 Scoring of results according to template for prehospital command and control. Score: Right decision and in right time. Correct = 2 points, Partly correct = 1 point, Incorrect = 0 points Methods There were 99 passengers and crew; played by trained personnel, onboard the jetliner. Prior to the exercise, the simuexercise), to implementation, and most likely, all the way to the application during real incidents. The aim of this strategy is to identify areas that need improvement. This also can be applied during large functional disaster exercises, which often are expensive to plan and execute. Therefore, results can be evaluated and compared objectively. When results are published, lessons learned from exercises can lead to improvements in other organizations or agencies. On 09 October 2008, a large-scale, functional, disaster exercise was conducted in order to practice a realistic airplane accident at a major airport in Sweden. The scenario involved a commercial passenger airplane with technical problems that crashed as it attempted to land. These types of exercises involving airplane crashes are governed by regulations and should be repeated at regular intervals. 5 The aim of this study was to increase the ability to learn from the results of disaster exercises by applying performance indicators in a full-scale, major incident exercise at two levels of command and control: (1) the local level (on-scene and at the hospital); and (2) the strategic level of command and con- 0 Performance Indicators: Strategic Management Measurable indicator Declaring major incident Deciding level of preparedness Decision on additional resources to scene Deciding on receiving hospitals Establishing contact with incident officers at scene Deciding on guidelines for referring hospitals Brief information to media Formulate general guidelines in accordance with guidelines from scene Make sure there is information for definitive referral guidelines Evaluated if capacity of own organization is sufficient Notify guidelines on referring hospitals Within min from alert Score 1 2 3 2 3 2 5 2 10 0 10 2 15 1 15 1 20 1 30 1 40 1 TOTAL 15 Approval level >11 Table 2 Scoring of results according to template for strategic command and control. Score: Right decision and in right time. Correct = 2 points, Partly correct = 1 point, Incorrect = 0 points trol. Templates of previously developed performance indicators were used. The results are summarized in Tables 1 4. Based on these results, the goal of this exercise was to clarify if performance indicators could be useful to point out weaknesses within the organization and thus, make it easier to improve the system. In the future, the current results should be compared with other similar disaster drills, and results from real incidents using performance indicators. March April 2010 http://pdm.medicine.wisc.edu Prehospital and Disaster Medicine

120 Aircraft Accident Exercise Performance indicators: Hospital Management Measurable indicator Within minutes from alert Score Decide on level of preparedness 3 0 Formulate guidelines for hospital response 15 2 Inform media 15 1 Give information about resources to strategic level 25 1 Ensuring that there is a medical officer in Emergency Operation 30 2 Estimate need of intensive care unit beds 45 2 First information to staff 60 2 Estimate endurance of staff 90 2 Evaluate and report estimated shortage of own capacity 120 1 Evaluate influence on the daily hospital activities 120 2 Information plan for patients with postponed appointments and operations 180 2 TOTAL 17 Approval level >11 Table 3 Scoring of results according to template for hospital command and control Score: Right decision and in right time. Correct = 2 points, Partly correct = 1 point, Incorrect = 0 points lated victims were classified according to injury that they should portray. Twenty-nine were classified as severely injured (category T1/T2), 61 as moderately injured (T3), and nine persons (manikins) were classified as dead. Colored priority tags were used for each category; T1 = red, require immediate treatment; T2 = yellow, indicates urgent or delayed; T3 = green, indicates walking wounded. The strategic level of command and control and the dispatch center were located together. After decision from the strategic level of command and control, the dispatch center referred the injured to three receiving emergency hospitals in the region. Emergency medical services (EMS) participated with 19 ambulances and two medical first response vehicles. Five mobile medical teams from hospitals (one anesthesiologist, two nurses per team) and one ambulance helicopter (one doctor, one nurse, and one paramedic) participated. Two disaster care units (with tents and medical equipment) also participated in the exercise. Other responding agencies included fire and police departments and the LFV group Arlanda (the State enterprise and associated companies at Arlanda). 5 Experienced observers used a template of performance indicators for the evaluation at the three different localities: (1) local command and control on-scene; (2) local command and control at the hospital; and (3) strategic command and control. 1 The indicators covered early decision-making in the medical command and control as well as staff procedure skills (Tables 1 4). 4,6,7 All participants knew the date of the exercise, but received no information or training in advance about the use of performance indicators or method of evaluating the results. All staff acted in their normal professional role, and were familiar with what ought to be done according to the disaster plan. The indicators were derived from concept and process modeling by the Swedish Board of Health and Welfare. 8 Important processes included the use of time stamps according to what could be considered as a golden standard. These standards were built from the process of evacuation of the first patient from the scene, indicator No. 10 (Table 1). The time stamps of all other performance indicators regarding command and control were constructed in regard to the overall goal to provide the correct resources within a timeframe that supported the patient processes. The scale 0, 1, or 2 was based on the completeness of fulfilling each process indicator as judged by a trained and experienced evaluator. A score of 0 meant that the standard was not met, 1 indicated that the standard was met, but not with adequate content or within specified timeframe. A score of 2 indicated correct performance within the correct timeframe. Staff procedure skills were evaluated using the template as reference, and the best-achieved result from each indicator was the basis for scoring. One or two observers were placed in each of the management groups as well as adjacent to the prehospital medical officer. Twenty-two points was the maximum score; 11 points was considered an acceptable score. Results The prehospital command and control scored three points. One point was given for establishing contact with a strategic level of command and control (within 18 minutes), but there still was a lack of the provision of crucial information until 40 minutes. One point was scored for establishing of type of medical ambition; no cardiopulmonary resuscitation (CPR) on pulseless and non-breathing persons, but the decision was late. One point was given for the evacuation of the first patient from the scene (within 43 minutes) (Table 1). The overall performance of Strategic Command and Control rated a total 15 points (Table 2). No contact was made with the incident officers at the scene. Only the decisions made to send additional resources to the scene, and decisions for providing guidelines for the referring hospitals, Prehospital and Disaster Medicine http://pdm.medicine.wisc.edu Vol. 25, No. 2

Gryth, Rådestad, Nilsson, et al 121 Strategic Staff Skills/Hospital Staff Skills Measurable indicators Time/Points Score Strategic Hospital Assigning functions to staff members Directly on arrival 2 2 Positioning in room in accordance to above Directly 2 2 Designated telephone numbers Directly 1 2 Introduction of arriving staff member Max one minute 2 1 Not 0 1 2 Whiteboard, flip-chart S/H Utilization of equipment (only if equipment is Fax S/H available) Computer S/H Other (specify) S Average: 2 2 Staff briefing Max eight minutes 2 2 Not 0 1 2 Reports from staff members S H Content of staff briefing Summarizing S H New assignments S H Next staff briefing S H Average: 1 2 Telephone discipline (during staff briefing) 1 2 Not 0 1 2 Staff briefings S H Content of staff schedule Media contacts S H Meals S H Staff relief S H Average: 1 2 Summary oral to staff members 1 2 Summary written to report 2 2 TOTAL S = Strategic 17 21 H = Hospital Approval level >11 Table 4 Scoring of results according to template in staff skills at the strategic (S) and the local level (hospital; H). Score: Right decision and in right time. Correct = 2 points, Partly correct = 1 point, Incorrect = 0 points and the selection of receiving hospitals were accomplished correctly and within the established timeframe. All of the other performance indicators were accomplished only in part. Evaluation of the performance of the Hospital Command and Control unit indicated that it failed to determine the level of preparedness required, and at least partially accomplished all of the other performance indicators (Table 3). All of the other performances were done correctly within the established standard for the time frame. It achieved a total score of 17. The performance of staff skills was best for the hospital staff with the only compromised performance occurring in the introduction of arriving staff members (Table 4). The major difference between the performance of the Strategic group and the Hospital group was related to telephone discipline. Discussion In 2008, the Aviation Safety Network recorded a total of 32 fatal multi-engine airliner accidents, resulting in 577 fatal- ities and 39 ground fatalities. 9 Therefore, safety and security among airline companies and involved authorities is a high-priority concern. Although air transportation is the safest method of transportation, 10 there always is a risk that human error or technical problems cause fatalities. Take-off and landing are especially risky moments in aviation. In this disaster drill, a crash involving a jetliner with 99 persons on-board, a quite realistic scenario, was simulated. In this exercise, the first and second reports from scene to the strategic level of command and control were not given according to the regional disaster plan. It is a cornerstone for the first EMS crew to immediately send information to the strategic level of medical command and control. Several things contributed to this failure. First, there was a misunderstanding of what had been reported from the initial medical incident command to the strategic level before the hand-over to the new medical incident officer. According to the disaster plan, the first ambulance that arrives on-scene immediately March April 2010 http://pdm.medicine.wisc.edu Prehospital and Disaster Medicine

122 Aircraft Accident Exercise should establish command and control until a staff member with a higher level of management and medical competence arrives and takes over. Second, during the first 30 minutes, the radio traffic was busy, and it became difficult to communicate and send the reports in time. There also were too many unauthorized users of available radio frequencies and a lack of radio discipline. Also, the communication was not clear and concise. Disturbing noise sometimes made it difficult to hear the calls on the radio. There also were some technical problems, but most of the problems were from handling the communication equipment. During a major incident, it is important that the medical incident command reports the information to the strategic level of management within stipulated timeframes. If not, this can cause the strategic management to not get an overall picture of the scene in order to dispatch the correct equipment and staff, and without this information, it often is difficult to decide the number and name of hospitals that must be involved and to refer patients to hospitals with adequate levels of care. Due to this lack of information, it was difficult for the strategic level of command and control to make the necessary decisions within the correct timeframe (they were blind the first 40 minutes ). One consequence was the lack of information further down in the medical chain (e.g., hospitals). This, in turn, delayed decision-making for a major incident deferred preparation for receiving injuries, i.e., stopping elective surgery. During a disaster and in the beginning of a major incident, there is a shortage of resources. To save life and limb, it is important to have a good organization and use resources in the best way. Sometimes, medical personnel must reduce the level of medical ambitions for saving as many lives as possible. In those situations, it can cost vast time and resources if the medical personnel put too much effort into trying to save injured patients who statistically, have a very limited chance of survival. That can lead to the preventable deaths of other patients. 11 This may be why the initial management was uncoordinated and delayed the evacuation process. Standby ambulances were positioned on-scene, but did not receive the order to start evacuation. The first patient was evacuated after 43 minutes (from the arrival of the first unit) despite of several ambulances being present on-scene. The medical commander must decide when to give priority to evacuation of patients and not wait for more resources. A general time for this may be difficult to establish; however in previous studies, the first evacuation in most cases took place within 15 minutes of the arrival of the first ambulance. This also is the standard set in the performance indicators. 1,4 In general, the goal for a severely injured patient is to refer the patient directly to an advanced trauma center within one hour after the accident, also named the golden hour. 12,13 During major incidents, the goal of fast evacuation still exists, because the scene times affect the survival rate. 14,15 In trauma care, it also is important to go directly to the most appropriate hospitals. Therefore, the strategic level of command and control must provide the best available information for each of the ambulances. 16 The performance described above scored three points out of possible 22 points for the prehospital command and control, which implies that the structure of field management must be improved. The difference between the achieved score of three and the average score of 16.6 reported in previous studies is too much to be explained by the fact the indicators were not known to the participants. 4 Special emphasis should be made to study the reasons for the poor performance of prehospital management. The score not being higher than 15 out of 22 for the performance indicators at the strategic level of command and control was the result of the difficulty of making proper and timely decisions when there was a lack of adequate information from the scene of the incident. However, there never was any problem to provide the correct destinations for the ambulances, since the evacuation process started quite late. However, this score is not different from the score reported in a previous study in which all participants were well informed about the use of performance indicators. 16 The score from the evaluation the hospital management group was higher than was previously reported and was evidenced in management skills and staff skills. 6,7 However, the score for the management and staff skills at the strategic level of command and control was on the same level as previously reported and could possibly be improved by more education and training. Limitations The limitations of the study were that although all involved units had been informed of the exercise and had had time to prepare, they may not have acted in the same way as in a real incident. Another limitation is that in the methodology, there is not an absolute distinction between a score of 1 and a score of 0. Other limitation was that the use of performance indicators is a quantitative (scaling) instrument and does not give obvious clues regarding reasons for failures or successes. Conclusions Like all disaster drills, the evaluation and lessons learned are important features. Some evaluations are expressed in general terms or inadequately, and sometimes, make it difficult to express what must be improved. 18 20 An evaluation also can be difficult to draw conclusions from if it is expressed in general terms. By using performance indicators for evaluation, it is possible to demonstrate results numerically. Performance indicators can be used for the evaluation of a full-scale disaster exercise and for finding strong and weak points of the medical command and control at different levels of responsibility. Performance indicators can also be used when comparing results from different exercises. Acknowledgements The authors thank the team of qualified observers Caroline Hybinette, Henrik Lidberg, Eva Bengtsson, Ingrid Björklund, all from (KMC) and Milka Dinevik who have supported as with the tables (SPC). Prehospital and Disaster Medicine http://pdm.medicine.wisc.edu Vol. 25, No. 2

Gryth, Rådestad, Nilsson, et al 123 References 1. Rüter A, Örtenwall P, Vikström T: Performance indicators for major medical management A possible tool for quality control. International Journal of Disaster Medicine 2004;2:52 55. 2. Rüter A, Vikström T: Indicateures de performance: De la théorie à la pratique. Approache scientifique à propos de la médecine de catastrophe. Urgence Pratique 2009;93:41 44. 3. Rüter A, Nilsson H, Vikström T: Prehospital command and control, a certifying training concept. Scand J Trauma Resusc Emerg Med 2004;12:148 149. 4. Rüter A, Örtenwall P, Vikström T: Performance indicators for prehospital command and control in training of medical first responders. International Journal of Disaster Medicine 2005;1 4. 5. LFV. Available at http://www.lfv.se. Accessed 22 March 2010. 6. Rüter A, Nilsson H, Vikström T: Performance indicators as quality control for testing and evaluating hospital management groups: A pilot study. Prehosp Disaster Med 2006;21(6):423 426. 7. Rüter A, Örtenwall P, Vikström T: Staff procedure skills in management groups during exercices in disaster medicine. Prehosp Disaster Med 2007;22(4):318 321. 8. Rüter A, Lundmark T, Ödmansson E, Wikström T: The development of a national doctrine for management of major incidents and disaster. Scandinavian Journal of Resuscitation Emergency Medicine 2006;14:189 194. 9. Aviation Safety Network. Available at http://www.aviation-safety.net. Accessed 05 February 2009. 10. Ernsting J, Nicholson A, Rainford D: Aviation Medicine, 3d Ed. Oxford: Butterworth Heinemann, 1999. 11. Rüter A, Nilsson H, Vikström T: Medical Command and Control at Incidents and Disasters From the Scene of the Incident to the Hospital Ward. Studentlitteratur ISBN 91-44-01951-3. September 2006. 12. National Association of Emergency Medical Technicians: Prehospital Trauma Life Support, 6th [rev.] ed. St. Louis: Elsevier Mosby, 2007. 13. American College of Surgeons: Advanced Trauma Life Support Program for Doctors, 7th ed. Chicago: American College of Surgeons, 2004. 14. Driscoll P, Kent A: The effect of scene time on survival. The effect of scene time on survival. Trauma 1999;1:23 30. 15. Murdock D: Trauma: When there s no time to count. AORN J 2008;87(2):322 328. 16. Demetriades D, Martin M, Salim A, Rhee P, Brown C, Chan L: The effect of trauma centre designation and trauma volume on outcome in specific severe injuries. Ann Surg 2005;242(4):512 517; discussion 517 519. 17. Nilsson H, Rüter A: Management of resources at major incidents and disasters in relation to patient outcome: A pilot study of an educational model. European Journal of Emergency Medicine 2008;15:162 165. 18. Hsu E, Jenckes M, Catlett C, et al: Effectiveness of hospital staff mass-causellty incident training methods: A systematic literature review. Prehospital Disast Med 2004;19:191 199. 19. Gebbie K, Vales J, Merrill J, Morse S: Role of exercises and drills in the evaluation of public health in emergency response. Prehospital Disast Med 2006;21:173 182. 20. Hersche B, Wenker O: Case report: Lasing mining accident. Internet Journal of Rescue and Disaster Medicine 2000;2(1). March April 2010 http://pdm.medicine.wisc.edu Prehospital and Disaster Medicine