Role of Exercises and Drills in the Evaluation of Public Health in Emergency Response

Transcription

1 ORIGINAL RESEARCH Role of Exercises and Drills in the Evaluation of Public Health in Emergency Response Kristine M. Gebbie, DrPH, RN; 1 Joan Valas, MS, RN; 1 Jacqueline Merrill, MPH RN; 1 Stephen Morse, PhD 2 1. Center for Health Policy, Columbia University School of Nursing, New York, New York USA 2. National Center for Disaster Preparedness, Mailman School of Public Health, Columbia University, New York, New York USA Correspondence: Joan Valas, MS, RN Columbia University School of Nursing Center for Health Policy 630 West 168th Street-Mail Code 6 New York, NY USA jv16@columbia.edu Keywords: agency; criteria; emergency response; evaluation; event; exercises; drills; performance; public health; readiness; standards Abbreviations: CDC = Centers for Disease Control and Prevention DHS = (US) Department of Homeland Security FEMA = (US) Federal Emergency Management Agency IAP = incident action plan IC = incident command ICS = incident command system JIC = jurisdiction-wide information center NACCHO = (US) National Association of County and City Health Officials PIO = public information officer PPE = personal protective equipment SOCO = single overriding communication objective UTL = universal task list Received: 22 June 2005 Accepted: 19 July 2005 Revised: 02 August 2005 Web publication: 21 June 2006 Abstract Introduction: Public health agencies have been participating in emergency preparedness exercises for many years. A poorly designed or executed exercise, or an unevaluated or inadequately evaluated plan, may do more harm than good if it leads to a false sense of security, and results in poor performance during an actual emergency. At the time this project began, there were no specific standards for the public health aspects of exercises and drills, and no defined criteria for the evaluation of agency performance in public health. Objective: The objective of this study was to develop defined criteria for the evaluation of agency performance. Method: A Delphi panel of 26 experts in the field participated in developing criteria to assist in the evaluation of emergency exercise performance, and facilitate measuring improvement over time. Candidate criteria were based on the usual parts of an emergency plan and three other frameworks used elsewhere in public health or emergency response. Results: The response rate from the expert panel for Delphi Round I was 74%, and for Delphi Round II was 55%. This final menu included 46 public health-agency level criteria grouped into nine categories for use in evaluating an emergency drill or exercise at the local public health level. Conclusion: Use of the public health-specific criteria developed through this process will allow for specific assessment and planning for measurable improvement in a health agency over time. Gebbie KM, Valas J, Merrill J, Morse S: Role of exercises and drills in the evaluation of public health in emergency response. Prehosp Disast Med 2006;21(3): Introduction The roles of exercises and drills as a mechanisms for education, experience, and evaluation of emergency response have increasingly become important for all response partners, including the public health system. 1 3 The ability of the public health community to perform exercises, drills, and emergency simulations for both internal and external evaluation of plan effectiveness has been improving, and includes greater involvement and more ties to other components of the response system. The objectives of the research are: (1) to clearly describe the levels of exercises and drills appropriate to levels of preparedness, the purpose of the drill (management and leadership, operational performance or systems performance), and the size of the agency and community served; and (2) to develop a standardized set of criteria for use by local public health agencies or others to evaluate an agency s response during a drill, exercise, actual emergency event, or after-exercise evaluation are described in this report. In addition, a subsequent phase of the same project was undertaken to develop a standardized method for recording a drill or exercise experience that can be used for the evaluation and documentation of public health readiness. The emergency response readiness of the public health system increasingly has been an important topic for research, development, and action in the May June Prehospital and Disaster Medicine

2 174 Role of Exercise and Drills United States since 1996, with the events of 11 September 2001 providing even more stimulus. 4 6 This research included: 1. the development of competencies in emergency preparedness for individual public health workers; 2 2. training for public health agency staff and their community partners; 4,7 3. improved electronic and other communication capacities at all levels of public health practice; 8,9 4. federal guidance to state public health agencies (and through state agencies to local public health agencies) on critical agency capacities needed for emergency response; 3 5. strengthened planning for emergency response throughout the public health community; 10 and 6. the creation of Project Public Health Ready, a system that certifies that a local public health agency has a staff competent in emergency response, a plan for emergency response, and has tested its preparedness through exercises or drills. 11 Need for Exercises Public health agencies have been participating in at least some emergency preparedness exercises for many years, (e.g., the participation of local public health agencies in emergency drills in the area surrounding nuclear power plants). 12 Many jurisdictions have regular experiences with emergencies caused by weather. 6 However, a poorly designed or executed exercise, or an unevaluated or inadequately evaluated plan, may do more harm than good if it leads to a false sense of security, resulting in poor performance during an actual emergency. Also, the public may be put at risk and confidence in the public health system undermined. The National Association of County and City Health Officials (NACCHO) has provided a useful tool, Bt Create, that guides an agency through the development of a locally relevant exercise or drill. 13 However, at the time this project began, public health had not developed its own specific standards for the public health aspects of exercises and drills, and no defined criteria for the evaluation of agency performance existed. Because exercises and drills are expected to provide a simulation of actual emergency experiences, the format for assessing response during a drill or exercise should be the template for evaluation of an actual response. 8,12 Evaluating the degree to which the exercise and its results are used for ongoing organizational development also is crucial. There currently are no published guidelines for assessing public health emergency response. General guidelines developed by the US Department of Homeland Security (DHS) have been developed, but are not specific for the public health agency role, both at the state and local levels. 19 It is essential that the evaluation of emergency response at the local level be consistent with expectations of emergency response at the state public health agency level, thus assuring that critical elements of response are developed in a consistent manner. Since this project began, several US organizations started initiatives that increase contribution from public health within the emergency response sector. One major challenge to this project was that public health agencies already had been conducting or participating in exercises and drills, using a wide range of resources and working with US national, state, and local emergency preparedness organizations, the national network of Academic Centers for Public Health Preparedness, and private consulting firms. This resulted in a lack of standardization that produced different vocabularies, expectations, and thus, different approaches in exercise assessment in the US. The purpose of this study was to describe the levels of exercises and drills appropriate to levels of preparedness and the purpose of these drills, and to develop a standardized set of criteria for use by local public health agencies to evaluate an agency s response during an exercise. Methods To develop criteria using expert opinions, this Delphi study was conducted at the Center of Health Policy at the Columbia University School of Nursing. The study was determined to be exempt from Institutional Research Board approval on 02 January The first step in this study was to define the types and levels of exercises based on critical review of existing national standard language developed by the DHS. 21 This review resulted in a document that was circulated in the public health community for feedback and comment. Subsequently, it was made available in its final form via the project Website, and is summarized in Table The second step was to identify criteria for evaluating local public health emergency exercises. This was achieved through a two-round Delphi survey utilizing a 26-member expert panel to gain insight and consensus to the development of public health agency criteria for the evaluation of emergency exercises. The experts for this panel were selected based on experience and expertise from representative public health professional associations (boards of health, local and state health officials), local and state public health agencies, the Centers for Disease Control and Prevention (CDC), the DHS, and the Federal Emergency Management Agency (FEMA). Special attention also was given to selecting a panel that included representatives from each of the ten geographic regions of the country. The composition of the panel is summarized in Table 2. Delphi Survey Round I Project staff reviewed an extensive range of materials from the DHS, the FEMA, the Centers for Public Health Preparedness (funded by the CDC), and local public health departments. In developing the Round I Delphi instrument, it was decided to present criteria for the evaluation of public health exercises in a manner consistent with and understandable to those working in the broader emergency management area. For that reason, the researchers chose to organize the document in categories identified by the major components of a public health agency emergency management plan as described by NACCHO, which is a format consistent with more general emergency plans. However, because other ways of describing public health performance or emergency response are in use, each of the Prehospital and Disaster Medicine Vol. 21, No. 3

3 Gebbie, Valas, Merrill, et al 175 Type of Exercise Definition Examples Discussion-Based Exercises The starting points in a building block approach to exercises. Primarily used to familiarize agency and staff with existing plans and capabilities. Led by facilitators and presenters to keep participants on track in meeting the objectives of the exercise. Seminar Workshop Tabletop Games Operation-Based Exercises Used for the purpose of assessing and validating emergency preparedness polices, plans, and procedures. They include drills, functional exercises, and full-scale exercises. There is an actual response and a mobilization of personnel and apparatus over a designated, extended period of time. Drill Functional Exercise Full-Scale Exercise (FSE) Table 1 Summary of definitions Expert Panel Respondents (n = 26) State or Local Public Health Official/Practitioner DHS/FEMA Regions National Response Agencies Public Health Professional Organizations Public Health Disaster Specialists/Academia Range of Representation 14 State and Local Public Health Agencies I, II, III, IV, V, VII, VII, IX, X Centers for Disease Control, FEMA, DHS Association for State and Territorial Health Officials, National Association of City and County Health Officials, National Association of National Boards of Health, National Environmental Health Association Public Health Practice, Risk Communication, Disaster Management Table 2 Professions represented in the expert panel composition (DHS = Department of Homeland Security; FEMA = Federal Emergency Management Agency; n = number) plan components was also compared to components of three other frameworks: (1) components used by general emergency management; (2) Public Health in America s essential services of public health; 24 and (3) the focus planning areas of the CDC state emergency plan guidance in use at the time. 25 The comparative framework was included in Round I for the purpose of explaining and grouping potential criteria the research team selected for consideration by the experts. For each category, the project team generated potential criteria that could measure actions likely to be taken by public health agencies during an emergency or a disaster. For example, under the category Initial Response Command and Control an initial draft criterion was: health department internal incident commander is identified immediately on notification that emergency management will be used in response to an incident. A total of 50 potential agency level criteria corresponding to 10 separate categories were originally sent out. In Round I of the Delphi survey, panel members were asked to read each potential criterion and respond whether it should be retained, modified, or eliminated. Additionally, they were asked to comment on the proposed criteria and suggest additional criteria. The Round I instrument was pilot-tested by four public health experts familiar with emergency response and emergency plan design, and slight adjustments in wording and format were made based on their feedback. The survey was administered by mail and electronically (reflecting panelist preferences). Suggested modifications focused on adjusting the agency response times stipulated in the potential criteria. Other suggested modifications primarily were to clarify language. Reasons for proposing rejection of criteria centered on redundant language, or panelists knowledge of a particular health department s non-participation in a specific activity, related to a criterion. Only four criteria were not retained by the panel. Delphi Survey Round II Based on the results of Round I, the Round II survey instrument grouped 46 potential criteria into nine cate- May June Prehospital and Disaster Medicine

4 176 Role of Exercise and Drills Figure 1 Exercise flow chart gories: (1) initial response command and control; (2) communication; (3) early recognition/surveillance and epidemiology; (4) sample testing; (5) evidence management; (6) mass prophylaxis, immunization, and pharmaceutical stockpiles; (7) mass-patient care; (8) mass-fatality management; and (9) environmental surety. All non-conflicting editorial suggestions from Round I were included. Where there were conflicting responses from the panel, changes suggested by the majority of panelists were included. Where there was no clear consensus, the research team decided on the best alternative for inclusion. The instrument also provided feedback to the expert panel consisting of: (1) the individual panelist s scores for each item in Round I; and (2) the range of scores for each criterion in all 10 categories of plan response. In response to questions asked by panelists, the research team developed a flow chart (Figure 1) to illustrate where in the process of developing a drill or exercise one or more criteria from the Round II menu of criteria could be selected for use in exercise evaluation. The Round II survey was pilot tested again with four public health experts, and minor adjustments were made before distribution. Again, the expert panelists were asked to review the criteria as they did in Round I. Round II also was administered by mail and . Suggested modifications were based primarily on language clarification and disputed response times. Criteria were rejected for similar reasons as in Round I, and once again, few criteria were rejected. Results The response rate for Delphi Survey Round I was 74% and 55% for Round II. Based on Round I responses, criteria were eliminated, reused, or others were suggested by the respondents, resulting in 46 potential criteria. These 46 criteria served as the basis for Round II. In Round II, >80% of respondents were in agreement with retaining all 46 criteria presented. Eighty percent of respondents agreed with retaining the potential criteria as written. The response rate for Delphi Survey Round II was 74%. In the last step of this research, the results of Round II were used as the basis for a final menu of criteria for use in evaluating an emergency drill or exercise at the local public health level. The research team selected the final criteria based on the majority opinion of the response panel. For purpose of clarity, the team separated criteria that contained composite actions, so that each resulting criterion had only one action verb and was stated in an observable format. This final document was circulated to the expert panel for a final round of comments. The final set of criteria is shown in Table 3. Discussion The US Homeland Security Exercise Evaluation Program provides exercise evaluation guidelines for conducting and observing exercise activities Within that document, the guidelines for public health lacked detail when this project began. Once validated, use of the public health-specific criteria that were developed through this research would allow for specific assessment and planning for measurable improvement over time in a health agency. The researchers plan to integrate these criteria into a format that can be used by exercise observers to record the events that take place in an exercise, and to use these observations in an analysis of agency performance. Preliminary application in a limited number of exercises suggests that using structured criteria minimizes subjective interpretation. Prehospital and Disaster Medicine Vol. 21, No. 3

5 Gebbie, Valas, Merrill, et al 177 I. Initial Response Command and Control 1. Health department internal incident commander is identified immediately on notification that emergency management will be used in response to an incident. 2. All health department internal ICS positions (PIO, Liaison, Safety Officer, needed section chiefs) are identified and activated* within 30 minutes regardless of time of day. 3. All initial, available, relevant information centralized to the agency s ICS Planning Section using a situation board or equivalent within 15 minutes. 4. Establish an IAP describing primary objectives and overall strategy to be accomplished by the health department in the first 24 hours within 30 minutes after the establishment of the department s EOC. 5. Activate appropriate ICS General Staff Sections within 60 minutes of initial activation of internal incident commander. 6. Portions of state/local public health law/code relevant to the specific emergency retrieved and available to ICS Command Staff. 7. Activate plan for operating/closing routine health department operations within 30 minutes of initial activation of internal incident commander II. Communication 1. Establish liaison with JIC within 15 minutes of notifying the incident commander that the JIC is operational. 2. Draft and approve public information for anticipated phases of response within two hours. Draft and approve public information for anticipated phases of response within two hours of establishment of agency s EOC. 3. Identify critical partners (e.g., other health jurisdictions, law enforcement, hospitals, etc.) and convey initial public health information to them immediately upon approval of information and no later than one hour after approval of information by agency JIC. 4. Establish a schedule to update partners regularly. 5. Test back-up communication procedures. 6. Identify a public health spokesperson appropriate to the emergency (e.g., Health Director, Epidemiologist). 7. SOCO developed, conveyed through ICS structure. 8. SOCO updated and distributed (e.g., posted to Situation Boards) as needed, but at least every four hours (maximum). Table 3 Menu of criteria for evaluation of local public health emergency drills and exercises (EOC = emergency operations center; IAP = incident action plan; ICS = incident command system; JIC = jurisdiction-wide information center; PIO = public information officer; SOCO = single overriding communications objective) *activated = notified and in communication, not necessarily on-site continued May June Prehospital and Disaster Medicine

6 178 Role of Exercise and Drills III. Early Recognition/Surveillance and Epidemology 1. Event/incident specific surveillance is established within 60 minutes of establishing the EOC. 2. Establish a preliminary case definition. 3. Adjustments to surveillance communicated as needed to and from the internal health public health IC. 4. Changes/enhancements in surveillance implemented within one hour of approval by the internal public health IC. 5. Epidemiology investigation plan including tools and forms completed within two hours of establishing the agency s EOC. 6. Field Staff appropriately prepared and protected prior to deployment. 7. Establish time interval for updating epidemiology investigation plan based on reassessment of information. IV. Sample Testing 1. Laboratory resources and requirements specific to agent/incident selected. 2. Specific protocols and procedures for collection of samples relevant to the specific emergency available within 60 minutes of notification of incident by IC. 3. Ensure case investigators appropriately prepared and protected in PPE prior to deployment. 4. Field workers deployed to sites within 60 minutes of final IAP. 5. Samples properly packed and shipped/transported consistent with laboratory requirements. Table 3 (continued from page 177) Menu of criteria for evaluation of local public health emergency drills and exercises (EOC = emergency operations center; IAP = incident action plan; IC = incident command; PPE = personal protective equipment) continued When combined with narrative accounts from participants and observers, the criteria can contribute to the production of a more detailed, robust After Action Report. More importantly, because criteria reflect measures of tasks and activities performed as part of an emergency response plan, they are likely to facilitate writing a more precise improvement plan and be easily adapted for use in updating emergency response plans. Recent publication of a Universal Task List (UTL) covering actions to be taken by the various emergency response sectors has raised questions about the use of the criteria reported here. The UTL organizes tasks according to the four US Homeland Security Missions: Prevent, Protect, Respond, and Recover. The criteria developed by this research can provide the metrics by which successful completion of tasks may be measured. The UTL tasks in Prehospital and Disaster Medicine Vol. 21, No. 3

7 Gebbie, Valas, Merrill, et al 179 V. Evidence Management 1. Evidentiary requirements confirmed with relevant agencies through jurisdiction-wide National Incident Management System partners, prior to collection of personal or environmental samples. 2. Coordinated arrangements for maintaining chain of evidence in place prior to collection of any samples/specimens. VI. Mass Prophylaxis and Immunization and Pharmaceutical Stockpiles 1. Generic mass dispensing strategy adapted to specific event within 60 minutes of notification. 2. Dispensing sites are staffed with adequate and appropriately staffed personnel for mass dispensing (including volunteer surge staff) prior planned to site opening time. 3. Needed just in time training including use of PPE identified and requested at least two hours prior to planned opening of sites. 4. System in place to restock supplies throughout duration of site operation. 5. System in place to rotate or relieve staff during site operation. 6. All needed record keeping supplies prepared and delivered for opening mass dispensing sites prepared and delivered to site coordinators by opening of site(s). VII. Mass-Patient Care 1. Generic mass care strategy adapted to specific event within 60 minutes of notification. 2. Mass care sites are staffed with adequate and appropriately staffed personnel (including volunteer surge staff) prior to opening. 3. Needed just in time training including use of PPE identified and requested at least two hours prior to planned opening of sites. 4. System in place to restock supplies throughout duration of site operation. 5. System in place to rotate or relieve staff during site operation. 6. All needed record keeping supplies prepared and delivered for opening mass dispensing sites prepared and delivered to site coordinators by opening of site(s). VIII. Mass-Fatality Management 1. If needed, identify suitable facility prior to dispatch of pick-up vehicles. 2. Develop plan for transport bodies, including routes and expected timeframes prior to dispatch of pick-up vehicles. IX. Environmental Surety 1. Develop/adapt strategy for control of environment rendered hazardous by this event within 30 minutes of site identification. 2. Identify specific characteristics of affected areas and report to incident commander within 30 minutes of arrival on site. 3. Establish PPE criteria for staff. Table 3 (continued from page 178) Menu of criteria for evaluation of local public health emergency drills and exercises (PPE = personal protective equipment) May June Prehospital and Disaster Medicine

8 180 Role of Exercise and Drills Function Universal Task List Item Local Public Health Criteria VI. Mass Prophylaxis & Immunization and Pharmaceutical Stockpiles 1. Generic mass dispensing strategy adapted to specific event within 60 minutes of notification. # Develop plans, procedures, and protocols to implement national pharmaceutical stockpile operations 2. Dispensing sites are staffed with adequate and appropriately staffed personnel for mass dispensing (including volunteer surge staff) prior to planned site opening time. 3. Needed just in time training, including use of PPE identifed and requested at least two hours prior to planned opening of sites. 4. System in place to restock supplies throughout duration of site operation. Misson: Protection (Pro.C.1.) #1.1.2 Develop laboratory plans, procedures, and protocols #1.1.3 Develop plans and procedures for disease outbreak 5. System in place to rotate or relieve staff during site operation. 6. All needed record keeping supplies prepared and delivered for opening mass dispensing sites prepared and delivered to site coordinator by opening of site(s). IV. Sample Testing 1. Laboratory resources and requirements specific to agent/incident selected. 2. Specific protocols and procedures for collection of samples relevant to the specific emergency available within 60 minutes of notification of incident by IC. 3. Ensure case investigators appropriately prepared and protected in PPE prior to deployment. 4. Field workers deployed to sites within 60 minutes of final IAP. 5. Samples properly packed and shipped/transported consistent with laboratory requirements. III. Early Recognition/Surveillance and Epidemiology 1. Event/incident specific surveillance is established within 60 minutes of establishing the EOC. 2. Establish a preliminary case definition. 3. Adjustments to surveillance communicated as needed to and from the internal health public health IC. 4. Changes/enhancements in surveillance implemented within one hour of approval by the internal public health IC. 5. Epidemiology investigation plan including tools and forms completed within 2 hours of establishing the agency s EOC. 6. Field Staff appropriately prepared and protected prior to deployment. 7. Establish time interval for updating epidemiology investigation plan based on reassessment of information. Table 4 Relationship of universal task list items to exercise evaluation criteria (EOC = emergency operations center; IAP = incident action plan; IC = incident command; PPE = personal protective equipment) continued Prehospital and Disaster Medicine Vol. 21, No. 3

9 Gebbie, Valas, Merrill, et al 181 Function Universal Task List Item Local Public Health Criteria VII. Mass Patient Care 1. Generic mass care strategy adapted to specific event within 60 minutes of notification. Misson:Response (Res.C.3) #1.1 Develop plans, procedures, and protocols for the immediate provision of mass care services to shelters for disaster victims to include shelter, feeding, basic first aid, bulk distribution of needed items, and other related services 2. Mass care sites are staffed with adequate and appropriate personnel (including volunteer surge staff) prior to opening. 3. Needed just in time training including use of PPE identified and requested at least two hours prior to planned opening of sites. 4. System in place to restock supplies throughout duration of site operation. 5. System in place to rotate or relieve staff during site operation. 6. All needed patient record keeping supplies prepared and delivered for opening mass care sites prepared and delivered to site cooordinator by opening of site(s). Table 4 (continued from page 180) Relationship of universal task list items to exercise evaluation criteria (PPE = personal protective equipment) two mission areas for which the public health criteria may be used for measurement are listed in Table 4. With any Delphi research approach, there is one perspective on the topic of interest. The researchers made every effort to acheive inclusion of those with an interest in emergency response in the local public health level, without regard to known perspective on the question of criteria. The range of responses to the first round indicates that this was accomplished. It is impossible to fully interpret the lower response rate to the second round. The non-respondents did not represent any one constituency or interest group, and the researchers observation is that this round coincided with a busy time of grant renewals and other demands for time from the experts involved with this study. The research team notes that limitations of this research may be attributed to the 74% response rate from the expert panel. Conclusion Public health agencies must be able to measure performance and identify areas for improvement. This can be done through ongoing training and emergency response exercising, and through the use of response exercises that include plans for evaluation. The criteria developed through this research are essential for those purposes. Preparedness and response are dynamic processes, as threats change and weaknesses are identified. Capacity is continuously built through planning, exercising, evaluating, and improving. References 1. Cope JR: Workforce Competencies for emergency preparedness NACCHO Exchange 2003;1(2): Centers for Disease Control and Prevention. Bioterrorism & emergency readiness: competencies for all public health workers. Available at Accessed 14 June Centers for Disease Control and Prevention. Capacity inventory. Available at 4. Allan S: The challenges of local preparedness for bioterrorism and other emergencies. NACCHO Exchange 2002;1(1): Firshien J: Public health preparedness for disaster: Perspective from Washington, D.C. J Urban Health 2002;79(1): Barbera JA, MacIntyre AG: The reality of the modern bioterrorism response. In: The Lancet Supplement Vol Available at 7. Morse A: Bioterrorism Preparedness for Local Health Departments. J Community Health Nurs 2002;19(4): Simmons SC, Murphy TA, Blanarovich A, et al: Telehealth technologies and applications for terrorism response: A report of the 2002 coastal North Carolina domestic preparedness training exercise. J Am Inform Assoc 2003;10(2): Centers for Disease Control and Prevention. Health alert network. Available at New York City Office of Emergency Management. Keeping New York prepared: An overview of the New York City Office of Emergency Management National Association of County and City Health Officials: Project Public Health Ready. Available at Ford JK, Schmidt AM: Emergency response training: Strategies for enhancing real world performance. Journal of Hazardous Material 2000;75(2 3): National Association of County and City Health Officials. Bt Create: A customizable bioterrorism tabletop exercise builder. Available at Gebbie KM, Merrill JA, Hwang I, et al: Identifying individual competency in emerging areas of practice: An applied approach. Qual Health Res 2002;12(7): May June Prehospital and Disaster Medicine

10 182 Role of Exercise and Drills 15. Turnock BJ: Roadmap for public health workforce preparedness. J Public Health Manag Pract 2003;9(6): Kerby DS, Brand MW, Johnson DL, Ghouri FS: Self-assessment in the measurement of public health workforce preparedness for bioterrorism or other public health disasters. Public Health Rep 2005;120(2): Qureshi KA, Gershon RR, Merrill JA, et al: Effectiveness of an emergency preparedness training program for public health nurses in New York City. Fam Community Health 2004;27(3): Columbia University School of Nursing, Center for Health Policy: Clinical competencies during initial assessment and management of emergency events. Available at US Department of Homeland Security, Federal Emergency Management Agency: State and local guide (SLG) 101: Guide of all-hazards emergency operations planning. Available at Columbia University School of Nursing Center for Health Policy: Defining emergency exercises: A working guide to the terminology. Available at US Department of Homeland Security, Office for Domestic Preparedness: Homeland security exercise and evaluation program Volume I: Overview and doctrine. Available at US Department of Homeland Security, Office for Domestic Preparedness: Homeland Security Exercise and Evaluation Program Volume II: Exercise Evaluation and Improvement. Available at docs/hseepv1.pdf. 23. US Department of Homeland Security, Office for Domestic Preparedness: Homeland security exercise and evaluation program Volume III: Exercise program management and exercise planning process. Available at Public Health Functions Steering Committee: Public health in American essential public health services. Available at Centers for Disease Control and Prevention: Continuation guidance for cooperative agreement on public health preparedness and response for bioterrorism Budget year five. Available at Prehospital and Disaster Medicine Vol. 21, No. 3