Root Cause Analysis Report RCA Example - Hawaii False Alarm (Revision 1) Problem Statement Report umber RCA Owner Chris Eckert Report Date 5/13/2018 RCA Facilitator Brian Hughes Focal Point: Widespread panic and confusion in the State of Hawaii, USA When Start Date: 1/13/2018 End Date: 1/13/2018 Start ime: 8:07 AM Unique iming Where Other Actual Impact End ime: 8:45 AM After Day Warning Officer misunderstood a drill for a real attack. Hawaii Emergency Management Agency (HI-EMA), Emergency Alert System (EAS), Wireless Emergency Alert System (WEA) Safety o safety impact $0.00 Customer Experience Reputation (External) Mass fear and confusion across the entire State of Hawaii egative impact on confidence that the State of Hawaii can responsibly manage the attack warning system Reputation (Internal) Workers required mental health counseling $0.00 Investigation Costs Frequency wo separate investigations took place, one at the state level and another at the federal level. Costs estimated. 1 times Overall $0.00 $0.00 $250,000.00 Actual Impact otal: $250,000.00 Potential Impact Safety People could have been injured or killed as a result of mass panic. Created with Causelink Page 1 of 10
Root Cause Analysis Report RCA Example - Hawaii False Alarm Executive Summary Report Summaries LISE UP! SOLOGIC DID O IVESIGAE HIS ICIDE I AY OFFICIAL OR PROFESSIOAL CAPACIY. HE SOLE PURPOSE OF HIS EXAMPLE IS O BE USED AS A LEARIG OPPORUIY O DEMOSRAE HOW A HIGH-VALUE, HIGH-RISK ICIDE COULD BE AALYZED USIG HE SOLOGIC MEHOD OF ROO CAUSE AALYSIS AD CAUSELIK SOFWARE. WE HAVE SIMPLY RESAED IFORMAIO PREVIOUSLY PUBLISHED BY OHERS. EJOY! his example is relatively simple. here are fewer than 30 causes and it is not very challenging to understand because it's not very technical. However, it offers some valuable lessons in how a simple event can have a massive impact if given the right platform. It follows the same pattern as a quality escape in that the "product" (in this case, the alert) was incorrect and it was "shipped" (via the statewide alert system) to the "customer" (several million people in the State of Hawaii). In order to solve it, we need to understand how the error was made, why the error was allowed to be broadcast, and also why the response was delayed by 38 minutes. here are many systemic pathways to explore. For instance: Why would the alert system's user interface be used for both drills and actual alerts? By design, a drill can be mistakenly broadcast as real. Why would such an alert be generated by a single individual employee? Why was there no plan for responding to a false alarm? Why was a reportedly "troubled" employee allowed such a great responsibility? Many questions remain and when the final FCC report is issued, we will update our RCA accordingly. Cause and Effect Summary On January 13, 2018 at 8:07AM a statewide alert was issued in Hawaii stating: BALLISIC MISSILE HREA IBOUD O HAWAII. SEEK IMMEDIAE SHELER. HIS IS O A DRILL. People naturally experienced terror as they rushed to shelter or to be with their loved ones. It is a wonder no one was hurt or killed during the ensuing panic, which lasted 38 minutes. Of course, we all know the rest of the story at this point: he alert was a false alarm. he Hawaii Emergency Management Agency is responsible for issuing alerts for disaster events, such as tsunamis, volcanos, earthquakes, and lately (due to tensions with orth Korea) for nuclear attack. Drills for a potential nuclear attack had taken place at least twice since the Fall of 2017. In this case, a night-shift supervisor decided to run a drill to test the incoming day-shift staff. He posed as a U.S. Pacific Command official and played a recorded message to the emergency workers which included the phrase Exercise, exercise, exercise. But it also included the phrase his is not a drill. he day-shift supervisor was aware of the drill, but thought it was intended to test the night-shift workers and was therefore not prepared to actually play a supervisory role in the drill. he day-shift Warning Officer heard most of the recording, but did not hear the Exercise, exercise, exercise part at Created with Causelink Page 2 of 10
the beginning of the warning. herefore he thought the warning was real and proceeded to issue the statewide alert. he Warning Officer is able to do this on his/her own without input from anyone else. he mistake was realized very quickly after the alert went out. However there an update was not issued for 38 minutes because there was no system in place to manage a false alarm. An approved message needed to be crafted and permission from FEMA had to be obtained prior to sending the update. An interim plan for the Governor of Hawaii to issue a correction via witter was delayed when he could not remember his password. Solutions: Hawaii authorities to require additional approvals before alerts and tests are transmitted. he state suspended emergency alert drills after the false alarm and will provide more warning before drills. Officials in Hawaii now require a second person to confirm sending out alerts. he FCC issued new requirements to enhance the geo-targeting of cellphone alerts. he FCC will require cellphone carriers to allow consumers to review any alert for up to 24 hours after they receive them. Created with Causelink Page 3 of 10
Root Cause Analysis Report RCA Example - Hawaii False Alarm Solutions SO-0001 Solution Hawaii authorities now require additional approvals before alerts and tests are transmitted. ote Day Warning Officer initiated the alert o additional notes. Assigned Brian Hughes Criteria Passed Due 5/17/2018 Status Completed erm short Cost $0.00 SO-0002 Solution he state suspended emergency alert drills after the false alarm and also plans to provide more warning before drills. ote Day Warning Officer initiated the alert o additional notes. Assigned Brian Hughes Criteria Passed Due 5/17/2018 Status Completed erm short Cost $0.00 SO-0003 Solution Officials in Hawaii now require a second person to confirm sending out alerts. ote Gaps in State alert system o additional notes. Assigned Brian Hughes Criteria Passed Due 5/17/2018 Status Completed erm short Cost $0.00 SO-0004 Solution he FCC issued new requirements designed to enhance the geo-targeting of cellphone alerts. ote his will make the distribution of alerts more accurate so that those outside of an emergency area will not receive warnings that do not affect them. Assigned Brian Hughes Criteria Passed Due 5/17/2018 Status Completed erm short Cost SO-0005 Solution he FCC will require cellphone carriers to allow consumers to review any alert for up to 24 hours after they receive them. Created with Causelink Page 4 of 10
Gaps in State alert system ote Carriers will have until ovember 2019 to implement the changes. Assigned Brian Hughes Criteria Passed Due 5/17/2018 Status Approved erm short Cost Created with Causelink Page 5 of 10
Root Cause Analysis Report RCA Example - Hawaii False Alarm Facilitator Brian Hughes brian.hughes@sologic.com Participants Melani Egbert melani.egbert@sologic.com Owner Chris Eckert Cory Boisoneau eam cory.boisoneau@sologic.com Created with Causelink Page 6 of 10
Root Cause Analysis Report RCA Example - Hawaii False Alarm Evidence EV-0001 Evidence Washington Post Article: "Hawaii s false missile alert sent by troubled worker who thought an attack was imminent, officials say." By Mark Berman and Brian Fung January 30, 2018 Location(s) Attachment(s) Contributor ype Correction not issued for 38 minutes Day Warning Officer believed an attack truly was imminent Day Warning Officer did not hear the message that it was a drill Day Warning Officer initiated the alert Day Warning Officer received a recorded message of an attack Day Warning Officer's job involves initiating alerts Drill initiated by ight Supervisor Gaps in State alert system Governor could not remember his witter password Heightened tensions with orth Korea Important for message to be effective - needed to get it right It was possible for a single person to issue a statewide alert o approved message was available o process in place to issue false alarm correction Other means of alerting to false alarm were delayed Recorded message system of alert is in place for real and drills State has been conducting drills of emergency alert system Statewide warning that a ballistic missile threat was "Imminent" Supervisors' job includes initiating surprise drills Day Warning Officer "pushed the wrong button?" Single drop menu in the interface to select real or drill System design Alert sent to millions of cell phones in the area Potential job performance gaps from this employee? Quality https://www.washingtonpost.com/news/the-switch/wp/2018/01/30/heres-whatwent-wrong-with-that-hawaii-missile-alert-the-fcc-says/? utm_term=.511cae925ac3 Brian Hughes URL EV-0002 Evidence FCC Report Location(s) Created with Causelink Page 7 of 10
Attachment(s) Contributor ype Quality FCC Report DOC-348923A1.pdf Brian Hughes Document EV-0003 Evidence Washington Post Article: "Hawaii missile alert: How one employee pushed the wrong button and caused a wave of panic." By Amy B Wang January 14, 2018 Location(s) Attachment(s) Contributor ype Correction not issued for 38 minutes Day Warning Officer believed an attack truly was imminent Day Warning Officer did not hear the message that it was a drill Day Warning Officer initiated the alert Day Warning Officer received a recorded message of an attack Day Warning Officer's job involves initiating alerts Drill initiated by ight Supervisor Gaps in State alert system Governor could not remember his witter password Heightened tensions with orth Korea Important for message to be effective - needed to get it right It was possible for a single person to issue a statewide alert o approved message was available o process in place to issue false alarm correction Other means of alerting to false alarm were delayed Recorded message system of alert is in place for real and drills State has been conducting drills of emergency alert system Statewide warning that a ballistic missile threat was "Imminent" Supervisors' job includes initiating surprise drills here was no actual ballistic missile emergency ime required for message to be crafted Want to ensure system works properly/ effectively Single drop menu in the interface to select real or drill https://www.washingtonpost.com/world/national-security/is-this-the-end-of-mylife-false-alert-of-missile-attack-sends-hawaiiscrambling/2018/01/13/78c86054-f8a0-11e7-b34a-b85626af34ef_story.html? utm_term=.d3cd5f5d5d8a&tid=a_mcntx Quality Brian Hughes URL EV-0004 Evidence Atlantic Article: "Pandemonium and Rage in Hawaii A false alert of an impending missile attack highlights just how unprepared the country is for nuclear disaster." By ALIA WOG, JA 14, 2018 Location(s) https://www.theatlantic.com/international/archive/2018/01/pandemonium-andrage-in-hawaii/550529/ Created with Causelink Page 8 of 10
Attachment(s) Contributor ype Quality Brian Hughes URL Created with Causelink Page 9 of 10
Root Cause Analysis Report RCA Example - Hawaii False Alarm otes O-0001 ote Reports said that this employee has a history of past performance gaps. Workers apparently did not trust his/her work. Additional information is being gathered - nothing has yet been released. As of the time of this report, he/she had been fired. Potential job performance gaps from this employee? Created with Causelink Page 10 of 10
RCA Example - Hawaii False Alarm imezone: America/Los_Angeles 1/13/2018 08:31 AM HI-EMA supervisor logs into alert system, begins to create false alert correction 1/13/2018 08:30 AM FEMA confirms HI-EMA s view on CEM; Hawaii Governor posts correction on Facebook 1/13/2018 08:20 AM HI-EMA posts on Facebook, witter O missile threat to Hawaii 1/13/2018 08:12 AM HI-EMA issues a cancellation, ceasing retransmission over EAS, WEA 1/13/2018 08:10 AM HI-EMA to U.S. Pacific Command and Honolulu PD: no missile launch 1/13/2018 08:09 AM HI-EMA notifies Hawaii Governor of false alert 1/13/2018 08:07 AM Day Shift Warning Officer enters alert. 1/13/2018 08:06:10 AM Day Shift Warning Officer believes it is not a drill. While other warning officers understand that this is a drill, the warning officer at the alert origination terminal claimed to believe, in a written statement provided to HIEMA, that this was a real emergency, not a drill. his day shift warning officer responds, as trained for a real event, by transmitting a live incoming ballistic missile alert to the State of Hawaii. 1/13/2018 06:00 AM HI-EMA s midnight shift supervisor begins a no-notice ballistic missile defense drill HI-EMA s midnight shift supervisor begins a no-notice ballistic missile defense drill at a shift change by placing a call, pretending to be U.S. Pacific Command, to the day shift warning officers. 1/13/2018 08:06 AM he day shift warning officers receive this recorded message on speakerphone. 1/13/2018 08:05 AM he midnight shift supervisor plays a recording over the phone. 1/13/2018 08:45 AM HI-EMA issues correction through EAS and WEA that there is no missile threat 1/13/2018 06:00 AM 8:06 AM 8:26 AM 1/13/2018 08:45 AM
Chart Key ransitory on-ransitory Heightened tensions with orth Korea ED erminated because: Out of scope ransitory Omission on-ransitory Omission Undefined Chart Quality Alert Focal Point State has been conducting drills of emergency alert system Evidence Solutions otes asks Drill initiated by ight Supervisor Want to ensure system works properly/ effectively ED erminated because: Desired State Day Warning Officer received a recorded message of an attack Supervisors' job includes initiating surprise drills ED erminated because: Other causal paths more productive Day Warning Officer believed an attack truly was imminent Recorded message system of alert is in place for real and drills ED erminated because: Other causal paths more productive OR Day Warning Officer did not hear the message that it was a drill? Day Warning Officer initiated the alert Day Warning Officer "pushed the wrong button?" Single drop menu in the interface to select real or drill? Solutions Statewide warning that a ballistic missile threat was "Imminent" Hawaii authorities now require additional approvals before alerts and tests are transmitted. Criteria Pass Status Completed o additional notes. he state suspended emergency alert drills after the false alarm and also plans to provide more warning before drills. Criteria Pass Status Completed o additional notes. It was possible for a single person to issue a statewide alert Day Warning Officer's job involves initiating alerts Potential job performance gaps from this employee? Gaps in State alert system Solutions Officials in Hawaii now require a second person to confirm sending out alerts. Criteria Pass Status Completed o additional notes. he FCC will require cellphone carriers to allow consumers to review any alert for up to 24 hours after they receive them. ED ED erminated because: Other causal paths more productive? erminated because: Likely systemic issue Criteria Pass Status Approved Carriers will have until ovember 2019 to implement the changes. Widespread panic and confusion in the State of Hawaii, USA Alert sent to millions of cell phones in the area System design ED erminated because: Desired State here was no actual ballistic missile emergency ED erminated because: Other causal paths more productive o process in place to issue false alarm correction ED erminated because: Likely systemic issue o approved message was available ED erminated because: Likely systemic issue Correction not issued for 38 minutes ime required for message to be crafted Other means of alerting to false alarm were delayed Important for message to be effective - needed to get it right ED erminated because: Other causal paths more productive Governor could not remember his witter password?