Southwest Regional Symposium 0n Business Continuity, Information Security, & IT Audit Converging on Information Assurance Lessons Learned From Hurricane Katrina 2006 SunGard Availability Services L.P. All rights reserved. Presented by Chuck Walts, CBCP, CRP Lead Consultant SunGard Professional Services
Denial is not a river in Egypt Hurricanes are an annual threat to the United States. They are one of the few major threats that announce their impending arrival days in advance, make known where they may hit, reveal the force and intensity with which they might strike, and allow time to prepare. The September 11, 2001 and the Florida hurricanes of 2004 should have raised disaster awareness and the advisability of comprehensive DR / BC planning. The people and businesses along the Gulf Coast ignored and / or underestimated Katrina s capability to cause devastation and disrupt the economy. The result was senseless and preventable losses. Measures of Katrina s Wrath Caused an estimated $200 billion in damage Did $600 million damage to the Telco infrastructure Left 3,000,000 people without power or phones Downed 11,000 utility poles and 1,000 wireless towers Knocked over 100 broadcast agencies off the air. Impacted 75,000 square miles in 5 Gulf Coast states 2.8 million gallons of oil spilled Displaced 500,000 citizens Closed 25 hospitals Damaged and closed roads and bridges No water, no fuel, no lodging, no sanitation, & no security The Consequences Left 350,000 people without homes or jobs An estimated 200,000 people have not returned to the area Wiped out 80,000 businesses Disrupted an already economically challenged region Major utility companies filed for bankruptcy No revenue, no taxes, no government 3,000 city of New Orleans employees laid off Political agencies overwhelmed due to bad planning Schools swamped by student consolidations Health care severely impacted Major security issues 40,000 military involved First responders unable to communicate across jurisdictions
Companies That Were Not Prepared - 1 Failed to Effectively Pre-Plan Let other business priorities take focus away from DR / BC planning Did not effectively engage Human Resources in planning Had no policy to handle staff Had no employee staff directory on site Made no decision on who works and who stays home Had no time-of-disaster compensation policy for staff Did not think through staff relocation (including families) Made no provisions for lodging or extended hotel stays Did not engage Business Units during IT plan development Planned for a short-term outage not a long-term disruption Had no formalized off-site storage arrangements Tested what plans they had once, and on a limited basis Did not exercise plans with external response agencies Companies That Were Not Prepared - 2 Failed to Effectively Respond Ignored or showed no concern for the warnings from authorities Panicked when the threat was imminent Failed to understand their limitations Realized too late that many things were out of their control Were affected by a community response that was not timely Scrambled to write system emergency shutdown procedures Made no provisions for staff exodus, including home issues Did not have ready access to emergency contact information Were denied access to their home site and had no alternate Failed to ship backup tapes Did not anticipate the extent of voice communications problems Companies That Were Not Prepared - 3 Experienced Recovery Problems Found that alternate sites / warehouse spaces were taken Learned that alternate site & equipment contracts were outdated Had difficulty gaining access to recovery facilities because no authorized personnel reported to the recovery site Were not ready for a mandatory evacuation Found that key recovery team members lived in evacuation zones Were not prepared for transportation gridlock Found that travel was slow and difficult, hotels were booked, and fuel was scarce or unavailable Found that employees familiar with the plan were not available Had to recover with inexperience, untrained staff Found that documentation lacked the detail for effective recovery Failed to identify replacement staff and worked recovery personnel around the clock. Were unable to effectively communicate
Companies That Were Prepared - 1 Engaged in Pre-Planning Built a resilient infrastructure including redundant Telco Included the corporate offices in planning Established automated notification systems & call trees Procured 800 numbers outside the affected area to update employee updates Set-up mirrored and remote data center operations Exercised their BC/ DR plans with Incident Management Tested their plan several times a year Scheduled hot-site tests in advance Referred to detailed documented recovery procedures Documented emergency shut-down procedures Identified several feasible meeting locations Authorized disaster funding to get money to people quickly (issued prepaid credit cards) Companies That Were Prepared - 2 Developed and Executed an Effective Response Monitored the storm s progress Called their IMT together to discuss impending disaster Developed and invoked an impending disaster plan Prepared staff and facilities for evacuation Rerouted their network & used redundant Telco Were able to make quick decisions facilitated by senior level management involvement Planned and mapped evacuation routes Arranged transportation & sent families with employees Evacuated company facilities prior to disaster Kept systems running at data centers and accessed remotely Transmitted immediate orders and periodic updates from senior management Companies That Were Prepared - 3 Recovered Successfully Automatically shipped tapes to hot-site; vendor started system restores Facilitated a special system backup pending the disaster Transferred critical operations to branch / regional office Had help available at alternate site for mental health issues Ensured that key people were available at the recovery site Had plans for remote user access to critical systems
Lessons Learned from the Failures of Businesses The Human Element If disaster strikes, recovery and continuity will be largely determined by employees Employees will be more concerned for themselves and their families than they are for the company Backup personnel need to be available to carry out plans if employees critical to plan execution are missing or can t travel Alternate staff at another location should be ready to engage Employee roles and responsibilities need to be assigned and tested Make provisions for families to go with employees Establish cash accounts with linked debit cards to ensure employees can cover expenses Provide for the safe travel and lodging of relocated employees Deploy key employees and their families at the first sign of trouble Map out alternate evacuation routes Address transportation issues such as rental cars and fuel Involve HR, corporate management, and local government in planning Lessons Learned from the Failures of Businesses Communications is Key Develop a backup communications plan Have an external communications plan Have a plan to keep employees informed Put the crisis and communication plan in place well in advance Consider alternate communications tools e.g. extra cell phones & batteries, satellite phones, text messaging, wireless cards for laptops VPN, a backup corporate e-mail address, and a crisis phone bridge Update local radio and TV stations with reports Develop a procedure for status reporting Maintain lists of vendor and local government contacts Plan for corporate headquarters to participate in recovery Lessons Learned from the Failures of Businesses Information Technology Recovery More than 50% of SunGard s customers were not prepared to recover Contracts were not current Hardware configurations were outdated Some clients had older technology Documentation and technical scripts were outdated Extended recovery times put information availability at risk Businesses had their tapes, DASD, and paper documents destroyed End users, reluctant to travel, generally won t travel far from home Delivery of tapes stored offsite were delayed because air travel was unavailable, highways were closed, and evacuations and curfews were enforced. Timely transfer of resources to a safe recovery facility was impaired Clients had not planned for an extended recovery
Business Considerations Develop / review / update Incident Management, BC and DR Plans Plan for the both the short and long-term Test plans frequently. Test the way you recover; recover the way you test Ensure adequate end user facilities are available nearby (about 50 miles) If event probability is high, activate the IMT; put BCT and DRT on standby Monitor the situation; heed warnings Develop plans of succession (every primary should have 2 alternates) Ensure monitoring vendors have current contact information Have communications plans (conference bridge, alternate e-mail, radios) Identify a Crisis Management Center Top off generators and arrange for fuel supply Follow company emergency response procedures If an event occurs, assess the situation and damage Activate recovery plans & notify service providers Implement support procedures Track incident status and recovery progress Develop plans to return to business as usual Technology Considerations Have a recovery strategy and solution Plan for an extended recovery Allocate connectivity with the plan Develop detailed recovery procedures and scripts Keep documentation and scripts up to date Create backup tapes and ship offsite Offsite storage critical Establish alternatives for accessing tapes, data, and documentation Establish RTOs and RPOs for all critical systems / applications Paper records and transactions may be totally lost/destroyed Evaluate the need / value of electronic journaling or critical applications hosting Ensure sufficient skill sets by cross-training; assigning backup roles Determine how end users will access recovered systems Assess the value of testing services and service types with vendors Other Considerations Know how to keep the business running and rebuild what was lost Understand your insurance coverage and entitlements Keep inventories current Be able to identify losses Establish a contract with an independent reviewer and test a variety of disaster scenarios to ensure disaster preparedness Continually review and update contact lists Commence damage assessment 24-36 hours after the disruption Engage emergency response and health agencies Involve elected leaders Include Corporate offices and Human Resources in IT planning Understand local government capabilities and restrictions If an area-wide disaster strikes, be ready to go it alone IT -- educate, inform, and support business units
Regional Disasters 1992-2005 Regional Disasters SunGard Alerts SunGard Declarations 1992 - Chicago Flood 10 5 1992 - Hurricane Andrew 18 0 1992 - Hurricane Iniki 6 0 1993 - World Trade Center Bombing 0 13 1994 - Northridge Earthquake 14 6 1995 - Hurricane Opal 26 6 1996 - East Coast Blizzard 6 2 1997 - Grand Forks Flood 0 4 1998 - Canadian Ice Storm 4 7 1998 - Hurricane Georges 75 25 1999 - Hurricane Floyd 189 58 2000 - Wall Street Bomb 0 2 2000 - Ft. Worth Tornado 0 2 2001 - Seattle Earthquake 6 4 2001 - September 11 Attacks 105 121 2002 - Pre-Winter Ice Storm 5 5 2003 Northeast Power Outage 155 66 2003 Hurricane Isabel 216 4 2004 - British Telecom Fire 11 7 2004 Hurricane Charley 111 10 2004 Hurricane Frances 231 37 2004 Hurricane Ivan 281 15 2004 Hurricane Jeanne 144 18 2005 Hurricane Dennis 97 7 2005 London Bombings 84 28 2005 Hurricane Katrina 128 32 2005 Hurricane Rita 153 27 2005 Hurricane Wilma 111 21 Totals 2386 532 &