August 25, 2010 MASTERING THE LOGISTICS OF A LARGE SCALE HAZUS PROJECT Shanna Michael, CFM, AECOM Stefan Zink, CFM, Michael Baker Jr., Inc.
Agenda Part 1: Project Overview Averaged Annualized Loss Estimation Study Complications and solutions Initial and secondary complications Approaches to solving the problems Part 2: Case Study Example Best Practices Part 3: Questions
Part 1 PROJECT OVERVIEW
Average Annualized Loss Study FEMA initiated the Average Annualized Loss (ALE) Study in 2009 to provide a Nationwide Loss Dataset Nation was divided up according to regions with each PTS contractor performing loss analysis for their assigned Regions Level 1 conducted on every county in the contiguous United States using HAZUS-MH version MR-4 Patch 1, for both Riverine and Coastal analysis Annualized Loss results were provided Annualized losses are maximum potential losses for a given year based on 5 return (10, 50, 100, 200, and 500yr) periods
Average Annualized Loss Study Level 1 is basic, mostly automated, level of analysis using default data and minimal inputs Riverine county inputs included: 10 sq mile drainage area, 30M USGS DEM, Default HAZUS inventory data Coastal county inputs included: weighted average stillwater elevation for coastline, 30M USGS DEM, and Default HAZUS inventory data HAZUS calculated the following return periods: 10, 50, 100, 200, and 500 year Loss results were annualized
BakerAECOM Counties Region 4: AL, FL, GA, KY, MS, NC, SC, TN Region 8: CO, MT, ND, SD, WY Region 9: AZ, CA, NV, UT 1117 Counties assigned to BakerAECOM team
Complications Issues/Problems Initially Identified at Beginning of Project General Time How long could it actually take to process data? Space Physical locations for rentals, circuits, AC, etc. Processing Can we reduce idle time? Specific to multiple locations Data Storage and transfer large amounts? Access To machines, data files Tracking Tracking progress in multiple locations with multiple people involved
Solutions - General Time & Space Over 1100 counties to be processed between Michael Baker, Jr. and AECOM in 3 months Divided work between two processing centers: Kansas City, MO (AECOM) Alexandria, VA (Baker) Required the rental of 50 computers at each processing center
Solutions - General Processing Increase processing rented high end machines Rental Computer Specs included: Intel Ci7-L motherboard, Intel Corei5-750 2.6X4 Quad core.8m bx, 4 G RAM, 640 G Hard drive, Gforce 9400gt video w/ 1G on board, 1G NIC, and Windows XP with current Service Packs and updates HAZUS-MH MR-4, ArcGIS 9.3 SP1, Spatial Analyst
Solutions General AECOM BAKER Processing Monitored machines locally Monitored machines remotely - VPN 2-3 staff on hand 8 analysts on rotation Computers were placed in open areas and available cubicles 10 different locations Machines located in Alexandria VA office Daily hours varied during production Weekly shifts: machines divided between 2 analysts every week, with shared spreadsheets to ensure smooth transition
Solutions General Example of Processing Station
Solutions Multiple Locations AECOM Used 1 TB external drive for storage of data and second external drive for back-up BAKER Used 1TB drive for storage on server (daily backups) Used ftp, CDs, DVDs, and portable drives for data transfer Local access to source data and produced data Set up workgroups between machines to share and access data rather than external Data and Accessing Source data and produced data stored on server All machines mapped to server location. Processed locally on machines and stored on external drive Staff monitoring Processed locally and then transferred to server upon completion Free remote software used to connect to multiple machines at once
Solutions Multiple Locations Tracking: Daily updates, spreadsheets for weekly meetings to update planning and tracking Used excel spreadsheet to track progress on assigned counties As project progressed, county production would shift processing centers
Solutions Tracking Main items tracked in spreadsheet: processing time, annualized loss ratio, building count, and number of problem reaches.
Solutions Tracking Final Summary for Each Region
Further Complications Needed a tracking mechanism other than the number of counties. Due to varying sizes, it became clear that using counties would not be a good indicator to track percent complete Region 9 counties were much larger areas than Region 4 counties Were not processing at same pace due to number of reaches Chose number of streams reaches Based off number of stream reaches created by HAZUS-MH with 10 sq mile drainage area on 30M DEMs Better representation of percent complete i.e. R9 county: 200+ stream reaches vs R4 county: 45 stream reaches
Further Complications Tracking
Further Complications Tracking Example of Compiled Data
Further Complications Tracking Tracking amount of time and number of streams became essential in determining schedule Data was used to calculate amount of time and number of computers necessary to complete the project Tracked total stream counts for regions Simple formula: #streams x 1hr (ave time)= total time
Further Complications - Problem Problem: Not processing enough streams fast enough to meet the 3 month deadline Memory fragmentation slows down processing after certain number of streams (approximately 40 on hydraulics) Process one maybe two streams a day Very big problem with Region 8 and Region 9 counties Had very large stream reach counts Very slow processing
Further Complications Solutions Solutions: Brought in 50 more machines into AECOM processing center in Kansas City Brought total to 150 machines processing Added additional staff member to monitor machines and continue processing Rebooting machines during Delineate Floodplain step Approximately every 40 reaches Splitting larger counties between multiple machines Mainly affected Region 8 and 9 Certain number of stream reaches determined if county would be divided SQL database limitation - HAZUSFloodHelp
Part 2 CASE STUDY
Case Study: Just How Big? Population (millions): 6.4 Total Area (square miles): 36,418 Number of Counties: 92 Number of stream reaches (HAZUS default setting): XXX
Case Study: Just How Big? Population (millions): 2.0 Total Area (square miles): 20,105 Number of stream reaches (HAZUS default setting): 1062
Case Study: San Bernardino How long is it going to take? Depth grid for one stream reach (all return periods) takes approx. 1.5 hours to process Advanced math: 1062 * 1.5 = 1593 hrs (about 66 days) Considerations? Exceeds capacity (hydrology) Memory fragmentation (hydraulics) Thanksgiving, Christmas, New Years are coming up!!! And the deadline, too.
Case Study: San Bernardino
Case Study: San Bernardino Hydrology Issue HAZUS reliably performs the hydrologic analysis for about 150 reaches Solution Split the study region into multiple parts Advanced math: 1062 / 150 = about 7 Hydrologic analysis performed in seven scenarios
Case Study: San Bernardino
Case Study: San Bernardino
Case Study: San Bernardino
Case Study: San Bernardino
Case Study: San Bernardino Hydraulics issue: IT TAKES TOO LONG!!! Solution: Run each partial scenario of 150 reaches from the hydrology step on a separate machine Advanced math: 150 reaches * 1.5 hrs = 225 hrs (about 9 days) X-Factor(s): depth grid merge, memory fragmentation, validity of results
Case Study: San Bernardino
Case Study: San Bernardino
Case Study: San Bernardino
Case Study: San Bernardino
Case Study: San Bernardino
Best Practices: Validation Concern: are the results going to be valid? Approach: Test split approach on a small county Process the same county the regular way Compare results Carson City, NV (13 stream reaches) Hydrology: identical Depth grids: identical Loss analysis: identical
Best Practices: Summary What is the split method? Split study region into multiple scenarios at the hydrology and hydraulics step When is the split method useful? For very large study regions When using HiRes terrain When expecting lots of reaches Does it work? YES!!! When there is not enough time
Summary Over 1100 counties to process in 3 months Processed over 90,000 stream reaches in approximately 3 ½ months Total numbers processed: Tracking Value Counties Processed 1117 Stream Reaches ~90,000
Questions? Shanna.Michael@aecom.com Szink@mbakercorp.com