Metrics in Research Measuring Readiness to Promote Resiliency Operational Metrics and Standards San Diego, CA December 3, 2015 Tiffany A. Radcliff, Ph.D. -- radcliff@tamhsc.edu
Emergency Management Considerations Operations vs. Research Operations Questions Research Questions What events are possible? Needed for better planning How is readiness assessed? Routine measures How will we respond in an event? Measure Resources Internal vs. External Personnel, timeframes, impact for each What events are most likely in various locations? How is readiness defined and measured? Any lessons learned from previous responses? Measure Resource Gaps available vs. needed Measure Response Gaps What occurred vs. what would be desired?
Readiness in theory
Readiness in Practice Source: tamu.edu and TEEX
Or this?
Today s Discussion: Outcomes measures for research on resiliency and recovery At VEMEC, researchers use a variety of data sources to capture measures of readiness and resiliency 1. VEMEC studies using VA data OEM direct measures: EMCAP assessment VA Administrative records: Missed Opportunities around Hurricane Ike 2. Federal Statistical Research Data Centers (RDCs): Disasters and the elderly (funded by National Academies of Sciences: Gulf Research Program)
EMCAP METRICS Phases 1 and 2 VEMEC and OEM analyzed methods, metrics and analyses to inform Phase 3 Examined capability metrics across various domains Counted frequency of by domain and assessment Phase 1: 69 capabilities, 6 domains Source: Recommendations for Phase III Emergency Management Capabilities Assessment Program (EMCAP): An Interim Report April 30, 2013, VEMEC
How we looked at EMCAP Outcomes Distribution of EMCAP assessment measures for each of 21 VA Integrated Service Networks Source: Recommendations for Phase III Emergency Management Capabilities Assessment Program (EMCAP): An Interim Report April 30, 2013, VEMEC
ONE DE-IDENTIFIED VISN Phase 1 Capabilities Assessment Summary Counts for all stations in a single VISN Phase 1 Capabilities Distribution for each station in a single VISN Phase 1 Capabilities Sparklines (win/loss) showing how each station was rated for each capability in a single VISN Source: Recommendations for Phase III Emergency Management Capabilities Assessment Program (EMCAP): An Interim Report April 30, 2013, VEMEC
EMCAP Outcomes Summary EMCAP Metrics to benchmark and understand general readiness indicators important, informative, and useful Next steps: Test whether relationship to overall performance across assessment phases remains consistent A well-prepared facility should fare well with both the old and new metrics, holding other factors fixed Determine if an overall score (weighting/indexing) measures is feasible for measuring readiness Others?
Resiliency & Recovery in VA VEMEC researchers studying realized access to care around major disasters Event studied: Hurricane Ike Gulf Coast near Galveston/Houston in September 2008 VA clinics in impacted station varied in proximity to storm 20 most-frequent clinic stop codes, grouped into clinic types (primary care, diagnostic services, mental health, specialty care, other clinician services) Desired properties of outcomes measures: improve understanding of impact on veterans (patients) use of care Existing/readily-available data Replicable to other storms or settings if deemed feasible/useful
Outcome Measure for Ike Study Resiliency Percent completed appointments Recovery For missed appointments, days to completing the same appointment type Considerations for Analysis week relative to Ike clinic type clinic location Reason for missed appointments Cancelled by clinic Cancelled by patient No Show No longer needed (excluded)
Preliminary Findings: Resiliency Resiliency: By Clinic Location Resiliency Summary Drop in % Completed Appointments the week prior to Ike s landfall Week 0: 0% completed in Beaumont, <10% in Galveston week 0 Beaumont exceeded other clinics by week 3; Galveston reached pre-storm % by week 6 Lufkin (far from coast) had minimal interruptions Source: Unpublished Data (IKE research in progress)
Preliminary Findings: Recovery Days to Complete a Missed Appointment before, during, and After Ike Source: Unpublished Data (IKE research in progress) Recovery timeframes varied across clinic types Average days to completing missed appointments was stable over timeframes Not shown, but similar: results by clinic location
Ike Outcomes Summary Resiliency: Lower % completed appointments around Ike was mostly due to clinic cancellations Operational considerations in cancellations: patient and staff safety internal and external infrastructure Rescheduling appointments Recovery: Time to completed appointments varied, but mean/median was not substantially higher around Ike Possible reasons: auto-rebooking and some flexibility in clinic locations for common appointments
Medicare (RDC) data studies Healthcare utilization around natural disasters Do disasters change use patterns of common healthcare services? Linking historical disasters data to Medicare claims records Possible Effects: delayed care, different mix of care, new location of care, etc. Preliminary hypotheses: Timeframe during disasters: more home health and ED less preventive care After disasters: more mental health services No results yet
Summary Primary and Secondary data metrics are useful for informing operations around disasters Readiness and resiliency metrics: better in theory or in practice? Depends on the intended use EMCAP can identify areas for targeted improvement before an emergency VA appointments data is more useful after an emergency to assess impacts Data resources for measuring outcomes VA EMCAP: specific to emergency management; requires dedicated resources Appointments data: more general and collected for another purpose Claims data: must be linked with dates and locations of emergencies This presentation captures 3 general metrics for readiness and recovery other metrics can be tailored to meet specific information needs