Orchestrated Testing Aggregate Data Michele Walsh, MD, MSE Through collaborative use of improvement science methods, reduce preterm births & improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
Percent of Infants with Greater than 10% Weight Loss from Birth Weight in First Seven Days De-Identified Formula Groups
Number of Finnegan Scores Greater than 12 in 24 Hours Prior to Starting Pharmacologic Treatment De-Identified Formula Groups
Highest Finnegan Score in 24 Hours Prior to Starting Treatment De- Identified Formula Groups
Average Length of Stay for All Opioid Exposed Infants De-Identified Formula Groups
Average Length of Opiate Treatment De-Identified Formula Groups
Average Length of Stay for Pharmacologically Treated Babies De- Identified Formula Groups
NAS-Orchestrated Testing Group 1 Theresa Ruby, MSN, RNC, IBCLC Southern Ohio Medical Center Through collaborative use of improvement science methods, reduce preterm births & improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
Group 1: 22 kcal /low lactose formula Akron Children s St Elizabeth/ Mahoning Valley/St Joseph Kettering NCH Dublin Methodist NCH Mt Carmel St Ann s NCH Riverside ProMedica Bay Park Rainbow Babies & Children s Cincinnati Children s Hospital Medical Center Soin Medical Center Southern Ohio Medical Center Southview Medical Center St Rita s Medical Center Trumbull Memorial UC Cincinnati ProMedica Bay Park Trumbull Memorial Rainbow Babies & Children s St. Rita s Medical Center Southview Medical Center Kettering NCH Riverside NCH Dublin Methodist NCH Mt. Carmel St. Ann s Akron Children s St. Elizabeth Mahoning Valley St Joseph s Soin Medical Center Cincinnati Children s UC Cincinnati Southern Ohio Medical Center
NAS-Orchestrated Testing Group 2 Gail Bagwell, DNP, APRN Nationwide Children s Hospital Through collaborative use of improvement science methods, reduce preterm births & improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
Group 2: 22 kcal /not low lactose formula Bethesda North Licking Memorial Hospital Dayton Children s Mercy Children s Hospital Good Samaritan Tri-Health Nationwide Children s Hospital Mercy Children s Hospital Dayton Children s Nationwide Children s Bethesda North Licking Memorial Good Samaritan Tri-Health Hospital
NAS-Orchestrated Testing Group 3 Mischel Balazs, NNP, CNP-BC Toledo Children s Hospital Through collaborative use of improvement science methods, reduce preterm births & improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
Group 3: 19 kcal /low lactose formula Akron Children s Hospital Medical Center Mercy Anderson Akron Children s Summa Mercy Medical Canton Akron Children s General Miami Valley Hospital Atrium Hospital Mt Carmel East Aultman Hospital Mt Carmel West Cleveland Clinic NCH Doctors Hospital Fairview Hospital NCH Grant Medical Fort Hamilton Hospital OSU Wexner Well Baby/OSU Wexner NICU Genesis Healthcare ProMedica Toledo Normal Newborn Hillcrest Medical Center ProMedica Toledo Children s Hospital Toledo Children s Toledo Normal Newborn Miami Valley Atrium Fairview Cleveland Clinic Hillcrest NCH Doctors OSU Well Baby OSU NICU NCH Grant Mt Carmel East Mt Carmel West Genesis Akron Children s ACH Summa ACH General Mercy Medical Canton Aultman Fort Hamilton Mercy Anderson
NAS-Orchestrated Testing Group 4 Tonia Deacon Through collaborative use of improvement science methods, reduce preterm births & improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
Group 4: 19 kcal /not low lactose formula Adena Regional Medical Center Mercy Health Fairfield Elyria UHCMC Mercy Health West Good Samaritan Dayton Mercy Regional Lorain Lima Memorial Hospital Springfield Medical Center Marion General Hospital The Christ Hospital MetroHealth Upper Valley Medical Center Lima Memorial Elyria Mercy Lorain MetroHealth Marion Good Samaritan Dayton Springfield Upper Valley Mercy Fairfield Medical Center Mercy West Christ Hospital Adena
Data Analysis of Orchestrated Testing Heather Kaplan, MD Through collaborative use of improvement science methods, reduce preterm births & improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
OPQC Phase II Orchestrated Testing
OT Analysis Team Maurizio Macaluso, MD, DPH Pierce Kuhnell, MS Heather Kaplan, MD, MSCE
Quality Improvement using the Model for Improvement Model for Improvement What are we trying to accomplish? How will we know that a change is an improvement? What change can we make that will result in improvement? A P S D Changes That Result in Improvement Implementation of Change Hunches Theories Ideas 55 A P S D Very Small Scale Test Follow-up Tests Wide-Scale Tests of Change Sequential building of knowledge under a wide range of conditions
Orchestrated Testing OT involves planned testing across multiple sites (within or across institutions) Can use factorial design to Be more systematic about simultaneous testing of different change ideas Look at the independent and combined effects of different changes Standardization of practices and reliable implementation is necessary Can result in faster and more efficient learning
OPQC OT Phase II Wide scale test of change examining the role of formula in non-pharmacologic care across 54 NICU/SCN sites Two change ideas (factors): Type of formula Calorie content of formula Two levels of each factor Standard Lactose vs. Low-Lactose Standard Calorie vs. Higher Calorie Factorial Design
OPQC Factorial Design (2 2 ) Group Calorie Content Formula Type A 19 kcal/oz Low-Lactose B 22 kcal/oz Standard C 19 kcal/oz Standard D 22 kcal/oz Low-Lactose
The Model for Improvement
Measures: Outcomes Primary Outcome LOS (pharmacologically treated infants) Secondary Outcomes LOS (all infants) LOTx Percent pharmacologically treated Percent infants requiring dose escalation Weight Loss >10% These are similar outcomes to the outcomes examined in the RCT of standard vs. 24 kcal formula posted on clinicaltrials.gov (weight loss, LOS, LOTx, Finnegan scores)
Measures: Background Variables Pharmacologic treatment approach* OCHA Morphine Cincinnati Methadone Other Morphine Other Methadone Rates of breastmilk feeding Percent outborn infants + Formula compliance Time of formula initiation (early vs. time of treatment) NAS Severity (Max Finnegan Score, Scores>12) Change in typical formula treatment from Phase I to II *Centers with extreme deviation from these protocols will not be included in analysis + Infants transferring between sites in different groups will be excluded
Analysis Main analyses are graphical Run and control charts Explore differences on background variables Examine impact according to groups Response plots
Run and Control Charts Explore Background Variables 1 2 3 4 5 6 Background Variable Groups: y, c, z, x
Run and Control Charts Examine Impact Group X Group Y Group Z Group C Factor 1 High High Low Low Factor 2 Low High Low High
Response Plots: Interactions No Interaction F2 High Moderate Interaction F2 High F2 Low F2 Low Low Factor 1 High Low Factor 1 High Moderate Interaction Strong Interaction F2 High F2 High F2 Low F2 Low Low Factor 1 High Low Factor 1 High
Replication Ultimate confirmation of findings from OT comes from replication across sites over time If positive results are seen for a change, process modification at additional sites with subsequent improvement further validates findings This is a critical step because conditions in the future (when the findings will be used) are likely different than the conditions during OT
How this works in real life SLUG Bug QI Project with OT 17 participating centers Examine 4 CLABSI prevention strategies Tubing change technique (sterile vs. clean) Hub care monitoring (yes vs. no) CVC access limitation (limitation vs. no limitation) CVC removal monitoring (tracking policy vs. no policy) Used factorial design (2 4-1 8 groups) Focused on reliability 16/17 centers achieved >75% reliability on these processes Piazza AJ, et al. Pediatrics. 2016
4 Factors 8 Groups Variation in Baseline Rates Piazza AJ, et al. Pediatrics. 2016
Run and Control Charts ** ** ** ** + + + + 3/4 Sterile Tubing Change ** Greatest decrease with addition of HC Monitoring+ Piazza AJ, et al. Pediatrics. 2016
Response Plots: Impact Sterile Tubing Change Decreased CLABSI Rates by an average of 0.51 Piazza AJ, et al. Pediatrics. 2016
Response Plots: Interaction The addition of hub care compliance monitoring with sterile tubing change created the greatest impact showing evidence of an interaction Piazza AJ, et al. Pediatrics. 2016
Replication Courtesy of Lloyd Provost, data unpublished
How this might look for OPQC Group Calorie Content Formula Type # Hospitals Baseline LOS Phase II LOS A 22 kcal LLF 11 17.36? B 22 kcal Standard 5 20.51? C 19 kcal LLF 17 20.00? D 19 kcal Standard 12 21.71?
Hypothetical Control Chart Type LLF Std LLF Std Calorie 22 kcal 22 kcal 19 kcal 19 kcal
Hypothetical Response Plots 19 kcal 22 kcal Std LLF Std 19 kcal 22 kcal LLF Calories LOS: Formula Type LLF Std 19 kcal 18 21 19.5 22 kcal 16 16 16 17 18.5
Keep in mind We are examining the effect of 2 factors in a complex system there will always be the possibility that other variables (known background variables or unknown variables) account for some of what we see BUT, the complexity is also a strength wide range of clinical conditions improves generalizability The strength of this OT approach is that is allows us to examine factors in real settings in a range of conditions (54 sites!) This is an new method of learning for OPQC and we have to figure out some things as we go (e.g., best approach to adjust for differences in baseline LOS) After June, we need to commit to implement the findings to achieve the full benefit of the OT approach
References for other geeks Pallotto EK, et al. Orchestrated Testing: An Innovative Approach to a Multicenter Improvement Collaborative. American Journal of Medical Quality. 2015 Oct 19, [Epub ahead of print] Piazza AJ, et al. SLUG Bug: Quality Improvement with Orchestrated Testing Leads to NICU CLABSI Reduction. Pediatrics. 2016 Jan;13(1):1-12. Moen RD, Nolan TW, Provost LP. Quality Improvement Through Planned Experimentation. 2 nd ed. New York: McGraw-Hill, 1999
Polaris C Polaris F Hallway 1 RB&C MetroHealth Dayton Children s Atrium Mercy Anderson Genesis Mt Carmel East Springfield Mercy Fairfield Elyria The Christ Hospital Licking Kettering NCH Riverside NCH Mt Carmel St Ann s UC Health St Rita s Medical Center Marion General Trumbull Hallway 2 Hallway 3 Nationwide Children s Aultman Hospital Toledo Children s Mercy Lorain Soin Medical Center Mercy West Gemini B Mt Carmel West Hillcrest Good Samaritan Dayton CCHMC Dublin Methodist Miami Valley Hospital Storyboard Rounds Steal Shamelessly; Share Seamlessly Bethesda North Fairview Hospital OSU NICU/OSU Well Baby Mercy Children s Southview Akron Children s/summa Gemini C Good Samaritan Tri-Health Mercy Canton NCH Grant Akron St E/Mahoning Valley Southern Ohio Medical Center Fort Hamilton **LUNCH is 12:30 1:15PM after Storyboard Rounds**