TRANSCRIPT FROM SEPTEMBER 24, 2012 ACHIEVING SUSTAINED IMPROVEMENT IN NURSING QUALITY WEBCONFERENCE

Transcription

1 September 25, 2012 TRANSCRIPT FROM SEPTEMBER 24, 2012 ACHIEVING SUSTAINED IMPROVEMENT IN NURSING QUALITY WEBCONFERENCE Good morning and thank you for joining us. I want to review some housekeeping items before we begin. This program is being recorded and will be made available on the NYSPEP website along with a transcript within five business days of this program. The handouts for this conference have been posted and are free to print on the NYSPFP website. To locate the handouts, search under the calendar tab, find the event that you are participating in, and click on see details. You can see the handouts closed captioning login and instruction there. This program has closed captioning. To access, click on the media viewer button located in the panel to the right of your screen, you will need to enter your name and facility inside the media viewer box and hit submit. Alternatively, you may access this by opening a new tab in your browser and going to If you look on your screen now I have this information on the whiteboard and any time you can go in there and copy and paste it into your browser. You also need your confirmation number, which is At any time during the presentation you can click back and forth to the tab get that information. We have muted all of our phones on our end. At the end of the program there will be time for questions and answers. There are two ways in which you can ask a question, you can type your question in the box located to the right of your screen, or you can click on the raise your hand button in the participant box which is at the bottom of your screen. We can then unmute you so you can ask your question. Due to the large group today in order to keep things as organized as possible, we ask that you minimize that box and only use the Q&A box to type in your questions. Please use the chat box if you need to convey technical difficulties you may be having. With that, I will turn the meeting over to Kathy. Good morning and I hope you're having a good Monday. It is a beautiful day in Albany, New York and you have the privilege of kicking off this program. In introducing the speakers and just by way of how I got here today along with Roseann, we have the privilege of leading the team relative to the New York State Partnership for Patients and I am the CNO at St. Peter s Health Partners here in the capital district. In one of our last meetings we were talking about the struggles with data and this is an outgrowth of that. I want to thank both the New York State Partnership for Patients, HANYS, and Greater New York for sponsoring this; I am looking forward to it today. So that sets the stage for how we got here today, and I would like to add to

2 our initiative relative to the Partnership for Patients. With that I am privileged to introduce you to our speakers today. I will quickly read their biographies. Our first speaker is Nancy Dunton; she is a research professor in the School of Nursing at the University of Kansas Medical Center and the School of Medicine s Department of Health and Policy Management. She has been the principal investigator of the NDNQI since it was first established in She has guided the growth and development of the program that has over 9,200 participating hospitals and is the nation's largest nursing quality database. She is an expert in the development of healthcare quality indicators; she has served on the advisory committee for the national quality forum, the agency for healthcare research and quality, and the joint commission. She has consulted with nursing organizations from several countries to help them understand nursing quality database development. Her scholarly work has included numerous presentations and publications. She is the co-editor of case studies on the use of nursing quality indicators for quality improvement published biannually by the American Nursing Association. She is a member of the editorial board of the American Nurses Association today. She was inducted as an honorary fellow of the American Academy of Nursing in 2011, and was the 2009 honorary alumni of the University of Kansas School of Nursing. She received her Ph. D in sociology from the University of Wisconsin. Joining Nancy today is Brandon Crosser, the reporting manager of the national database of nursing quality indicators. He oversees the collection, analysis, and recording of quarterly quality indicators, as well as the RN survey. Recently Brandon has been involved with the design, development, and implementation of the new reporting system for NDNQI using stateof-the-art business intelligence software and visualization techniques. He received his Master of Arts in mathematics and statistics from the University of Missouri, Kansas City and he has extensive experience teaching and tutoring courses ranging from developmental to advanced mathematics. Joining both Nancy and Brandon is Michael Grove our account manager for NDNQI who handles inquiries from the new hospitals that are interested in becoming a NDNQI participant. Welcome to all three of you. Thank you. Thank you. I don't think any good presentation should begin without a round of applause or gracious thanks to the team at the New York State Partnership for Patients. This is Michael Grove speaking and I am the account manager for NDNQI. I want to specifically thank Kathy and Lorraine for the invitation for us to speak on this important topic today and a big thanks to Mary Therriault for bringing all the pieces together on today's presentation. Also, very importantly, I want to express the appreciation to Jack Jordan and thank Jack for showing tremendous leadership behind this remarkable Partnership for Patients program. We are honored at NDNQI to be an important contributor to the program s goals. Lastly, I would like to thank the New York hospitals that are actively participating in the NDNQI database with approximately 70% of New York hospitals using NDNQI, the opportunity is greater to achieve, if not exceed, the Partnership for Patient s goals in reducing hospital acquired conditions. We are thrilled and honored to be presenting today and I will hand the baton off to Nancy and Brandon. Thank you. Page 2

3 Thank you Michael. This morning we are very happy to be with you all and we know that a great many of you have been members of NDNQI for some time, and there will be people who are less familiar with NDNQI, or this is their first exposure. Today's presentation will be a review for some or a refresher and new information for others. The National Database of Nursing Quality Indicators is a program of the American Nurses Association and it is very likely the largest national database on nursing quality in the world. It is at the unit level, which is the appropriate level to look at outcome indicators for quality improvement for nurses, since the care provided by the unit determines the quality of service delivered. NDNQI, I have to say it is not only large and unit level but one of the features that we take pride in is the quality of data we produce. It is thoroughly cleaned and analyzed before reports are distributed to hospitals. In addition, we are always pleased to talk about what you can do with the database. So today we are going to talk about our NDNQI products and services. We are also going to talk about the math behind them and some uses of the data. Then we will have open time for discussion. So let me begin by saying a few words about our NDNQI products and services. Again we provide unit level performance reports with indicators that align with quality improvement models which looks at structures of care, processes of care, and outcomes of care. For many measures within NDNQI there are both nursing structural measures and nursing process measures. We will talk more about those and how they are used in implementing the information in the reports for quality improvement in your hospitals. The reports contain comparison data for similar units and we have a fairly high standard for how each unit is classified, so you get a homogeneous group so it is clearly apples to apples comparisons. We know that it is not usually nurses first job, unless they are in the quality office to collect data and use data so we provide an exceptional amount of customer service and user support for hospitals. So our nurse liaisons answer phone and messages at the rate of 1,000 to 1,500 a month and the response time is very low at about 20 minutes on average. So when we do customer satisfaction surveys we find that the liaisons are probably the most valued part of the NDNQI web service in addition to the reports which contain comparison data. We also have an annual quality conference each year at which participating hospitals send their staff and look at what other hospitals are doing, and to hear national speakers. I find it very satisfying as somebody who has worked with data my entire life, to know that someone is actually using the data, it is always a treat. NDNQI supports the Partnership for Patients goals due to the fact that we had collected for a long time measures that are part of the Partnership for Patients goals. Specifically, we collect date on 5 of the 10 PFP measures. These measures we collect are catheter associated urinary tract infections, central-line associated bloodstream infections, ventilator associated pneumonia, injuries from falls, serious injuries from falls, and hospital acquired pressure ulcers. Stars there indicate these measured are endorsed by the National Quality Forum as the kinds of measures and specifications that could be collected by hospitals across the nation so that we are all engaged in measuring the same items. Today, we are going to look at and understand the structure and process measures that are in the database, as well as, the interpretation of reports and statistics that are developing quality improvement. First, we will begin by understanding the measures that are in the NDNQI. The structure and process measures include, nurse staffing levels, which sometimes are called the ratios, and a skill mix, which is a percent of hours supplied by RN s, who are generally the most Page 3

4 skilled nurses on the unit. So that measures the richness of staffing as well as RN education and certification. The process measures we collect for indicators include whether or not a risk assessment was done, whether the patient was at risk, and if patients who are at risk had a prevention protocol in place at the time of the adverse event. This kind of information, once you look at the outcome measures that you drill down into these areas of structure process to look at how you are doing on factors that are related to the outcomes. Again, we have a pretty rigorous routine for classifying units into appropriate types and we have quite a number of unit types including adult and pediatric critical care, step down and immediate care, medical and surgical units, rehabilitation, psychiatric units and recently, we have added mixed acuity units in which more than one level of patient is on a unit so it could be a combination of step down and medical, or critical care and step down, or critical access units, or bone marrow transplant units, burn units etc. We provide 67 categories of hospital types so you can find hospitals very similar to yours including dimensions, an academic medical center, a teaching hospital or a community hospital, magnet status, we have information by states, so you can look at New York as a whole and other locational units such as metropolitan small town or larger urban region. Interpreting reports and statistics. Now Brandon is going to tell you more about the reports. Thank you Nancy. So, whenever you receive your reports from NDNQI, the first thing you want to do is go into it with a question. Whenever you are formulating your questions you want to think, how are we doing and compared to whom? So are we in the top 25% or the bottom 10 percentile of like units and like hospitals? So once you have your questions, then to approach the tables with a question, it is always good to have a plan of attack. So I will layout one way to go about that and there could be many ways to read it, but just to have a plan of attack, and then we will have a quick review of some of the statistics that are presented in the reports. So one way to approach the reports is with a counterclockwise motion or a spiral and you can start at the very top with the title of the report. This will always give you a very good indication of what is being measured. In the example that we see here, it is clear that we are talking about unassisted falls per 1,000 patient days and with that in mind, we want to go around the spiral and the next step is to figure out what units we are looking at. In this case, we're looking at adult medical units and you can start to think about the structures of care on medical units and the types of patients they are caring for and would you expect falls there or not. As you move down you want to look at who are we being compared to and how many of them? In this example, we see this one medical unit is being compared to over 800 units and teaching facilities. That gives us a really high level of confidence that the data is very stable and we have a large sample size to compare against. The last piece you want to figure out is where in the percentiles is this unit performing? This unit s unassisted fall rate is between the 10th and 25th percentile and since lower fall rates are a good outcome this is a fairly good outcome on this unit. So, once you have an approach to the table, there is a lot of statistics so what we will review today is some of the basic ones that are presented here as far as the mean and standard deviation. How to use the median and percentiles and interpret them properly and what some subtle differences are between rates such as percentages and ratios. To start off, we want to touch on mean and standard deviation so first we will talk about the mean and the mean is the measure of expected value and on average that is where the normal Page 4

5 range is so the mean is the mathematical average. If you have five observations it is the sum of all those observations divided by five. For example, if you have a set of five numbers, 1,1,1,3, and 4, the sum of those is10, divided by 5 and the mean is two. The next thing that will be presented is the standard deviation. The standard deviation gives you an idea of the scatter of the data or how tightly grouped the data is. Conceptually the way you can interpret the standard deviation is the average distance that each observation is away from the mean. Another way to tie the standard deviation and the mean together is with confidence intervals. For example, if you want to know where 95% of the data is in a normal distribution it is the mean standard deviations between that and the mean standard deviations. The next couple of concepts are very related. One is a special case of the other with the median and percentiles. The idea of a median is if you line the data up in a row, it is the very middle value; basically, it is the midpoint of a distribution. So if we were to look at the same five numbers as we did before, 1, 1, 1, 3, 4 we will see that the middle observation is 1. The median of this small data set is 1 and shows that when you are given the same set of numbers the mean and the median are not the same. So the median is a special case of a percentile. So if we were to grow that concept and maybe instead of where just the middle is we want to know where the bottom 10% is, you would go about it in a similar method. Order all of the observations from low to high and you count how many observations you have. If you divide those into groups like the bottom 10% or the bottom 25% or top 25% those give you your percentiles. For example if you have 50 observations in a data set and you order them in a row, the 5th observation is going to be where your 10th percentile is since 5 is 10% of 50. So now that we know we have two different ways to describe the expected value of a distribution, the question becomes when to use which. There are two cases depending on the data. In this picture we see two different distributions the one on the left, if it is kind of symmetrical and bell shaped, these are more normal distributions. An example of these could be staffing rates on a particular unit. They are fairly bell shaped; they're about the same amount of data above and below the mean, so the mean is a good measure of the central tendency or expected value for these types of distributions. However, many distributions can be heavily skewed to the left or the right and they no longer look bell shaped as in the picture on the right. For these distributions the median will give you a better idea of the expected value. So an example of this would be fall rates on a critical care unit. Almost all of them will be zero, so in that case it is very heavily skewed toward 0, so we suggest the median is a much better measure for the expected value of those. To touch on a little bit of the subtlety between some of the rates, the differences between percentages and ratios, so the idea of percentages is we have a specific number of events divided by all observations that were measured. So for example, we survey hundred patients for pressure ulcers and twenty-five of them did indeed have a pressure ulcer, then we are measuring the same thing the numerator and denominator where the numerator the special case in which they met the criteria for being included in the percentage. So 25 divided by 100 gives you a 25% pressure ulcer rate. This is a little bit different than ratios used for example, in fall rates, in the numerator we will still be talking about the number of events and in the fall rate would be the number of falls. However, in the numerator is not the same thing being measured it is a standardized population, so for a fall rate we want to take into account the number of patient days on that unit so that evens the playing field and different units of different sizes will come out on the same scale. So for example if we have 7 falls on a unit in 500 patient days, if we take 7 divided by 500, and to make the numbers a little bit nicer we have a scaler of 1,000 so that would come out to an even whole number for this fall rate. So now that we Page 5

6 have learned a little bit about how the data is presented and what some of the statistics mean, the next step is to start developing your QI plan. Thank you, Brandon. To begin with as with all uses of NDNQI reports, the best thing we can do is to start with a set of questions. The things that are answered directly in the NDNQI tables are how does the unit compare to other units of the same type? Are we getting better or worse over time? The first one you can tell is how your unit compared to other similar units by looking at the mean or median and which percentile are you in. We recommend for outcomes looking at the percentile distribution to see where you are in the distribution. Due to the fact that it is a skewed distribution as Brandon said and it is important to know how you are progressing, you can track yourself by looking through the percentiles. So that is related to the second question which is, is my unit getting better or worse over time? The third question is, what goals should we set, and with all quality improvement text that you find, you should set an achievable goal to start so people don't get discouraged. If you find out according to the percentile distribution what the rate is for the best 25% of peer units, unless you decide there should be zero tolerance for a particular outcome on the unit in which case your goal would not be a percentile or a rate it would be zero for the rate. So if you look at the NDNQI reporting for total falls, you see in the top row there are medical adult units that have a fancy name of medical adult and you have 8/4 of data to track the unit over time, did not report for the first three quarters so the ND is no data. Then the rate went from 5.09 to 1.87 so it got a lot better over time and as you can see in the red boxes in the comparison data for teaching facilities you can see that the unit was in the 75 th percentile in the first quarter of 2011, and got progressively better in each of the next quarters. You can use the reports to track how you are doing in regards to an implementation of a fall prevention plan. You can also use the reports to get a better understanding of the problem and its causes on the unit. In some circumstances the data we have seen can examine related nursing workforce measures or examine process related measures. So this is what we have diagrammed, a flow process for how the fall prevention process measures are working on this unit. You can use this to identify steps you can take to improve the process. For example, if you look at all of the people that fell on this unit and identify if they had a prior risk assessment or not, you will see that 83.9% of patients had a prior risk assessment and 16% didn't, which means that 16% of your population that fell you are not assessing for risk. Therefore, you might want to think about expanding the population in your unit that is subject to a fall risk assessment, expanding the age range, or something about the medical condition that would allow you to include more people in the risk assessment tool. Among those that had a prior risk assessment, the anonymous risk assessment tool identified 79% at risk and 21% not at risk and these are people are the ones that fell, so the tool you are using did not pick up 21% of the patients who had a fall who were determined to not be at risk. So there is an issue with sensitivity of that measure. If this is the case, you might want to look at other risk assessment tools or modify the tool that you have to be more inclusive of reasons why people fall. Look at case reviews of the falls and find out what it was that led to the patient fall. Among those at risk, 92% of the patients had a prevention protocol in place and 8% didn't and I can't think of any reason why 8% didn't it must be they just didn't get the protocol in place in time for the fall. So we turned to an example of a hospital that had sustained improvement in fall prevention. Rush-Copley Medical Center, which I believe is in Illinois, looked at their NDNQI reports and they decided that falls were a problem. Not that they were the worst rates ever but that there was Page 6

7 inconsistent performance. There were high rates in some quarters and low rates and others and they celebrated when they had a reduction but then they realized it was due to chance because they could not sustain it. So they set the goals to be in the best 25% of hospitals in this particular unit and to be in the top 25% to have among the lowest 25% of all rate. They wanted to reduce the variance in risk assessment and interventions similar to the flow chart that we just looked at in which more patients got risk assessment and more interventions were introduced. For their quality improvement plan they used the rapid cycle plan to study protocol which is a standard quality improvement protocol. They did involve the top leadership and they decided to change the culture in the hospital so they embarked on a staff engagement campaign in which instead of oh falls will happen, everybody started to see everywhere the slogan of no falls will happen. They went to the literature to identify evidence based practice methods for fall prevention and for those of you who are into falls; there is not a lot of evidence about specific sets of interventions that work. So they went through a period of time trying a number of different interventions to see what would work best with their patient population. They introduced staff accountability so if there was a fall and a nurse was involved with the patient at the time of the fall, and perhaps did not plan an intervention; they would question this person so they could learn as a group more about fall prevention. They adopted the just culture approach on the unit so that people weren't penalized necessarily but they went into continuous learning culture modality and what was the result? They had no falls with major injury in nearly 2 years on this unit so they were very successful at sustaining their improvement and they did this by monitoring how the progress happened and maintaining the interventions that they had initially put in place. Sometimes a problem with the quality improvement intervention plan is that the focus of the goals fades and people revert to normal behaviors, but not in this instance. I want to change to just a side detour and as Michael said, we have over 1,900 hospitals out of the 5,000 or 6,000 hospitals in the United States participating in NDNQI and 75% of the hospitals in New York State. In soils across the nation, we find that NDNQI hospitals tend to be larger and more likely to be an academic medical center, and tend to be in an urban or metropolitan location. We want to address briefly what the benefits of NDNQI participation are for small and rural hospitals because I think that in the perception of many, the benefits of participation are seen as minimal, and in fact participation in this program could be quite useful. We do collect data as I mentioned several times by collecting information on the characteristics of the hospital so we have comparison data, solid comparison data for hospitals under 100 beds that and for critical access hospitals. We have adopted the mixed acuity unit type which is more prevalent in smaller rural hospitals. Also, starting in 2013 we are going to have staffing data for emergency perinatal units and many of these are central business areas for smaller rural hospitals that handle emergency cases and in some cases transfer to larger hospitals or are the regional birthing centers, etc. There are no special IT requirements for NDNQI. We have a broad range of setups in hospitals across the United States are able to use whatever operating systems they have. To give some idea of the size of the comparison data for small hospitals, there are 243 units in 51 hospitals located in rural areas, and that is a small number that we certainly want it to grow, but it is a large enough number that the comparison data is stable and doesn't jump about because of a small sample size. We have 1,100 units in hospitals in cities between 10,000 and 50,000 people which are mid-level towns and we have reports based on fewer than 100 beds based on the comparison data of 2,000 units in these under 100 bed hospitals. So the comparison data is stable, there are no special IT requirements, we are beginning to get measures and the Page 7

8 common areas for practice in rural facilities and critical access hospitals participating in NDNQI has grown over time. Brandon is going to tell you about some exciting new things that are coming in Thank you, Nancy. So, just around the corner, we have been developing new types of reporting systems for the end-user. So earlier we took a look at some of those data intensive tables and one of the ideas here is to make the data much easier to consume. So just a guide of where we are headed in this direction, we have a couple pictures, and this is just proof of concept at this point, an actual product is in development right now. The idea here is it is much more intuitive to navigate around, you have options to interact with the data, so you can change the unit type you are looking at on a particular dashboard, and you can change the comparison group or the timeframe you are looking at. You can go through your data and visually see how you compare to a mean or a median or whichever you choose to compare against. So, we have organized this to have multiple levels so you could have a dashboard that is for your hospital level data and we have dashboards specific to unit type that will show the entire step down units compared to each other's step down units. Even drilling further down into a specific unit showing their measures of structure process and outcome for an indicator and how those interactions occur on a specific unit. We also want to make it much easier for hospitals on the magnet journey so we want to give you the ability to change your comparison group on the fly and change between the mean and the median. We can tell you how many previous quarters you have out-performed that measure and that is where we are headed in the future so it is an exciting time and we are going to be happy to hear the feedback on this and keep growing and developing these dashboards even more. With that, I think I can turn it back over to Michael and he will let you know where you can find more information. Thank you, Brandon. We are wrapping up and coming to the end and we want to share with everyone that the NDNQI has created a link for additional resources if you are interested and we have created a micro-site specifically for today's program and you can see the link to the URL, which is What we have put there are some quick videos, all about learning and we have put a case study and made that available on a fall prevention case study based on NDNQI data and so we would welcome you to write that down and visit at your convenience. The last slide before we get into Q&A, again, thank you to everyone who participated in this outstanding presentation and thank you Nancy and Brandon. With that, I would like a round of applause to them for all of their hard work and for putting today's program together. I think we should move to Q&A at this point. Thank you. That was wonderful. I have one question and the question is, should we be assessing all individuals for fall risk to have a baseline assessment? I am not sure I understand the question. Should everybody get a fall risk assessment? That is correct. That is what I believe. Page 8

9 I don't think so. I think that there are certain populations that you don't expect to have falls. We are expanding the falls measure next year as well to include intentional falls such as what might occur on psychiatric units and we have a number of categories for falls on pediatric units so you could separate out developmental falls for those purposes. Some falls are clearly the result of an immediate physiological event or the result of environment problems such as slippery floors and we don't collect information so we don't know what the proportions of each are but I think you should let your data guide you in terms of the population on each unit to see if you are, over time, capturing most of the people that fall if most of them have had a prior risk assessment you are fine. One of the measures that NDNQI has is the recency of this assessment and for pressure ulcers in particular, it really has a fall risk assessment at least every 24 hours is beneficial, you have lower fall rates and lower pressure ulcer rates if you assess daily. I believe that Kathy has a question. I was wondering if you could talk about the structure or staffing and what other nursing reports measures you use in addition to hours per patient days. NDNQI has a large number of staffing measures available and I really think of them as characteristics of the nursing workforce as opposed to staffing measures because when you say staffing you think there's nurse-patient ratios which we capture as nursing hours per patient day which is algebraically the same thing. We also collect data on licensure status and how many RN s and LPN's are disappearing over time. We capture years of experience on the survey and we capture education which is useful for the institutes of medicine tracking progress towards 80% of the hospital nurses will have BSN s by 2020 which is a very tall order for hospitals but at least you can track your progress on that measure. Certainly there is evidence in the literature for better patient outcomes with BSN prepared nurses and we also look at the number of agency nurses and contract nurses who have been linked in some cases to adverse events. Although the directionality of that measure is in question so you don't know if agency nurses have as much group knowledge of the hospital or knowledge of the team members on the nursing staff as the others. We also look if there is a problem on the unit and at any rate, that is another characteristic and we look at other features out of the survey such as where the nurses were trained, U.S. or abroad, if there is information on years of experience on the unit, years of experience in nursing, and then a lot of job satisfaction and environment scales. We are doing a set of research that shows some of those factors are related to adverse events as well. In particular, we are publishing a paper that looks at RN and M.D. communication and one of the infection rates. You can go to the nursing quality.org website and see the list of publications and presentations and that is an easy way to capture some of the factors that have been associated with nursing and give you a guide to what to look for your review reports besides just staffing levels. Alissa, this is Mary, and I just want to make sure there are no questions that you are seeing, that I am not seeing on your end. Thank you, this is Alissa from the NYSPFP team and I want to thank our speakers from NDNQI for joining us today. This has been helpful. I just wanted to note, I have been getting a few e- mails and some questions about the PowerPoint slide presentations and the website address that Page 9

10 our speakers have provided on the second to the last slide. I just want to let everyone know the PowerPoint presentation is up on the New York State Partnership for Patients website for your access located within the calendar section of today's webinar. It is there and if you have not gotten it already I encourage you to visit that area of the website for more information. There is one question that I see on the screen, are you going to let the NYSPFP take the data directly out of your database? We are working on that, you have to sign a waiver for that to happen. So it would not happen without your consent. Just to reiterate, we sent out a notice about this specifically and are excited that we will be able to access hospital data with their permission for use in Partnership for Patients throughout the data collection methodology rather than hospitals having to reenter that information through the online data portal. We have heard it was causing a little bit of a burden or duplicate action for hospitals. So we are going to be sending out further information about this in the coming weeks and we will provide you with the specific information and next steps that hospitals have to do, including signing this waiver to allow us to access that data directly from the NDNQI database. We are excited about that and more information will come over the next couple of weeks. Thanks. Perfect. I don't see any more questions and on behalf of the NYSPFP, this is Mary Therriault, and I want to thank Nancy, Brandon, Michael, and Kathy for joining us today and for all of you that were able to listen in. Have a great day and we will talk to you soon. Goodbye. Page 10