Bariatric Surgery Registry Outlier Policy 1
Revision History Version Date Author Reason for version change 1.0 10/07/2014 Wendy Brown First release 1.1 01/09/2014 Wendy Brown Review after steering committee Feedback from RBH 1.2 02/03/2015 Monira Hussain, Wendy Brown, Dianne Brown 1.3 21/05/2015 Monira Hussain, Wendy Brown, Dianne Brown Review after steering committee Feedback Review after steering committee Feedback 1.4 15/04/2016 Brittany Smith Review after steering committee Feedback Sections modified 5 All the sections All the sections 2, 3.1 & 3.2 Amendments in this release Section Title Section Number Amendment Summary Stratified Response Strategy 5 Clarification of contact protocols in the event of alerts being triggered The whole document All the definitions have been added and clarification of the proposed actions. Definitions, Data Collection, Data Validation 2, 3.1 & 3.2 Amended 30 day to perioperative, and adverse event to Defined adverse event 2
The Bariatric Surgery Registry Outlier policy 1 Introduction This policy document sets out the guideline for outlier monitoring, investigation and reporting in the Bariatric Surgery Registry (BSR). This policy is designed to provide transparency about data handling and analysis, and outlines a robust process for managing surgeons, hospitals and/or devices with indicator values that fall outside the expected range of performance (i.e. are flagged as an outlier ). The primary role of the BSR is to ensure the safety and quality of bariatric surgery in Australia. It does this by providing high-quality, robust clinical data. It is anticipated that outliers will be rarely detected and that a regular reporting cycle will help to drive up clinical quality. Where outliers are found, the BSR team will seek to provide additional help to providers wanting to review data entry and quality. Surgeons with concerns about data quality are urged to contact the BSR team at the School of Public Health and Preventive Medicine, Monash University, Australia at the earliest opportunity to discuss them. All feedback will be confidential. 1.1 Responsibility The BSR Steering Committee is primarily responsible for overseeing the implementation of the outlier policy which is designed to respond to surgeons, hospitals or devices that are found to be performing in a sub-optimal fashion. 1.2 Purpose of the outlier policy The purpose of an outlier policy is to ensure that all providers of data patients, surgeons, hospitals are aware of how the data will be collected and analysed. It also clarifies who will be undertaking this analysis and who is responsible for responding to issues identified. In this way all stakeholders can have confidence in the BSR s ability to ensure the safety and quality of bariatric surgery in Australia. Key features of the outlier policy are the focus on early warning as well as the provision of opportunity to review the factors which may have contributed to being an outlier or exception being identified. These factors may include data not being sent in a timely manner, or the casemix of the surgeon. The outlier policy is in line with the recommendations of the Australian Commission for Quality and Safety in Healthcare in their Framework for Australian clinical quality registries. 3
2 Definitions Performance indicators - Performance indicators are intended to provide a valid measure of a provider s (surgeon, device and hospital) quality of care. In the case of the BSR these vary according to the entity that is being assessed: Surgical performance: individual surgeon perioperative clinical indicators (defined below), mortality and need for reoperation will be measured. As the Registry matures, we envisage we can also benchmark the weight loss achieved by individual surgeons when compared to peers. However, our first priority will be safety. Device performance: the need for devices to be replaced or repaired, as well as mortality and perioperative clinical indicators for each device. Hospital performance: the perioperative clinical indicators for each hospital will be measured. Expected Performance Level The expected performance level of a performance indicator is determined by the mean of the population. For example, BSR will compare a surgeon s/ device s performance against all other surgeons or devices in the same class after case-mix 1. Clinical Indicators - measures reported at 20-90 days after surgery and annually that are recognised by the Royal Australasian College of Surgeons (RACS) as objective, measurable markers of performance. Perioperative Indicators unplanned readmission to hospital, unplanned ICU admission, unplanned reoperation, death Annual Indicators - need for reoperation, weight loss, change in diabetes status Outlier If the value of a performance indicator is more than a specified number of standard deviations (SD) from the expected performance level, over a specified period of time, it will be considered an outlier. For instance, those surgeons or providers who fall between 2 SD and 3 SD from the expected level of performance will be considered as an alert. If in subsequent two reporting periods the surgeon or the provider falls beyond 3 SD from the expected level of performance, will be flagged as an outlier 2. Bariatric Surgery Registry (BSR) Steering Committee The body that oversees the governance of the BSR, provides strategic direction and ensures the agreed deliverables of the Registry are achieved. A full description of the structure and function of the steering committee is provided in the relevant Terms of Reference (ToR) document. Investigation Committee this is a sub-committee of the BSR that will be formed to ensure that the results of any Level 3 Alert are investigated. The membership of the Investigation Committee will be decided by the Chair of the steering committee of the BSR. They will select members according to the skills required for the specific situation that is being monitored, and will always ensure there is independent expert representation. Investigation Committee members will commonly be Steering Committee Members, however, the Chair has the power to appoint any person they feel has the appropriate skills to contribute to the best resolution of the situation 4
3 Procedures 3.1 Data collection Currently data are collected through paper based patient information sheets as well as direct entry through the Bariatric Surgery Registry Interface Responsibility for data collection and provision of data to the BSR rests with local surgeons or surgical team performing bariatric surgery. These data are required to comply with the standards for data quality and format as described by the BSR in the data dictionary. Data should be sent to the BSR perioperatively and annually. The Perioperative Follow Up Data Window is any data that is taken between 20 days from day of surgery and 90 days from day of surgery). Annual reporting,i.e. 12 Month Follow Up Data Window, is any data that is taken between 3 months from day of surgery and 15 months from day of surgery and 24 Month Follow Up Data Window is any data that is taken between 15 months from day of surgery and 27 months from day of surgery and so no). Day of Surgery Perioperative Follow Up 12 month Follow Up* 24 month Follow Up* DOS 20 days from DOS 90 days from DOS Perioperative Follow Up Data Window is any data that is taken between 20 days from day of surgery and 90 days from day of surgery 3 months from DOS 12 Month Follow Up Data Window is any data that is taken between 3 months from day of surgery and 15 months from day of surgery 15 months from DOS 24 Month Follow Up Data Window is any data that is taken between 15 months from day of surgery and 27 months from day of surgery 27 months from DOS Figure 1: Schematic diagram of follow-up for primary patients 5
3.2 Data Validation Three aspects of data quality will be validated, namely: Case ascertainment: This is the number of patients entered into the BSR compared to the number eligible, derived from external data sources (ICD 10 codes). The BSR will also receive information on relevant ICD10 codes from contributing hospitals and eventually from the state government at regular intervals to ensure complete data capture of all bariatric procedures. The collection of complete information is critical to the long term success of the Registry. The BSR aims to capture more than 95% of cases. Experience from other Registries, including the National Joint Replacement Registry 3 and the Canadian Stroke Registry 4, clearly demonstrate that to be considered accurate and to avoid bias, at least 95% of case should be captured by a quality Registry. Data completeness: this refers to the completeness of the data submitted by surgeons for each patient. Internal validation will occur in the database whereby mandatory data will need to be provided for the record to be complete. These fields include: Patient Surname Patient First Name Date of Birth Indigenous status Name of surgeon State Date of operation Procedure type Height Weight at surgery and follow-up Diabetes status at surgery and follow-ups Date of revision Planned or Unplanned operation Mortality Defined Adverse event Unplanned return to theatre Unplanned ICU admission Unplanned re-admission to hospital Complete data is required for accurate analysis and reporting. Without complete data, indicator values for units may be unrepresentative of actual practice. Data accuracy: this will be tested using consistency and range checks, as well as external validation through regular reviews of ICD10 discharge codes for bariatric surgery against public hospital data, Births Deaths and Marriages cross checks and ICU admission registry cross checks. It may also involve other methods of validation such as peer review. BSR data manager will internally check the data for obvious discrepancies such as duplicate data entries or an age or BMI in excess of human limits. After such 6
internal checks, if inconsistency persists, the data will be sent back to the individual surgeon so that they can then check again whether or not the data are correct, and a time limit is given within which to respond. After this process of validation the data may be corrected, and the individual surgeon will be written a letter confirming this. It should be kept in mind that while the BSR has a duty to report on the data it holds, the BSR is not responsible for the accuracy and completeness of the data submitted. This responsibility rests with the clinical teams providing the service to patients. Issues with clinical audit data (either case ascertainment or data quality) must be addressed by the surgeon, device manufacturer or hospital concerned. The role of the BSR is to provide consistent analysis and case mix adjustment of data received from participating surgeons and hospitals and to make reports on the process and outcome of care publically available. 3.3 Data Analysis 3.3.1 Process of data analysis De-identified data will be provided to an independent biostatistician employed by the Monash University, Department of Epidemiology and Preventive Medicine by secure file transfer for review on a 6 monthly basis. The results will be provided to the Project Manager. Only Monash University registry staff, who are non-clinical will have access to identifiable data. All data to be used in reports or publications are to be cleaned and verified prior to analyses. Routine, re-analysis of data will be undertaken by the Project Manager of the BSR to ensure findings are reproducible. 3.3.2 Case-mix (risk) adjustment Operative outcomes for each individual surgeon need to be interpreted with great care as the case-mix of surgical practice can vary. For example some surgeons may only take on patients needing more difficult, revisional surgery, and others may operate predominantly on higher risk patients having primary surgery. As a result, the comparison of outcomes must carefully consider the patient characteristics that are beyond the control of the surgeons. This typically will involve taking account of a patient s age, sex, body mass index (BMI) and the existence of any other co-morbidity. Other confounders that should be considered are the socioeconomic status (SES), demographic location and complexity of the surgery. In cases where clear exceptions exist, as identified by validated biostatistical means, individuals may be excluded from sample data to ensure results are not misleading. Documentation of decisions to remove cases from samples for data analysis purposes is to be maintained by the Data Manager. The number and percent of such observations will be reported to the steering committee. The BSR will report on details of the risk-adjustment model and its performance characteristics. Where possible, the risk-adjustment process will be based on published, 7
validated data. In the absence of such data, the person in the BSR responsible for generating reports (currently the Post Doctoral Fellow) will use the national data to develop appropriate risk-adjustment methods for each procedure. 3.4 Outliers 3.4.1 Methods of finding potential outlier Over recent years the use of the funnel plot to display non-normal values or outliers graphically has been advocated as the standard method 5. A funnel plot comprises the plotting of an outcomes summary statistic from each individual institution against a specified target, together with upper and lower control limits. Usually two sets of control limits are displayed: typically 95% (2 SD) alarm limits and 99.8% (3 SD) action limits 2, 6. Data points falling outside the control limits are seen as potential outliers and it is recommended that possible causes for this are investigated. 3.4.2 Stratified response strategy The steering committee will work on an early warning model with all stakeholders including surgeons, device manufacturers and hospitals. The Registry will provide information that should minimise patient harm should a device prove to be deficient. The Registry could also provide a resource to assist with patient contact should a recall be necessary. No monitoring system can reliably separate true inliers from the true outliers. This is because of the complexity and variations inherent in patients and healthcare services and the inaccuracy of monitoring methodologies. Therefore a three-level graded response system is proposed with escalation to the next stage if an alert has occurred during the previous stage. The following table indicates the three stages that will be followed in managing a potential outlier, the actions that need to be taken, the people involved and the maximum time scales. It aims to be feasible and fair to providers identified as potential outliers and sufficiently rapid so as not to unduly delay the publication of comparative information. Definition Level 1 alert Level 2 alert Level 3 alert Two standard deviations below the mean; OR Statistically significant deterioration in outcomes between reporting period (annual reports) Three standard deviations below the mean; OR Two reporting period at two standard deviations below the mean, OR if a patient dies during or as a consequence of the bariatric surgical procedure. Two reporting periods at three standard deviations below the mean OR continued performance at two standard deviations below the mean despite corrective measures 8
Action by BSR Level 1 alert Level 2 alert Level 3 alert Surgeon, device or hospital flagged as level 1 alert will not be subject to the review process. This is because this size of difference from the national average may occur simply from random variation alone. Data will be checked for major errors e.g. validate against hospital records and devices, ensure data entry are correct. Data checked for accuracy, major shift (case-mix) and other potential confounders. Assess whether there are case-mix factors peculiar to this situation that may explain the observed variations. Check with the Surgeon/hospital whether the submitted data is correct. If not request correct data. Expected Outcome No case to answer: Submitted data in BSR revised, updated results show provider is not an outlier Case to answer: data in BSR records revisited, reanalysis shows potential outlier status persists Chair of steering committee convenes investigation committee. New data checked and old data rechecked for accuracy, major shift (case-mix) and other potential confounders. No case to answer: Submitted data in BSR revised, updated results show provider is not an outlier Case to answer: Appropriate pathway decided by Investigation Committee including reporting to appropriate body 9
Reporting Level 1 alert Level 2 alert Level 3 alert To support regular local review of data submissions and clinical practice, the BSR will notify surgeons of their alert status. The surgeon and the hospital where he is practicing should be notified of the finding Support will be offered to surgeons. If a device is raises a level 2 alert, the device manufacturer should be notified of the finding. If performance is persistently at the level 3 stage and the Investigation Committee is satisfied with the validity of the data, reporting to the appropriate regulatory body by the Chair of the Steering Committee will be mandated. For example A device will be notified to the Therapeutic Goods Authority (TGA) A surgeon will be reported to the Royal Australasian College of Surgeons (RACS) A hospital will be reported to The Department of Health or the regulator in each state 10
3.5 Confidentiality If a situation arises where an investigation is required, all individuals will be subject to binding confidentiality obligations before they are allowed to access data which identifies particular people or organisations for the purpose of the investigation. Whilst data will be analysed in a deidentified manner, BSR may require using or disclosing particular information about an individual, procedure or surgeon for investigation or reporting purposes as per section 3.4. BSR believes that the use or disclosure is necessary to lessen or prevent serious consequences related to bariatric surgery and to ensure the safety and quality of the procedure. References 1. de Steiger RN, Miller LN, Davidson DC, Ryan P, Graves SE. Joint registry approach for identification of outlier prostheses. Acta Orthop 2013; 84(4): 348-52. 2. Seaton S, Manktelow B. The probability of being identified as an outlier with commonly used funnel plot control limits for the standardised mortality ratio. BMC Med Res Methodol 2012; 12(1): 98. 3. Australian orthopaedic association national joint replacement registry. Annual report. Adelaide:Aoa; 2011. 4. Tu JV, Willison DJ, Silver FL, et al. Impracticability of informed consent in the Registry of the Canadian Stroke Network. N Engl J Med 2004; 350(14): 1414-21. 5. Seaton SE, Barker L, Lingsma HF, Steyerberg EW, Manktelow BN. What is the probability of detecting poorly performing hospitals using funnel plots? BMJ Qual Saf 2013; 22(10): 870-6. 6. Spiegelhalter D. Funnel plots for comparing institutional performance. Stat Med 2005; 24(8): 1185-202. 11