Improving transparency and reproducibility of evidence from large healthcare databases with specific reporting: a workshop Shirley V Wang PhD, ScM Division of Pharmacoepidemiology and Pharmacoeconomics, Brigham and Women s Hospital Department of Medicine, Harvard Medical School ISPOR Boston, May 2017 1 Disclosures Dr. Wang is: Funded by contracts with the FDA Sentinel Initiative, Agency for Healthcare Research and Quality (AHRQ), and an investigator initiated grant from Novartis Consultant to Aetion, Inc. a software company This presentation reflects the views of the presenter and should not be construed to represent FDA s views or policies 2 1
Joint Task Force ISPOR/ISPE Improving the Confidence of Decision-Makers in Utilizing Real World Evidence Through Increasing Transparency and Reproducibility ISPE ISPOR Transparency What did you actually do? Public reporting of specific details of study implementation What did you intend to do? Pre-registration Public posting of study protocols Facilitates replication, assessment of validity RCT focus on data and code sharing Protects against data dredging, cherry picking Data use agreements Decisions remain hidden 4 2
Reproducibility Objective Catalogue key operational parameters defined when creating a customized study population from a healthcare database to facilitate replication and validity assessment 5 Reproducibility Data Source Methods Direct replication Reproduction of a specific study Same Same Conceptual replication Reproduction of a finding for the exposure (and comparator), outcome and estimand of interest Different Same Different Same Different Different 6 3
Transparency and reproducibility of healthcare database research relies on data provenance: Transparency of steps to clean, pre-process Transparency of operational decisions to define and create a temporally anchored study population Transparency in analysis choices 7 Methods Catalogued key parameters defined when creating a customized study population 2 Step Process: 1. Review sample of software systems¹ designed to support healthcare database research explicit flexible, user specified options that reflect scientific decisions behind creation of study populations 2. Discussion with international database experts to expand and fine tune list of parameters ¹ FDA Sentinel CIDA+PSM, Aetion, OHDSI Atlas + CohortMethod, OMOP RICO 8 4
Important points Fully transparent study scientifically valid study Fully transparent study replication, validity assessment possible We do not recommend specific software tools suggest that studies conducted with such tools are more valid than studies based on de novo code 9 Step 2 Parameters for creation of study population Comprehensive catalogue contains 9 sections: A. Data source B. Design diagram C. Inclusion/exclusion criteria D. Exposure definition E. Follow up definition F. Outcome definition G. Covariates H. Control sampling I. Software 10 5
Step 2. Critical temporal anchors Temporal Anchors Base anchors (calendar time): Data Extraction Date Source Data Range First order anchors (event time): Study Entry Date Second order anchors (event time): Enrollment Window Covariate Assessment Window Follow-Up Window Exposure Assessment Period Event Date Washout Period (Exposure) Washout Period (Outcome) 11 Step 2. Design Diagram 12 6
Step 2 Attrition table Huybrechts, KF; BMJ 2012 13 Dive into the details Feedback on catalogue in Table 2 Synonyms What do you think? What needs to be clarified? Pros and cons of more comprehensive reporting policies Burden on researchers Incentives to be more transparent? 14 7
ISPE led task force paper Catalogued specific parameters corresponding to scientific decisions defining a study population Reporting these parameters will facilitate replicability and assessment of validity Catalogue will grow and change over time Consensus within task force that a limited number of parameters are absolutely necessary to recreate a study population Which? 15 Future efforts to improve database research transparency? What to report? Evaluate prevalence and impact of transparency (or not) of specific operational parameters Inform policies, guidelines on reporting Parameters infrequently reported and influence on replicability could be prioritized in policies, standards, guidelines How to report it? Words used for same concepts vary across groups Develop shared terminology, structured reporting template? 16 8
Description A. Reporting on data source should include: A.1 Data provider A.2 Data extraction date (DED) A.3 Source data range (SDR) A.4 Data sampling Data source name and name of organization that provided data The date (or version number) when data were extracted from the dynamic raw transactional data stream (e.g. date that the data were cut for research use by the vendor). The calendar time range of data used for the analysis. Note that the implemented study may use only a subset of the available data. The search/extraction criteria applied if the source data accessible to the researcher is a subset of the data available from the vendor. Medicaid Analytic Extracts data covering 50 states from the Centers for Medicare and Medicaid Services. The source data for this research study was cut by [data vendor] on January 1st, 2017 and included administrative claims from Jan 1st 2005 to Dec 31st 2015. The Medicare extract included data from Jan 2005- Dec 2015 for enrollees that were 1) included as part of the 5% random sample in 2005 and 2) had a diagnosis for diabetes (ICD9: 250.xx) in 2005. 17 Description A. Reporting on data source should include: A.5 Type of data A.6 Data cleaning The domains of information available in the source data, e.g. administrative, electronic health records, inpatient versus outpatient capture, primary vs secondary care, pharmacy, lab, registry Transformations to the data fields to handle missing, out of range values or logical inconsistencies. This may be a the data source level or the decisions can be made on a project specific basis. The administrative claims data include enrollment information, inpatient and outpatient diagnosis (ICD9/10) and procedure (ICD9/10, CPT, HCPCS) codes as well as outpatient dispensations (NDC codes) for 60 million lives covered by Insurance X. The electronic health records data include diagnosis and procedure codes from billing records, problem list entries, vital signs, prescription and laboratory orders, laboratory results, inpatient medication dispensation, as well as unstructured text found in clinical notes and reports for 100,000 patients with encounters at ABC integrated healthcare system. Global cleaning: The data source was cleaned to exclude all individuals who had more than one gender reported. All dispensing claims that were missing day's supply or had 0 days supply were removed from the source data tables. Project specific cleaning: When calculating duration of exposure for our study population, we ignored dispensation claims that were missing or had 0 days supply. We used the most recently reported birth date if there was more than one birth date 18 reported. 9
Description A. Reporting on data source should include: A.7 Data model conversion A.8 Data linkage, other supplemental data A.9 Duration of observation time Format of the data, including description of decisions used to convert data to fit a Common Data Model (CDM) Data linkage or supplemental data such as chart reviews or survey data not typically available with license for healthcare database. Median person-time observed within data source The source data were converted to fit the Sentinel Common Data Model (CDM) version 5.0. Data conversion decisions can be found on our website (http://ourwebsite). Observations with missing or out of range values were not removed from the CDM tables. We used Surveillance, Epidemiology, and End Results (SEER) data on prostate cancer cases from 1990 through 2013 linked to Medicare and a 5% sample of Medicare enrollees living in the same regions as the identified cases of prostate cancer over the same period of time. The median duration of enrollment in the insurance plan was 2 years. The data source captures all medical encounters within COUNTRY from birth until migration or death. 19 Description B. Reporting on design should include: A figure that contains 1st and 2nd order temporal anchors and See example figures in manuscript. B.1 Design diagram depicts their relation to each other. 20 10
Description C. Reporting on inclusion/exclusion criteria should include: C.1 Study entry date The date when subjects enter the cohort (SED) The time window prior to SED in which an C.2 Enrollment window individual was required to be contributing to the (EW) data source The algorithm for evaluating enrollment prior to C.3 Enrollment gap SED including whether gaps were allowed C.4 Washout for exposure C.5 Washout for outcome C.6 Person or episode level study entry C.7 Sequencing of exclusions The period used to assess whether exposure at the end of the period represents new exposure. The period prior to SED or ED to assess whether an outcome is incident. The type of entry to the cohort. For example, at the individual level (1x entry only) or at the episode level (multiple entries, each time inclusion/exclusion criteria met). The order in which exclusion criteria are applied, specifically whether they are applied before or after the selection of the SED(s). Patients entered the cohort on the date of their first dispensation for Drug X or Drug Y after at least 180 days of continuous enrollment (30 day gaps allowed) without dispensings for either Drug X or Drug Y. Patients were excluded if they had a stroke within 180 days prior to and including the cohort entry date. Cases of stroke were excluded if there was a recorded stroke within 180 days prior. We identified the first SED for each patient. Patients were included if all other inclusion/exclusion criteria were met at the first SED. We identified all SED for each patient. Patients entered the cohort only once, at the first SED where all other inclusion/exclusion criteria were met. We identified all SED for each patient. Patients entered the cohort at every SED where all other inclusion/exclusion criteria were met. 21 Description C. Reporting on inclusion/exclusion criteria should include: The exact drug, diagnosis, procedure or lab codes C.8 Codes used to define inclusion/exclusion criteria. The temporal relation of codes in relation to each other as well as the SED. When defining temporality, be clear whether or not the SED is C.9 Temporality of codes included in assessment windows (, e.g. occurred on the same day, codes for A occurred within 7 days of codes for B during the 30 days prior to and including the SED) C.10 Diagnosis position (if relevant) C.11 Care setting The restrictions on codes to certain positions, e.g. primary vs. secondary diagnoses The restrictions on codes to those identified from certain settings, e.g. inpatient, emergency department, nursing home. Exclude from cohort if ICD-9 codes for deep vein thrombosis (451.1x, 451.2x, 451.81, 451.9x, 453.1x, 453.2x, 453.8x, 453.9x, 453.40, 453.41, 453.42 where x represents presence of a numeric digit 0-9 or no additional digits) were recorded in the primary diagnosis position during an inpatient stay within the 30 days prior to and including the SED. Invalid ICD-9 codes that matched the wildcard criteria were excluded. 22 11
Description D. Reporting on exposure definition should include: D.1 Type of exposure D.2 Exposure risk window (ERW) D.2a Induction period¹ D.2b Stockpiling¹ D.2c Bridging exposure episodes¹ D.2d Exposure extension¹ D.3 Switching/add on D.4 Codes, temporality of codes, diagnosis position, care setting The type of exposure that is captured or measured, e.g. drug versus procedure, new use, incident, prevalent, cumulative, time-varying The ERW is specific to an exposure and the outcome under investigation. For drug exposures, it is equivalent to the time between the minimum and maximum hypothesized induction time following ingestion of the molecule. Days on or following study entry date during which an outcome would not be counted as "exposed time" or "comparator time" The algorithm applied to handle leftover days supply if there are early refills The algorithm applied to handle gaps that are longer than expected if there was perfect adherence (e.g. non-overlapping dispensation + day's supply) The algorithm applied to extend exposure past the days supply for the last observed dispensation in a treatment episode. The algorithm applied to determine whether exposure should continue if another exposure begins. Description in Section B. We evaluated risk of outcome Z following incident exposure to drug X or drug Y. Incident exposure was defined as beginning on the day of the first dispensation for one of these drugs after at least 180 days without dispensations for either (SED). Patients with incident exposure to both drug X and drug Y on the same SED were excluded. The exposure risk window for patients with Drug X and Drug Y began 10 days after incident exposure and continued until 14 days past the last days supply, including refills. If a patient refilled early, the date of the early refill and subsequent refills were adjusted so that the full days supply from the initial dispensation was counted before the days supply from the next dispensation was tallied. Gaps of less than or equal to 14 days in between one dispensation plus days supply and the next dispensation for the same drug were bridged (i.e. the time was counted as continuously exposed). If patients exposed to Drug X were dispensed Drug Y or vice versa, exposure was censored. NDC codes used to define incident exposure to drug X and drug Y can be found in the appendix. 23 D. Reporting on exposure definition should include: D.5 Exposure Assessment Window (EAW) A time window during which the exposure status is assessed. Exposure is defined at the end of the period. If the occurrence of exposure defines cohort entry, e.g. new initiator, then the EAW may be a point in time rather than a period. If EAW is after cohort entry, FW must begin after EAW. We evaluated the effect of treatment intensification vs no intensification following hospitalization on disease progression. Study entry was defined by the discharge date from the hospital. The exposure assessment window started from the day after study entry and continued for 30 days. During this period, we identified whether or not treatment intensified for each patient. Intensification during this 30 day period determined exposure status during follow up. Follow up for disease progression began 31 days following study entry and continued until the firsst censoring criterion was met. 24 12
Description E. Reporting on follow-up time should include: E.1 Follow-up window (FW) The time following cohort entry during which patients are at risk to develop the outcome due to the exposure. FW is based on a biologic exposure risk window defined by minimum and maximum induction times. However, FW also accounts for censoring mechanisms. E.2 Censoring criteria The criteria that censor follow up F. Reporting on outcome definition should include: F.1 Event date - ED The date of an event occurrence. F.2 Codes, temporality of codes, diagnosis Description in Section B. position, care setting Follow up began on the SED and continued until the earliest of discontinuation of study exposure, switching/adding comparator exposure, entry to nursing home, death, or end of study period. We included a biologically plausible induction period, therefore, follow up began 60 days after the SED and continued until the earliest of discontinuation of study exposure, switching/adding comparator exposure, entry to nursing home, death, or end of study period. The ED was defined as the date of first inpatient admission with primary diagnosis 410.x1 after the SED and occurring within the FUP. 25 Description G. Reporting on covariate definitions should include: G.1 Covariate assessment window (CW) G.2 Comorbidity/risk score The time over which patient covariates are assessed. The components and weights used in calculation of a risk score. We assessed covariates during the 180 days prior to but not including the SED. See appendix for example. Note that codes, temporality, diagnosis position and care setting should be specified for each component when applicable. G.3 Healthcare utilization metrics The counts of encounters or orders over a specified time period, sometimes stratified by care setting, or type of encounter/order. We counted the number of generics dispensed for each patient in the CAP. We counted the number of dispensations for each patient in the CAP. We counted the number of outpatient encounters recorded in the CAP. We counted the number of days with outpatient encounters recorded in the CAP. We counted the number of inpatient hospitalizations in the CAP, if admission and discharge dates for different encounters overlapped, these were "rolled up" and counted as 1 hospitalization. Baseline covariates were defined by codes from claims with service dates within 180 days prior to and including the SED. G.4 Codes, temporality of codes, diagnosis position, care setting Description in Section B. Major upper gastrointestinal bleeding was defined as inpatient hospitalization with: At least one of the following ICD-9 diagnoses: 531.0x, 531.2x, 531.4x, 531.6x, 532.0x, 532.2x, 532.4x, 532.6x, 533.0x, 533.2x, 533.4x, 533.6x, 534.0x, 534.2x, 534.4x, 534.6x, 578.0 - OR - An ICD-9 procedure code of: 44.43 - OR - 26 A CPT code 43255 13
Description H. Reporting on control sampling should include: H.1 Sampling strategy H.2 Matching factors H.3 Matching ratio The strategy applied to sample controls for identified cases (patients with ED meeting all inclusion/exclusion criteria). The characteristics used to match controls to cases. The number of controls matched to cases (fixed or variable ratio). We used risk set sampling without replacement to identify controls from our cohort of patients with diagnosed diabetes (inpatient or outpatient ICD-9 diagnoses of 250.xx in any position). Up to 4 controls were matched exactly to each case on length of time since SED (in months), year of birth and gender. I. Reporting on statistical software should include: I.1 Statistical software program used The software package, version, settings, packages or analytic procedures We used: SAS 9.4 PROC LOGISTIC Cran R v3.2.1 survival package Sentinel's Routine Querying System version 2.1.1 CIDA+PSM¹ tool Aetion Platform release 2.1.2 Cohort Safety 27 Our questions 1. Data cleaning/curation 2. Terminology Reproducibility direct/conceptual/robustness Exposure assessment period Bridging exposure/episode extension Exposure risk window vs follow up 3. How would you use this guidance? What steps do you think would be effective to increase the impact and reach? 14
Data cleaning and curation At data source level Extremely curated (OMOP) Minimal curation of raw data (Sentinel) Others At project level Cleaning missing or out of range values and creation of measures specific to project needs Recommendations for transparency? Whose responsibility to report what? Step 1. Source Data Preparation and Pre-processing Data cut (timing) Shifts in PCP participation in CPRD Shifts in administrative services only population Final adjudication Data cleaning How are missing, inconsistent demographics, out of range values handled? Project specific versus global decisions? Data model Common Data Models (CDM) Sentinel, PCORnet, OMOP Different structure and granularity How is raw/clean data transformed to fit CDM? 30 15
Parameters Defining Follow-Up Period Description Exposure start date Minimum induction period (for exposure) Maximum induction period (for exposure) Administrative grace period Episode gap/ Extension Grace period Exposure risk window Follow-up period Comments The point in time when exposure started. The hypothesized minimum delay for first exposure to cause initiation of an outcome event The hypothesized maximum time following last exposure for exposure to cause an outcome event Number of days that are added to the (reported) days of drug supply dispensed due to incomplete adherence which stretches the tablet supply. = Admin grace period + maximum induction time. The grace period is added to each dispensing. Hypothesized biologic time at risk begins after minimum induction and continues for dispensed days supply plus grace period. The analytic follow up for an individual patient begins at minimum induction but may be shorter than the exposure risk window due to patient-specific censoring. * Rothman AJE 1981 31 Follow up versus exposure risk (AT) As treated analysis: grace period may be added to accommodate lapsed days plus a drug-outcome-specific exposure risk window Analytic follow up may be less than hypothesized biologic exposure risk due to censoring Minimum induction time Censored FUP Exposure risk Rx Rx Rx Rx CED = ESD (Exposure start date) = Days supply as dispensed = Grace period : For this patient the FUP is defined by 4 dispensings, a grace period was used 3 times. 32 16
Follow up versus exposure risk (ITT) Once exposure status is defined, it is carried forward until FU ends. Discontinuation of (drug) exposure does not lead to FU end. FU is often a specified length, e.g. 180 days Analytic FU may be greater than hypothesized biologic exposure risk if adherence low Analytic FU may be less than hypothesized biologic exposure risk due to censoring for reasons other than exposure status Minimum induction time FU Exposure risk Rx Rx Rx Rx CED = ESD (Exposure start date) 180 day ITT follow up = Grace period : For this patient the FUP is defined by 4 dispensings, a grace period was used 3 times. = Days supply as dispensed 33 17