Quality of Electronic Pathology (E-path) Records: A Function of Time, X Factors and One Constant Jovanka Harrison, Ph.D. New York State Cancer Registry North American Association of Central Cancer Registries Annual Conference, Albuquerque, New Mexico, June 16-23, 2017
July 13, 2017 2 Objective 1 Utilize a previously developed data quality assessment tool, the Quality Report Card, to evaluate change in data quality of pathology records received in the NAACCR Standards Volume V format, submitted by five hospital-based laboratories* to the NYSDOH Electronic Clinical Laboratory Reporting System (ECLRS). * Receiving e-path services from the same vendor. The services included daily electronic transmission of HL7 messages from the reporting facility to ECLRS.
July 13, 2017 3 Objective 2 Utilize the same assessment tool, covering the same time period, to evaluate change in data quality of pathology records received in the NAACCR Standards Volume V format, submitted by five independent laboratories.
July 13, 2017 4 Background ECLRS is a NYSDOH developed Web-based reporting system, within the Health Commerce System (HCS); a secure infrastructure for Mandatory Reportable Conditions (e.g., HIV/AIDS, Communicable Diseases, Lead poisoning and Cancer). Supported (Cancer) formats: HL-7 (v. 2.x), NAACCR ASCII, and Web data entry (HTML).
July 13, 2017 5 Background Continued The NYSCR Cancer-ECLRS Initiative started in 2001. Its mission: to recruit labs for e-path reporting and thereby improve compliance and timeliness. High volume reporters (>1,000 reportable cancer cases per year) are encouraged to use the NAACCR Standards for Cancer Registries Volume V (HL7 v. 2.x) format.
July 13, 2017 6 Background Cont d Initial recruiting efforts focused on onboarding independent pathology laboratories, with the aim of capturing cancer cases usually diagnosed in a nonhospital setting, such as malig. melanomas of the skin or prostate cancer. In recent years, the scope has been expanded to include hospital based laboratories.
July 13, 2017 7 140,000 120,000 100,000 80,000 60,000 40,000 20,000 At a Glance: Number of E-Path Lab Reports Received by NYSCR Cancer- 0 ECLRS, by Data Format: 2004-2016 Web File Total 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
July 13, 2017 8 Onboarding of Hospital-based Labs Lab B 1/22/2015 The TimeLine Lab D 4/9/2015 Dec 2014 Jan 2015 Mar 2015 Apr 2015 Jun 2015 12/17/2014 Lab A 3/17/2015 Lab C 6/15/2015 Lab E
July 13, 2017 9 Background Cont d An important part of the onboarding of labs is the certification process. The goal is to ensure good quality data from the go live date (date certified for reporting into production). Subsequent monitoring (incl. through the QA Report Card) assures high data quality.
July 13, 2017 10 Certification Process The certification process involves two phases: 1) pre-certification: Is the HL7 message structure correct? (may use dummy data in the first test file/s); 2) certification= validation of content and structure (real data), optionality/usage, and reportability. Specifically, electronic and paper submission of pathology reports to include a sample of sites (e.g., prostate, breast, incl. FISH, ER/PR analyses, colon, malig.mel.of skin, hemes.)
July 13, 2017 11 Dissecting the Path Report Looking for LOINCs (Logical Observation Identifiers Names and Codes) in OBX-3 ( question code ) of the HL7 message (1) Relevant (Clinical) History 22636-5 (2) Site of Origin 22633-2 (3) Gross Description 22634-0 (4) Microscopic Observation 22635-7 (5) Final Diagnosis (narrative) 22637-3 (6) Pathologic Findings (by CAP) 33746-9 (7) Comments (Interpretation) 22638-1 (8) Supplemental Reports 22639-9
July 13, 2017 12 Evaluation of Cancer Lab Reporting: The Quality Report Card The QA Card a tool to monitor and review data quality of specific required and recommended data items in a systematic way. For each reporting lab, and for a selected period, the Card shows the number reports received, and calculates percentage of missing (no value submitted) and invalid values per data item.
July 13, 2017 13 The Quality Report Card Required data items: Patient Demographics (patient name, address, DOB, gender, race, SSN) Tumor-related Information (specimen collection date, relevant/clinical history, nature of specimen/site of origin, final dx) Recommended data items: Ordering Provider/Facility Information (name, address)
July 13, 2017 14 Methods The onboarding of five hospital-based labs within a short period, all using the same vendor ( the constant ) to create their HL7 feed was considered a controlled experiment, which provided an opportunity to evaluate whether data quality for specific data items changes of over time. For each lab, The Card was run at four different time points: 1) 30 days from certification, 90 days, and 18 months later. This process was repeated for the independent* labs. * Timeline starting point, the middle point of onboarding of hospital based labs.
July 13, 2017 15 Select Results: Percentage Missing Clinical History- Five Hosp.-based Labs Reporting to NYSCR: Four Time Points from Go Live % 20 30 days 90 days One Year 18 Months 15 10 5 0 Lab A Lab B Lab C Lab D Lab E
July 13, 2017 16 Select Results: Percentage Missing Clinical History- Five Independent Labs Reporting to NYSCR: Four Time Points from Go Live % 100 30 days 90 days One Year 18 Months 80 60 40 20 0 Lab AA Lab BB Lab CC Lab DD Lab EE
July 13, 2017 17 Select Results: Percentage Missing SSN- Five Hosp.-based Labs Reporting to NYSCR: Four Time Points from Go Live % 80 70 60 50 40 30 20 10 0 30 days 90 days One Year 18 Months Lab A Lab B Lab C Lab D Lab E
July 13, 2017 18 Select Results: Percentage Missing SSN- Five Independent Labs Reporting to NYSCR: Four Time Points from Go Live % 80 70 60 50 40 30 20 10 0 30 days 90 days One Year 18 Months Lab AA Lab BB Lab CC Lab DD Lab EE
July 13, 2017 19 Select Results: Percentage Invalid Values for SSN- Five Hosp.-based Labs Reporting to NYSCR: Four Time Points from Go Live % 35 30 25 20 15 10 5 0 30 days 90 days One Year 18 Months Lab A Lab B Lab C Lab D Lab E
July 13, 2017 20 Select Results: Percentage Invalid Values for Ord. Facility Name- Five Independent Labs Reporting to NYSCR: Four Time Points from Go Live % 40 35 30 25 20 15 10 5 0 30 days 90 days One Year 18 Months Lab AA Lab BB Lab CC Lab DD Lab EE
July 13, 2017 21 Summary of Select Results There is great variability among labs in terms of data quality. Patient demographics, especially address and race/ethnicity is more often provided by hospital-based laboratories versus independent path labs. Reference laboratories (e.g., those that perform genomic tests) tend to have less patient demographics. For example, 3 of 5 independent path labs, had 100% missing patient address. Two of the three are genomic labs and one is a dermatopathology lab. Labs sending local version of the ordering facility name, e.g. Lab B at Union Street 52, will get dinged by the algorithm.
July 13, 2017 22 Lessons Learned Quality Assurance (QA) processes to be re-evaluated and updated, if necessary, at the least on a yearly basis. The number and nature of on-boarding facilities/clients will drive the QA process. The more varied the facilities in terms of data quality (and type of information submitted), the more complex the evaluation. (E.g., algorithms for invalid values.) The more timely the submission of pathology reports (e.g., the path report is received by the central cancer registry within a day of it being entered into the submitting lab LIS), the greater the need for frequent QA.
July 13, 2017 23 Lessons Learned Continued The higher the e-path reporting volume, (even with generally good data quality) the greater the need for resources at the registry to assure data quality and consistency; for example, 2% of 100 path reports missing patient address is trivial, but 2% of 120,000 is 2,400 reports which require a different type of attention and resources. More frequent QA feedback to reporting labs, not just error driven re-education, might facilitate improved buy-in by labs.
July 13, 2017 24 Next Steps Improve the algorithm for handling invalid values Evaluate patterns of variable and/or decreasing data quality among labs Continue collaboration with NYS DOH in developing a comprehensive Timeliness and Compliance Report Card for path labs Conduct targeted QA education of labs with known issues Consider de-certification of non-compliant labs
July 13, 2017 25 Special Thanks To: My colleagues at the NYSCR & NYSDOH: Maria Schymura, Ph.D. (Director, Bureau of Cancer EPI), Amy R. Kahn, M.S., CTR Todd Szwetkowski, CTR Jovan Ormsby, CTR The Analysis and the Output Unit, NYSCR Staff (our invaluable CTRs), The NYS DOH Electronic Clinical Laboratory Reporting System Staff
July 13, 2017 26 Acknowledgement This work was funded in part by CDC s Cooperative Agreement U58/DP003879 awarded to the New York State Department of Health. The contents are solely the responsibility of the New York State Department of Health and do not necessarily represent the official views of the Centers for Disease Control and Prevention.