Innovations in Primary Care Education was a

Use of Medical Chart Audits in Evaluating Resident Clinical Competence: Lessons Learned from the Development and Refinement of a Study Protocol (Implications for Use in Meeting ACGME Evaluation Requirements) Camille Proden, 1,2 Laura Carravallah 2,3 and D. Kay Taylor 2,4, * 1 Innovations in Primary Care Education, Flint Area Medical Education 2 College of Human Medicine, Michigan State University 3 Combined Medicine/Pediatrics Education, Hurley Medical Center 4 Hurley Medical Center, Department of Research, One Hurley Plaza, Flint, MI 48503, USA Summary Medical chart auditing, one strategy that may be employed to assess resident clinical competence, represents both a tool to evaluate performance, and a vehicle for change. It has been demonstrated that an ongoing chart audit and feedback system can successfully affect resident compliance with practice guidelines and is thus associated with improvements in performance. In the study described here, however, the vehicle for change was an enhanced curriculum, the success of which was subsequently measured via chart audits. Regardless of the primary purpose of such audits, the study investigators gleaned important information about the conduct of the audits that can be applied to current efforts by medical educators to incorporate chart audits into a comprehensive *Correspondence to: D. Kay Taylor, Department of Research, Hurley Medical Center, 8 West, One Hurley Plaza, Flint, MI 48503, USA. E-mail: ktaylor@hurleyme.com plan to meet new Accreditation Council for Graduate Medical Education (ACGME) program evaluation mandates. Copyright 2002 John Wiley & Sons, Ltd. Key words medical chart audits; clinical competence; resident education Background Innovations in Primary Care Education was a 3-year State-funded project designed to improve resident-physician patient care in the ambulatory setting through the development and implementation of an evidence-based curriculum and corresponding practice guidelines. The project involved faculty and residents from two primary care training programs (Family Practice and Combined Internal Medicine/ Pediatrics) located at two community hospitals affiliated with the Michigan State University College of Human Medicine. DOI: 10.1002/qaj.190

176 Camille Proden, Laura Carravallah and D. Kay Taylor Over the period of the project, practice guidelines were developed for four chronic diseases: diabetes mellitus, hypertension, hyperlipidemia and congestive heart failure. In addition, guidelines were created for both adult and pediatric prevention clinic visits. Guideline development was achieved through a committee structure that entailed comprehensive working meetings conducted on a weekly basis over the course of the 3-year project. Recommended practice guidelines were based on best evidence and best practice standards. These were developed and refined through the participation of faculty and residents from the two primary care training programs. Furthermore, final approval was required of faculty from both institutions (obtained at faculty development forums and/or faculty meetings), and the residency quality assurance committees. Feedback from these groups was regularly brought back to the Curriculum and Guidelines Development Committee and recommendations were incorporated into the guidelines. Medical Record Audits The retrospective medical record audit was instituted to evaluate resident performance before and after the new practice guideline implementation, and to assess the health status of the patient population that received resident-physician services for specified chronic diseases and preventive care. In order to determine resident adherence to the guidelines, it was necessary to develop audit tools that corresponded to the new practice guidelines. Staffing A certified medical assistant (CMA) had been previously trained by the Family Practice program to audit patient medical records according to physician-developed protocols for quality indicators and Health Plan Employer Data and Information Set (HEDIS) measures. She had 3 years of experience with good performance and acceptance by the residents, faculty and staff and therefore she was recruited to conduct the project audits for this study. Project staff subsequently trained and added additional auditors to this model to accommodate the volume of work. Refinement of Audit Tool The project began with implementation of the diabetes curriculum and guidelines. A flow sheet was developed and approved, along with the guidelines. This sheet, which was designed to aid the resident in providing appropriate patient care, represented a snapshot of the parameters of the new practice guidelines. Initially, the audit tool was developed to follow the flow sheet and the perceived need to collect a broad base of data for later evaluation. Discussion at committee meetings resulted in the delineation of such parameters as the study population, time frame for study, location for recording of each parameter in the chart, and the definition of each parameter. Meeting minutes acted as reference for the CMA auditors. The auditors completed a retrospective chart review for diabetics for the pre-intervention period and a 3-month review of the same charts post-implementation. There were a total of 262 patient charts in the audit. After follow-up discussion with the auditors it was determined that some key items of the audit tool in the preliminary audit were interpreted by the CMAs in different ways. As a result, it was necessary to void or discard this early audit. A more formalized audit protocol, which was very detailed and specific, was necessary to obtain accurate results. Once the new audit tool was completed, the CMA auditors reviewed the same population of patient s charts with the revised audit tool and protocol. Additionally, a physician audited the charts with the CMA audit team until such time as she was convinced of the accuracy of the audit process. In the development of all subsequent audit tools, a solid protocol was defined. For each item of the audit tool, clear and objective scoring instructions were developed such that only minimal interpretation by the CMA audit team was necessary. The auditors collected a maximum

Use of Medical Chart Audits in Evaluating Resident Clinical Competence 177 amount of data including demographics, medications noted on the progress notes, test results, referrals made, consultant reports returned, copies of the master problem list and medication list, the first progress note and/or history and physical examination results. Committee members, including the CMA auditors, completed a pilot audit of 20 random charts. Each member s audit was compared to a colleague s assessment of the same record. Issues of methodology could then be addressed in the development stage, avoiding later conflicts in methods of data collection. In the process of auditing, the committee determined that appropriateness of medication and laboratory test ordering were important components in the assessment of resident compliance with guidelines. The CMA auditors, however, were not capable of executing this type of subjective clinical judgment. Although the auditors had collected comprehensive descriptive data, the clinicians would need to be responsible to collect interpretive data (including the significance of patient co-morbidities and complications, appropriateness of referrals, laboratory tests ordered as well as appropriate medication ordering). Clinically specific audit tools and protocols were developed and completed by physicians and nurses. This process aided in the development of a collaborative relationship between the clinicians and the CMA auditors. With a careful protocol, the time necessary for clinicians to audit was minimized. Data Entry and Analysis Staff from the hospital s Research Department was responsible for computer data entry and analyses. The project committee worked closely with the department staff to query the database for answers to resident performance questions. This process provided valuable and reliable data regarding resident compliance with accepted practice guidelines as well as the health status of patients receiving care in these primary care residency training programs. Type II diabetes was chosen as the template for the chronic disease audit process for two major reasons. First, the patient population served had a high incidence of diabetes with poor control. Second, due to the pervasive nature of the disease affecting all aspects of the patient, it seemed an ideal choice. For all patients in the study group, information was collected on demographics, comorbidities and complications, process and outcome. Monitoring resident-physician compliance with the new practice guidelines revealed important applications for education. These included appropriateness of care and use of chart audit tools provided (Table 1). Outcome measures (laboratory tests results and other clinical indicators) revealed minimal success in improving patient health status due primarily to the few number of office visits during the limited time of the study. Overall, for the chronic diseases and preventive care patients studied, resident behavior changed minimally pre- and post-implementation of the new curriculum and guidelines. Residents were not given the results of the audits concurrently due to time or scheduling constraints. (Again, however, the intent of the audit was to assess the impact of a new curriculum and not serve as a vehicle for change.) Some improvement was noted in the care of patients with multiple co-morbidities. Additionally, the use of flow sheets by the residents improved their compliance with accepted parameters of care. In general, though, compliance with preventive care guidelines was determined largely to be resident specific. Some resident physicians were consistently good in following guidelines, while the performance of other residents could be described only as being fair (Table 2). Summary and Recommendations Auditor turnover was a challenge to tight control during the initial phase of the project. Eventually this was rectified. The rationale for using CMA auditors was twofold. First, in order to complete the volume of audits required by the scope of the project and remain fiscally viable, it was necessary to employ non-clinical staff. Also, the quality audit process had been previously established

178 Camille Proden, Laura Carravallah and D. Kay Taylor Table 1. Innovations in primary care education diabetes mellitus AUDIT TOOL Sample items Pt Name Med rec # DOB Gender Race Date Date initial Auditor (last, first) diagnosed visit (this facility) Comorbidities (circle): HTN, CAD, CHF, Chol, Thyroid, A-fib, COPD, Asthma, Obesity, Pshyc/Substance abuse, DX, Smoking, Other Complications (circle): Retinopathy, Nephropathy, Neuropathy, Foot Ulcer Visit Pre Post Date Resident ID for visit Flow sheet utilized Y N NIC a Self monitoring Y N Y N Smoker Y N ND b Y N ND b Smoking Education Y N NA Y N NA Diet Y N Y N Exercise plan Y N Y N Feet Y N Y N Blood pressure Weight Labs ordered HgbA1C Y N Y N Microalbumin Y N Y N LDL Y N Y N Referral diabetic Ed Y N Y N Referral ophthalmol Y N Y N a Not in chart. b Not documented. and credible at one of the residency training program sites and was therefore functional. Collaboration of project staff resulted in finetuning of the skills that each person brought to the effort. Data entry by staff with no medical background was successful due to well-defined protocols. They did not require knowledge of medicine or medical terminology. Likewise, the clinicians and CMA auditors were not required to be computer proficient. The group members offset each other s deficiencies. On the other hand, co-operative collaboration was a slow process for a variety of reasons. The residency cultures at the two teaching hospitals were dissimilar in multiple ways. The philosophy of care provided by differing hospital cultures, types of residency programs (Family Practice and Combined Medicine/ Pediatrics, allopathic and osteopathic), and multiple medical and non-medical disciplines provided challenges to the progress of new guideline development and implementation as well as the acceptability of the audit process. This, along with a history of non-cooperation between these residency training programs (who competed for interns and faculty) slowed collaborative efforts at the outset. Ultimately, trust was established but only after a year and a half of work. To ensure confidence in the accuracy of the audit, physician oversight is desirable. All clinical judgments must be made by physicians and/or nurses competent to make such determinations. When the residency program directors provided

Use of Medical Chart Audits in Evaluating Resident Clinical Competence 179 Table 2. Corresponding protocol for chart auditors Resident ID for visit Flow sheet utilized Self monitoring Smoker Smoking education Diet Exercise plan Feet Blood pressure Weight Labs ordered HgbA1C Microalbumin LDL Referral diabetic education Referral ophthalmology Name and number of the resident who signed the progress note. Is documentation on the flow sheet for same date? Make a copy if utilized at all. In the progress note or the flow sheet did the resident document review of self monitoring for blood sugar? Is patient a smoker as documented in the progress note or flow sheet (either resident or vital signs documentation)? Was the patient advised to quit? (N/A if non-smoker) In the progress note or flow sheet did the resident address diet with the patient (any mention of it)? In the progress note or flow sheet did the resident address exercise with the patient (any mention of it)? In the progress note or flow sheet did the resident document that they looked at the patient's feet? (Need any words such as L foot, R foot, mention of lesions, mention of sensory testing) Record value (lowest one of visit). Record value. HgbA1C at this visit? microalbumin at this visit? LDL at this visit? referral to diabetic education at this visit? referral to ophthalmologist at this visit? feedback to their residents, credibility of the audit findings was established through this method. This could occasionally be seen as a challenge when the feedback provided did not meet perceptions that the faculty and residents had regarding their compliance with guidelines. The actual scope of the project was much greater than planned and the project investigative team took on much more than is advisable in any one study endeavor. The team had taken a position that broad data collection would eventually lead to answers not only about resident compliance with practice guidelines, but would also provide needed insight into the health status and care of a high-risk patient population. While there was some validity to this approach, it also resulted in a collection of information that was not particularly relevant for resident feedback nor likely to have an impact on their training. It is therefore recommended that data collection be limited to select variables representing key resident performance indicators. The impact that the audit process has had on training in the ambulatory setting is the provision of a place to start. The quality improvement/ management process has begun and the next step is for ongoing evaluations to determine if the care provided by primary care residents is meeting accepted standards of practice. Additionally, the information gathered through the audit process can be used to evaluate readiness to practice in an ambulatory setting without supervision. Eventually, if expanded to additional ambulatory training situations, a measure of readiness for graduation into private practice could be obtained through this process.

180 Camille Proden, Laura Carravallah and D. Kay Taylor Implications for Use in Meeting Accreditation Council for Graduate Medical Education (ACGME) Evaluation Requirements As the faculty from these two community teaching hospitals work toward the inclusion of medical record auditing in their overall plan to address ACGME evaluation mandates, the lessons learned during the course of this study were somewhat sobering. First, the auditing process is extremely labor intensive. After identifying the specific clinical competency areas to be evaluated, five key tasks must be undertaken: (1) development of chart audit tools; (2) collection of chart data; (3) computer data entry; (4) analysis of data; and (5) resident and faculty feedback, with subsequent tracking of this process and outcome data. Each of the above tasks necessitates additional steps and thus additional work. For example, the collection of chart data entails the recruiting, training, and monitoring of chart auditors. Second, the successful execution of medical chart auditing requires the active involvement of multiple disciplines. The initiative described here included faculty and resident physicians, nurses, chart auditors, data entry persons, and research faculty with an understanding of statistics and evaluation methodology. Third, physician participation and oversight are needed from the very beginning when the chart audit tools are being developed (and on to the actual chart abstraction) to the feedback and tracking of data. The substantial faculty time commitment required to conduct such audits has been cited previously as a disadvantage for using medical record audits to assess resident competence [1,2]. Yet the ACGME has identified records reviews in its Suggested Best Methods for Evaluation for five resident required skills and the ACGME has cited records reviews as the most desirable method for evaluating two of these key skills: preventive health services (=a patient care competency) and resident use of evidence from scientific studies (=a practice-based learning and improvement competency). Our experience in developing and refining a study protocol, which included medical chart audits, has heightened our awareness of the difficulties or obstacles, and increased our appreciation of scientific rigor in producing useful and important evaluation data for resident education. In mandating program evaluation changes, the ACGME has identified political need for better measures of quality as one of the primary reasons for adopting a new education assessment model. The Institute of Medicine report that estimated 98 000 Americans die annually as a result of preventable medical errors certainly heightened the demand for greater accountability [3]. Our involvement in this study has convinced us of the importance of incorporating medical chart audits in a resident education evaluation model. The audits provided compelling data on resident adherence to practice guidelines. For example, both the baseline and follow-up audits revealed that less than half of the resident physicians provided counseling or patient education in the areas of smoking cessation, diet/nutrition, and exercise as detailed in our new practice guidelines for diabetic patients. If the ACGME premise of Whatever we measure we tend to improve. holds true, then the use of medical chart audits will certainly be of great benefit to residency training programs. References 1. Holmboe E, Gross R, Hawkins R. Chart review for residents. Acad Med 1996; 71 (11): 1139. 2. Sliwa JA, Kowalske KJ. Assessing resident clinical competence. Am J Phys Med Rehabil 2000; 79 (5): 468 473. 3. Kohl LT, Corrigan JM, Donaldson MS, et al. To Err is Human: Building A Safer Health System. National Academy Press: Washington, DC, 2000.