178 March 2004 Family Medicine Assessing Resident Competency in an Outpatient Setting Andrea L. Wendling, MD Background and Objectives: The Grand Rapids Family Practice Residency Program has been using an ongoing spot check method to evaluate global competencies of family medicine residents in the ambulatory setting. This paper describes and assesses this evaluation method. Methods: During each clinic session, preceptors evaluate residents on core ambulatory clinical skills. Evaluations emphasize direct observation of behaviors. Summative feedback is provided on a quarterly basis. Both residents and faculty were surveyed regarding impressions of this system. Results: Of the residents surveyed, 88% felt this feedback regarding performance was useful, and 75% were comfortable with the process. Sixty percent felt the assessment was an adequate representation of outpatient skills. Eighty-eight percent of residents wanted to continue the evaluation process. After institution of the program, precepting faculty reported an improvement in their assessment of overall resident performance in an outpatient setting. Faculty also self-reported an increase in direct observation of resident encounters. Conclusions: Assessing global competencies in an outpatient setting is a way for family medicine programs to evaluate residents, provide feedback on an ongoing basis, and monitor growth in key areas. This method was well accepted at our program and has been shown to change self-reported preceptor behaviors regarding direct observation of residents. (Fam Med 2004;36(3):178-84.) From the Department of Family Medicine, Michigan State University. Dr Wendling is currently in private practice at Charlevoix Area Hospital, Boyne City, Mich. Demonstrating resident competency is a challenge of all family medicine residency programs. The Accreditation Council for Gradua te Medica l Education (ACGME) requirements for competency 1 include several areas that can be demonstrated in an ambulatory care situation, and, for several reasons, family medicine residency programs are well designed to demonstrate their competencies using observation of ambulatory encounters. First, all family medicine residency programs include longitudinal ambulatory care experiences with dedicated faculty preceptors, providing an ideal setting for observation. In addition, residents are required to spend increasing amounts of time in the ambulatory setting throughout their residency careers. This allows for the demonstration of progressive improvements in resident performance over time, a requirement of the ACGME Outcomes Project. 1 Third, it has been shown that when evaluation and observation are increased, feedback increases and is more useful. 2 Since the ambulatory patient care setting is the cornerstone of family medicine, feedback and teaching centered in this setting can be undertaken by all programs. The Grand Rapids Family Practice Residency has been using an ongoing spot check method of evaluating resident progress in ambulatory patient care skills for the past 2 years. This article describes the ambulatory care competency assessment tool, provides the results of a resident survey regarding usefulness and comfort with this method, and reviews faculty surveys demonstrating self-reported changes in preceptor behavior after institution of this evaluation system. Methods Setting The Peter M. Wege Health and Learning Center is an inner-city-based continuity clinic staffed by 30 family medicine residents and five practicing faculty physicians. Each year, there are approximately 26,000 patient visits to residents. A faculty preceptor is assigned to one of two residents patient care areas during each half-day clinic session. These dedicated preceptors are responsible for overseeing resident patient care, teaching, and resident evaluation. Twenty-four preceptors provide this service on a rotating basis. Twelve preceptors are faculty members, and 12 are family physicians from the community. Of the 12 faculty members, five are women. Eight of the faculty members have had more than 5 years of teaching experience; four have had less than 5 years. All faculty members precept in the resident clinic 1 or 2 half days every week; community family physicians precept 1 to 4 half days each month. Physicians are not involved in direct patient care while precepting.
Special Articles Assessing Competence Vol. 36, No. 3 179 Defining Core Ambulatory Skills and Development of Evaluation Forms The described ambulatory evaluation system was developed by the author with input solicited from 12 faculty members and two resident representatives during faculty meetings. Discussions regarding the evaluation system occurred monthly during the development process and quarterly during the first year of use. We began the development process by defining measurable ambulatory skills, a process that included three steps. The first step was the determination of global skills that were central to all ambulatory patient encounters. These skills included history taking and physical exam skills, assessment and plan development, presentation skills, patient education skills, and time management. The next step was to divide each skill into key components, beginning with a very basic level of competency and advancing to mastery of the skill. Each component needed to be both distinct and measurable. When taken together, these components needed to reflect the progression to proficiency in each area. The faculty and resident representatives then assigned each skill a basic or advanced level of training. Basic skills were defined as those related to information gathering; these skills are evaluated during the intern year. Advanced skills involve interpretation and processing of information; these are evaluated during residents second and third years. Using this approach, two separate evaluation forms were developed, one for the intern level and one for the second- and third-year levels of training. See Appendix A for sample evaluation forms. Evaluative Process During each half-day office session, a precepting faculty physician is expected to complete one evaluation form for each resident involved in patient care. The information recorded is based only on encounters observed during the specific clinic session. Each question is meant to be answered based on a single observed encounter chosen randomly by the precepting physician, rather than a summation of multiple encounters. Multiple questions may be completed using the same encounter, or, if necessary, a different encounter observed during the same session may be used to complete each question. If specific evaluative categories are not directly observed, these areas are left blank. Evaluation forms are collected, and the preceptor s numerical ratings are recorded. Written comments, if included, are transcribed for each resident. After the information is transferred, individual feedback forms are destroyed. After the system had been in place for 15 months, data regarding completion of forms were recorded for one quarter. During the quarter, 252 evaluation forms were completed, representing 73.5% of scheduled resident clinic half days. On average, third-year residents each received 12 evaluations (range 4 19), second-year residents each received nine evaluations (range 5 14), and interns received five evaluations (range 2 11). Each precepting faculty member completed an average of 23 forms during the quarter; this ranged from 3 to 47 forms. Feedback At the end of every quarter, a numerical summary of preceptor ratings is provided to each resident for evaluated areas. Summary percentages are provided for previous quarters, and a quarterly class average percentage is listed so residents can compare progress with peers. Numerical preceptor ratings are presented anonymously; written comments from precepting faculty are listed with the dates of specific encounters and with the evaluating faculty member s name. This combination of anonymous and non-anonymous feedback was chosen to provide a way to reduce reporting bias, yet still provide residents specific non-anonymous comments to foster discussion with faculty. Preceptors are also encouraged to discuss ratings and comments with residents prior to the end of each clinic session. The summary report is placed in the resident s folder and is reviewed by the individual resident and faculty advisor during regularly scheduled resident-faculty review meetings. See Appendix B for a sample feedback report. Instrument To evaluate the program, a survey instrument was distributed to 13 faculty members. Community physicians who provided less-regularly scheduled precepting services were not surveyed. Faculty surveyed included 11 family physicians, one geriatrician-family physician, and one behavioralist. The first administration of the survey was completed prior to beginning the ambulatory evaluation process and the second after the process had been in place for 18 months. Specific questions on the survey instrument are shown in Table 1. All responses were expressed on a 5-point numerical scale with 1 representing agree with statement and 5 representing disagree with statement. For clarity in reporting, responses of 1 or 2 were then grouped to represent agree with statement and responses of 4 or 5 were grouped representing disagree with statement. A similar instrument was distributed to 18 secondand third-year residents after the evaluation process had been in place for 12 months. Survey responses were expressed using several scales, including a 4-point scale, a 3-point scale, and simple yes/no responses. All surveys were created by the author, and survey responses were recorded anonymously. Data Analysis Analyses were performed using SPSS for MS Windows version 6.0. The average numerical response from the faculty pre-project survey was compared to
180 March 2004 Family Medicine Table 1 Faculty Survey Questions and Results P values represent comparison between average numerical ratings before and after project. Values shown with an asterisk (*) are statistically significant with a P value <.05.
Special Articles Assessing Competence Vol. 36, No. 3 181 that of the post-project survey using analysis of variance (ANOVA). The resident survey responses when grouped by academic year and gender were compared using the paired t test (Levene s Test for Equality of Variances). Differences were considered significant at P<.05. Results Faculty Survey Of the 13 faculty members surveyed prior to beginning the ambulatory assessment process, 11 returned surveys, for a response rate of 82%. The 13 faculty members were again surveyed after the process had been in place for 18 months; 13 surveys were returned, for a response rate of 100%. The initial and subsequent faculty survey results are shown in Table 1. Preceptors were first surveyed regarding their assessment ability in several categories. Almost all preceptors felt comfortable assessing residents presentation skills on both initial and subsequent surveys. On initial survey, only a small percentage of preceptors felt they were able to assess well the history taking and physical exam skills of residents. On the post-evaluation survey, there was no significant change in this category. Preceptors did report a statistically significant improvement in their ability to assess residents schedule management. Preceptors also showed statistically significant improvements in their self-reported ability to communicate impressions of resident behavior and their perceptions of the tool used to evaluate performance. The most significant changes occurred in the area of self-reported preceptor behavior regarding resident observation. Although video-assisted observation and direct observation of residents was available and encouraged prior to the evaluation process, no preceptors reported regularly using these methods on the initial survey. On the post-evaluation survey, however, 46% of preceptors agreed that they often entered patients rooms or used video to listen to residents obtain a history. Fifty percent of preceptors agreed with the statement that they often watched videotapes of resident performance (compared with 18% on the initial survey), and 42% of preceptors agreed that they had often observed residents providing patient education (compared with 18% on the initial survey). Significantly more faculty reported post evaluation that they were often repeating residents examinations to confirm physical exam findings (P<.05). Despite changes in the faculty s reported assessment skills and behaviors, faculty impressions of resident performance did not change. Although faculty scored residents higher on several behaviors such as timely arrival to clinic, presentation skills, and ability to formulate specific questions for preceptors, none of the changes were statistically significant. Resident Survey Of the 11 second-year residents and seven third-year residents surveyed after the evaluation process had been in place for 12 months, 16 returned surveys. This was a response rate of 89%. Eighty-eight percent of residents felt the information provided regarding outpatient skills was useful. Seventy-three percent of residents felt their individual percentage mastered scores were useful, and 75% felt the comparison with class averages was useful. Seventy-five percent of residents felt comfortable knowing preceptors were evaluating performance. Ninety-four percent of residents were comfortable discussing performance feedback with their faculty advisors. Only 60% of residents, however, felt the evaluation feedback was an adequate representation of their outpatient skills. Most of the written comments that were included with returned surveys addressed this issue. Residents mainly felt their recorded performance regarding time management did not accurately represent their skills due to lack of control over schedules, patient cancellation and no-show rates, and late patient arrival times. Eighty-one percent of residents felt that the current amount of feedback, using the evaluation system, was appropriate. Thirteen percent would have liked more feedback; 6% would have liked less feedback. When surveyed, 88% of residents responded that they would like to continue the evaluation process. Written comments from those residents who did not wish to continue the process reflected discomfort knowing preceptors were also evaluating during instructional clinic time and a feeling of receiving too much feedback with the quarterly reporting system. There was no statistically significant difference when resident responses were compared by either program year or gender. Discussion The results of this study indicate that following implementation of an outpatient resident evaluation system, faculty reported positive changes in their ability to accurately assess residents time management, felt themselves in possession of a tool to communicate impressions of residents performance, and changed their behavior by increasing direct observation of resident-patient encounters. Most residents found that the system provided them with an appropriate amount of useful feedback. Many sources have shown that clinical teaching in residency and medical school can be enhanced by directly observing learners interactions with patients. 3-5 Unfortunately, most clinical evaluations are still based on end-of-rotation summary recollections of presentations, notes, and discussions, rather than on contemporaneously observed assessments of clinical abilities. 3,6 Ongoing, structured, point-of-care evaluation systems
182 March 2004 Family Medicine such as the system described in this article offer an opportunity to assess and document resident clinical skills using individual patient encounters throughout the training period. A formal evaluation system for family medicine residency ambulatory settings is important for several reasons. First, as residents progress through family medicine programs, an increasing percentage of patient care time is spent in the outpatient rather than the inpatient setting. Formal evaluation processes are frequently used during inpatient rotations, but similar systems are not always used for outpatient continuity clinics despite the importance of outpatient care to the practice of family medicine. Second, family medicine patient care clinics provide a setting in which residents and faculty members have dedicated time to interact on a patient-by-patient basis. All intern patient visits and many upper-level visits are precepted immediately, affording multiple opportunities for daily evaluation and feedback. As described by Bell et al in 1997, multiple resident and faculty encounters such as these are the cornerstone of any formative evaluation process. 5 A family medicine resident continuity clinic setting is an ideal site for focused evaluation and continued educational growth. This venue for competency evaluation already exists at virtually all family medicine residency programs. By using the described system to formalize the evaluative process, we were able to increase the amount of recorded feedback without increasing the number of precepting physicians. Our current preceptors use direct video to view portions of resident-patient interactions whenever possible during regularly scheduled precepting time. When these interactions are observed, those portions of the forms requiring direct observation are completed. Other evaluative categories, not requiring direct observation, can be completed following a normal resident patient presentation and preceptor interaction. The majority of expense involved with the evaluation system at our program is administrative. An administrative secretary spends approximately 1 hour per week entering data and faculty comments. Each quarter, one faculty member spends approximately 4 6 hours summarizing and printing data sheets for all residents. Residents review data sheets individually with faculty advisors during regularly scheduled educational review meetings; these meetings had occurred prior to beginning the outpatient evaluation system and did not represent additional expense. Finally, the ambulatory patient care experience is the cornerstone of family medicine. By directly observing multiple patient encounters, preceptors can increase the amount of spontaneous teaching and feedback. The literature has shown that increased feedback can improve learners performance. 6,7 Through experiential patient tips, access to reference sources, and discussions surrounding difficult issues, even those residents performing at a high level can benefit if encounters are observed. We have shown that after instituting this system, faculty members at our program felt they were better able to communicate their feedback regarding resident performance, and the majority of residents felt this feedback regarding outpatient skills was useful. Limitations This study has several limitations. First, residents and faculty of a single program were surveyed, and the small numbers limited the statistical power of the findings. Second, assessment of changing preceptor behavior was based on preceptor impressions rather than on direct observation, a significant limitation. Third, although preceptors felt they were better at assessing resident performance, their general impressions of such performance did not significantly change, raising the question of whether direct observation in this setting is necessary to formulate an impression of resident behavior. Fourth, preceptor assessment of resident performance was not compared with any other measurement tool, so results of evaluation techniques cannot be validated. Finally, although surveys were anonymous, residents and faculty knew faculty members were collecting responses, which may have biased data. Summary ACGME has challenged teaching programs to design better methods to assess and evaluate resident performance and document resident competency in core areas. The evaluation method described in this report outlines one approach to documenting competencies in patient care, medical knowledge, interpersonal and communication skills, and components of professionalism in the ambulatory care setting. Our results indicate that this method was well accepted by residents and, based on self-reported data, has improved preceptor behaviors regarding direct observation of resident performance. Correspondence: Address correspondence to Dr Wendling, 11133 Summerhill Way, Charlevoix, MI 49720. awendling@pol.net. REFERENCES 1. ACGME. Competencies and outcomes assessment. Minimum program requirements language. Approved by ACGME September 28, 1999. www.acgme.org/outcome/comp/compmin.asp. Accessed April 3, 2003. 2. Ende J. Feedback in clinical medical education. JAMA 1983;250:777-81. 3. Kassebaum DG, Eaglen RH. Shortcomings in the evaluation of students clinical skills and behaviors in medical school. Acad Med 1999;74:842-9. 4. Quattlebaum TG. Techniques for evaluating residents and residency programs. Pediatrics 1996;98(6):1277-83. 5. Bell HS, Kozakowski SM, Winter RO. Competency-based education in family practice. Fam Med 1997;29(10):701-4. 6. Bowen JL, Irby DM. Assessing quality and costs of education in the ambulatory setting: a review of the literature. Acad Med 2002;77:621-80. 7. Holmboe E, Scranton R, Sumption K, Hawkins R. Effect of medical record audit and feedback on residents compliance with preventive health care guidelines. Acad Med 1998;73:901-3.
Special Articles Assessing Competence Vol. 36, No. 3 183 Appendix A Sample Evaluation Forms
184 March 2004 Family Medicine Appendix B Sample Feedback Form, Including Summary Data Sheet and Written Faculty Comments