ACAD EMERG MED December 2002, Vol. 9, No. 12 www.aemj.org 1397 CLINICAL PRACTICE Comparison of Triage Assessments among Pediatric Registered Nurses and Pediatric Emergency Physicians Sylvie Bergeron, MD, Serge Gouin, MDCM, Benoît Bailey, MD, Hema Patel, MD, MSc Abstract Objectives: To compare triage level assignments, using simulated written case scenarios, in a pediatric emergency department (ED) among registered nurses (RNs) and pediatric emergency physicians (PEPs) and to compare the triage level assignments among RNs and PEPs with a consensus criterion standard. Methods: This was a cross-sectional mailed questionnaire survey. The study was conducted at a pediatric tertiary care center with more than 65,000 annual patient visits. Participants were PEPs and RNs working in the ED. Dillman s Total Design Method, with three mailouts, was used for questionnaire construction and implementation. The survey included 55 case scenarios of patients presenting to the ED. Participants were instructed to assign triage level on each case, using the following four-level triage scale: 1 = resuscitation/emergent, 2 = urgent, 3 = less-urgent, and 4 = non-urgent. A priori, all cases were assigned a triage level by consensus agreement of three PEPs, using established triage guidelines from the RNs teaching manual. Kappa statistics (95% CI) and the mean percentage of correct responses ( 1 SD) were calculated. Results: There was a 100% response rate (39 RNs, 24 PEPs). The kappa level of agreement (95% CI) was 0.453 (0.447 to 0.459) among the RNs and was 0.419 (0.409 to 0.429) among the PEPs. The mean percentage of correct responses ( 1 SD) for the RNs was 64.2% ( 8.0%) and for the PEPs was 53.5% ( 8.1%, p < 0.01). There was no significant difference within groups by experience level (<10 vs. 10 years) or by the type of work schedule (day vs. evening vs. overnight) or full-time vs. part-time status. Conclusions: The level of agreement and accuracy of triage assignment was only moderate for both RNs and PEPs. Triage, a crucial step in emergency care, requires improved measurement. Key words: pediatric; emergency medicine; triage; clinical decision rules. ACA- DEMIC EMERGENCY MEDICINE 2002; 9:1397 1401. The now well-known medical decision-making process of triage had its military origins in the 1700s. 1,2 Then, triage was used to provide medical services for those wounded in battle such that the limited medical resources could be optimized. The sickest, but still treatable, soldiers were evaluated first. 2 Today, triage is a part of every busy emergency department (ED), as the struggle to provide efficient service with finite resources continues. 3 Guidelines for triage vary from center to center, but the components of the ideal system are constant and reflect the delicate balance of prioritizing patients complaints based upon brief, selected information. From the Division of Emergency Medicine, Department of Pediatrics, Hôpital Sainte-Justine, Université de Montréal (SB, SG, BB), and Intensive Ambulatory Care Service, Department of Pediatrics, The Montreal Children s Hospital, McGill University (HP), Montreal, Quebec, Canada. Received March 6, 2002; revision received June 10, 2002; accepted June 21, 2002. Presented in part at the Ambulatory Pediatric Association annual meeting, Baltimore, MD, May 2001, and at the SAEM annual meeting, Atlanta, GA, May 2001. Address for correspondence and reprints: Dr. Sylvie Bergeron, Hôpital Sainte-Justine, Division of Emergency Medicine, 3175 Côte-Sainte-Catherine, Montreal, Quebec, Canada, H3T 1C5. Fax: 514-345-2358; e-mail: liber@videotron.ca. The ideal strategy must safely identify truly ill or injured children. Even when the signs of illness may be subtle, the triage categorization should be sensitive enough to accurately and reliably identify this group. At the same time, an excessively sensitive triage system, which falsely overidentifies sick children, is not useful. This sort of system would unnecessarily quickly overwhelm the assessment capabilities of most pediatric EDs. 1,2,4 Thus, the ideal triage system must also be highly discriminatory. 5 It should be able to accurately categorize children with less acute medical problems who can safely wait for further evaluation or even be diverted to other facilities for less acute care. Finally, as with any robust measurement tool, reproducibility is a clear component of an effective triage system. 2 That is, assessments of triage level should be similar among health care professionals working in the ED setting. In the past, discrepancies in triage level between nurses and physicians have been reported. 2,5 10 These differences may be, in part, due to the time lag between the registered nurses (RNs ) and the pediatric emergency physicians (PEPs ) assessments. 11 In the interim, many children have re-
1398 Bergeron et al. COMPARISON OF TRIAGE ASSESSMENTS ceived antipyretics, analgesics, or hydration, any of which may affect the overall appearance of the child. As well, inter-nurse assessments have also varied significantly. 5,8,10,12 In 1993, the Australian Emergency Medical Committee adopted a single national triage acuity scale. 13 This new five-category triage scale was shown to improve triage assignment. 14 More recently, the Canadian Association of Emergency Physicians has proposed new triage guidelines: the Canadian Pediatric Triage and Acuity Scale. 15 These new guidelines have five levels of triage similar to the Australian model, instead of the previous fourlevel system. The use of a greater number of categories may allow for greater precision in determining who needs the most urgent care, but the additional category may also be cumbersome and confusing. 2 With the imminent implementation of these new triage guidelines in some Canadian EDs, it is important to record, a priori, the degree of agreement in triage level by different health care professionals with the current four-level triage. Thus, it will be possible to assess whether the new guidelines will actually improve the agreement between medical and nursing staff. Reliability and accuracy studies of pediatric triage methods are limited. Most previous studies have been conducted with a small number of participants and involved adult patients. 5,7,10,16 Therefore, we believed it would be of interest to examine the use of the current triage acuity scale in a busy pediatric ED. We hypothesised that there would be wide variability and moderate reproducibility in the way RNs categorized patients level of acuity. The objectives of the study were to compare triage level assignments, using written case scenarios, in a pediatric ED among RNs and PEPs and to compare the triage assignments between simulated patient presentations among the RNs and PEPs with a consensus agreement of assignment of the case scenarios. METHODS Study Design. A mailed study questionnaire of 55 case scenarios was sent to RNs and PEPs. The participants were instructed to assign a triage level for each scenario, using the following scale: 1 = resuscitation/emergent, 2 = urgent, 3 = less urgent, and 4 = non-urgent. This study was approved by the scientific committee and the institutional review board of our hospital. Study Setting and Participants. Our tertiary pediatric referral center has approximately 65,000 annual ED patient visits. All the PEPs working in the ED with a minimum of two shifts (eight hours/ shift) per a four-week period were included. All the RNs working full time and part time in the ED with a minimum of one year of experience in our pediatric ED and at least six months triage experience were included. Survey Content and Administration. The study questionnaire including the 55 case scenarios was distributed to all eligible PEPs and RNs. All participants were asked to rate each patient scenario. In order to maximize response rate, a modified Dillman s Total Design Method was implemented for survey mailouts. 17 This method included three mailouts and standardized questionnaire construction and implementation. All the data collected were entered in a database using Microsoft Excel (Microsoft Inc., Redmond, WA). The 55 case scenarios described patients (0 to 18 years of age) coming to an ED. Each case scenario provided background information of the patient s symptoms and important clinical signs including vital signs (temperature, heart rate, respiratory rate, and blood pressure) and mode of arrival to the ED. The 55 cases chosen were not validated, but were found to represent typical cases of the most frequent complaints for each category of triage. A patient described as resuscitation/emergent required immediate attention; any delay in his or her care would be detrimental. A patient was classified as urgent if his or her condition was associated with a possibility of deterioration within a short time, but without immediate compromise to vital functions. A patient was categorized as less urgent if he or she required care in the ED, but a delay in his or her care was not a critical factor. Finally, a non-urgent patient was one who did not require emergency care. The study cases were similar to those described in the RNs triage teaching manual. This manual, developed ten years ago, explains and illustrates the triage level for different pathologies frequently encountered in the ED. The four-level triage guidelines were introduced at the same time. The guidelines for triage assignment were developed by a group of PEPs and RNs working in the ED. All the RNs were taught with this manual as part of their initial ED training. A priori, a criterion standard triage category was assigned to each case scenario. This proxy criterion standard was developed from the guidelines of the RNs manual by a consensus of three nonparticipating PEPs. The primary outcome was the comparison of the level of agreement among the RNs and the PEPs of the 55 designated case scenarios. Secondary analy-
ACAD EMERG MED December 2002, Vol. 9, No. 12 www.aemj.org 1399 ses included comparisons of agreement between the consensus criterion standard and the RNs and the PEPs. For both RNs and PEPs, results of agreement were also stratified by years of experience (<10 or 10 years). For the RN group, agreement was also examined by the type of shift routinely worked, and for the PEP group, by their status: part-time vs. full-time. Data Analysis. The data were analyzed using SAS software (SAS Institute Inc., Cary, NC). A p-value of less than 0.05 was considered significant. Student s t-test was used for continuous variables. The rates of agreement among the PEPs and among the RNs were calculated using kappa statistics (weighted kappa) (95% CI). The following guidelines for interpretation of kappa statistics were used: less than 0.40, poor to fair; 0.41 to 0.60, moderate; 0.61 to 0.80, substantial; and greater than 0.80, almost perfect. 18 The rates of concordance between the RNs, between the PEPs, and with the established criterion standard were calculated by the mean percentage of correct responses ( 1 SD). RESULTS Of the 63 eligible participants (39 RNs, 24 PEPs), a response rate of 100% was achieved. As shown in Table 1, 28 of 39 of the RNs had more than ten years of experience of working in the ED, similar to the PEPs (17 of 24). Nursing shift assignment was equally spread (day, evening, or night), reflecting the permanent nature of the shift type worked by the RNs. The overall kappa level of agreement (95% CI) in assigning a triage level to the case scenarios was 0.453 (0.447 to 0.459) among the RNs and was 0.419 (0.409 to 0.429) among the PEPs. In Table 2, the levels of agreement of the RNs are stratified by the number of years of experience and the type of shifts. Table 3 shows the levels of agreement of the PEPs stratified by the number of years of experience and full- vs. part-time status. There appears to be no difference between interrater reliability of RNs with regard to experience or shifts worked. However, PEPs had improved reliability with experience and full-time status. The mean number of correct responses compared with the criterion standard for the RNs was 35.3 of 55 (64.2% 8%) and for the PEP was 29.4 of 55 (53.5% 8.1%) (p < 0.01) (Table 4). There was no significant difference by stratifying the RNs and the PEPs by experience level (<10 vs. 10 years) or by the type of shift work (day vs. evening vs. overnight) or full-time vs. part-time status. As per the criterion standard, ten case scenarios TABLE 1. Characteristics of the Participants Registered Nurses (n = 39) Pediatric Emergency Physicians (n = 24) 10 years of experience 28 (72%) 17 (71%) <10 years of experience 11 (28%) 7 (29%) Day shift 15 (39%) Evening shift 14 (36%) Night shift 10 (25%) Full-time 16 (67%) Part-time 8 (33%) TABLE 2. Levels of Agreement among Registered Nurses Kappa 95% CI Overall (n = 39) 0.453 0.447, 0.4459 10 years of experience 0.446 0.438, 0.454 <10 years of experience 0.483 0.459, 0.507 Day shift 0.435 0.419, 0.451 Evening shift 0.501 0.483, 0.519 Night shift 0.445 0.419, 0.471 TABLE 3. Levels of Agreement among Pediatric Emergency Physicians Kappa 95% CI Overall (n = 24) 0.419 0.409, 0.429 10 years of experience 0.545 0.521, 0.569 <10 years of experience 0.396 0.382, 0.410 Full-time 0.517 0.487, 0.547 Part-time 0.373 0.359, 0.387 TABLE 4. Percentage of Correct Responses (Mean 1 SD) of Registered Nurses (RNs) and Pediatric Emergency Physicians (PEPs) versus Criterion Standard RNs PEPs Overall 64 8% 54 8% 10 years of experience 65 8% 53 9% <10 years of experience 61 7% 54 9% Day shift 65 7% Evening shift 66 7% Night shift 61 10% Full-time 51 7% Part-time 58 8% were assigned as resuscitation, 15 as urgent, 20 as less urgent, and ten as non-urgent. Overall, the RNs had a rate of undertriage (i.e., assigning a less severe category than the one attributed by the criterion standard) of 26.7% vs. for 38.8% for the PEPs. Rates of overtriage (i.e., assigning a triage category more critical than the one attributed by the criterion standard) were similar between the RNs and the PEPs: 9.1% vs. 7.7%. Table 5 indicates the direction and magnitude of disagreement between groups.
1400 Bergeron et al. COMPARISON OF TRIAGE ASSESSMENTS TABLE 5. Agreement in Triage Categorization for Registered Nurses (RNs) and Pediatric Emergency Physicians (PEPs) Compared with the Criterion Standard Criterion Standard Non-urgent Less Urgent Urgent Resuscitation RNs Non-urgent 313 (14.6%) 270 (12.6%) 15 (0.7%) 4 (0.2%) Less urgent 73 (3.4%) 425 (19.8%) 89 (4.1%) 13 (0.6%) Urgent 4 (0.2%) 85 (4.0%) 448 (20.9%) 182 (8.5%) Resuscitation 0 (0%) 0 (0%) 33 (1.5%) 191 (8.9%) PEPs Non-urgent 192 (14.6%) 265 (20.1%) 18 (1.4%) 2 (0.2%) Less urgent 47 (3.6%) 195 (14.8%) 130 (9.9%) 24 (1.8%) Urgent 1 (0.1%) 19 (1.4%) 178 (13.5%) 73 (5.5%) Resuscitation 0 (0%) 1 (0.1%) 34 (2.6%) 141 (10.7%) DISCUSSION As hypothesized, we have demonstrated that the level of agreement and accuracy of triage assignment was only moderate for both RNs and PEPs. The low reliability of assigning a similar triage level by using a triage acuity scale was reported in the past. Our results are in keeping with the ones of Brillman et al., 5 who reported an overall kappa level of agreement of 0.452 for the RNs and the physicians in using a four-level-of-triage acuity scale. Similarly, Wuerz et al. had found a kappa level of agreement among the RNs of 0.35. 3 Recently, Gorelick et al. commented that the triage level assignment had a significant impact for admission and resources utilization. 19 It seems that the quest for a reliable tool for triage level categorization continues. Different reasons may explain why we obtained only a moderate level of agreement in triage assignment, among both RNs and PEPs. We did not assess specifically whether some of the triage inconsistencies were attributable to the training provided to the triage nurses, but all participants had at least one year of experience. It is possible that RNs require more formal training in the implementation of the established triage guidelines. Furthermore, they may need more formal supervision in the constant application of the guidelines. We are not surprised that the RNs performed better than the PEPs, since the RNs are trained using the criterion standard and the PEPs are not. Interestingly, the full-time status and the more experience level of some RNs do not seem to improve their performance. It is plausible that the nursing experience is captured in clinical assessments of real patients and the playing field is leveled by giving them the patient s clinical condition in a written vignette. Of interest, we had found a rate of undertriage of 26.7% by the RNs and of 38.8% by the PEPs. Undertriage was observed mostly in the less urgent cases. However these patients could be put at risk if their status deteriorated, especially in a busy ED where the RNs do not routinely reassess less urgent patients. As well, it is thought that the pediatric population might be more vulnerable than the adult population because children often present with only subtle signs of serious illness. As shown in Table 5, we are concerned to find that some cases that were assigned a resuscitation level of triage by the criterion standard were given only a non-urgent or less-urgent level by some RNs and PEPs. However, it occurred in a small proportion. We are unsure whether the clarity of some case scenarios or the fatigue of filling the questionnaire could have been factors. The 100% response rate is a strength of this study. Furthermore, the number of case scenarios is large and represents common patients presentations to the ED with different levels of acuity. To our knowledge, this is the first study that has assessed the agreement of triage levels for an exclusively pediatric population. LIMITATIONS In spite of the small number of participants, testing of all involved PEPs and RNs was attempted. Previous studies had only a selected number of participants. Although we limited the triage categorization to written case scenarios, this method has previously been shown to successfully test triage techniques. 20 It still remains unclear whether a written scenario with symptoms and vital signs is an accurate reflection of how a person would be triaged. The logistics of a real-time clinical assessment of actual patients by 63 reviewers would be substantial, particularly in the setting of a busy ED.
ACAD EMERG MED December 2002, Vol. 9, No. 12 www.aemj.org 1401 Another limitation of the study is the use of a proxy criterion standard. No RNs were involved in the creation of the criterion standard. It is possible that the three designated PEPs gave overall a lesser severity level to the case scenarios, since our findings have shown the PEPs have generally a higher rate of undertriage. Future directions may include the decision to modify the present triage guidelines and to improve ease of application. Since we obtained only a moderate level of agreement with the present guidelines, we may introduce the new proposed pediatric triage guidelines proposed by the Canadian Association of Emergency Physicians. 15 We plan to re-evaluate our levels of agreement with this new five-level acuity scale. If a reliable and valid triage scale exists, it would be an important tool in comparing the performances of different EDs and might help to determine the division and utilization of resources. 19 CONCLUSIONS Triage, a crucial step in emergency care, requires improved measurement. Our present triage guidelines appear to have only moderate interrater reliability. A revised pediatric triage acuity scale would help us to improve the delivery of timely care. References 1. Wiebe A, Rosen LM. Triage in the emergency department. Emerg Clin North Am. 1991; 19:491 502. 2. Kennedy K, Aghababian RV, Gans L, Lewis CP. Triage: techniques and applications in decision making. Ann Emerg Med. 1996; 28:136 44. 3. Wuerz R, Fernandes CMB, Alarcon J. Inconsistency of emergency department triage. Ann Emerg Med. 1998; 32: 431 5. 4. Christopher C. Pediatric triage [letter]. Acad Emerg Med. 1996; 3:2 3. 5. Brillman JC, Doezema D, Tandberg D, et al. Triage: limitations in predicting need for emergent care and hospital admission. Ann Emerg Med. 1996; 27:493 500. 6. George S, Read S, Westlake L, Fraser-Moodie A, Pritty P, Williams B. Differences in priorities assigned to patients by triage nurse and by consultant physicians in accident and emergency departments. J Epidemiol Community Health. 1993; 47:312 5. 7. Gill JM, Reese CL 4, Diamond JJ. Disagreements among health care professionals about the urgent care needs of emergency department patients. Ann Emerg Med. 1996; 28:474 9. 8. McDonald L, Butterworth T, Yates DW. Triage: a literature review 1985 1993. Accid Emerg Nurs. 1995; 3:201 7. 9. O Brien GM, Shapiro MJ, Fagan MJ, Woolard RW, O Sullivan PS, Stein MD. Do internists and emergency physicians agree on the appropriateness of emergency department visits? J Gen Intern Med. 1997; 12:188 91. 10. Fernandes CMB, Wuerz R, Clark S, Djurdjev O. How reliable is emergency department triage? Ann Emerg Med. 1999; 34:141 7. 11. Brillman JC, Doezema D, Tandberg D, Sklar DP, Skipper BJ. Does a physician visual assessment change triage? Am J Emerg Med. 1997; 15:29 33. 12. George S, Read S, Westlake L, Williams B, Fraser-Moodie A, Pritty P. Evaluation of nurse triage in a British accident and emergency department. BMJ. 1992; 304:876 8. 13. FitzGerald G. Emergency department triage [thesis]. Queensland, Australia: University of Queensland, 1989. 14. Dilley SJ, Standen P. Vistorian nurses demonstrate concordance in the application of the National Triage Scale. Emerg Med. 1998; 10:12 8. 15. Canadian Association of Emergency Physicians. Implementation of Canadian Pediatric Triage and Acuity Scale. Can J Emerg Med. 2001; 3(4):1 32. 16. Beveridge R, Ducharme J, Janes L, Beaulieu S, Walter S. Reliability of the Canadian Emergency Department Triage and Acuity Scale: interrater agreement. Ann Emerg Med. 1999; 34:155 9. 17. Dillman DA. Mail and other self-administered questionnaires. In: Handbook of Survey Research. San Diego, CA: Academic Press, Inc., 1983, pp 359 77. 18. Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977; 33:159 74. 19. Gorelick M, Lee C, Cronan F, Kost S, Palmer K. Pediatric Emergency Assessment Tool (PEAT): a risk-adjustment measure for pediatric emergency patients. Acad Emerg Med. 2001; 8:156 62. 20. Landis SS, Benson NH, Whitley TW. A comparison of four methods of testing emergency medical technicians triage skills. Am J Emerg Med. 1989; 7:1 4.