ARTICLE. Appropriateness of Urgent Referrals by Nurses at a Hospital-Based Pediatric Call Center

ARTICLE Appropriateness of Urgent Referrals by Nurses at a Hospital-Based Pediatric Call Center Allison Kempe, MD, MPH; Catherine Dempsey, MPH; Julie Whitefield, MD; Joan Bothner, MD; Todd MacKenzie, PhD; Steven Poole, MD Objectives: To evaluate (1) the appropriateness of the after-hours referral (AHR), (2) clinical characteristics of urgently referred patients, and (3) parental compliance with disposition recommendations by the After-Hours Call Center of the Children s Hospital, Denver, Colo. Subjects: Patients of Denver, Colo, pediatricians who called after office hours were triaged by nurses using a computerized triage system, and were referred for afterhours evaluation to 1 of 5 urgent care sites (N = 409). Design: Data were collected at each site for 2-week periods every 3 months, from October 1996 to October 1997 (total study period, 10 weeks). Appropriateness of AHRs (diagnoses, clinical interventions, and final dispositions) were determined by a questionnaire that was completed by the evaluating physician. Parental compliance with the AHR was assessed by review of the after-hours site patient database and by telephone survey. Results: Of the referred patients, 339 (82.9%) complied with the recommendation for AHR and, of these, physician questionnaires were completed for 332 (97.9%). The mean percentage of evaluated patients judged appropriate was 90.7% and did not differ statistically by site or by physician training. A history indicating a potentially serious condition was the most common reason for judging a referral appropriate (80.1%), followed by patient discomfort (53.3%), findings from a physical examination (42.5%), parental anxiety (41.5%), and an urgent need for diagnostic test or therapy (34.7%). Of evaluated patients, 37.0% had a diagnostic test, and in 43.5% of cases, the evaluating physician thought a therapeutic intervention was necessary that night. Of the total sample, 93.4% were discharged and 6.6% were admitted to the hospital. The major reasons given by families for noncompliance were lack of understanding of the disposition recommendation and disagreement with the need for urgent referral. Conclusions: Approximately 90% of patients who complied with a referral for urgent evaluation by the After- Hours Call Center were judged by the evaluating physician to have been appropriately referred. The appropriateness rate for all referrals may be lower if there is significant self-selection in those families who do not comply. Arch Pediatr Adolesc Med. 2000;154:355-360 Editor s Note: I wonder what would have happened if a parent directly accessed the computerized algorithms first, made a decision, and then did it all over again with the nurse asking the questions. Surely some money-managing mogul would pay for that study. Catherine D. DeAngelis, MD From the Departments of Pediatrics (Drs Kempe, Bothner, MacKenzie, and Poole) and Preventive Medicine and Biometrics (Drs Kempe and MacKenzie), University of Colorado Health Sciences Center, Denver; and the Research Institute at The Children s Hospital, Denver (Drs Kempe and Whitefield and Ms Dempsey). IN THE PAST, it was estimated that 30% of a pediatrician s weekly time was spent on the telephone giving medical advice 1,2 and, in an era of managed care, telephone care in pediatrics is expected to increase. 3-6 Despite the increasingly important role of telephone care, 30% to 50% of pediatricians report that it is the most difficult part of their practice, 7-9 particularly the after-hours calls. 10 In response to the problem of physician burnout and the necessities of increasing the consistency of telephone advice and care, improving documentation to decrease medicolegal risk, and providing telephone care in a cost-effective manner, systems for managing after-hours telephone triage and advice have been evolving. In recent years, centralized call centers, often using computer-accessed triage systems, have been established to manage after-hours calls from many physicians. Estimates suggest that at least 25% of all general pediatricians currently sign out their after-hours calls to a call center. 11 Despite the rapid growth of call centers using triage and advice systems, there has been little evaluation of the clinical validity of this type of after-hours care. 355

PATIENTS AND METHODS THE AFTER-HOURS CALL CENTER AND THE COMMUNITY-BASED PEDIATRIC URGENT/EMERGENT CARE NETWORK The After-Hours Call Center of the Children s Hospital in Denver is staffed by trained pediatric nurses who use the Pediatric Triage and Advice System to answer after-hours calls from patients of subscribing pediatricians. The Community-Based Pediatric Urgent/Emergent Care Network (CBPEN) consists of 5 sites in the Denver metropolitan area where approximately 90% of patients referred by the call center for urgent after-hours evaluation are seen (Table 1). The sites are staffed primarily by board-certified pediatricians or pediatric emergency medicine physicians who are salaried faculty at the University of Colorado, Denver, or the Children s Hospital, Denver, and do not depend on referrals from the Denver After-Hours Call Center as a source of income. At the Denver After-Hours Call Center, pediatric nurses identify the presenting complaints of the caller, access the appropriate computerized algorithms, and ask a series of questions dictated by the algorithm. Based on the responses, the algorithm gives a recommended disposition. Possible dispositions include (1) the patient should be referred for an urgent after-hours evaluation, (2) the patient should be referred for evaluation the next day at his or her primary care physician s office, or (3) the patient should be seen at a later date or requires home care and advice only. In addition, if the nurse feels that the caregiver is unable to give an adequate history or if there are worrisome features to the history that are not captured by the algorithm, the nurse may override the algorithm and upgrade the disposition decision to a more urgent disposition category. Likewise, a parent who feels uncomfortable waiting for an evaluation at a later date may override a nurse s disposition and be seen for an after-hours evaluation. If an after-hours evaluation is recommended, the patient is referred to 1 of the 5 CBPEN sites and a fax including the algorithm reason for after-hours referral is automatically sent to the after-hours site and the primary care physician. Patients are referred to sites outside the CBPEN approximately 10% of the time on the basis of physician or parent preference, payer constraints, or geographic factors. STUDY POPULATION The study population included patients of subscribing Denver pediatricians who called after office hours regarding a medical problem, were triaged by the After-Hours Call Center nurses, and were referred to 1 of the 5 urgent/emergent care sites. We collected data for 2 weeks at each of the 5 sites. The 2-week data collection periods were distributed equally over a 1-year period from October 1996 to October 1997. Because we collected data at only 1 site for each 2-week period, the total volume of urgent calls for the 5 sites together was representative of the volume of urgent calls in one 2-week period for the entire Denver metropolitan area. In determining the urgent referral rate during the study period, the numerator was the total of urgent referrals at the 5 sites and the denominator was estimated by calculating the mean of the total calls in metropolitan Denver during the five 2-week periods of data collection. If a child was referred more than once during the study period, only the first referral was included. Calls in which the parents overrode the algorithm disposition were not included, as they were not representative of triage recommendations of the After-Hours Call Center. DATA SOURCES AND OUTCOME DEFINITIONS Appropriateness of referral was determined by a questionnaire completed by the attending physician evaluating the patient at the CBPEN site. The questionnaire, given in Table 1, was developed and pilot-tested by a group of general pediatricians and pediatric emergency medicine physicians and was designed to capture factors important to practicing pediatricians in determining urgency of care. If physicians responded positively to any of the questions or if the patient was admitted to the hospital, the referral was judged to be an appropriate referral. In addition, to assess reasons for referrals judged inappropriate, we asked the question Was the history obtained by the after-hours nurse that resulted in same-night disposition accurate (please refer to reasons on the after-hours fax)? Diagnoses, clinical interventions, and final dispositions were collected by the evaluating physician at the time of evaluation. Results of tests or final disposition data not available on the night of evaluation were collected from review of the medical records the morning after evaluation. Parental compliance was assessed by review of the after-hours site patient database. Reasons for noncompliance were assessed by telephone interview with the person who originally called the After-Hours Call Center 24 to 48 hours after the referral. The questionnaire gathered information about demographics, parental satisfaction, and compliance and consisted of 41 closed-ended questions and 2 open-ended questions. The survey was adapted from the Behavioral Risk Factor Survey. 12 Our study protocol was approved by the Colorado Multiinstitutional Review Board, Denver. STATISTICAL METHODS Proportions, as well as contingency tables and 2 analysis, were employed. For proportions of particular interest, 95% confidence intervals are calculated, using the exact binomial method. All analyses were performed using commercial software (SAS software version 6.12; SAS Institute, Cary, NC). The after-hours program in Denver, Colo, currently provides after-hours telephone triage and advice for the patients of 91% of private practice pediatricians in the Denver metropolitan area and 90% of private practice pediatricians in the state of Colorado. Nurses at the Denver After-Hours Call Center use the Pediatric Triage and Advice System (Access Health Group, Phoenix, Ariz), a computerized system with clinical decision-making algorithms. The objectives of the present study were to evaluate after-hours telephone care given by the Denver After-Hours Call Center with respect to (1) the appropriateness of urgent after-hours referrals, 356

Table 1. Questionnaire for Evaluating Physicians Did the history you obtained alone (without physical examination) suggest the possibility of a condition that could medically compromise the patient if not evaluated or treated tonight? Did the patient s physical examination suggest a condition that could medically compromise the patient if not evaluated or treated tonight? Was the patient in sufficient discomfort or sufficiently symptomatic to warrant treatment tonight? In your opinion, were the parents so anxious that an evaluation tonight was warranted for their comfort? In your opinion, were there characteristics of the parents reaction to their child s illness that made them unable to adequately evaluate or care for patient? Was there a problem unrelated to the presenting complaint that required same-night evaluation or treatment? Were there other reasons that the patient should have been evaluated tonight? Did the patient require therapy or a test or procedure tonight that could not have been given at home without evaluation? (2) the clinical characteristics of urgently referred patients, and (3) parental compliance with a recommendation for urgent evaluation. RESULTS The mean number of total calls to the After-Hours Call Center during the study period was 2223, which was representative of the volume of calls for a 2-week period for patients in the Denver metropolitan area. Of these, 509 (22.9%) were referred for urgent evaluation and 454 (89.1% of urgent referrals) were referred to 1 of the 5 CBPEN sites. A total of 409 (90.0%) were eligible for our study. Reasons for ineligibility included parental overrides of less urgent disposition categories (n = 15), repeat cases (n = 12), or age older than 18 years (n = 1). In addition, 17 subjects were excluded because they had walked in to the hospital and were subsequently triaged over the phone. This latter group was excluded because of concern that the nurse s disposition decision might be affected by knowledge that the family was already at the hospital. STUDY POPULATION DEMOGRAPHICS Telephone interviews of the person who telephoned the After-Hours Call Center were completed for 79.9% (n = 362) of the referred sample. Most callers were women (87.0%) and were aged 25 to 44 years (76.5%); of this group, most had at least 1 year of vocational school or college (75.1%), were unemployed (68.8%), and had another wage earner in the household (75.1%). APPROPRIATENESS OF REFERRAL Of the 409 eligible patients referred, 70 (17.1%) of the families did not comply and were not evaluated after hours. Of the 339 families that complied, evaluating physician questionnaires were completed for 332 (97.9% of those evaluated and 81.2% of all eligible cases). As shown in Table 2, the percentage of referrals judged to be appropriate varied between 81.8% and 95.1% at the sites, Table 2. Characteristics of Urgent or Emergent Care Network Site* The Children s Hospital Littleton Hospital Lutheran Medical Center Aurora Urgent Care St Anthony s North Setting Urban children s hospital emergency department Pediatric emergent clinic attached to suburban hospital emergency department Pediatric emergent clinic attached to suburban hospital emergency department Suburban freestanding pediatric clinic Suburban hospital emergency department Pediatric Patient Visits per Year, October 1996 to October 1997 26 124 8367 7495 6063 5791 *All sites were in the metropolitan area of Denver, Colo. Estimated yearly figure, as clinic was closed for a portion of the year. % of Referrals Judged Appropriate 100 90 80 70 60 50 40 30 20 10 0 80.1 53.3 42.5 41.5 34.7 with a mean for all sites of 90.7%. The differences between sites were not statistically significant. Of the total evaluations, 51.2% were completed by physicians boarded in general pediatrics, 30.4% by physicians boarded in pediatric emergency medicine, 16.3% by physicians boarded in general emergency medicine, and 2.1% by family practice physicians or dentists. The percentage of referrals rated appropriate varied from 87.3% for pediatric emergency physicians to 91.8% for general pediatricians, and there were no statistically significant differences by specialty. To assess the maximum bias that could have been introduced by noncompliant families, we also calculated the rate of appropriateness based on the assumption that all noncompliant patients were inappropriately referred. This assumption decreased the appropriateness rate for all urgent referrals to 74.9%. The Figure shows the percentage of referrals judged appropriate by physicians on the basis of different factors. Overall, 295 cases were judged appropriate on the basis of history, patient discomfort, or parental anxiety, and only 5 cases were judged appropriate without any 7.6 4.6 History Patient Discomfort Parental Anxiety Physical Findings Urgent Test or Intervention Parental Inability to Evaluate or Care for Patient Other Percentage of referrals judged appropriate by physicians on the basis of different factors (nonexclusive categories) (N = 332). 357

Table 3. Appropriateness of After-hours Referral* Site Evaluations (N = 332) No. of Evaluations by Site (%) of these factors. Physicians judged that the information obtained by the nurse that resulted in urgent disposition was inaccurate in 12.7% of the cases (n = 42). These cases accounted for 53.1% of inappropriate referrals. CLINICAL CHARACTERISTICS OF REFERRED CASES Final diagnoses and the percentages of each diagnostic category judged appropriate are given in Table 3. Of the total sample, 37.0% (n = 123) had a diagnostic test, including a laboratory test in 22.0% (n = 73), a radiographic examination in 19.9% (n = 66), and a subspecialty consultation in 5.0% (n = 5). In 43.5% (95% confidence interval, 38.0%-48.9%; n = 144) of the cases, the evaluating physician felt a therapeutic intervention was necessary that night, including suturing or other surgical interventions (22.9%), nebulized medications (11.1%), intravenous fluids (9.7%), intravenous or intramuscular medications (9.0%), orthopedic procedures (6.2%), or other therapies (steroids, removal of foreign bodies, oxygen, resuscitation, or oral antibiotics) (14.5%). Of the total sample, 93.4% (n = 310) were discharged and 6.6% (n = 22) were admitted to the hospital. COMPLIANCE WITH RECOMMENDATIONS FOR AFTER-HOURS EVALUATION Of the 70 families who did not have a documented visit, 80.0% (n = 56) were reached for telephone interview. Of these families, 26.7% (n = 15) said they had complied despite no documentation of their visit, and 12.5% (n = 7) said they had complied but had gone to a facility other than the one to which they were referred. Of families who said they did not comply (n = 34), major reasons given were confusion about what was being recommended (n = 16), disagreement with the need for a visit (n = 12), improvement in the child s condition (n = 3), weather conditions (n = 2), or not wanting to go to an emergency room (n = 1). COMMENT Appropriate Evaluations, % 95% Confidence Interval The Children s Hospital 101 (30.4) 87.3 80.2-92.9 Littleton Hospital 93 (28.0) 91.4 85.1-96.2 Lutheran Medical Center 83 (25.0) 95.1 88.1-98.0 Aurora Urgent Care 33 (9.9) 93.9 79.8-98.1 St Anthony s North 22 (6.6) 81.8 59.7-92.2 Mean for all sites NA 90.7 NA *NA indicates not applicable. No significant differences by site ( 2 = 0.199). All sites were in the metropolitan area of Denver, Colo. In recent years, there has been dramatic growth in afterhours call centers using triage systems to handle patient calls from multiple physicians in a community and, sometimes, from distant sites. Hospital-based call centers and service bureaus are currently providing after-hours coverage for millions of children in the United States. 11 Despite this profound change in the way after-hours health care is being administered, there has been little assessment of the appropriateness of referrals made by call centers using triage and advice systems. To our knowledge, our study is the first to assess the appropriateness of urgent referrals of pediatric patients triaged by pediatric nurses using a computerized triage and advice system. Our results show that evaluating physicians judged urgent referrals to have been appropriate for approximately 90% of patients who complied with the recommendation for urgent evaluation. Approximately 80% of the families complied, with major reasons given for noncompliance being lack of understanding of the recommendations or disagreement with the need for evaluation. Previous studies have shown that without prior telephone triage, substantial numbers of patients seek care in the emergency room for nonurgent problems that would be better managed in a primary care setting. 13-28 Estimates of the proportion of inappropriate or nonurgent pediatric visits to an emergency room range from 25% to 75%. 23-32 The wide range in estimates of appropriateness is due not only to differences in the period, populations, and settings studied, but also to a lack of consensus about the definition of an appropriate urgent or emergent care visit. Some studies have defined appropriateness retrospectively based entirely on final diagnoses, patient disposition, 23,27,33 or assessment of the patient s course. 20,22,34 Purely retrospective assessments, especially those based on diagnosis alone, cannot be used to determine whether a patient should have been seen. Prospective assessments of the need for care must assess the potential for serious illness, often based on limited information, rather than the ultimate outcome after evaluation. 17 Classifying an emergency visit appropriate only if, in retrospect, there was threat to life or limb 35,36 is a particularly flawed approach, although a survey of medical directors of health maintenance organizations in 39 states found that 92% used life-threatening emergency in defining emergency room access policies. 37 Other studies have attempted to define appropriateness prospectively by applying criteria based on the presenting complaint and vital signs. 17,38 DeAngelis et al 26 used a combination of retrospective and prospective factors and included a category incorporating legitimate, acute parental concerns, such as extreme discomfort in the child. Our criteria incorporated prospective factors that practicing pediatricians take into account in determining need for urgent care. We also included retrospective factors, such as physical findings and the urgent need for a test or therapy or hospital admission, but these factors identified few patients who were not identified by prospective criteria. Although our methods differ slightly, our data are most comparable to the study by DeAngelis et al, 26 who observed an appropriateness rate of approximately 67.5% for children who were not triaged prior to arrival at the emergency department. There are limited data about rates of urgent referrals by physicians taking calls directly with which to compare our findings. A study 358

Table 4. Final Diagnosis for Evaluated Patients Evaluations (N = 332) % of Total Appropriate Diagnosis No. Evaluations Evaluations, % Trauma 83 25.1 91.5 Upper respiratory 80 infections Otitis media 25 Pharyngitis 25 Viral upper 16 24.1 88.8 respiratory infection Croup 13 Sinusitis 2 Gastrointestinal disorders 58 Gastroenteritis 36 or dehydration Abdominal pain 15 17.5 87.9 Constipation 5 Other (n 1) 2 Other infectious diseases 43 Cellulitis 4 Conjunctivitis 5 Viral syndrome 21 12.9 91.3 Fever 7 Urinary tract infection 4 Other (n 1) 2 Lower respiratory 23 infections and asthma Asthma/RAD* 9 4.2 100 Pneumonia 10 Bronchiolitis 4 Other surgical problems 11 Foreign body 9 3.3 100 Hernia 2 Dermatological problems 7 2.1 100 Miscellaneous 24 Headache 4 Dental 3 Allergic reaction 3 7.2 82.6 Chest pain 2 Other 12 *RAD indicates reactive airway disease. by Caplan et al 39 in 1983 reported that physicians taking after-hours calls recommended after-hours evaluation for only 6.4% of their patients, although an additional 5.6% went to the emergency room without their consent. The initial referral rate of 22% by the After- Hours Call Center nurses in this study is significantly higher, although only 10% were judged by the evaluating physicians to have been overreferrals. Although direct comparisons between the findings by Caplan et al and our findings are confounded by differences in the populations, methods used, temporal factors, the differences in referral rates may indicate that our data underestimate the rate of overreferrals or that physicians taking calls directly in the study by Caplan et al underreferred in some cases. The conditions of the patients referred by the afterhours program were moderate in severity, with approximately 7% being admitted to the hospital, more than 40% requiring a therapeutic intervention on the night of the call, and almost 40% having a diagnostic test performed. The most frequent diagnostic categories seen were similar to those reported in a recent study of pediatric visits to a general emergency department in which the prior triage status was not specified. 40 Our study has several limitations: (1) Because we were unable to make a determination of appropriateness for referred patients who did not comply, the rate of appropriateness we report most accurately pertains only to patients who came in for evaluation. It is possible that secondary triage by the patient s parents decreased the rate of inappropriate referrals in our study. The actual rate of appropriateness for all urgent referrals is probably close to 90%, but could be as low as 75%, in the extreme case that every noncompliant patient was assumed to have been inappropriately referred. (2) The method used to assess appropriateness of referral in this study has limitations. The physicians completed their forms with knowledge of the diagnosis and, in some cases, ultimate outcome of the patient. Previous studies have shown that knowledge of patient outcome affects assessments of appropriateness of care. 41 However, the fact that the rates of referrals judged appropriate did not differ substantially by category of final diagnosis (Table 4) suggests that the potential confounding of determination of appropriateness by outcome was not a major factor. (3) The determinations of appropriateness in this study also represent the subjective opinions of a group of physicians without a uniform clinical approach. (4) Physician bias either for or against afterhours call centers also could have influenced decisions about appropriateness of referrals. Any bias would not be expected to be financial, however, as evaluating physicians have no financial connections to the Denver After-Hours Call Center and are all salaried. In addition, patient volumes at the CBPEN sites are high, providing little incentive to underreport overreferrals. None of the evaluating physicians were involved in the development of algorithms used in the Pediatric Triage and Advice System or in running or supervising the After-Hours Call Center. (5) Our study did not address the important issue of potential underreferral of urgent cases by after-hours call centers. (6) Finally, the socioeconomic status of our study population was relatively high and most parents were married; therefore, our results may not be generalizable to all populations. Because many after-hours call centers do not employ nurses with pediatric training, do not have backup coverage by pediatricians, and do not use algorithms developed by pediatricians, our results may not be generalizable to all after-hours call centers. The marketing of after-hours triage systems and call centers has progressed much more rapidly than the investigation into the clinical effectiveness of this type of care or the regulation of the quality of after-hours telephone care. The newly created Provisional Section on Pediatric Telephone Care and the Committee on Practice and Ambulatory Medicine 11 of the American Academy of Pediatrics, Elk Grove Village, Ill, has recently developed guidelines for the safe operation of pediatric call centers. Hopefully, such guidelines will result in the development of standards of care in this area and, possibly, accreditation requirements for call centers. Further investigation is needed to examine underreferral (espe- 359

cially in high-risk or special-needs populations), the appropriateness of nonurgent referrals, and the effect of this type of after-hours care on the patient-physician relationship. Accepted for publication September 9, 1999. This study was supported by grants from the Research Institute at The Children s Hospital, Denver, Colo (Dr Kempe); and the Center for Research in Demand Management and Telephone Care, Denver (Dr Kempe). Corresponding author: Allison Kempe, MD, MPH, Children s Hospital, 1056 E 19th Ave, B032, Denver, CO 80218. REFERENCES 1. Hessel S, Haggerty R. General pediatrics: a study of practice in the mid-1960s. J Pediatr. 1968;73:271-279. 2. Curtis P, Talbot A. The telephone in primary care. J Community Health. 1981;6: 194-203. 3. Nazarian L. A look at the private practice of the future. Pediatrics. 1995;96:812-816. 4. Lewin L. Adapting your pediatric practice to the changing health care system. Pediatrics. 1995;96:799-804. 5. Poole S. Pediatric healthcare trends: 1995 and beyond. Pediatr Demand Manage. 1995;1:1-6. 6. Poole S. Clinical strategies for adapting pediatric practice to managed care and capitation. Pediatr Demand Manage. 1995;2:1-5. 7. Fosarelli P, Schmitt B. Telephone dissatisfaction in pediatric practice: Denver and Baltimore. Pediatrics. 1987;80:28-31. 8. Katz H, Pozen J, Mushlin A. Quality assessment of a telephone care system. Am J Public Health. 1978;68:31-38. 9. Caplan S, Orr S, Skulstad J, et al. After-hours telephone use in urban pediatric primary care centers. AJDC. 1983;137:879-882. 10. Poole S, Schmitt B. After-hours telephone coverage: the application of an areawide telephone triage and advice system for pediatric practices. Pediatrics. 1993; 92:670-679. 11. American Academy of Pediatrics. Pediatric call centers and the practice of telephone triage and advice: critical success factors. In: Provisional Section on Pediatric Telephone Care and Committee on Practice and Ambulatory Medicine. Elk Grove Village, Ill: American Academy of Pediatrics; 1998:1-6. 12. Centers for Disease Control and the Colorado Department of Public Health. Behavioral Risk Factor Survey. Denver: Colorado Dept of Public Health; 1995. 13. Halfon J, Newacheck PW, Wood DL, St Peter RF. Routine emergency department use for sick care by children in the United States. Pediatrics. 1996;98:28-34. 14. Buesching DP, Jablonowski A, Vesta E, et al. Inappropriate emergency department visits. Ann Emerg Med. 1985;14:89-93. 15. Wolcott BW. What is an emergency? depends on whom you ask. JACEP. 1979; 8:241-243. 16. Brillman JC, Doezema D, Tandberg D, et al. Triage: limitations in predicting need for emergent care and hospital admission. Ann Emerg Med. 1996;27: 493-500. 17. Derlet RW, Kinser D, Ray L, et al. Prospective identification and triage of nonemergency patients out of an emergency department: a 5-year study. Ann Emerg Med. 1995;25:215-223. 18. Gifford MJ, Franaszek JB, Gibson G. Emergency physicians and patients assessments: urgency of need for medical care. Ann Emerg Med. 1980;9:502-507. 19. Guterman JJ, Franaszek JB, Murdy D, et al. The 1980 patient urgency study: further analysis of the data. Ann Emerg Med. 1985;14:1191-1198. 20. Baker DW, Stevens CD, Brook RH. Determinants of emergency department use by ambulatory patients at an urban public hospital. Ann Emerg Med. 1995;25: 311-316. 21. Shaw KN, Selbst SM, Gill FM. Indigent children who are denied care in the emergency department. Ann Emerg Med. 1990;19:107-110. 22. Gill JM, Reese CL, Diamond JJ. Disagreement among health care professionals about the urgent care needs of emergency department patients. Ann Emerg Med. 1996;28:474-479. 23. Wingert WA, Friedman DB, Larson WR. Pediatric emergency room patients: a comparison of patients seen during the day and at night. AJDC. 1968;115:48-56. 24. Bergman AB, Haggerty RJ. The emergency clinic. AJDC. 1962;104:68-76. 25. Robinson GC, Kinnis C, Anderson DO, Argue C, Miller HS. Use of a hospital emergency service by children and adolescents for primary care. CMAJ. 1969;101: 69-73. 26. DeAngelis C, Fosarelli P, Duggan AK. Use of the emergency department by children enrolled in a primary care clinic. Pediatr Emerg Care. 1985;1:61-65. 27. Ziv A, Boulet JR, Slap GB. Emergency department utilization by adolescents in the United States. Pediatrics. 1998;101:987-994. 28. Kahn L, Anderson M, Perkoff GT. Patients perceptions and uses of a pediatric emergency room. Soc Sci Med. 1973;7:155-160. 29. Dershewitz RA, Paichel W. Patients who leave a pediatric emergency department without treatment. Ann Emerg Med. 1986;15:717-720. 30. Franco SM, Mitchell CK, Buzon RM. Primary care physician access and gatekeeping: a key to reducing emergency department use. Clin Pediatr (Phila). 1997; 36:63-68. 31. Halperin R, Meyers AR, Alpert JJ. Utilization of pediatric emergency services. Pediatr Clin North Am. 1979;26:747-757. 32. Bass LW, Cohn RL. Ostensible versus actual reasons for seeking pediatric attention: another look at the parental ticket of admission. Pediatrics. 1982;70: 870-874. 33. Pane GA, Farner MC, Salness KA. Cost implications of delayed access to care in patients admitted through the emergency department. Ann Emerg Med. 1991; 20:730-733. 34. Oberlander TF, Pless IB, Dougherty GE. Advice seeking and appropriate use of a pediatric emergency department. AJDC. 1993;147:863-867. 35. Weinerman ER, Ratner RS, Robbins A, et al. Yale studies in ambulatory medical care: determinants of use of hospital emergency services. Am J Public Health. 1966;56:1037-1056. 36. Vayda E, Gent M, Hendershot A. Emergency department use at two Hamilton hospitals. CMAJ. 1975;112:961-965. 37. Kerr HD. Access to emergency departments: a survey of HMO policies. Ann Emerg Med. 1989;18:71-74. 38. Derlet RW, Nishio DA. Refusing care to patients who present to an emergency department. Ann Emerg Med. 1990;19:75-80. 39. Caplan SE, Tangerose Orr S, Skulstad JR, Charney E. After-hours telephone use in urban pediatric primary care centers. AJDC. 1983;137:879-882. 40. Zimmerman DR, McCarten-Gibbs KA, DeNoble DH, et al. Repeat pediatric visits to a general emergency department. Ann Emerg Med. 1996;28:467-473. 41. Caplan RA, Posner KL, Cheney FW. Effect of outcome on physician judgments of appropriateness of care. JAMA. 1991;265:1957-1960. 360