A nursing qualitative systematic review required MEDLINE and CINAHL for study identification

Journal of Clinical Epidemiology 58 (2005) 20 25 A nursing qualitative systematic review required MEDLINE and CINAHL for study identification Mireia Subirana a,b, *, Ivan Solá b, Josep M. Garcia b, Ignasi Gich a, Gerard Urrútia a,b a Epidemiology and Public Health Department, Hospital de la Santa Creu i Sant Pau, Sant Antoni Maria Claret 171, 08041 Barcelona, Spain b Iberoamerican Cochrane Centre, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain Accepted 2 June 2004 Abstract Objective: Analyze the number and the relevance of references retrieved from CINAHL, MEDLINE, and EMBASE to perform a nursing systematic review. Study Design: A search strategy for the review topic was designed according to thesaurus terms. The study analyzes (1) references with abstract, (2) overlap between databases, (3) reference relevance, (4) relevance agreement between experts, and (5) reference accessibility. Results: Bibliographic search retrieved 232 references: 16% (37) in CINAHL, 68% (157) in MEDLINE, and 16% (38) in EMBASE. Of these, 72% (164) were references retrieved with an abstract: 14% (23) in CINAHL, 70% (115) in MEDLINE, and 16% (26) in EMBASE. Overlap was observed in 2% (5) of the references. Relevance assessment reduced the number of references to 43 (19%): 12 (34.3%) in CINAHL, 31 (19.7%) in MEDLINE, and none in EMBASE (Z 1.97; P.048). Agreement between experts achieved a maximum Cohen s κ of 0.76 (P.005). References identified in CINAHL were the most difficult to obtain (χ 2 3.9; df 1; P.048). Conclusions: To perform a quality bibliographic search for a systematic review on nursing topics, CINAHL and MEDLINE are essential databases for consultation to maximize the accuracy of the search. 2005 Elsevier Inc. All rights reserved. Keywords: Bibliographic databases; MEDLINE; CINAHL; EMBASE; Systematic review; Nursing 1. Introduction Systematic reviews are undertaken to answer specific clinical questions. They take all relevant research studies into account and sometimes include those that have not been published. Following specific methodology to overcome many of the limitations of narrative reviews, systematic reviews identify all the relevant studies and include strategies to minimize bias and to maximize precision. Quality systematic reviews are generally considered the most efficient means to provide qualified, explicit, specific, synthesized, and up-to-date information. The first step in the systematic review process is to establish a well-crafted question and the second step is to design the search strategy to answer it. A clearly defined question guides and directs the literature search. The search to locate relevant studies follows a defined strategy and should be as comprehensive and exhaustive as possible. Consequently, the databases selected for consultation play a key role in the quality of a systematic review. * Corresponding author. Tel.: 34-93-2919526; fax: 34-93-2919525. E-mail address: msubirana@santpau.es (M. Subirana). To identify the highest number of relevant publications in a specific issue for a systematic review, the initial sources used are bibliographic databases such as MEDLINE. Bibliographic databases vary in scope, currency, accessibility, and cost. In recent years, the production of scientific literature and the number of databases available to researchers have increased dramatically. Moreover, improvements in information technology have facilitated the retrieval of information from electronic database [1 5]. Once the database has been chosen, the different components of the research question should be translated into the key search terms, to design the search strategies. The databases most likely to contain the highest number of references relevant to nursing topics are CINAHL (Cumulative Index to Nursing and Allied Health Literature), MEDLINE (Medical Literature, Analysis, and Retrieval System Online), and EMBASE (Excerpta Medica Database) [6]. CINAHL, produced by Cinahl Information Systems (Glendale, CA, USA), was established in 1982. It includes subjects related to nursing and health, as well as other topics such as education, behavioral sciences, and health administration. Since 1984, it has also included publications of the National League for Nursing and the American Nurses Association. CINAHL indexes 1,620 biomedical, 0895-4356/04/$ see front matter 2005 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2004.06.001

M. Subirana et al. / Journal of Clinical Epidemiology 58 (2005) 20 25 21 social sciences, and consumer health journals; 492 of these are nursing titles, of which 144 are also indexed in MED- LINE [7]. This database uploads 300,000 new references annually. Titles are mainly in English, but articles in other languages are also included. CINAHL incorporates thesaurus terms with hierarchically organized descriptors, 70% of which agree with MEDLINE s medical subject headings (MeSH terms) [8]. The U.S. National Library of Medicine produces MED- LINE, which began in 1966; older literature is continually added to the database, back to 1950 and before. It includes publications of the Index Medicus, the Index to Dental Literature, and the International Nursing Index. References from all areas of biomedicine are indexed, exclusively from journals. Of the 4,600 indexed biomedical journals, 8% are nursing journals [8]. The database uploads some 500,000 references annually. Approximately 69% of the references include an abstract [9], with 12 descriptors for each reference [7]. EMBASE, established in 1974, is produced by Elsevier Science Publishers in the Netherlands. This database covers biomedical publications; its main areas of interest are toxicology and drug literature. It includes the fields of biomedicine, health administration and economy, and forensic sciences. Some 3,600 journals are indexed, including a number of nursing titles [9]. About 375,000 references are added per year. Original articles account for 95% of the EMBASE content; doctoral theses, monographs, handbooks, and conference and congress proceedings are also included. The thesaurus language, with a total of 38,000 terms, is hierarchically organized with descriptors called EMTREE. In previous studies of bibliographic databases, several authors assessed the overlap between CINAHL and MED- LINE [7 12], and also compared the policies of descriptor inclusion [7] and journal coverage [11]. Burnham and Shearer [8] also included EMBASE in their comparison. Brenner [7] compared the allocation of descriptors in MED- LINE and CINAHL, and found that MEDLINE includes twice as many descriptors as CINAHL, thereby allowing a more specific search. Okuma [11] compared MEDLINE and CINAHL from the perspective of student nurses and recommended CINAHL as the database of choice for nursing students. Watson and Perrin [10] focused their study on the overlapping of the journals retrieved in the search; they concluded that consulting both databases is necessary because each database includes unique journal titles. Brazier and Begley [9] analyzed the accessibility in Ireland of the retrieved references for each database. Their results indicated that it is necessary to consult both databases for an exhaustive search of literature; furthermore, they found for threequarters of retrieved articles a copy could be obtained in a maximum of two days [9]. Coma et al. [12] did a similar comparison of the databases analyzing accessibility in Spain. Their results suggested that CINAHL retrieved a higher appropriate record number for the field of nursing, although the degree of difficulty regarding accessibility was greater. In spite of the variety of results gathered in previous studies on this subject, all authors recommend consultation of both CINAHL and MEDLINE for a more complete and detailed nursing bibliographical search. Brenner [7] also emphasized the importance of adapting the search strategy to fit the thesaurus terms of each database, because these may differ slightly. In 2002, the authors were commissioned to conduct a systematic review of the instruments that measure nursing workload and establish the impact of nursing staff on patient outcomes. Whereas the studies cited all based their bibliographic searches on several nursing topics, our objective was to present, compare, and discuss the amount and quality of the literature extracted from each database on a single topic. 2. Materials and methods 2.1. Search strategy The search strategy was designed to retrieve validation studies on nursing workload measurement instruments and studies that establish the impact of nursing staff on patient outcomes. Bibliographical databases consulted were CINAHL, MEDLINE, and EMBASE. The databases were searched from their inception until September 2002, with no limit regarding original language of the article. The CINAHL database was consulted through the SilverPlatter distributor available in the network of libraries at the University of Barcelona, MEDLINE was consulted through the free electronic access PubMed, and EMBASE by means of the Ovid distributor, also available at the University of Barcelona. The search strategies, designed by a librarian, included terms related to the condition and to the intervention on the systematic review topic and were developed according to the relevant descriptors for each database [7]. Table 1 presents the search strategy for each database. 2.2. Assessment A number of aspects were examined for each reference obtained from each database: (1) references with abstract, (2) overlap between databases, (3) reference relevance, (4) relevance agreement between experts, and (5) accessibility and cost of obtaining each references. To assess the relevance of the retrieved references in each database, three external experts on the subject of the systematic review were consulted. Each expert received a list with the title (and abstract, if available) of each retrieved reference, excluding source, author, and year of publication. The experts scored the relevance of each reference in relation to the subject of the systematic review, using the following classification: (1) Irrelevant: very poor in relation to the review topic; (2) Slightly irrelevant: poor in relation to the review topic; (3) Somewhat relevant: good in relation

22 M. Subirana et al. / Journal of Clinical Epidemiology 58 (2005) 20 25 Table 1 Search strategy for the three databases CINAHL MEDLINE EMBASE #1: personnel staffing and scheduling / #1: Personnel Staffing and Scheduling [MAJR] #1: Personnel management/ #2: job description/ #2: Job Description [MAJR] #2: Personnel management/ or #3: personnel turnover/ #3: Personnel Turnover [MAJR] personnel staffing #4: workload/ #4: Workload [MAJR] OR workload[text Word] and scheduling.mp. #5: workload measurement/ #5: Nursing Staff [MESH] #3: PERSONNEL STAFFING #6: workload.tw. #6: Nurses [MESH] AND SCHEDULING.mp. #7: nursing workload.tw. #7: Nursing [MESH:NOEXP] #4: Job analysis/ #8: 4 or 5 or 6 or 7 #8: Nursing Care [MESH] #5: Workload/ #9: nursing staff, hospital/ #9: Nursing, Team [MESH] #6: Nursing staff/ #10: nurses/ #10: Nursing Process [MESH] #7: Nurse/ #11: staff nurses/ #11: Nursing Services [MESH] #8: Nursing/ #12: nursing care/ #12: Nurse-Patient Relations [MESH] #9: Health care delivery/ #13: team nursing/ #13: Nursing Service, Hospital [MESH] #10: Nurse patient relationship/ #14: nursing service/ #14: Treatment Outcome [MESH:NOEXP] OR #11: Treatment outcome/ #15: nursing process/ outcome[text Word] #12: 1 or 2 or 3 or 4 or 5 #16: nurse-patient relations/ #15: Outcome Assessment (Health #13: 6 or 7 or 8 or 9 or 10 #17: nursing outcomes/ Care) [MESH:NOEXP] #14: 11 and 12 and 13 #18: treatment outcomes/ #19: outcome.tw. #20: 18 or 19 #21: outcome assessment/ #16: Outcome and Process Assessment (Health Care) [MESH:NOEXP] #17: #16 OR #15 OR #14 #18: #13 OR #12 OR #11 OR #10 OR #9 #22: process assessment (health care) / OR #8 OR #7 OR #6 OR #5 #23: 1 or 2 or 3 or 8 #19: #4 OR #3 OR #2 OR #1 #24: 9 or 10 or 11 or 12 or 13 or 14 #20: #18 AND #17 AND #19 or 15 or 16 #25: 17 or 20 or 21 or 22 #26: 23 and 24 and 25 to the review topic, and (4) Relevant: very good in relation to the review topic. Scores from two of the experts were compared to assess agreement. In cases of discrepancy, the third expert was consulted to determine the final relevance value for each reference. Study quality assessment was based on quality criteria for assessment of experimental studies [13]. Accessibility cost of the references relevant to the review were taken into account because funding was limited, as is the case in the majority of research projects. We established accessibility cost criteria for relevant references as follows: (1) journals available locally (free of charge); (2) journals available somewhere else in Spain (6 /9V/10$) and (3) journals not available in Spain (15 /22V/25$). 2.3. Statistical analysis Descriptive results are shown as number of cases and percentages. The χ 2 test was used to perform comparisons between percentages. Agreement between experts was assessed by calculating Cohen s kappa. This test was chosen for expected chance agreement [14,15]. To compare the final classification of relevance for each database, Mann Whitney nonparametric tests were used, considering the ordinal nature of the variable to compare, and the comparison performed between two independent groups (we excluded EMBASE for this analysis). All the analyses were two-tailed. The level of significance was the standard of 5% (α.05). All analyses were performed with the SPSS package (v. 10.0). 3. Results The bibliographic search of the three databases retrieved 232 references, of which 229 were articles; two of the remaining references corresponded to handbooks and one to a doctoral thesis. From the total number of references (n 232), 16% (37) were retrieved in CINAHL, 68% (157) in MEDLINE, and 16% (38) in EMBASE. There was an overlap in 2% (5) references; 2 retrieved in both CINAHL and MEDLINE and 3 in both MEDLINE and EMBASE. Omitting the overlapping references, the total was 222 references: 15.8% (35) retrieved in CINAHL, 68.4% (152) in MEDLINE, and 15.8% (35) in EMBASE. In view of the low overlap found (2 between MEDLINE and CINAHL; 3 between MEDLINE and EMBASE), an author search from the references found in each database was conducted in the other two databases. The aim of this check was to exclude the possibility that the references were in actual fact in the other databases; 163 further duplicated references were found, 16 of which were found in all databases These results are given in Table 2. From the total number of references, 72% (164) included an abstract. Of these, 14% (23) were in CINAHL, 70% (115) were in MEDLINE, and 16% (26) in EMBASE.

M. Subirana et al. / Journal of Clinical Epidemiology 58 (2005) 20 25 23 Table 2 Overlap between databases from 232 references retrieved and after checking citations between databases References retrieved, Overlap from bibliographic search Overlap after checking citations between databases Comparison 1st database, N n % n % CINAHL MEDLINE 37 2 5.4 15 40.5 CINAHL EMBASE 37 0 0 3 8.1 MEDLINE CINAHL 157 2 1.3 99 63 MEDLINE EMBASE 157 3 1.9 22 14 EMBASE CINAHL 38 0 0 6 15.7 EMBASE MEDLINE 38 3 7.9 18 47.4 The chi-square test reported no significant differences when compared databases in the number of retrieved references with abstract (P.38). Figure 1 shows the total number of references retrieved from each database, showing percentages with and without abstract. Agreement between experts in the relevance of the retrieved references was analyzed using Cohen s kappa test, obtaining κ 0.56, which rose to κ 0.76 (P.005) when intermediate relevance scores (Slightly irrelevant and Somewhat relevant) were considered jointly with the extreme scores (Irrelevant and Relevant). After reweighting the relevance scores, a very similar trend in classification was observed between the two experts, who classified the same number of references as Relevant or Irrelevant. Figure 2 shows the relevance score for each database according to the two experts. Finally, four cases of disagreement were resolved with the scores of the third expert. A total of 19% of the references (43 of the total retrieved) were considered relevant to the systematic review topic (Table 3). Only CINAHL, with 34.3% (12) references, and MEDLINE, with 19.7% (31), retrieved relevant references. CINAHL retrieved the highest number of relevant references (Z 1.97; P.048). Table 4 summarizes accessibility of the relevant references. Although 63% (27) of references could be obtained directly through libraries in Spain, the remaining third had to be requested from abroad. References retrieved from CINAHL were the most difficult to obtain (χ 2 3.9; df 1; P.048). This implied a substantially higher cost (which we estimated at approximately 230 /345V/392$). 4. Discussion The results show that for the bibliographic database search of our systematic review topic, MEDLINE provided not only the highest number of references but also more references with an abstract than did CINAHL or EMBASE. Using the above-mentioned search strategies, the overlap observed between databases appeared to be minimal compared with the results of other studies, which showed an overlap between the databases of 16% [12], 18% [4], and even 20% [9]. In the present study, the citations were checked against each database to assess the real overlap, because we suspected there would be some differences in the controlled vocabulary between databases. According to our findings, the increase in duplicated references compared with the initial low values was found to be due to differences in indexing and also in quantity and quality between descriptors in each database (MEDLINE used more accurate descriptors than CINAHL and EMBASE, although CINAHL appeared to use a greater number of descriptors). N=227 27.8% (63) N=157 26.8% (42) 72.2% (164) N=35 34.3% (12) 65.7% (23) 73.2% (115) N=35 25 8% (9) 74.2% (26) CINAHL MEDLINE EMBASE TOTAL Articles with Abstract Articles without Abstract Fig. 1. Total number of references retrieved from each database, with and without abstract.

24 M. Subirana et al. / Journal of Clinical Epidemiology 58 (2005) 20 25 Relevant References 19 18 16 17 53 54 4 30 4 30 76 76 Irrelevant References 102 101 148 148 CINAHL MEDLINE EMBASE TOTAL κ=0.76 Percentage of agreement=96% Expert 1 Expert 2 Fig. 2. Relevance scores according to experts criteria. The second aspect could be related to the search strategy. In the strategy used for MEDLINE we realized that only nursing staff[mesh] was used. When this was checked with the references retrieved from CINAHL and EMBASE duplicated in MEDLINE, we observed that all these reference included nursing staff, Hospital [MeSH], which is not the same as what we had searched for. In CINAHL, the reason why we did not retrieve all the references was because only one subject heading of the search strategy was included in the reference. Although it was the same for EMBASE when only one subject heading was in the reference, this reference was retrieved, possibly explaining the large number of irrelevant references retrieved from EMBASE. Account should also be taken of the number of references retrieved with an abstract (14% in CINAHL, 70% in MED- LINE, and 16% in EMBASE), given the importance of the information provided by the abstract at the moment of deciding whether or not to include a reference in the systematic review. Our results were generally similar to those obtained by Coma et al. [12], who retrieved 74% of the references in MEDLINE with abstracts, and those obtained by Brazier and Begley [9], who retrieved 69% of MEDLINE references with abstract; however, our bibliographic search retrieved more references with an abstract from CINAHL than did these other studies. In our bibliographic search, CINAHL and MEDLINE retrieved the references most relevant to the search topic. As shown in the Burnham and Shearer study [8], and as supported by other findings [7], there is a need to consult both CINAHL and MEDLINE for an exhaustive bibliographic search. Burnham and Shearer [8] also indicate that EMBASE did not provide a great amount of additional information. The relevance assessment led us to balance the scores in cases of discrepancy, thereby obtaining 43 relevant references for the systematic review. Although this is a low percentage, the specificity of the topic for the systematic review and the comprehensiveness of the bibliographic search design must again be taken into account. The references from CINAHL were not easily available in Spain. Of the total references retrieved, only two thirds of those that were relevant were available in Spanish libraries; the remaining third had to be obtained from abroad. Research costs consequently rose substantially (by 230 /345V/392$). Although some references were obtained at a minimal cost, it should be kept in mind that most research projects have a limited budget. As Petticrew emphasized, performing a good systematic review requires skill in the design of search strategy [5]. Furthermore, assessment of retrieved references in a search plays an important role in ensuring a quality systematic review. For all these reasons, we recommend that both Table 3 Final classification of relevance for each database CINAHL MEDLINE EMBASE Total Relevance n % n % n % n % Relevant 12 34.3 31 19.7 43 18.9 Somewhat relevant 7 20 27 17.2 6 17.1 40 17.6 Slightly irrelevant 8 22.9 45 28.7 7 20 60 26.4 Irrelevant 8 22.9 54 34.4 22 62.9 80 37.1

Table 4 Accessibility to relevant references for each database CINAHL M. Subirana et al. / Journal of Clinical Epidemiology 58 (2005) 20 25 25 MEDLINE Accessibility n % n % Total, n Available locally 4 33.3 18 58.1 22 Available in Spain 5 16.1 5 Not available in Spain 8 66.6 8 25.8 16 Total 12 100 31 100 43 CINAHL and MEDLINE be consulted when planning a bibliographical search related to nursing topics. Although it is best to search as many databases as possible, if cost is an issue the most useful database is MEDLINE; it is free of charge and available simply through an Internet connection. In spite of the attractiveness of CINAHL for nursing research, it should not be the only database consulted. Extending the search to MEDLINE will provide the complement needed to optimize the quality of the search, particularly as nursing research today is increasingly adopting an interdisciplinary approach. Attention should also be paid to other sources of evidence. The option of searching gray literature has been discussed [16 18]. We did not include a gray literature search in this study, due to limitations of time and costs, but we are aware that some instrument validation studies have not been published, or have been published only as hospital internal reports. All three databases change incrementally over time; the details noted here apply to the time of our study. Current information on journals indexed and thesaurus vocabulary and structure is available online (www.cinahl.com, www. ncbi.nlm.nih.gov, and www.embase.com). Such updates may immediately affect the specifics of any given search, but not the general tenor of our findings and recommendations. In conclusion, an effective search of nursing care literature on clinical topics requires the use of specific parameters. A content analysis of the subject headings assigned in each database could determine the reasons for the lack of commonality in the indexing terms assigned. Finally, a combination of CINAHL and MEDLINE is recommended to obtain optimal results, as the differences between the two are striking. Acknowledgments The authors acknowledge and thank Professor Claire A. Hale for her helpful suggestions and comments, Teresa Mas for her advice, and Mirem Fernández for designing and performing the bibliographic search. We are also grateful to Carolyn Newey for editorial assistance with the English text. This project was financed by AETS Ministry of Health no. 01/10089. The authors are members of the Red Temática de Medicina Basada en la Evidencia (FIS G03/090). References [1] Needleman IG. A guide to systematic reviews. J Clin Periodontol 2002;29(Suppl 3):6 9. [2] Magarey JM. Elements of a systematic review. Int J Nurs Pract 2001; 7:376 82. [3] Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D. Methodology and reports of systematic reviews and metaanalyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998;280:278 80. [4] Deville WL, Bezemer PD, Bouter LM. Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol 2000;53:65 9. [5] Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ 2001;322:98 101. [6] Glanville J, Haines M, Auston I. Getting research finding into practice: finding information on clinical effectiveness. BMJ 1998;317:200 3. [7] Brenner SH. CINAHL and MEDLINE: a comparison of indexing practices. Bull Med Libr Assoc 1989;77:366 71. [8] Burnham J, Shearer B. Comparison of CINAHL, EMBASE, and MEDLINE databases for the nurse researcher. Med Ref Serv Q 1993; 12(3):45 57. [9] Brazier H, Begley CM. Selecting a database for literature searches in nursing: MEDLINE or CINAHL? J Adv Nurs 1996;24:868 75. [10] Watson MM, Perrin R. A comparison of CINAHL and MEDLINE CD- ROM in four allied health areas. Bull Med Libr Assoc 1994;82:214 6. [11] Okuma E. Selecting CD-ROM databases for nursing students: a comparison of MEDLINE and Cumulative Index to Nursing and Allied Health Literature (CINAHL). Bull Med Libr Assoc 1994;82:25 9. [12] Coma I, de Dios R, Montcusí C. Fuentes de información para enfermería: comparación entre las bases de datos CINAHL y MEDLINE. Metas 1999;14:21 7. [13] Khan SK, ter Riet G, Glanville J, Sowden AJ, Kleijnen J, editors. Undertaking systematic reviews of research on effectiveness. CRD Report no. 4. 2nd ed.: York, UK: NHS Centre for Reviews and Dissemination, University of York; 2001 [Available at http://www. york.ac.uk/inst/crd/report4.htm] [14] Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37 46. [15] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159 74. [16] Conn VS, Valentine JC, Cooper HM, Rantz MJ. Grey literature in meta-analyses. Nurs Res 2003;52:256 61. [17] Egger M, Smith GD. Bias in location and selection of studies. BMJ 1998;316:61 6. [18] Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts: a meta-analysis. JAMA 1994;272: 158 62.