414..418 International Journal of Nursing Practice 2010; 16: 414 418 TIPS AND TRICKS Critical appraisal of systematic reviewsijn_1863 Dónal P O Mathúna PhD Senior Lecturer in Ethics, Decision-Making and Evidence, School of Nursing, Dublin City University, Glasnevin, Dublin, Ireland The aim of this Tips and Tricks column is to help readers of the CNCF Supplement get the most out of systematic reviews. The first column examined what systematic reviews are and how they differ from other types of reviews. 1 This column will focus on the critical appraisal of systematic reviews. Readers are welcome to suggest ideas for future columns by emailing topic proposals to the column author. Systematic reviews are found at the top of various hierarchies of evidence. 2 Different types of research questions require different types of study design. The Cochrane Collaboration has chosen to focus on systematic reviews of treatments and interventions. At the same time, systematic reviews of diagnostic studies are being developed, and some qualitative data will be included in reviews. However, interventions remain the focus, including pharmaceutical, educational, organizational and others. The best study design to evaluate the effectiveness of an intervention or treatment is the randomized controlled trial (RCT). This column will focus on critical appraisal of systematic reviews of RCTs, although the general principles apply to other reviews. WHAT IT IS Although we would like to think that all published research is of the highest quality, this is not always the case. Systematic reviews themselves can be of varying quality. 3 Clinicians and users of evidence must therefore evaluate the quality of systematic reviews before applying their results to practice or policy development. Critical appraisal is the process of carefully and transparently Correspondence: Dónal P. O Mathúna, School of Nursing, Dublin City University, Glasnevin, Dublin 9, Ireland. Email: donal.omathuna@ dcu.ie evaluating the quality of published research reports and their relevance to a particular context. 4 Critical appraisal is both central to the systematic reviewing process and required to assess the quality and applicability of systematic reviews themselves. The critical appraisal process involves asking three types of questions: 5 1. Are the results of the study valid? 2. What are the results? 3. Will the results help me in caring for my patients? The first question focuses on whether the study was undertaken in a way that its results are more likely to be accurate and valid. Inappropriate methods are sometimes used, such as when a randomized trial assigns people to groups by a method that is not truly random (such as allocating people by their date of birth or day of admission). Research methods should be chosen that minimize the risk of bias in conducting studies. Bias can be introduced into systematic reviews, and identifying potential bias is part of their critical appraisal. One goal of critical appraisal is to identify ways in which the findings might be due to the way the systematic review was designed and conducted, rather than reflecting the actual effectiveness or harmfulness of the intervention. No study will be totally free of bias, but the extent to which it minimizes bias is called its internal validity. 4 The second question addresses the findings of the study and is only worth considering if the first question has been answered in the affirmative. If the systematic review was not conducted with valid methods, and was hopelessly biased, the results are not worth considering. The results will often be presented as odds ratios, effect sizes or numbers needed to treat. All of these will be examined in detail in future columns. The third question addresses the issue of clinical relevance or generalizability. A practitioner might be doi:10.1111/j.1440-172x.2010.01863.x
Critical appraisal of systematic reviews 415 considering the suitability of an intervention for patients in the community and find a systematic review that included only studies with nursing home residents. The critical appraisal process includes determining whether the review s results are applicable to those whom the reader cares for. This stage also examines whether the systematic review included all important outcomes. A systematic review might have concluded that a particular intervention was effective but failed to collect data on adverse effects. Information on many types of outcomes might be sought by clinicians and patients, but might not be available in a systematic review. CHECKLISTS Because systematic reviews and meta-analyses are playing an increasingly bigger role in evidence-based practice, their quality is being examined more carefully. Studies in the 1980s found that many systematic reviews were reported poorly. The situation had not improved greatly by 1996, when reviewers met to develop guidelines to improve the situation. This led to the QUOROM Statement (QUality Of Reporting Of Meta-analyses), published in 1999. 6 This provided 18 items that should be described in meta-analysis reports to minimize bias. Although these headings were designed for those conducting meta-analyses, they provided a detailed checklist for those critically appraising a meta-analysis. These guidelines were completely revised and published in 2009 as the PRISMA Statement (Preferred Reporting Items for Systematic reviews and Meta-Analyses). 3 This is a 27-item checklist, which can be downloaded with the published article. Given the level of detail in the PRISMA Statement, the 10-item Critical Appraisal Skills Programme checklist is likely to remain popular. 7 This condensed checklist has advantages for those in clinical practical who want to carry out a rapid appraisal of a systematic review. The framework for this checklist has recently been revised to give the 11-question AMSTAR tool (Assessment of Multiple Sys- TemAtic Reviews). This appraisal tool has been validated, with good interrater reliability, and found to take about 15 min to complete. 8 The basic AMSTAR questions are in Table 1, whereas the instrument elaborates on each. CRITICAL APPRAISAL When a systematic review is first located, it should be clear from the title or the abstract if the report fits the clinical question at hand. However, it should be noted Table 1 The basic AMSTAR questions for assessing systematic reviews 8 1. Was an a priori design provided? 2. Was there duplicate study selection and data extraction? 3. Was a comprehensive literature search performed? 4. Was the status of publication (i.e. grey literature) used as an inclusion criterion? 5. Was a list of studies (included and excluded) provided? 6. Were the characteristics of the included studies provided? 7. Was the scientific quality of the included studies assessed and documented? 8. Was the scientific quality of the included studies used appropriately in formulating conclusions? 9. Were the methods used to combine the findings of studies appropriate? 10. Was the likelihood of publication bias assessed? 11. Were potential conflicts of interest included? that sometimes what are called systematic reviews in their titles do not meet the criteria to be systematic reviews, and sometimes systematic reviews are described as integrative reviews. 9 Decisions about whether to use a report require going beyond an examination of the abstract alone. It is also important to distinguish between systematic reviews, meta-analyses and integrative (or narrative) reviews. This was the focus of the previous Tips and Tricks column. 1 A systematic review should begin with a clearly defined question. This is often presented in PICO format, sometimes extended to PICOT. 10 The acronym stands for patient population (P), intervention or issue of interest (I), comparison intervention (C), outcome(s) of interest (O) and (less commonly used) the time for the outcome(s) to be achieved (T). A particular comparison intervention might not be predetermined by the review, or might include a placebo. An example of a PICO question would be, In nursing home residents (P), are behavioral interventions (I) more effective than pharmaceuticals (C) in reducing the incidence of falls (O)? A less-focused question that could lead to problems in conducting the systematic review would be, What are the best ways to reduce falls among the elderly? By writing a PICO-style question to begin with, the search strategy will be more focused and it will be easier to see if the located evidence is more or less clinically relevant.
416 DP O Mathúna The types of studies included in the systematic review must be considered next. A RCT is the best type of study to address the effectiveness of an intervention. However, sometimes RCTs might not be available for an intervention (as can happen with an educational programme or a complementary therapy), might be difficult to conduct (when treatments have a distinctive taste or well-known side-effect), or might be unethical to conduct (as with some surgical interventions). Some systematic reviews will therefore include quasi-randomized studies, cohort studies or other study designs. Systematic reviews that include these study designs raise additional methodological questions, which will not be considered further here, but are discussed in detail in Chapter 13 of the Cochrane Handbook. 11 Critical appraisal should consider whether the included studies address the systematic review s question and if they have the appropriate design. Assuming both of these issues are addressed adequately, the detailed methods of the systematic review should be examined for internal validity. The literature search should be as broad as feasible to ensure all relevant studies are found. The search should include major electronic databases (such as MEDLINE, Embase, CINAHL and the Cochrane Library). The key words and MeSH terms should be stated, and, if feasible, the search strategy itself listed. Depending on the topic, significant journals or conference proceedings might need to be handsearched, and so-called grey literature evaluated. The latter could include unpublished dissertations, government reports or textbooks. The bibliographies of all publications should be examined for further studies. More thorough systematic reviews will examine research registers for ongoing studies and contact prominent researchers or manufacturers for information on other studies. All this should be described in the review. As part of the search, restrictions might be placed on language or years of publication. A balance must be sought between the breadth of searching and the time and resources available. A good systematic review will only restrict its search with a good rationale, not just for convenience. Publication dates should be limited based on when an intervention became available, and are less justified when the search duration is arbitrarily set at, say, 5 years. Similarly, other languages should not be excluded for convenience. After the search strategy identifies potential studies, predetermined eligibility criteria should be used to include or exclude studies. These criteria should be explicitly reported, and how they were applied. It is preferable to have duplicate review (two independent reviewers) to assess each study for inclusion, with a method described for resolving discrepancies. Reasons for excluding studies should also be reported. The whole study selection process should be summarized in a flow diagram. 3 How the data in the included studies were extracted should be described. Extraction sheets will normally need to be developed, and should have been piloted. This stage should also be done by two people working independently. This step might be done with the reviewers blinded to the names and affiliations of the original research authors. This is more important if the reviewers collaborate or compete with any of the researchers included in the review. The systematic review should critically appraise all the included studies. The criteria used should be objective and known to influence study results. In Cochrane reviews, this is reported as the risk of bias. The main criteria are sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting and any other sources of bias. 11 When a number of outcomes are reported, each outcome can be examined separately. A number of summary scales are available for quality or risk of bias (such as the Jadad Scale), but their use is explicitly discouraged in Cochrane reviews for several reasons, including their unreliability. 11 Instead, a more complete discussion of the various risks of bias is expected. Publication bias is one of a number of reporting biases that should be considered in systematic reviews. Several studies have shown that intervention trials with beneficial findings are more likely to be published than those with negative findings. 11 Such positive studies are more likely to be published in English (language bias) and to be published more quickly (time lag bias). More comprehensive searches might help reduce publication bias, but it cannot be eliminated completely. A funnel plot is a method that systematic reviews can use to identify possible publication bias. 12 Symmetry in these plots suggests the absence of such bias, although asymmetry is not always due to publication bias. Critical appraisal of a systematic review should include an assessment of whether the reviewers considered such bias. They should also disclose any conflicts of interest and whether funding was obtained for the review. Critical appraisal should assess whether such factors increased the risk of bias.
Critical appraisal of systematic reviews 417 Systematic reviewers must decide whether to combine the results of some (or all) of the included studies into a meta-analysis. This is justified when the studies are similar but not justified when there is significant heterogeneity. A number of methods are available to access heterogeneity. One rough method is to display the results of the studies in a forest plot (the type of diagram upon which the Cochrane logo is based). Each study gets a box with a vertical line representing the 95% confidence interval (CI). When the CI lines have little overlap, heterogeneity can be estimated as relatively high. Various statistical methods, such as the I 2 test, give a more precise estimate of heterogeneity. 13 In critical appraisal, the important point is that heterogeneity should be examined and reported. The results of the systematic review should be presented clearly and allow the reader to assess their reliability. The characteristics of the included studies should be presented, at least in table form. For each of the outcomes considered by the reviewers, summary data should be presented for each intervention group in each study, and effect estimates with CIs. These are often given as odds ratios or effect sizes, and presented in forest plots. These statistical terms are fully described in the Cochrane Handbook, and will be the subject of future columns in this series. 11 If heterogeneity is relatively low, the results will be combined into a meta-analysis. The combining of the results of individual studies provides a more precise estimate of the effect. Meta-analysis will be considered in more detail in a subsequent column. However, a rough appraisal of a meta-analysis can be done by examining the data on the individual studies combined. Clinicians are well suited to appraise whether the patients, inventions, controls and settings were similar enough to warrant combining the results. 12 The final stage in critically appraising a systematic review is to apply the results to practice. The systematic review should provide a general conclusion discussing the implications of the review for practice and further research. However, the review s readers will need to determine if their patients and clinical settings are similar enough to those of the included studies. 5 Other local factors might make it less feasible to implement the findings, such as cost or organizational restraints. Some of the relevant outcomes might not have been included in the review. Even if the results are directly applicable to practice, the patients being cared for might choose not to accept these interventions. The final application of the review is thus dependant on many other factors. THE BOTTOM LINE Systematic reviews and meta-analyses continue to increase in importance in health care. Clinicians use them to remain up-to-date in practice and they are often used to develop clinical practice guidelines. 3 They are increasingly being required by funding agencies to justify research projects. Patients and consumers are using them to become more aware of potential treatment options. At the same time, their quality can vary significantly. Although Cochrane reviews are consistently found to be of higher quality than other systematic reviews, they have room for improvement also. 14 The bottom line is that clinicians need to critically appraise systematic reviews and meta-analyses, just as they should any individual research study. Although the task might seem daunting at first, with practice, it can quickly increase one s confidence in reliably applying evidence to practice. One approach would be to find a systematic review relevant to your area of practice. With a few colleagues, use the full AMSTAR measuring tool from Reference 8 to appraise the review. This will likely lead to invigorating discussions about using evidence in practice and quickly develop everyone s critical appraisal skills. REFERENCES 1 O Mathúna DP. Tips and tricks: The role of systematic reviews. International Journal of Nursing Practice 2010; 16: 205 207. 2 Oxford Centre for Evidence-based Medicine. Levels of evidence. 2009. Available from URL: http:// www.cebm.net/index.aspx?o=1025. Accessed 1 June 2010. 3 Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine 2009; 6: e1000097. 4 Burls A. What is critical appraisal? 2009. Available from URL: http://www.medicine.ox.ac.uk/bandolier/ painres/download/whatis/what_is_critical_appraisal.pdf. Accessed 1 June 2010. 5 O Mathúna DP, Fineout-Overholt E, Johnston L. Critically appraising quantitative evidence. In: Melnyk B, Fineout- Overholt E (eds). Evidence-Based Practice in Nursing and Healthcare: A Guide to Best Practice, 2nd edn. Philadelphia, PA, USA: Lippincott Williams & Wilkins, 2010; 81 134. 6 Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF, for the QUOROM Group. Improving the quality of reporting of meta-analysis of randomized controlled trials: The QUOROM statement. The Lancet 1999; 354: 1896 1900.
418 DP O Mathúna 7 Public Health Resource Unit. 10 questions to help you make sense of reviews. 2006. Available from URL: http:// www.phru.nhs.uk/doc_links/s.reviews%20appraisal% 20Tool.pdf. Accessed 1 June 2010. 8 Shea BJ, Hamel C, Wells GA et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology 2009; 62: 1013 1020. 9 Krainovich-Miller B, Haber J, Yost J, Jacobs SK. Evidencebased practice challenge: Teaching critical appraisal of systematic reviews and clinical practice guidelines to graduate students. The Journal of Nursing Education 2009; 48: 186 195. 10 Stillwell SB, Fineout-Overholt E, Melnyk BM, Williamson KM. Evidence-based practice, step by step: Asking the clinical question: A key step in evidence-based practice. The American Journal of Nursing 2010; 110: 58 61. 11 Higgins JPT, Green S (eds). Cochrane Handbook for Systematic Review of Interventions. Version 5.0.2. Chichester, UK: Wiley, 2009; Available from URL: http://www.cochranehandbook.org. Accessed 1 June 2010. 12 Goodacre S. Critical appraisal for emergency medicine: 6 systematic reviews. Emergency Medicine Journal 2009; 26: 114 116. 13 Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statistics in Medicine 2002; 21: 1539 1558. 14 Delaney A, Bagshaw SM, Ferland A, Laupland K, Manns B, Doig C. The quality of reports of critical care metaanalyses-in the Cochrane Database of Systematic Reviews: An independent appraisal. Critical Care Medicine 2007; 35: 589 594.
Copyright of International Journal of Nursing Practice is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.