Appraising the Evidence Appraising the Evidence Objectives 1. Describe the process for appraising evidence 2. Review types of evidence 3. Review tools for evaluating evidence TCHP Minneapolis VAHCS Kim Wallick, MSN RN Kimberly.wallick@va.gov 2 2 Goal Make a recommendation to change or not to change a practice. 3 4 CODE OF ETHICS FOR NURSES Provision 4: The nurse has authority, accountability, and responsibility for nursing practice: makes decisions and takes action consistent with the obligation to promote health and to provide optimal care. Keeping up with new health information The NIH (National Institutes of health) lists clinical trials of diseases and conditions all over the world Cancer Alzheimer's Infectious disease Neurological disorders ClinicalTrials.gov lists 273,201 research studies with locations in all 50 states and in 204 countries. WE are accountable for our practice, WE are responsible for decisions made and actions taken. American Nurses Association (2016). National Institutes of Health (2016). 5 6 1
review PICO Use PICO to generate search Terms First Read/Edit omay need to review ~50 articles ois this literature worthy of the clinical practice question you have posed? osearch keywords Patient - Adult over 65 y Population - Hospitalized Problem- Cardiac Arrest Intervention- CPR Comparison-Manual Compression vs. Chest Compression System Outcome-Mortality or Survival Rate 7 8 Abstracts oa short statement that describes a larger work okeywords, scope, purpose, or results onot a review oshould not evaluate the study omay indicate relevance, but a personal perspective may be included odon t over-rely on abstracts. APPRAISE Assess Evaluate Judge Rate Review Gauge Value Consider Size up 9 10 Appraising the Evidence APPRAISING Was it done well? A summary of the medical literature that uses explicit methods to systematically search, critically appraise and synthesize the literature on a specific topic. What are the study results? Are the results valid? Are the results of this study important? Is it relevant to your patient care goals? Centre for Evidence-Based Medicine Toronto. (2016). 11 12 2
CAVEAT Types of Evidence Not all evidence is created equal The purpose of searching for evidence is to get an answer to a question; use different types of evidence to gain more knowledge. 13 14 Types of Evidence-research Research Meta-analysis Randomized control trial Quasi-experimental Non experimental Qualitative Meta-synthesis Clinical research Research that either directly involves individual people or uses materials of human origin, such as observed behavior, answers to questions or tissue samples, obtained through direct contact with a particular living person that volunteers and agrees to participate in a research study. Hirschfeld, S. MD PHD. (n.d.). 15 16 meta analysis Combines and analyzes quantitative research o Proso Is treatment/procedure useful for patients? o Is there enough clinical impact to justify the cost? o Saves time by reading 1 article of combined results o Conso Potential for bias o May have lack of understanding the basic issue o Evaluator may overstate the results or misunderstand the variables Randomized Controlled Trials (RCT) Strongest true experimental design o Proso Strongest research evidence using traditional scientific method o All participants are broadly identical o Supports cause and effect relationships o Subjects randomly assigned to control or experimental groups o Key requirements: random, manipulation, control o Conso Cost paid participants, experimental medications o Ethical consideration withholding a treatment with a known benefit. 18 17 3
Quasi Experimental Study Evaluates the benefits of interventions o Proso Pre & post-intervention studies o Causal relationships between intervention and outcome o Used when RTC is not feasible, ethical, or practical o Conso Not considered a TRUE experiment o No randomization o Results may not imply cause and effect; question if there are alternative explanations for the results. 19 Non Experimental Study Most published studies related to health care questions are non-experimental. o Proso Descriptive designs that explain the uniqueness of events o Systematic review of data related to past occurrences o Interrelationship between variables o Prospective, retrospective o Conso No randomization o No manipulation (intervention/treatment) o Little control 20 Qualitative Study Collection of data in non-numeric form o Feelings, not numbers o Perceptions and opinions o Exploratory ; used as a starting point for questions, research o Not random o No manipulation o Little control o Examples: personal interviews, surveys, focus groups meta synthesis o Technique that synthesizes and analyzes qualitative research. o Identifies key concepts o Interprets and translates findings Qualitative and quantitative research designs are often used together 21 22 Types of Evidence Non-research Non Research o Systematic review of research evidence o Clinical Practice Guidelines o Expert opinion o Human & organizational experience o Practitioner experience & expertise o Patient /family values & preferences Systematic Review o The basis for evidence-based medicine. o Summarizes research evidence related to a specific question. o Shows which treatments and prevention methods have been proven to work and what remains unknown. PubMed Health. (2015). 24 23 4
Clinical Practice Guidelines o A form of systematically developed statements that can help practitioners and patients make decisions about care. o These are developed by a group of experts who combine evidence from three sources: Research findings Clinician experts Patient preferences Expert Opinion This evidence can be found in: o Published commentaries o Position statements o Case reports o Letters to the editor o Written or verbal communication with a known expert Agency for Healthcare Research and Quality. (2016) 25 26 Organizational Experience This evidence can be found within the organization or from published reports from other organizations. o Quality improvement reports o Financial data o Program evaluations o Root Cause Analysis (RCA) Patient Experience Evidence but not research: o Values, beliefs, experiences, resources, family, culture, religion o Patient s right to be engaged, ask questions, make informed choices 27 28 tools Rating tools Levels of Evidence Appraise the evidence for strength & quality relative to the PICO question. o This consists of critiquing, leveling, & synthesizing the evidence. o The process is applied to both research and non-research evidence. o Provides a structured way to decide which available evidence to use. 3 0 29 5
Leveling Evidence Types of Rating Scales o Johns Hopkins Rating Scale uses levels I - V Level I is the strongest evidence RCT or Meta-analysis Level V is the least convincing evidence Expert opinion o AACN: Critical Care Nursing Levels A-E; M = manufacturer s information o Joanna Briggs More in-depth several levels and subcategories John Hopkins Strength of evidence Level I Experimental study/randomized controlled trial (RCT) or meta analysis of RCT Level II Quasi-experimental study Level III Non-experimental study, qualitative study, or meta-synthesis. Level IV Opinion of nationally recognized experts based on research evidence or expert consensus panel (systematic review, clinical practice guidelines) Level V Opinion of individual expert based on non-research evidence. (Includes case studies; literature review; organizational experience e.g., quality improvement and financial data; clinical expertise, or personal experience) Newhouse R, Dearholt S, Poe S, Pugh LC, White K. The Johns Hopkins Nursing Evidence-based Practice Rating Scale. 2005. 31 32 John Hopkins quality of evidence A High Research: Consistent results with sufficient sample size, adequate control, and definitive conclusions; consistent recommendations based on extensive literature review that includes thoughtful reference to scientific evidence. Summative reviews: Well-defined, reproducible search strategies; consistent results with sufficient numbers of well defined studies; criteria-based evaluation of overall scientific strength and quality of included studies; definitive conclusions. Organizational: Well-defined methods using a rigorous approach; consistent results with sufficient sample size; use of reliable and valid measures Expert Opinion: Expertise is clearly evident 33 John Hopkins quality of evidence B Good Research: Reasonably consistent results, sufficient sample size, some control, with fairly definitive conclusions; reasonably consistent recommendations based on fairly comprehensive literature review that includes some reference to scientific evidence Summative reviews: Reasonably thorough and appropriate search; reasonably consistent results with sufficient numbers of well defined studies; evaluation of strengths and limitations of included studies; fairly definitive conclusions. Organizational: Well-defined methods; reasonably consistent results with sufficient numbers; use of reliable and valid measures; reasonably consistent recommendations Expert Opinion: Expertise appears to be credible. 34 John Hopkins quality of evidence C Low quality or major flaws Research: Little evidence with inconsistent results, insufficient sample size, conclusions cannot be drawn Summative reviews: Undefined, poorly defined, or limited search strategies; insufficient evidence with inconsistent results; conclusions cannot be drawn Organizational : Undefined, or poorly defined methods; insufficient sample size; inconsistent results; undefined, poorly defined or measures that lack adequate reliability or validity Expert Opinion: Expertise is not discernable or is dubious. Critically Appraise the Literature Second step in the process o Critique research using a checklist or tool. o Critiquing the literature is the evaluation part of the process. o Various tools exist 35 36 6
APPRAISING THE EVIDENCE There are three elements to measure when appraising any research evidence Are the study findings valid? What are the results and are they important? Will the results help in caring for patients? Fineout-Overholt, E., Baldwin, C. Greenberg, E. (2005). Using Evidence-Based Practice for Clinical Decision Making in the Holistic Nursing Process. Retrieved from http://www.ahna.org/research/web-library/evidence-based-practice. 37 APPRAISING THE EVIDENCE o Validity Authenticity of the conclusions -results were based on sound scientific methods? o Reliability Are results great enough statistically and clinically to make a difference when applied to practice? Total number of participants in the study (N=x) Number of participants in each comparison group The difference or lack of difference between the groups o Applicability Are the results of the study appropriate for the particular patient or group of patients described in the PICO question? 38 JHNEBP Appraisal Tools Ten Questions Tool for Literature Evaluation Johns Hopkins Nursing Evidence-Based Practice Research Evidence Appraisal Johns Hopkins Nursing Evidence-Based Practice Non-Research Evidence Appraisal The tools are available in Johns Hopkins Nursing Evidence-Based Practice Model and Guidelines What are these questions really asking? The use of rating tools to determine the strength and quality of the evidence provides a structured way to determine which of the available evidence to use. A first step in determining a study s applicability to a clinical problem Adapted from Beyea, SC & Nicoll, L.H. (1997) Newhouse, et al.,2007 39 40 Synthesize the Evidence Final step in the process Combining all of your literature appraisals Side by side appraisal of the literature. Provides summary of your literature Decision making, discussion needed, recommendations. An example of the HCMC Review Matrix. Additional tools are available on the HCMC nursing web site to assist with this process. HCMC Review Matrix for ( subject here ) Authors, Title, Journal Evidence Level: Quality Rating: Evidence Level: Quality Rating:. Evidence Level: Quality Rating: Year Purpose Question Type of Study/Design Sample Findings Use in Practice Comments Evidence Level: Quality Rating: 41 42 7
The Final Decision Is there enough evidence to change our practice based on clinical relevance, scientific merit, and implementation potential? Strong compelling evidence Translation of Evidence into Practice Translation is the primary reason for having conducted a review of evidence. Good and consistent evidence; consider pilot testing change or a research study Good but conflicting evidence; no indication for practice change; consider research study Little or no evidence; no indication for practice change; consider research study 43 Recommendations from the evidence direct translation. 44 Some Studies That I Like To Quote Questions? https://www.youtube.com/watch?v=ij8bpx8iing 46 References Agency for Healthcare Research and Quality. (2016) National Guideline Clearinghouse. Retrieved fromhttp://www.guideline.gov/ American Nurses Association. (2016). Code of Ethics for Nurses with Interpretive Statements. Retrieved from http://www.nursingworld.org/documentvault/ethics-1/code-of-ethics-for- Nurses.html Centre for Evidence-Based Medicine Toronto. (2016). Are the results of this study valid? Retrieved from http://www.cebm.utoronto.ca/practise/ca/therapysr/ Clinicaltrials.gov.(2016). Retrieved from https://clinicaltrials.gov/ Fineout-Overholt, E., Baldwin, C. Greenberg, E. (2005). Using Evidence-Based Practice for Clinical Decision Making in the Holistic Nursing Process. Retrieved from http://www.ahna.org/research/web-library/evidence-based-practice. References Hirschfeld, S. MD PHD. (n.d.). Clinical Research Definitions and Procedures. U.S. Department of Health and Human Services. National Institutes of Health. Retrieved from https://www.nichd.nih.gov/health/clinicalresearch/clinicalresearchers/references/documents/research_definitions_procedure.pdf National Institutes of Health. (2016). NIH Clinical Research Trials and You. Retrieved from https://www.nih.gov/health-information/nih-clinical-research-trialsyou Newhouse, R.P., Dearholt, S.L., Poe, S.S., Pugh, L.C., White, K.M. (2007) Johns Hopkins Nursing Evidence-Based Practice Model and Guidelines. Sigma Theta Tau International. Indianapolis: ID PubMed Health. (2015). About Clinical Effectiveness Research. Retrieved from http://www.ncbi.nlm.nih.gov/pubmedhealth/aboutcer 47 48 8