Best Practices for Writing and Editing CME Needs Assessments

Best Practices for Writing and Editing CME Needs Assessments Donald Harting, MA, ELS, CHCP Manager, CE Grant Writing and Outcomes National Comprehensive Cancer Network Fort Washington, Pennsylvania Katherine Molnar-Kimber, PhD President KMK Consulting Services of Kimnar Group LLC Worcester, Pennsylvania Nathalie Turner, MS, ELS Senior Grant Developer Medscape Education Newtown, Pennsylvania 1

Learning Objectives Upon completion, participants will be better able to: Write a needs assessment (NA) that reflects current best practices, as determined by a nationwide survey of professionals working in the field of accredited continuing medical education (CME) Discuss the relative importance of various sources of evidence for a NA, along with key criteria for inclusion of entries in the reference list Identify and address key practice barriers commonly encountered by novice writers of NAs 2

What Is a Needs Assessment (NA)? Actual Practice (measure) The Gap (causes) Optimal Practice (standards) Queeney D. Assessing Needs in Continuing Education. 1995. 4

Value of NAs Essential tool for quality improvement [a] Used in program planning [b] Basis for developing learning objectives Justify agenda and content Required for accredited programs [c] [d] a. Queeney D. Assessing Needs in Continuing Education. 1995. b. Moore D. J Contin Educ Health Prof. 1998;18:133-141. c. ACCME website. Accreditation criteria. d. Harting D, et al. AMWA Journal. 2016;31:128-131. 5

Core requirements for activities: criterion 2 The CME activity must address an educational need (knowledge, competence or performance) that underlies the professional practice gaps of that activity's learners The key for compliance is to be able to show ACCME that planning included the identification of a professional practice gap from which needs were identified ACCME website. AMA and ACCME Announce Simplification and Alignment of Credit and Accreditation Systems. 6

Part of a Larger Whole Audience Generation Plan Needs Assessment Value of CME Provider Program Agenda Credit Info Outcomes Measurement Plan Budget & Schedule TYPICAL CME GRANT PROPOSAL

Study Background Variability noted in 2011 pilot study of NAs written for private clients Editor of Journal of Continuing Education in the Health Professions (JCEHP) suggested article with "scholarly" approach Simple descriptive research study planned 8

Annual Series of Surveys Every fall in back-to-school time frame 2014, 2015, 2016, 2017 Mixed methods Quantitative data (from survey) Qualitative data (eg, focus groups, interviews) Survey link distributed via email and social media Everyone who takes survey receives raw results Additional results dissemination via: Live workshops, webinars, newsletter & journal articles, posters 9

2016 Survey Logistics Opened September 16, closed October 8 10 questions (some old, some new) Promoted to investigators' professional networks N=118 survey starts Adjusted N=108, including partial responses 10

2016 Qualitative Method June 2017 focus group in Philadelphia area for in-depth discussion of responses to survey question 4: "When you read a needs assessment written by someone else, what are a few things you quickly check to help you determine whether the writer did a good job? An example might be, 'How old are the references?' but you can write anything you wish. (You may list up to 3 quality indicators below.)" 158 responses from veteran writers were analyzed using the nominal group technique (NGT) American Society for Quality website. Nominal group technique. 11

Preliminary Set of Quality Indicators After 6 rounds of NGT, an expert focus group ranked the following 5 of 24 preliminary quality indicators to be the most essential and measurable. These indicators are listed in descending order: Accuracy [Currency/timeliness/freshness/age of evidence]* Readability [Clear statement/clarity]* Data source *Similar qualities that were condensed into a single line and voted on as a single quality indicator in the 6th round of NGT. Molnar-Kimber et al. AMWA 2017. Poster 12

2016 Quantitative Methods 8 of 10 questions covered demographics and aspects writers encounter writing NAs Analysis Percentage of survey takers Mean (average) Median (middle value, with range) Mode (most frequent response) Rankings (most important to least important for group) 13

2016 Survey Highlights 14

Q1: When you write needs assessments today, do you work as a staff employee or as a freelancer? Type of Employment of Survey Takers Staff employee 38 Freelancer 44 Both roles 7 I am not currently writing NAs 9 I have not written any NAs 3 0 10 20 30 40 50 Percentage of Survey Takers 15

Q2: Approximately how many needs assessments have you written in your entire career and in 2015? 1 to 5 NAs 6 to 25 NAs Career 26 to 50 NAs 2015 >50 NAs 0 10 20 30 40 50 60 Percentage of Survey Takers 16

Q3: Which of the following would your clients or employer consider most essential for inclusion in a first-rate needs assessment? (Choose your top 3.) Most Essential Evidence Clinical practice guidelines 65 Review of medical literature 60 Interview with key opinion leaders 47 Learning outcome data 34 Basic epidemiology of disease state 24 0 10 20 30 40 50 60 70 Percentage of Survey Takers 17

Q5. Please rank the following barriers in terms of their relative importance to your professional practice. Ranking of Barriers to Practice Not enough lead time 7.1 Poor match between gaps & outcomes 7 Out-of-date clinical practice guidelines 6.95 Few published data on topic 6.81 Appearance of bias 5.92 Unclear assignment 5.87 0 1 2 3 4 5 6 7 8 Ranking by Survey Takers 1 = least important 10 = most important 18

Q7: Please tell us a bit more about the way you choose references to cite. Indicate the relevancy of the following characteristics. Most Important Characteristics of Cited Publications Content relates to the assigned topic 80 Peer-reviewed publication vs not 50 Date of publication 41 Content explains basic science 24 Phase of clinical trial 23 Mentions supporter's product 12 0 10 20 30 40 50 60 70 80 90 Percentage of Survey Takers 19

Q8: Properties of Typical NAs Number of References Statistic # of References Median number of references 30 Full range 3, 145 Interquartile range 15, 41 (25%, 75%) Mode (10.3% of survey takers) 25 Other common numbers of references 7.7% survey takers 35 6.4% survey takers 14 5.1% survey takers 12 Molnar-Kimber et al. AMWA 2017 Poster. 20

Q8: Typical Lead Time and Document Length for Assignments Lead Time Statistic Median lead time (range) # of Days 14 (1 to 90) Mode (20%) 14 Other common lead times 14% survey takers 30 9% survey takers 7 Document Length Statistic # of Words Median (range) 2500 (78 to 7850) Mode (8%) 1500 Other common lengths 7% survey takers 3500 5% survey takers 2500 4% survey takers 2000, 3000, & 4000 Molnar-Kimber et al. AMWA 2017. Poster 21

Study Limitations Not a random sample of writers Biased toward members of investigators' professional networks (LinkedIn, Twitter, AMWA, Alliance, etc.) Dinner focus group respondents were survey takers and biased toward residents of Philadelphia area No way to break out NAs written for commercial support versus NAs written for other purposes (eg, academic, hospital, or government work) No way to mathematically calculate a response rate 22

Insights on Best Practices Clinical practice guidelines and a medical literature review were considered the 2 most essential types of evidence. The 3 most relevant characteristics for citing a reference were Alignment with assigned topic, Peer review, and Recent publication date. Median lead time for assignments is 14 days. Preliminary set of 5 quality indicators includes accuracy, timeliness, readability, clarity, and data source. 23

Future Research: 2017 Survey Best way to write NAs for interprofessional education (eg, physicians and nurses and pharmacists)? Use of storytelling devices Narrative Anecdotes Quotations Pictures, graphics, charts Writers' professional memberships/self-identity References: How old is too old? 24

Evidence-Based* Recipes for NAs Basic Better Deluxe *Based on first 2 years of survey data. Recipes are cumulative: better includes basic, and deluxe includes better. 25

Basic NA Ingredients 1. Medical literature review 2. Practitioner survey 3. Reference to clinical practice guidelines 4. Key opinion leader interview(s) 5. Alignment chart with columns labeled "Learning Objective" "Practice Gap" "Desired Outcome" Harting D, et al. AMWA Journal. 2016;31:128-131. 26

Better NA Added ingredients 6. Perspective from a patient or patient advocacy group regarding patient-level gaps 7. Text or chart showing outcomes data* 8. Evaluation reports from participants in previous activities* *Some practitioners consider these to be part of a basic NA. Harting D, et al. AMWA Journal. 2016;31:128-131. 27

Deluxe NA Added ingredients 9. Reference(s) to national healthcare quality standards* 10.Evidence of change measured against a validated quality benchmark *Some practitioners consider these to be part of a basic NA. Published by National Quality Forum, Agency for Healthcare Research and Quality, Patient- Centered Outcomes Research Institute, or similar agency. Harting D, et al. AMWA Journal. 2016;31:128-131. 28

Tips for a Good NA/Gap Analysis Identify the target audience Indicate what isn't being done (knowledge, competence, performance) Describe why it's not being done (evidence) Describe what needs to be done or known Indicate expected outcomes from closing the gap(s) Keep it concise Visovsky C. J Adv Pract Oncol. 2015;6:279-280. 29

Acknowledgments (2016 survey only) Co-investigators Donald Harting and Katherine Molnar-Kimber Focus group on quality measures Clifford Cui, Michelle Dalton, Barry Fiedel, Nathalie Turner Survey beta testers Andrew Bowser, Greselda Butler, Lauren Ero, Nathalie Turner, Ruwaida Vakil, Deb Whippen Promotion and dissemination: Roxane Bollinger, Greselda Butler, John Craine, Victoria White, many officers of AMWA's 20 local chapters 31

Questions? Donald Harting, MA, ELS, CHCP Twitter: @CME_Scout Email: don@hartingcom.com Katherine Molnar-Kimber, PhD Website: www.kmk-how-to-write.com Visit our AMWA poster during this meeting to view more data at your leisure. Best Practices for Writing CME Needs Assessments 2016 Email: KMK997@KMK-How-to-Write.com Nathalie Turner, MS, ELS LinkedIn: linkedin.com/in/nathalie-turner-ms-els-b6724b5 Email: nturner7890@gmail.com 32

Small Group Activity 33

Bloom s Taxonomy 34