Into the Abyss: The Tale of Curriculum Evaluation to Improve Standardized Test Scores Mary Ellen Glasgow, PhD, RN, ANEF, FAAN Dean and Professor Golden Triangle
Golden Triangle at Night City of Bridges
City of Champions Duquesne University
1. Explore a school s culture, curriculum, and student performance as a new dean 2. Address data-based curriculum changes 3. Examine a School of Nursing s (SON) Curricular Evaluation Plan 4. Analyze and evaluate the use of standardized testing to improve NCLEX-RN scores Objectives Into the Abyss
Abyss a deep or seemingly bottomless chasm: a rope led down into the dark abyss bottomless pit a wide or profound difference between people; a gulf: the regions of hell conceived of as a bottomless pit: Satan's dark abyss a catastrophic situation seen as likely to occur: teetering on the edge of the abyss of a total political wipeout Oxford Dictionaries My Story I was a faculty member, Chair, and Associate Dean at Drexel University for many years Drexel NCLEX-RN Pass Rate 98.7% from 2002-2011 for 2,173 BSN graduates I became the Dean at Duquesne in August 2012 Duquesne had a communityconcept based curriculum Duquesne NCLEX-RN Pass Rate 80.9% from 2007-2012 for 545 BSN graduates
An Urgent Situation The PA State Board of Nursing required an 80% first-time NCLEX-RN pass rate beginning on October 1, 2010. The National Council of State Boards of Nursing increased the NCLEX-RN in difficulty on April 1, 2013. The First 90 Days The Dean s 90 Day Assessment Whether challenged with taking on a startup, a turn around (both very different), or inheriting a high-performing unit, a new leader s success or failure is determined within the first 90 days on the job.
90-Day Assessment Interviewed 34 Faculty 17 Staff 10 Chief Nursing Officers 50 students attended open office hours and students forums 6 parents called or made appointments 5 Duquesne senior administrators Alumni Board Questions to Faculty What are the biggest challenges the SON is facing or will face in the future? Why is the SON facing (or going to face) these challenges? What are the most promising unexploited opportunities for growth in terms of teaching/programmatic development, research/scholarship, service, and reputation? What needs to happen for the SON to take advantage of these opportunities? If you were me, where would you focus your attention?
Questions to Staff What are the biggest challenges the SON is facing or will face in the future? Why is the SON facing (or going to face) these challenges? What are the most promising unexploited opportunities for growth in terms of student services, academic operations, and public relations? What needs to happen for the SON to take advantage of these opportunities? If you were me, where would you focus your attention? Feedback from Faculty and Staff Multiple faculty in BSN courses (e.g. 3 faculty in a 3-credit course) Need for more coordination and communication Difficult for students to adjust to several faculty in one course Who is following up on concerns? Fixing problems as they arise is an issue Be empowered to make changes and not be bystanders Too much community and no specialty courses They do not learn what is on NCLEX-RN Prepare students for their first job
Feedback from BSN Students First two years are not challenging, junior year is very challenging Too much community - we sit around and take blood pressures and could not practice nursing skills Community facility is not clear on the role of the nursing student Dissatisfaction with the clinical aspects of the program Stuck in the same clinical site for one to two years Feedback from BSN Students Lack of exposure to pediatrics and OB Would not recommend program to students who want to be in pediatrics, OB, or critical care Older adult community visits works in theory but not in practice very time consuming for what you learn Concerned about preparation for NCLEX
Feedback from Chief Nursing Officers Low NCLEX Scores are a concern I had 4 Duquesne grads fail NCLEX last year The NCLEX scores dropped when Duquesne went to the community curriculum Duquesne students are very nice and respectful Duquesne grads have a harder time acclimating to practice Strong foundation in pathophysiology Feedback from Parents They did not prepare my daughter for practice or the NCLEX. (Senior parent) My child was in a school all last semester and now is walking around neighborhoods. I sent her to Duquesne to be a nurse, not a social worker. (Sophomore parent) Has not had proper clinical experiences.. She does not feel prepared to work in a hospital. (Junior parent)
Findings NCLEX and BSN curriculum revision are priorities Academic Programs need leadership Low to moderate student satisfaction related to BSN community curriculum students remain in the same neighborhood No clinical specialty rotations (OB, pediatrics, psychiatric-mental health, gerontology, critically ill) Organize curriculum in a manner that students can digest and apply information in the clinical area Cultural Assessment Private, Catholic university with 10,000 students Caring, Service-oriented More Midwest than East coast in culture and demeanor Heavy focus on Student Evaluation Surveys SES Average SAT score 1186 but less ambitious with respect to study habits Helicopter parents
Helicopter Parents Initial Curricular Recommendations New Curriculum based on The Future of Nursing Report and The Carnegie Report on Nursing Education One faculty per course specialty Chair who oversees day-to-day operations and logistics who provides leadership and responds to issues in a timely manner Chair who is very involved with students Real life simulation based on Harvard model Quality diverse clinical experiences (focus on clinical preparation, clinical teaching, and clinical evaluation)
Organizational Leadership Clarity Processes and procedures for a mid-size SON Chain of command Real course accountability Looking at the Big Picture Use of metrics to ensure greater accountability Course Chairs (Each faculty member owns a course or set of courses to oversee quality) How does a nursing program actually Change and Improve their long-standing clinical experiences?
Improving the Clinical Experience Partner more closely with clinical agencies Host Chief Nursing Officer Think Tank Meetings every year Examine the student s entire clinical experience for quality and diversity Expand greater infusion of technology and simulation such as e-books to enhance point-of-care technology, as well as, and online clinical communication centers Develop innovative adjunct supervision model to mentor, orient, and manage quality for adjunct faculty per term to provide cost-effective, high quality clinical education Online clinical evaluation tools such as ProjectNurse (now ProjectConcert) Increase diversity of clinical experiences The Devil is in the Data
Making Sense of Evaluation Data Issue: Major, Minor, Urgent Faculty leaders and administrators charged with setting priorities Incremental improvements in student performance, faculty productivity, or in external operations may not be sufficient today. Do not ignore anecdotal data, but trend-data is essential to make highimpact decisions. BSN Curriculum Revision Addition of specialty courses (OB, Pediatrics, Psychiatry and Critical Care) Technology Focus (e-books, apps, ipads, ExamSoft, etc.) Ethics NCLEX Task Force (stay convened) HESI Exams incorporated into courses Organizational Structure Change Major Changes
Culture & Courageous Leadership Culture eats strategy over breakfast, said Peter Drucker, and it is only when you fully understand what this means, that you'll be successful as a leader Changing the Culture From process to outcomes. It is kind to be honest Crucial Conversations with faculty, students, & staff Quantitative assignments in major clinical courses Passing grade from C (70) to C (74) to C+ (78) Efforts made to increase student ambition
Additional Refinements Hiring of a NCLEX-RN Remediation Coordinator/Success Coach, Sr. Mary Meyers Focused reviews Live NCLEX Reviews Skills and Simulation Lab sessions Smaller sections for Role Preparation course BSN Curriculum Nursing Courses Freshman Year UPNS 103 Nutrition for Wellness UPNS 108 Freshmen Seminar UPNS 111W Contemporary Professional Nursing UPNS 112 Informational Technology and Nursing Practice UPNS 113 Human Development and Health Promotion across the Lifespan UPNS 114 Service Learning Health in Communities Sophomore Year UPNS 200 Fundamentals of Nursing and Clinical Ethics UPNS 218 Health Assessment UPNS 219 Pharmacology for Nursing Practice I UPNS 323 Adult Health and Illness
BSN Curriculum Nursing Courses Junior Year UPNS 325 Pharmacology for Nursing Practice II UPNS 326 Pathophysiology for Nursing Practice UPNS 337 Nursing for Women and Children s Health UPNS 354 Behavioral and Community Health UPNS 324 Cultural Applications in Clinical Practice UPNS 338 Gerontological Nursing Practice UPNS 339W Genetics in Nursing and Health Senior Year UPNS 400 UPNS 411W UPNS 420 UPNS 423 Critically Ill Adult Nursing Ethics Across the Lifespan Role Preparation I Evidence-Based Approaches to Professional Nursing Practice BSN Curriculum Nursing Courses Senior Year UPNS 422 Role Preparation II UPNS 426 Leadership and Management in Clinical Environments ****** Elective
Curriculum Evaluation Plan Assessment and evaluation are integral to the success of the Duquesne University School of Nursing. With this premise in mind, the Evaluation Committee has endeavored to create a comprehensive evaluation plan that will provide feedback in a number of different areas that will allow for continued improvements to the learning being facilitated within the school. How will changes be made to your Evaluation Plan to better inform your curriculum? Evaluative Purpose The purpose of the evaluation is to show competency on the part of the School of Nursing in the following areas: Student achievement of program outcomes Student achievement of course objectives University conferment of degrees on qualified graduates Promotion of student learning and development
Evaluative Activities on an Annual Basis Program outcomes Individual courses, across all curricula Clinical evaluation Student evaluation surveys Student program exit surveys Faculty peer review Alumni surveys Student affairs evaluations Community focus group and survey results Student grades and standardized test scores Student certification pass rate results Educational technologies efforts and initiatives Justification for Evaluative Activities Using the criteria presented by the Commission on Collegiate Nursing Education (CCNE) and the National Task Force Criteria for Evaluation of Nursing Practitioner Programs (NTF), the Evaluation Committee developed methods of data collection that will allow the faculty to compare expected outcomes to actual outcomes. Action plans will be developed as necessary when specific data points fall short of expected thresholds.
Annual Calendar of Evaluative Activities The evaluation committee coordinates the review of all program outcomes over the course of a four or five year period. By separating the outcomes over the course of multiple years, a more in-depth review of each outcome can be achieved. The outcomes for the programs are scheduled to be evaluated on the SON evaluation calendar. BSN Outcomes Integrates clinical judgment skills when implementing care for individuals, families, groups, and community. Role models ethical, legal, and professional standards into one's professional nursing practice when acting as a moral agent. Displays a caring attitude in all aspects of one's practice. Institutes collaborative efforts for the improvement of care to individuals/families, and communities. Demonstrates the ability to utilize integrated systems analysis for personal and professional navigation of the health care delivery systems. Integrates culturally congruent care in caring for individuals/families of diverse populations.
BSN Outcomes Engages in evidence based practice. Incorporates teaching/learning process into all aspects of one's practice. Participates in ongoing educational activities that enhance professional growth and improve health care. Utilizes evolving information technologies to enhance professional nursing practice. Communicates effectively both verbally and in writing to all individuals and groups. Uses leadership abilities to influence professional nursing practice. Evaluates the interrelationship of nurse competencies and the patient characteristics to patient outcomes. BSN Review Year of Review First Outcome Second Outcome Third Outcome Fourth Outcome 2016/2017 #3 Caring Attitude # 10 Information Technologies #6 Diverse Populations #13 Interrelationship 2017/2018 #4 Collaborative Efforts #5 Integrated Systems Analysis #7 Evidence-Based Practice 2018/2019 #1 Clinical Judgement #8 Teaching/Learning #12 Leadership 2019/2020 #2 Moral Agent #9 Ongoing Education #11 Communication
Collection of Data Timeline October Program Committees review the outcomes to be evaluated and assign responsibilities for review to members of the committee December Program Committee members review the data points they intend to use for evaluation and determine resources for review. January March Review of outcomes based on defined data points conducted by members of the program committee March Program outcome report is reviewed and recommendations based on the results are recorded. Final results are submitted to the Evaluation Committee for review April August Action plan based on the results is developed with implementation goals for next academic year Administrative Course Reviews The administrative course review is conducted based on the evaluation calendar to review the following data points: Information contained in the course syllabus Evaluation of student evaluation surveys Evaluation of instructor course review Analysis of two key learning outcomes from the course Review of specific pedagogies used in instruction
Semester of Review Fall 2016 Spring 2017 Fall 2017 BSN Course Review First Course Second Course Third Course Fourth Course Fifth Course UPNS 326: Patho (SD) UPNS 108: Freshman Seminar (BSN) UPNS 103: Nutrition (BSN) UPNS 124: Cult. Issues (SD) UPNS 426: Leadership (BSN) UPNS 200: Fundamentals (SD) UPNS 111: Contemporary Prof. (SD) UPNS 423: EBP (RN- BSN) UPNS 400: Critically Ill Adult (BSN) UPNS 325: Pharmacology II (BSN) UPNS 337: OB/Peds (SD) UPNS 420: Role Prep I (BSN) UPNS 327: Advocacy in Older Adults (RN-BSN) UPNS 119: Contemporary Pro. + Tech : Info Tech (SD) Spring 2018 UPNS 218: Health Assessment (BSN) UPNS 422: Role Prep II (BSN) UPNS 309: Pharmacology (SD) UPNS 323: Adult Health (SD) Summer 2018 UPNS 360: Perioperative Nurs. (BSN) Fall 2018 UPNS 222: Exam Contem. Nurse (RN- BSN) UPNS 430: Gerontology (SD) UPNS 339W: Genetics (SD) UPNS 411: Ethics (BSN) UPNS 112: Info. Tech (BSN) UPNS 114: Service Learning (BSN) Spring 2019 UPNS 324: Cultural App (BSN) UPNS 354: Behavioral and Comm. Health (SD) UPNS 113: Human Dev. (BSN) UPNS 223: Pop. Based Health (RN-BSN) Summer 2019 UPNS 428: Nursing Leadership (RN-BSN) Evaluation Clinical Evaluation Surveys Review of Student Evaluation Surveys (SES) Faculty Peer Review Review of Program Exit Surveys Student Affairs Evaluation Community Focus Groups
Focus Groups Calendar Academic Year Location 1 Location 2 Location 3 Location 4 Location 5 Location 6 2016-2017 UPMC Shady Side Canonsburg Hospital (DNP) Heritage Valley Health System (DNP) UPMC McKeesport (DNP) UPMC Presbyterian UPMC East 2017-2018 UPMC Magee (2 nd Attempt) VA (2 nd Attempt) UPMC McKeesport (2 nd Attempt) UPMC St. Margaret s Jefferson Regional 2018-2019 West Penn UPMC Mercy St. Clair Hospital UPMC Children s AGH 2019-2020 UPMC East Western Psych Institute and Clinic Forbes Regional Hospital UPMC Passavant Review of Student HESI Results The Commission on Collegiate Nursing Education (CCNE) require schools of nursing to demonstrate a systematic program evaluation. HESI Summary Reports provide content area scores that can be used to evaluate curricular strengths and weaknesses.
HESI Schedule Sophomores Class of 2020 Fall 2017 Fundamentals V-1 Final Exam UPNS 200 Spring 2018 Med-Surg Custom (75 items) Final Exam - UPNS 323 Fall 2018 Custom Peds/OB (75 items) Final Exam UPNS 337 Custom Behavioral/Community (50 Items) Final Exam UPNS 354 Pharmacology V-1 Final Exam UPNS 219 Spring 2018 Fundamentals V 2 10% Gero Grade UPNS 338 Fall 2019 Critical Care Final Exam UPNS 400 Med Surg V1 Role Prep I UPNS 420 Med- Surg V2 Role Prep I - UPNS 420 Spring 2020 Diagnostic HESI Role Prep II - UPNS 422 Pharmacology V2 Role Prep II - UPNS 422 Hesi Exit Role Prep II - UPNS 422 Hesi Exit Role Prep II - UPNS 422 Lx & Management Final Exam UPNS 426 Review of Certification Pass Rates An acceptable first-time pass rate for a university In Pennsylvania is 80%. The reporting period for this rate is from October 1 st of any given year until September 30 th of the following year. As part of committee meetings, faculty will review the pass rate and make recommendations as necessary.
The Annual Evaluation Report is released at the end of the academic year that includes: Year-to Year outcome comparisons Proposed modifications and goals The final report is reviewed by faculty as part of the first committee meetings of the year. Final Reporting
Standardized Testing Debate Standardized testing that accurately predicts student success while identifying weak content areas has become commonplace (Barton, Willson, Langford, & Schreiner, 2014, p.68). Program-exit testing characterized as High-Stake testing is pejorative. Competency exam or high competency exam is a more accurate description and less inflammatory. Standardized tests scores are generally more stable (reliable) and have higher construct validity that teaching-made exams. Multiple attempts are associated with mastery learning Standardized Testing Debate State Boards of Nursing have regulations with respect to school s first-time pass rates. (The PA State Board of Nursing required an 80% first-time NCLEX-RN pass rate or the school is placed on probation). The cost of NCLEX-RN failure to the student, school, hospital/clinic, and society When knowledge is low, medical errors tend to be higher in new graduates (Failure to Rescue)
Supportive Literature, Pro-Testing Barton, L., Willson, P., Langford, R., & Schreiner, B. (2014). Standardized Predictive Testing: Practices, Policies, and Outcomes. Administrative Issues Journal: Education, Practice, and Research, 4(2), 68-76. Nibert, A., & Morrison, S. HESI Testing (2014). A History of Evidence-Based Research. Journal of Professional Nursing, 29(2), S2-S4. doi:10.1016/j.profnurs.2012.06.004 Young, A. & Willson, P. (2012). Predicting NCLEX-RN Success: The Seventh Validity Study HESI Exit Exam. CIN: Computers, Informatics, Nursing, 30(1), 55-60. Zweighaft, E. L. Impact of HESI Specialty Exams: The Ninth HESI Exit Exam Validity Study. Journal of Professional Nursing, 29(2), S10-S16. doi:10.1016/j.profnurs.2012.06.011 The Critics Spurlock, D.R., & Hanks, C.E. (2004). Establishing progression policies with the HESI Exit Examination: A review of the evidence. Journal of Nursing Education, 43, 539-545. Spurlock, D.R. (2006). Do no harm: Progression policies and high-stakes testing in nursing education. Journal of Nursing Education, 45, 297-302. Spurlock, D.R., & Hunt, L.A. (2008). A study of the utility of the HESI Exit Exam in Predicting NCLEX-RN outcomes. Journal of Nursing Education, 47(4), 157-166. Spurlock, D. R. (2013). The Promise and Peril of High-Stakes Tests in Nursing Education. Journal of Nursing Regulation, 4(1), 4 8. National League for Nursing High Stakes Testing Statement There is no debate about the need to protect the public through standardized evaluation measures of nursing competence and there is a clear understanding that licensure exams are themselves high-stakes for students, faculty, and schools of nursing. It is the prevalent use of standardized tests to block graduation or in some other way deny eligibility to take the licensing exam that is most concerning to the NLN. Select State Boards of Nursing
The Debate: Refuting the Findings from Spurlock and et al. At the Post-Secondary Levels, professional schools, are different than K- 12 standardized testing in fundamental ways: By matriculating at a university, you are consenting to the rules for obtaining the degree. If you have been informed of the program policies, Have been given opportunities to learn the content, Have been provided with adequate resources to learn the content, the test content and course content are aligned, and afforded multiple opportunities to pass the test, the organization is not in violation of the basic standards. K-12 standardized test system has no core opt-in or opt-out system. And the tests are used to evaluate the teachers mainly, not the students. The Debate Continues The author s 2006 focus is on the student and not the future patient. The do no harm phrase comes from medical physicians (Inman or Sydenham) and not the Hippocratic Oath (Smith, 2005). The focus of do no harm was not designed for students taking a test, but for patients. Assessing students in health professions programs which has a focus on minimum competency so that patients do not die thus do not harm (to the patients) Nursing is a high-stakes profession! Faculty set the competency score bar that supports your respective academic program outcomes.
An Underlying Factor Given that some graduates can practice before taking the NCLEX-RN, and do not have to take NCLEX-RN for theoretically up to one year (in my state), Do we want graduate nurses who have not retained and synthesized nursing knowledge for whatever reason caring for patients? The Fundamental Question: How does one improve standardized test scores? There is no single answer to address this complex issue of improving standardized test scores, namely, NCLEX-RN. There are different curricula, students (aptitude versus effort versus time), faculty, educational resources, etc. Faculty need to evaluate their own data and make recommendations based on that data.
Combine four nursing courses to two courses Consider Spanish for Health Care Professionals and Conversational Spanish Telehealth Consider innovative nursing electives such as BME/BSN MakerSpace, Creative Arts and Health, Honors Projects Futures Plan What does society want, expect, or deserve from academic nursing programs? Safe practice first at the core of everything Ethical practice Communication skills Expert clinical reasoning Technology competencies What else does your school particularly value? Final Points
Summary Curriculum evaluation is an essential phase of curriculum development. Through evaluation a faculty discovers whether a curriculum is fulfilling its purpose and whether students are actually learning and graduating with the requisite knowledge, clinical reasoning skills, psychomotor skills, communication skills and the necessary ethics/values to practice SAFELY. Questions
References Alameida, M. Prive, A.., Davis, H., Landry, L., Renwanz-Boyle, C., & Dunham, M. P. (2011). Predicting NCLEX-RN Success in a Diverse Student Population. Journal of Nursing Education, 50(5), 261-267. Barton, L., Willson, P., Langford, R., & Schreiner, B. (2014). Standardized Predictive Testing: Practices, Policies, and Outcomes. Administrative Issues Journal: Education, Practice, and Research, 4(2), 68-76. Carr, S. M. (2011). NCLEX-RN Pass Rate Peril: One School's Journey Through Curriculum Revision, Standardized Testing, and Attitudinal Change. Nursing Education Perspectives, 32(6), 384-388. Davis, P., Grinnell, N., & Niemer, N. (2013). Laying a Foundation for Evaluating Curricular Performance: Tools of the Trade. Journal of Nursing Education, 52(12), 671-679. Giddens, J. F., & Morton, N. (2010). REPORT CARD: An Evaluation of a Concept-Based Curriculum. Nursing Education Perspectives, 31(6), 372-377. Hyland, J. R. (2012). Building on the evidence: Interventions promoting NCLEX success. Open Journal of Nursing, 2(3), 231-238. References Johnson, T., Sanderson, B., Wang, C.-h., & Parker, F. (2017). Factors Associated With First-Time NCLEX-RN Success: A Descriptive Research Study. Journal of Nursing Education, 56(9), 542-545. Koestler, D. L. (2015). Improving NCLEX-RN First-Time Pass Rates with a Balanced Curriculum. Nursing Education Perspectives, 36(1), 55-57. Langford, R., & Young, A. Predicting NCLEX-RN Success With the HESI Exit Exam: Eighth Validity Study. Journal of Professional Nursing, 29(2), S5-S9. McCarthy, M., Harris, D., & Tracz, S. (2014). Academic and Nursing Aptitude and the NCLEX-RN in Baccalaureate Programs. Journal of Nursing Education, 53(3), 151-159. Nibert, A., & Morrison, S. HESI Testing (2014). A History of Evidence-Based Research. Journal of Professional Nursing, 29(2), S2-S4.
References Schooley, A. & Kuhn, J. (2013). Early Indicators of NCLEX-RN Performance. Journal of Nursing Education, 52(9), 539-542. Sosa, M.-E., & Sethares, K. A. (2015). An Integrative Review of the Use and Outcomes of HESI Testing in Baccalaureate Nursing Programs. Nursing Education Perspectives, 36(4), 237-243. Young, A., Rose, G., & Willson, P. Online Case Studies: HESI Exit Exam Scores and NCLEX-RN Outcomes. Journal of Professional Nursing, 29(2), S17-S21. Young, A. & Willson, P. (2012). Predicting NCLEX-RN Success: The Seventh Validity Study HESI Exit Exam. CIN: Computers, Informatics, Nursing, 30(1), 55-60. Zweighaft, E. L. Impact of HESI Specialty Exams: The Ninth HESI Exit Exam Validity Study. Journal of Professional Nursing, 29(2), S10-S16.