Annual Report on Degree Program Assessment of Student Learning 1 University Assessment Committee Office of Academic Assessment Purpose: The purpose of the Annual Report on Degree Program Assessment of Student Learning is to provide information about progress in assessment efforts for each degree program within your academic unit. Only one report is requested of each academic unit, as this report will accommodate multiple degree plans. (You can still submit separate reports if you prefer.) The report will be made available publicly at the Office of Academic Assessment website and will be available to appropriate accrediting agencies. It is recommended that your unit use your assessment report and results to celebrate achievements of student learning as well as to identify potential areas for future curriculum improvement. Please email this completed form as an attachment to d-oaa@jan.ucc.nau.edu. CONTACT INFO: Academic Unit: School of Nursing Date: November 30, 2008 Name: Debera Thomas, DNS, RN, FNP/ANP Title: Dean and Professor Email: debera.thomas@nau.edu Phone: 928-523-2656 NAU Box: 15035 Degree Program(s) reported here: Undergraduate Northern Arizona University School of Nursing Program Evaluation / Assessment Report Annual Report 2007-08 AY Prepared by Dr. Debera Thomas Preparation Assistance by Dr. Susan Rieck and Ms. Donna Sutton Major Accomplishments of the Program for this year: (Listed by CCNE Standards): CCNE Accreditation self study completed; Site visit November 3-5, 2008 Arizona Board of Nursing Approval visit: November 3-4, 2008 Mission, Goals, Outcomes Reviewed by faculty for CCNE accreditation 07-08 (see Annual Report 2007-2008 attached). Resources for Program Managed by the Interim Director and Business Manager within budget. (See attached Annual Report 07-08). Curriculum Developed curricular template to foster curricular improvements. Assessment of Program outcomes was the basis for the template development. (see attached template).
Effectiveness of Program (Question 1-5) 2 Question 1: Summarize your assessment activities during the past year for each degree program. (e.g. faculty discussions, new survey design, data collection, revised assessment plans or learning outcomes, etc.). Assessment Activities**: A. Satisfaction--EBI Exit Survey-Report: Data collected on program satisfaction and ability to achieve intended program outcomes, Survey by Graduating Seniors evaluated against benchmark. B. Critical Thinking Ability--CCTDI critical thinking exit survey: Data collected evaluated against benchmark and national data from similar programs. Survey by Graduating Seniors. Added the HESI AACN category of critical thinking benchmark is a mean score of >850 in this category C. Clinical competency--nclex national licensing exam, report on first time test takers/new graduates. Evaluated against national and state data. D. Clinical competency--hesi achievement exam for exiting seniors, knowledge and application exam similar to national licensing exam, predictive of NCLEX pass rates. HESI specialty exams were initiated (Medical-Surgical Nursing I and II, Fundamentals, Obstetrics/Pediatrics, and Psychiatric/mental health nursing). They were completed on a voluntary basis Fall 2007 and mandatory assessment of students started Spring 2008. E. Teaching/Learning Evaluation criteria previously established were evaluated and changes were made. The outcome is now evaluated using the HESI exit exam teaching questions, SON outcome item on the EBI exit survey, alumni survey and employer survey. G. Leadership/Management Evaluation criteria were developed. Criteria include the NCLEX exam results in this category, HESI leadership and management of care categories, questions related to this category on the EBI exit survey, alumni survey and employer survey E. Outcome Assessment.--Program Evaluation Committee Assessment of individual Program outcomes. Strands evaluated during the 2007-2008 academic year were clinical competency, critical thinking, teaching/learning, and leadership/management. F. Graduation Rates (traditional, fast-track & RN), (see Annual Report 07-08) G. Formal Complaints BS and MS Student complaints evaluated and report generated for provost. ** See Program Evaluation Model at the end of the document. Question 2: Describe specific assessment findings related to the learning outcomes assessed for each degree program, including any pertinent context surrounding the findings. Please include the learning outcomes themselves. (e.g. 77% of seniors performed at the proficient level of competency in problem solving, which is where we aimed to be this year using a new scoring rubric ) Assessment Findings: The results and analysis for the outcomes evaluated this year (critical thinking, teaching/learning, and leadership/management) can be found in the attached tables. A. Satisfaction--EBI Exit Survey-Report: Data are attached in a report labeled EBI Exit Survey for both fall 2007 and Spring 2008. Overall findings indicate that we are nearly
meeting our benchmark of 95% satisfaction in the teaching/learning program outcome strand (Fall 82.7% Spring 95.5%). Our scores in the accountability criterion were (Fall 78.9%; Spring 88.3%) which is lower than our last year s results and lower than our benchmark of 95%. This criterion relates to the student s ability to integrate ethical & legal, political awareness and practice standards. We are addressing these issues in our Health Policy Course as well as integrating content in other courses. We nearly met our benchmark in most of the other areas; communication (Fall 92%; Spring 92.7%), and cultural awareness (Fall 88.3%; Spring 91.1%) strands. We are not meeting our benchmark for the critical thinking (Fall 79.2%; Spring 89.9%) program outcomes strand. Various cohorts of students [Traditional undergraduate, Accelerated/Fast Track, and RN/BSN], had slightly different degrees of satisfaction. Additional results can be viewed in the report. a. On the critical thinking question, one group of students, the RN-BSN Fall 2007 group exceeded the benchmark of 95% at 100%. Our student representative felt that because the students don t get the integration of information required for critical thinking, it does not occur until late in the program at the end of the capstone course. Whereas, the students in the RN-BSN program think at that level from day one. b. Management/leadership Fall 2007 results were 63.8% and 79.3% on each question. The outcome statements in the survey were not exactly the program outcome statements, therefore, there may have been some confusion as to the meaning of the statement. This is being revised Fall 2008. c. Teaching/learning The benchmark was met Spring 2008 (95.5%) and Fall 2007 was 82.7% over all, but 100% in the RN to BSN group. B. Critical Thinking Ability--CCTDI critical thinking exit survey: Data collected are evaluated against benchmark and national data from similar programs. The CCTDI scores and results tables are attached. Overall, for the fall 2007 semester we surpassed our desired outcome of > 40% of students achieving at a score of 320 or greater; 51.7% scored higher. In the Spring 2008 semester we neared our benchmark in that 34.6% scored higher. RN-BSN students met the benchmark for all years, Fall 2007 all BSN tracts met the benchmark. Both traditional groups evaluated met the benchmarks in fall but not in spring (taken the semester they graduate). The HESI critical thinking category was used. Only the fast-track cohorts met the HESI critical thinking category. It was difficult to get data on 1 st time test takers only for the other groups. The committee is developing a plan to have current data pulled when the students first take the exam. C. Clinical competency--nclex national licensing exam, report on first time test takers/new graduates. Results are included in the attached NCLEX results report. The benchmark is an 85% of first-time test takers would pass the exam within the first 12 months of graduation. Overall the first time test-taking pass results have increased to 84.62% over the previous year (82%). However, The students graduating in December 2007 had a pass rate of 90.38% while those graduating in spring of 2008 had a pass rate of 82.76%. Those students who graduated in May 2007 and did not take the test in the first year had a pass rate of 50% and effectively bringing down the average. We need additional data to further evaluate our results; however, additional actions are being considered and will be described under question 4. D. Clinical competency--hesi achievement exam for exiting seniors, knowledge and application exam is similar to national licensing exam and predictive of NCLEX pass rates. Please see the attached clinical competency Outcome Report. The HESI 3
benchmark was not met. The clinical competency report contains recommendations for improvement. E. Outcome Assessment.--Program Evaluation Committee Assessment of individual Program outcomes. Strands evaluated during the 2007-2008 academic year were clinical competency, critical thinking, teaching/learning, and leadership/management. Detailed and final outcome reports are attached for each strand. The assessment strategy can be followed on the BS grids. A goal of the PE committee this year (08-09) is to have the plan completed for evaluation of the graduate program outcomes. The surveys are currently being revised. F. Graduation Rates (traditional, fast-track & RN), (see Annual Report 07-08, page 11) G. Formal Complaints BS and MS Student complaints evaluated and report generated for provost and School of Nursing Executive Director. Complaint log is kept with complaint and resolution and provided to the Provost on request. Only a couple of student submitted complaints. Each was resolved shortly after identified. 4 Question 3: Describe how assessment feedback has been provided to students, faculty, and staff. (e.g. report for faculty, executive summary for the dean, web page for students, alumni newsletter, discussion with students in class or club event, etc.) Assessment Feedback: The Interim Director met with NAU SON students around the state and shared feedback about the following topics. A. Satisfaction--EBI Exit Survey-Report: The School of Nursing Dean and the Program Evaluation Committee are provided outcome data for evaluation. A summary report is emailed to faculty with details as appropriate. B. Critical Thinking Ability--CCTDI critical thinking exit survey: Students are able to see their results on completion of the exam and results are shared through email to the faculty and to the School of Nursing standing committees on curriculum and program evaluation. C. Clinical competency--nclex national licensing exam. Students are able to see their results on completion of the exam. Results are standardized and comparisons can be made by states and cohorts. Again results are shared through email to the faculty and to the School of Nursing standing committees on curriculum and program evaluation. D. Clinical competency--hesi achievement exam. Students are able to see their results on completion of the exam and results are shared through email to the faculty and to the School of Nursing standing committees on curriculum and program evaluation. In addition, student s individual advisors are specifically provided results when student remediation is needed to support their success in passing the exam. E. Outcome Assessment.--Program Evaluation Committee Assessment of individual Program outcomes. Data from the four strands of Critical Thinking, Clinical Competency, Teaching Learning and Leadership/Management were evaluated in 2007-2008. The attached reports were distributed to all faculty and to were sent to curriculum committee for their review and suggestions for improvement in these areas. The results were discussed during a faculty meeting. In addition, all data and reports are posted on the Program Evaluation website for faculty to review and use for committee and school interactions. The intent is to keep the process transparent. There was an undergraduate student and a graduate student on the committee. Feedback from them was important in the reviewing and analyzing of data.
F. Graduation Rates (traditional, fast-track & RN): Rates are reported, tracked and evaluated by the School of Nursing Dean and Student Services, by the AZ State Board of Nursing and by our accrediting agency American Association of Colleges of Nursing. Only if there is a problem with inadequate graduation rates do any of these agencies become involved with the school. G. Formal Complaints BS and MS Student complaints. Results go to the provost and school executive director. Faculty involved in the complaints were provided related information. 5 Question 4: In what ways have you used assessment findings to celebrate student achievements and/or to improve the curriculum this past year? (e.g. prizes to students, hosting student parties, changes to curriculum, student projects, learning goals, assessment strategies, etc.) Using Assessment Findings: A. Satisfaction-- EBI Exit Survey-Report: Data collected on program satisfaction and ability to achieve intended program outcomes is reviewed and discussed with faculty. Suggested program changes are considered and based on outcomes. Some clinical assignment/situations have been changed based on student comments in evaluations. B. / C. / D. Critical Thinking Ability/ Clinical competency-- CCTDI / NCLEX / HESI results are all evaluated to understand the success of the program. Some score decreases in the clinical competency indicators have prompted the school to change the HESI testing strategy. Standardized course testing was implemented during the last AY in response to concerns with lower scores over the preceding year. The intent is to assist the students and faculty to clearly identify levels of learning achievement related to standardized content. In addition, students need the opportunity to practice critical thinking skills in the way in which they will be held accountable for demonstrating critical thinking, ie: on standardized tests that admit them to practice (NCLEX). The HESI specialty exams were implemented Spring 2008. Data were used for improving course content and teaching strategies for identified weaknesses. The Faculty have been trained on how to remediate students who have not passed the HESI exams. The committee will be looking at tracking specific area scores in the mid-curricular exams. E. Outcome Assessment Program-- Evaluation Committee Assessment of the individual program outcomes on clinical competency, critical thinking, leadership/management and teaching/learning has been distributed and curriculum committee is now evaluating and leveling these strands more. In addition, a faculty in-service was held on January 8 th, 2008 on implementing culture in courses. Course coordinators are looking at the culture strand in detail for each course and analyzing for inclusiveness and how it is being implemented across the curriculum. F. Graduation Rates-- (traditional, fast-track & RN) Current program success does not warrant changes. G. Formal Complaints-- In the past this report has not been shared with the Program Evaluation Committee. The administrative team is working to restructure the report so that it is in a format that protects the confidentiality of the students and instructors involved in the complaints, but contains constructive information for program evaluation.
Question 5: Describe any changes to your assessment plans, or any challenges or educational experiences with the assessment process this past year that you would like to share. (Please submit any revised/updated assessment plans to the Office of Academic Assessment along with this report.) Assessment Changes: A. / B. / C. No change anticipated. The Exit survey was revised Fall 2008 so that the outcomes have consistent wording with our current outcomes. The one year alumni survey is sent out each spring. Outcome wording is being changed on this survey as well. A letter was sent to the students prior to the opening of the assessment to aid them in understanding the importance of their feedback. The return on the One Year Alumni survey and the Employer survey continues to be low although it was slightly better in May 2008 than in previous years. D. Clinical competency--hesi achievement exam for exiting seniors: This assessment is expanding to include course content exams. At the end of each of the major nursing courses students will be taking a standardized exam. The exam allows them to demonstrate course competency and to prepare for the exit exam. Getting clean data from the first time test takers only has been a challenge. We are working to improve this process so that our results are accurate. E. Outcome Assessment.--Program Evaluation Committee Assessment of individual Program outcomes for Accountability, Caring, Communication and Culture were evaluated in AY 2006-2007. The outcomes of clinical competency, critical thinking, teaching/learning and leadership/management were evaluated in 2007-2008. Four of the eight strands are evaluated each year; the other four are evaluated in alternate years. This is described in our current evaluation plan. F. / G. No change anticipated. Additional evaluations: H. Clinical competency The one year Alumni/Employer Survey had poor return results for the second year in a row. We sent 102 surveys in May 2008 and received 16 first year alumni and 6 employer surveys. There were only 3 surveys returned as having incorrect addresses. The committee will be looking at ways to educate the graduating students on the importance of completing these surveys. I. Graduate outcomes Data that are collected from preceptor evaluations of graduate students will be evaluated this year in relation to graduate program outcomes. J. Course Coordinator Reports are complied by the Program Evaluation Committee to document the changes that have been completed in each course. This is done to assist in the evaluation of data. K. Revise teaching/learning outcome assessment strategy. With the new curriculum some assignments have changed in relation to the teaching/learning program outcome. The Program Evaluation Committee is in the process of evaluating the impact of the curricular change on the program outcomes. 6
7 Northern Arizona University School of Nursing Program Evaluation Model Administration, Faculty and Curriculum Committee II Institutional Curriculum III Commitment & Quality Resources (Curriculum Committee, (Executive Director, Associate Faculty, Administration) Director, Business Manager) Program Evaluation Grid I Mission & Goals (School Administration & Faculty Governance) Program Outcomes BS & MS HESI, NCLEX Required Surveys EBI, CCTDI Formal Complaints Program Effectiveness IV (Program Evaluation Committee) Recommendations Program Assessment Plan - CCNE Standards I IV - Program Outcomes, MS, BS - Program Evaluation Committee - CCTDI, EBI, HESI, NCLEX
8