Radiography Program: Outcomes and Assessment for the Class of 2016 Cohort Group 2014-2016 6004 S County Road G, Janesville, WI 53546 (608) 758-6900 info@blackhawk.edu blackhawk.edu BTC is an EO/AA educator/employer. For more information, go to blackhawk.edu.
Page 1 PROGRAM MISSION: The Mission of the Associate Degree Radiography Program is to prepare the Student to Practice Entry-Level Diagnostic Medical Radiography. PROGRAM GOALS: GOAL 1: STUDENTS WILL PERFORM COMPETENT RADIOGRAPHY GOAL 2: STUDENTS WILL COMMUNICATE EFFECTIVELY GOAL 3: STUDENTS WILL USE CRITICAL THINKING AND PROBLEM SOLVING SKILLS GOAL 4: STUDENTS WILL DEMONSTRATE PROFESSIONALISM PROGRAM OUTCOMES OUTCOME 1: CARRYOUT THE PRODUCTION AND EVALUATION OF RADIOGRAPHIC IMAGES OUTCOME 2: APPLY COMPUTER SKILLS IN THE RADIOGRAPHIC CLINICAL SETTING OUTCOME 3: PRACTICE RADIATION SAFETY PRINCIPLES OUTCOME 4: PROVIDE QUALITY PATIENT CARE OUTCOME 5: MODEL PROFESSIONAL AND ETHICAL BEHAVIOR CONSISTENT WITH THE A.R.R.T. CODE OF ETHICS OUTCOME 6: APPLY CRITICAL THINKING AND PROBLEM SOLVING SKILLS IN THE PRACTICE OF DIAGNOSTIC RADIOGRAPHY
Page 2 GOAL 1: STUDENTS WILL PERFORM COMPETENT RADIOGRAPHY Outcomes OUTCOME 1: CARRYOUT THE PRODUCTION AND EVALUATION OF RADIOGRAPHIC IMAGES Measurement Tool 1. Clinical Competency Form/Unsuccessful Competency Form Benchmark Timeframe Responsible Party Results 1. All students successfully complete 85% of clinical competency attempts on first attempt. 2. Lab Rubric 2. All students successfully complete 75% of laboratory competency attempts on first attempt 1. Clinical 2, 3 & 5 End of Semester 2. Fall Y1 & Spring Y1 1. All radiography Semester 2: 76% Semester 3: 84% Semester 5: 98% 2. Course instructors Fall Y1: 97.1% Spring Y1: 99% 3. Image Evaluation Tests 3. Average score of 80% for cohort 3. Fall & Spring Y1 3. Course instructor Fall: 87% Spring: 82% OUTCOME 3: PRACTICE RADIATION SAFETY PRINCIPLES 4. Program Technical Skills Attainment (TSA) Assessment 1. Clinical Competency Form/Unsuccessful Competency Form 2. Clinical radiation safety policy infractions 4. Successful completion of 100% of completing cohort 1. Less than 10% of unsuccessful competency attempts due to rad. safety issues for 100% of students. 2. 10% or less students incurring clinical infractions for radiation safety issues. 4. Spring Y2 Exhibits Supporting Goal 1: Exhibit 1: Clinical Competency Form Exhibit 2: Unsuccessful Competency Form Exhibit 3: Procedures Lab Rubric Exhibit 4: Procedures 1 & 2 Image Evaluation Tests Information Exhibit 5: Program Technical Skills Attainment (TSA) Exhibit 6: Clinical Radiation Safety Infraction Information See Next Page for Action Plan for Goal 1 1. Clinical 2, 3 & 5 End of Semester 2. Clinical 2, 4 & 6 End of Semester 4. Program Director 1. All radiography 2. All radiography 100% Clin 2: 21.5% (14 of 65) Clin 3: 14.8% (7 of 47) Clin 5: 30% (3 of 10) Clin 2: 12% (2 of 17) Clin 4: 0% (0 of 13) Clin 6: 0%: (0 of 13)
Page 3 Action Plan: Goal 1 Outcome 1: Measurement Tool 1 Competency/Unsuccessful Competency Forms Unsuccessful competency rate calculated by dividing total competency attempts (successful attempts + unsuccessful attempts) divided by successful competency attempts: Semester 2: 204/204 + 65 = 76% Semester 3: 400/400 + 47 = 89% Semester 5: 494/494 + 10 = 98% Benchmark not met for semesters 2 & 3; The program is monitoring reasons for this increase and information seems to be pointing at 2 factors: Change of responsibility of completion of form from technologist to student Better compliance with completion of these forms by both students and Based on this information, the program will continue to monitor for next cohort & consider revising benchmark Outcome 1: Measurement Tool 2: Lab Rubric As with the competency forms identified above, while the program tracked the rate of unsuccessful laboratory attempts, good data related to the reasons for unsuccessful laboratory attempts were not tracked and compiled. A laboratory rubric was developed, but this form was returned to the student without data being collected, and was not used at all for the second semester radiographic procedures course, resulting in no data being available for the spring semester course. Based on this information, the program has made the following changes: The new laboratory rubric for radiographic Procedures 1 was implemented fall semester 2015. For the Procedures 2 course (spring 2016) this rubric is replaced by the combination of clinical competency/unsuccessful competency forms to better reflect the clinical competency experience. Outcome 1: Measurement Tool 3 Image Test There was a concern that having a benchmark that reflected that that the average score of the group was a passing score that this could allow for the benchmark being met without all students passing. Based on this information the program will: Change the benchmark for the class of 2017 to assure that all students receive a passing for the fall and spring semester image evaluation examinations. Outcome 1 : Measurement Tool 4 Technical Skills Attainment (TSA) As the TSA tool is used to verify technical skills when all clinical education graduation requirements have been met, to this point all students have met this requirement since its implementation. As this tool was derived as a Wisconsin Technical College System form to be used by all radiography programs, the tool cannot be changed should flaws be found without such a change impacting all WTCS programs. Based on this information, the program will continue with this assessment while perhaps looking for a tool better suited for affecting meaningful change.
Page 4 Outcome 3: Measurement Tool 1 Competency/Unsuccessful Competency Forms Unsuccessful competency rate calculated by dividing total competency attempts (successful attempts + unsuccessful attempts) divided by successful competency attempts: As was indicated in the assessment plan for the class of 2015, the program collects information regarding unsuccessful competency attempts by exam, facility & student, and has implemented with the class of 2017 a system of structured remediation for students that show patterns of unsuccessful competency attempts. During this process, it became apparent that there was a need for structured remediation for unsuccessful attempts at the image evaluation component of the clinical competency evaluation. Based on this information, the program has redesigned the unsuccessful competency form to have a second side for use during the image evaluation process. Students will be required to perform structured remediation of this part of the clinical competency evaluation process following the 2 nd unsuccessful attempt at image evaluation on the same procedure, or at the discretion of program. This revised form will be implemented with the new cohort starting summer semester 2016 (class of 2018.) Outcome 3: Measurement Tool 2: Radiation Safety Policy Infractions As reported in assessment plan for the class of 2015, owing to the design of the both the clinical competency and unsuccessful competency forms, the program was unable to collect meaningful data related to demonstrated competency of radiation safety practices. Based on this information, the program has revised both forms to reflect radiation safety as a separate category for evaluation and data collection. These revised forms will be implemented for the incoming cohort starting summer semester 2016 (class of 2018). However, in accordance with program policy any documented incidents of violation of radiation safety policy are documented using anecdotal notes or unsuccessful competency form, as appropriate.
Page 5 GOAL 2: STUDENTS WILL COMMUNICATE EFFECTIVELY Outcomes OUTCOME 4: PROVIDE QUALITY PATIENT CARE Measurement Tool 1. Affective Evaluation #8: Appropriately interacts with patients (courteous, thoughtful, empathetic, displays patience and non-judgmental) Benchmark Timeframe Responsible Party Results 1. Average score of 4 or better: 1-5 scale 1. Clinical 2, 3 & 5 1. All radiography Clinical 2: 4.0 Clinical 3: 3.8 Clinical 5: 4.4 2. Clinical Competency Form/Unsuccessful Competency Form 2. Less than 10% of unsuccessful competency attempts identified in Patient Preparation and History category for 100% of students 2. Clinical 2, 3 & 5 2. All radiography Clinical 2: 4.6% (3 of 65) Clinical 3: 2.1 % (1 of 47) Clinical 5: (0 of 10) 3. Critical Thinking Journal Blind Assessment 3. 90% of Journal 1 demonstrate positive reflection 3. Clinical 4 3. Faculty assigned specific clinical course Clinical 2: 95% Clinical 4: 94% OUTCOME 5: MODEL PROFESSIONAL AND ETHICAL BEHAVIOR CONSISTENT WITH THE A.R.R.T. CODE OF ETHICS 1. Affective Evaluation #10: Communicates effectively within the healthcare setting (communicates appropriate information, applies confidentiality, uses appropriate medical terminology) 1. Average score of 3 or better: 1-5 scale 1. Clinical 2, 3 & 5 1. All radiography Clinical 2: 3.9% Clinical 3: 3.8% Clinical 5: 4.4% 2. Clinical Assessment Activities Clinical Semesters 2 & 4: Critical Thinking Journal. Blind Assessment of Affective Evaluation #9 (Conducts himself/herself in an ethical and professional manner; displays integrity, sincerity and applies discretion) 2. Average score of 3 or better: 1-5 scale on Journal 1 Rubric Exhibits Supporting Goal 2: Exhibit 7: Affective Evaluation through Dataarc (www.dataarc.ws) Exhibit 1: Clinical Competency Form Exhibit 2: Unsuccessful Competency Form Exhibit 8: Critical Thinking Journal Information Exhibit 9: Trauma Lab Rubric See Next Page for Action Plan for Goal 2 2. Clinical 2 & 4 End of Semester 2. Faculty assigned specific clinical course Clinical 2: 4.94 Clinical 4: 5.0
Page 6 Action Plan: Goal 2 Outcome 4: Measurement Tool 1: Affective Evaluation Statement # 8 Evaluation demonstrates that benchmarks are largely being met with respect to statement 8 of the affective evaluation which address those employability (soft) skills, or as Blackhawk Technical College has identified them, Core Abilities. As identified in the assessment plan for the class of 2015, the benchmark for this outcome was increased from 3 to 4. Even with this increase, the benchmark was met in 2 of 3 semesters evaluated, with the semester that it was not met only falling short by 0.2%. Based on this information the program has/will be implementing the following changes. is currently developing a number of rubrics for the assessment of core abilities in the classroom. For this reason, no major changes are being at the program level for radiography. Outcome 4 Measurement Tool 2 Competency/Unsuccessful Competency Forms As was stated in the assessment plan for the class of 2016, as procedures laboratory sessions progressed through the first semester, and even more so into the second semester, the atmosphere become more relaxed and activities such as history taking are not as stringently practiced and evaluated. With the fall semester 2015, a new member has taken over the instruction of the Procedures 1 & 2 courses. There has been a conscious attempt to assure that the level of rigor in the demonstration of all aspects of laboratory competency evaluation remains high through the duration of these courses Based on this information, the program has revised laboratory procedures in these courses to include: Continued emphasis on the practice and evaluation communication skills in the laboratory setting necessary to perform clinical radiography. Switch from the laboratory rubric in Procedures 1 Course to Clinical Competency Form & Unsuccessful Competency form in Procedures 2, to more closely reflect the clinical competency experience Outcome 4 Measurement Tool 3: Critical Thinking Journal Blind Assessment There has been a concern that while valuable, the critical thinking journal does not provide adequate information related to of assessment of critical thinking especially as it relates to patient care skills - in the clinical setting. Based on this information, the program will continue to utilize the journaling. The program has expanded the use of trauma simulation lab sessions and a rubric has been developed for these experiences. One area of the rubric addresses specifically is that of patient care and assessment. This document was implemented fall semester 2015 and data from this tool will be available for assessment for the class of 2017. The program currently utilizes 3 trauma labs (2 fall semester, 1 spring semester) that utilizes this tool. Outcome 5 Measurement Tool 1 Affective Evaluation Statement 10 As communications skills are part of employability skills, this is adequately addressed above in Outcome 4: Measurement Tool 1 Affective Evaluation Statement # 8. Program changes as outlined above are designed to address this program outcome as well. Outcome 5 Measurement Tool 2: Critical Thinking Journal. Blind Assessment of Affective Evaluation #8 This remains a good tool to evaluate professionalism, especially as a blind assessment where students believe that they are being evaluated on their writing skills. Based on this information, the program will continue to utilize the journaling as a method of assessing professionalism.
Page 7 GOAL 3: STUDENTS WILL USE CRITICAL THINKING AND PROBLEM SOLVING SKILLS Outcomes OUTCOME 6: APPLY CRITICAL THINKING AND PROBLEM SOLVING SKILLS IN THE PRACTICE OF DIAGNOSTIC RADIOGRAPHY Measurement Tool 1. Affective Evaluation #11: Efficient planning and management of time (prioritizes work, adapts to changing workload and completes assignments on time) Benchmark Timeframe Responsible Party Results 1. Average score of 3 on 1-5 scale for Clinical 2 & 3 2. Average score of 4 on 1-5 scale for Clinical 5 1. Clinical 2, 3 & 5 End of Semester 1. All radiography Clinical 2: 3.4 Clinical 3: 3.7 Clinical 5: 4.4 2. Clinical Competency Form/Unsuccessful Competency Form 3. Less than 10% of unsuccessful competency attempts due to critical thinking/problem solving issues for 100% of students 2. Clinical 2, 3 & 5 End of Semester 2. All radiography Clinical 2: 12% (8 of 65) Clinical 3: 3% (3 of 47) Clinical 5: 30% (3 of 10) 3. Clinical Assessment Activities Clinical Semesters 2 & 4: Critical Thinking Journal 1 4. Average score of 2 or better on 1-3 scale 3. Clinical 2 & 4 End of Semester 3. Faculty assigned specific clinical course Clinical 2: 100% Clinical 4: 100% OUTCOME 2: APPLY COMPUTER SKILLS IN THE RADIOGRAPHIC CLINICAL SETTING 1. Imaging 2 Experiments 1. All students will complete 15 of 20 digital imaging experiments as assigned 1. Spring Semester Y1 1. Course instructor 100% 2. Clinical Orientation Checklist: Brief Orientation to PACS system 2. Completed by 100% of placed students 2. Clinical Semesters 2, 4, & 6 2. All radiography 100% Exhibits Supporting Goal 3 Exhibit 7: Affective Evaluation through Dataarc (www.dataarc.ws) Exhibit 1: Clinical Competency Form Exhibit 2: Unsuccessful Competency Form Exhibit 8: Critical Thinking Journal Information Exhibit 10: Imaging Experiments Information Exhibit 11: Clinical Orientation Checklist See Next Page for Action Plan for Goal 3
Page 8 Action Plan: Goal 3 Outcome 6 Measurement Tool 1: Affective Evaluation: The affective behavior evaluation through Dataarc remains a very good method of collecting data related to critical thinking skills and the program will continue to rely on this data as a primary method of assessing these skills. Outcome 6 Measurement Tool 2: Clinical Comp Form and Unsuccessful Comp Form: The unsuccessful competency attempt form in particular proved a very valuable tool to assess critical thinking. The technologists documented primarily though their comments - that students struggled with pediatric, geriatric, trauma and obese patients. In evaluating the data, it was noticed that the benchmarks were not met for semester 2 & 5, with semester 5 showing a 30% failure rate owing to critical thinking skills. This has prompted the following issues to be evaluated: With better compliance of the documentation of unsuccessful competency attempts, a benchmark of 90% pass rate, especially in the first year of the program might be unrealistic. While semester 5 shows a 30% fail rate for issues of critical thinking, this number is based on a total of 10 unsuccessful competency attempts, meaning that more than one unsuccessful attempt would result in the benchmark not being met. Based on this information, the program will be making the following changes: Increase the benchmark from 10% to 20% for semester 2 only, understanding that there are more unsuccessful competency attempts in the first year of the program. Collect data related to unsuccessful competency attempts as percentage of the total attempts for the time in the program as opposed to those in a particular semester, keeping the numbers large enough to be statistically significant. These changes will be implemented with the cohort starting summer semester 2016 (class of 2018.) Outcome 6 Measurement Tool 3 Critical Thinking Journal: The program has expanded the use of journaling as a method of collecting data related to critical thinking about and reflecting on the importance of critical thinking in the clinical education setting. The program has no immediate plans to change this current assessment. Outcome 2 Measurement Tool 1: Exposure Experiments: As was indicated in the assessment plan for the class of 2015, the program has revised scoring rubrics for exposure experiments that include evaluation of the use of them DICOM viewing software, and CR system interface. These new rubrics were implemented in the fall semester 2015 and data collected will be utilized for the assessment document for the class of 2017. Additionally, as the program will be moving in new laboratory facilities fall of 2016, these new facilities will include a new DR system. Following this move rubrics will be revised to reflect assessment of the use of this equipment. This new assessment will be implemented for the fall semester 2016, and utilized for the assessment document for the class of 2018. Based on this information, the program has made the following changes: Increase the number of experiments for Imaging 2 as well as to refine existing experiments. The course instructor has also expressed the desire to improve on the existing rubric for these experiments to better collect meaningful data. This same instructor has subsequently been assigned the analog Imaging 1 class the preceding semester. Seeing the value of the digital experiments as a tool for critical thinking assessment, the experiments in this class have been expanded as well. Outcome 2 Measurement Tool 2 Clinical Orientation Checklist: The clinical orientation checklist provides good documentation that students have been provided with minimally an overview of the PACS/RIS systems when they are placed at a new clinical affiliate. As importantly, all clinical affiliates had input into the design and content of this document and are very happy with its completion as part of the clinical orientation process. The program has no immediate plans to change this current assessment.
Page 9 GOAL 4: STUDENTS WILL DEMONSTRATE PROFESSIONALISM Outcomes OUTCOME 4: PROVIDE QUALITY PATIENT CARE Measurement Tool 1. Clinical Competency Form/Unsuccessful Competency Form Benchmark Timeframe Responsible Party Results 1. Less than 10% of unsuccessful competency attempts identified in Other Positioning Criteria category for 100% of students 1. Clinical 2, 3 & 5 1. All radiography Clinical 2: 21% (14 of 65) Clinical 3: 15% (7 of 47) Clinical 5: 10% (3 of 30) 2. Clinical Assessment Activities Clinical Semester 5: Patient Care/Empathy Video 2. Score of 90% or better by all students. 2. Clinical 5 End of Semester 2. Faculty assigned to specific clinical course 85%: (11 of 13) OUTCOME 5: MODEL PROFESSIONAL AND ETHICAL BEHAVIOR CONSISTENT WITH THE A.R.R.T. CODE OF ETHICS 1. Affective Evaluation #1: Professional appearance (cleanliness, grooming and proper attire) 2. Affective Evaluation #9: Conducts himself/herself in an ethical and professional manner (displays integrity, sincere and applies discretion. 1. Average score of 3 on 1-5 scale 1. Clinical 2, 3 & 5 2. Average score of 3 on 1-5 scale 2. Clinical 2, 3 & 5 1. All radiography 2. All radiography Clinical 2: 4.6 Clinical 3: 4.3 Clinical 5: 4.9 Clinical 2: 3.9 Clinical 3: 3.8 Clinical 5: 4.2 3. Midterm and Final Advising Form: Rubric for the Role Plays on Accepting Criticism 3. Cohort average score > 90 See rubric 3. Fall & Spring Y1 & Y2 Midterm and 3. All radiography 90% (72 of 80) Exhibits Supporting Goal 4: Exhibit 1: Clinical Competency Form Exhibit 2: Unsuccessful Competency Form Exhibit 12: Patient Care/Empathy Video (www.health.clevelandclinic.org) Exhibit 7: Affective Evaluation through Dataarc (www.dataarc.ws) Exhibit 13: Rubric for the Role Plays on Accepting Criticism Exhibit 14: Professional in the Classroom Rubric See Next Page for Action Plan for Goal 4
Page 10 Action Plan: Goal 4 Outcome 4 Measurement Tool 1 Clinical Competency/Unsuccessful Competency Forms: As with the assessment of critical thinking through the use of the unsuccessful competency forms, the same types of issues have become evident with respect to the benchmarks being met: A 90% pass rate benchmark is likely too high for students in the first year of the program As identified in the 2015 assessment plan, Other Positioning Criteria included among other things, all of the criteria related to radiation safety. With the cohort starting summer semester 2016 (class of 2018), the clinical competency form and unsuccessful competency form have items related to radiation safety in a separate category. It is hoped that the unsuccessful attempt rate will drop when this category area does not include the evaluation or radiation safety practices, which are now evaluated separately. Based on this information, the program will : Increase the benchmark from 10% to 20% for semester 2 Continue to monitor unsuccessful attempt rate for subsequent semesters to evaluate effect of transferring items related to radiation safety of separate category. Outcome 4 Measurement Tool 2 Patient Care/Empathy Video: Although the benchmark for this activity was not met, this is due to the fact that a number of students failed to complete the assignment on time, resulting in a lower grade. The program feels that the activity remains valid and has no plans at this time to make any changes to this assessment activity. Outcome 5 Measurement Tool 1 Affective Evaluation #1; Professional Appearance: Data collected related to professional appearance demonstrated a decreasing score as students progressed through the program. However at a subsequent meeting of the clinical instructors, this point was addressed and no individuals at the meeting stated that students appeared less professional as they progressed through the program. Based on this information, the program will continue to monitor professional appearance and remind students of program dress code annually with input from the Clinical Instructor meetings. Official dress code policy has been, and will continue to be included in the syllabus of each clinical education course, as well as the program student handbook. Outcome 5 Measurement Tool 2 Affective Evaluation #9; Conducts self in a professional manner. As indicated in the 2015 assessment plan, all BTC radiography students continue to be active HOSA members and participate in a community service day activity in the final semester before graduation. The program has identified no immediate need to change or expand this assessment activity. Outcome 5 Measurement Tool 3 Midterm Advising Form: In the 2015 assessment plan, student empathy with respect to the clinical education experience was perceived as decreasing. Additionally this has appeared to correspond to a decreased level of professionalism in the classroom and laboratory settings. Based on this information, the program will be making the following changes: As identified earlier in this document, increased rigor in the procedures laboratory sessions with evaluation and assessment of all aspects of laboratory competency. Development and implementation a professionalism in the classroom rubric. Both of these changes will be implemented with the cohort starting summer semester 2016 (class of 2018).
Page 11 Name Complete Program 2 Y. Associate Degree Radiography Program Graduate Completion Worksheet Class of 2015 Passed ARRT 1 st Attempt Passed ARRT Subsequent Placed in Field Within 1 Year X Placed in Field Subsequent Not Actively Seeking Employment X X Summary: Class of 2015 Program Completion: 72%; 13 of 18 ARRT Pass Rate 1 st Attempt: 77%; 10 of 13 ARRT Pass Rate Subsequent: 85%; 11 of 13 Placed in Field 1 Year Following Graduation: 92%; 11 of 12 Placed in Field Subsequent: Notes * Summary: BTC Total Program Completion 5 Year Average: 71%; 64 of 90 ARRT 1 st Attempt 5 year Average: 95%; 60 of 63 ARRT Pass Rate Subsequent: 97% 61 of 63 Placed in Field 1 Year Following Graduation 5 Year Average: 86%; 49 of 57