Utah Fire and Rescue Academy 2014 2015 Quality Report Introduction The Utah Fire and Rescue Academy (UFRA) Quality Report consists of feedback collected from students who attended UFRA Direct Delivery training courses throughout the fiscal year. Approximately 90% of all Direct Delivery courses were evaluated and 82% of all students that attended courses between July 1, 2014 and June 30, 2015, completed surveys. This reflects a 14% increase of completed evaluations as compared to the previous fiscal year. This increase is a direct result of program managers working with course intructors to gather feedback and evaluate each course. An ongoing goal of UFRA is to complete evaluations on 100% of all Direct Delivery courses. Evaluations are completed by the students on survey forms provided at or near the end of the training course. Data is gathered in three specific sections: Section 1: Course Material, Section 2: Instructor Performance and Section 3: Additional Comments / Recommendations. Completed evaluations are processed by the Quality Assurance (QA) Office and all information is analyzed and stored into a database that is later used to generate reports. Original hard copy documents are stored at UFRA for a period of 2 fiscal years. The Annual Quality Report is a tool where the collected data is presented and can be used in strategical planning meetings. The measureable, qualitative data that was compliled throughout the fiscal year is placed within this document as the primary source of documentation used for overall quality. Evaluation Purpose and Goal The director of UFRA utilizes feedback to identify strengths and areas for improvement regarding training courses, instructors, products and services. One of UFRA s primary goals is to continually review, evaluate and make changes in order to seek ways to improve and provide excellent service to the firefighters of Utah. UFRA strives to meet the growing demands and needs of all firefighters by providing the most cost effective and state-of-the-art training available. Course evaluations serve as one method to determine what customers need and how to best deliver it. In order to measure quality, a yearly baseline-scoring threshold value has been set for each scoring category. If information in any categorical section within a survey receives a score lower than the baseline threshold for an individual report, a red flag is triggered in order to draw attention to that item. The triggered event requires mandatory follow 1
up by, first, the Program Manager - Training, second, the Assistant Director - Training and, if necessary, the UFRA Director. Currently, the annual baseline threshold is set at 90% for quality of course material and 93% for instructor approval. The baseline was determined by averaging the scores for course material quality and instructor approval for fiscal years 2009 through 2013 (see Baseline Chart on page 6), then courses at the end of the fiscal year are measured against that established baseline. Next, a minimum quality threshold score for any individual course total is set at 95% and 85% for any qiven question within that survey. Regardless of the baseline, if an individual evaluation receives a no regarding safety, then the reason for the no answer is immediately investigated and the Assistant Director Training is notified. For courses and survey questions that fall below the 95% or 85% thresholds respectively, then the QA Office completes an investigation to answer the who, what, where, when and why. Documentation is collected and stored in the QA Office database. This year, of the 336 courses evaluated, 53 required a quality assurance investigation with 8 of those requiring immediate action**. All 8 of the immediate action events were prioritized, addressed and corrective action was applied. In most cases, immediate action triggered events received a response in less than one business day following receipt of the feedback. In one example, students attending the Fire Inspector I course provided negative feedback stating that the course is targeting the wrong audience. The QA Office learned that most of the students taking the Fire Inspector I course did not volunteer to take the course. They were required to take the course in order to test for an interdepartmental promotion. Currently, the scope of the Fire Inspector I course targets a higher level student who is required to certify in duties most commonly needed to fill the position of Fire Marshal, agency specific Fire Inspector or Plans Examiner. Following discussions with UFRA s Assistant Director Training and the Curriculum Development Program Manager, it was determined that the scope of the company officer requires a slightly different approach. Therefore, a Company Officer Fire Inspector course is being developed and expected to be available in early 2016. This course will offer the basics of what is needed at the fire company level to be successful in identifying priority safety items and reporting duties while conducting inspections within any given jurisdiction. Several fire department administrations were interviewed and they stated they will change their promotion requirements to allow a fire company level inspection course in lieu of requiring their candidates to take the more comprehensive Fire Inspector I course. Finally, the additional 45 action required events were addressed and/or corrected by 30 June 2015**. **Contact UFRA QA Office for detailed information regarding action items and results. 2
Highlights 2014 2015 proved to be another very successful year with all UFRA Direct Delivery courses receiving an overall approval score of 97%. The number of students that received training increased which is primarily due to the increase of several core course deliveries and the addition of the Emergency Apparatus Driving Simulator (EADS) training course. Since the pilot delivery of the EADS course in January 2015 through June 30, 2015, approximately 675 students have attended this new offering. The demand for this course has far exceeded the goal of UFRA planners and the expectation is that this demand will continue for years. The success of the course is well documented with an average approval score of 97% for both curriculum and instructor performance. Comments provided by the students state that this is one of the best courses that UFRA has ever offered. Next, the demographic makeup of students who attended UFRA courses ran an interesting pattern from the FY 2009 through 2013 (see graph below titled Volunteer vs. Career). During that period of time, volunteer firefighters attending courses averaged 33%, while career firefighters who attended averaged 65% and the remaining 2% attending were listed as others. NOTE: The chart below reflects the percentages for volunteer vs. career and does not account for other in the totals. It is intended that this comparison is between those two categories only. In FY 2013 2014 an interesting change occurred in which the number of both volunteer and career firefighters attending UFRA Direct Delivery courses was nearly equal. Of particular interest, a significant downward trend has occurred where the number of career firefighters attending UFRA courses has declined. The chart below illustrates a decrease in attendance from 68% in FY 2012 2013 to 33.6% in FY 2014 2015. Several factors may be attributed to that trend such as; shifts in training budgets, career promotions, UFRA course selection, etc. PERCENTAGE OF ATTENDANCE Volunteer (Percentage) 80 70 60 50 40 30 20 10 0 Full-Time / Career (Percentage) 60.21 38.06 UFRA Course Attendance Volunteer vs. Career 67.5 66 68 31.5 31.5 30 45.02 52.05 FY 09-10 FY 10-11 FY 11-12 FY 12-13 FY 13-14 FY 14-15 43.9 33.6 38.06 31.5 31.5 30 45.02 52.05 60.21 67.5 66 68 43.9 33.6 3
Another factor affecting demographics in course attendance is Winter Fire School. Generally, Winter Fire School draws a higher percentage of volunteer firefighters vs. career firefighters. As illustrated in the chart below (titled Attendance Volunteer vs. Career, Winter Fire School) you can see that in the fiscal years shown, volunteers led the overall attendance by slightly over 50%. However, notice that in the same reporting period two converse spikes occurred: one in 2010 where 100% of the students that completed evaluations indicated they were from volunteer fire departments, and the other spike occurred in 2013 in which 87% of those completing surveys indicated they were from career departments. In 2013, only 12% of the students indicated they were from volunteer departments. The QA Office will monitor trends and report them to UFRA Management in order to proactively keep valuable training accessible to every firefighter in Utah regardless of their affiliation as a volunteer, career, part-time paid or seasonal firefighter. PERCENTAG OF ATTENDANCE 120 100 80 60 40 20 0 Volunteer (Percentage) Full-Time / Career (Percentage) 100 Attendance - Volunteer vs. Career Winter Fire School 76 0 24 24 75 FY 09-10 FY 10-11 FY 11-12 FY 12-13 FY 13-14 FY 14-15 100 76 75 12 63.33 72.28 0 24 24 87 30.47 22.06 87 12 30.47 63.33 22.06 72.28 Comparative Scoring Table and Baseline Charts The following table provides a comparative analysis of all direct delivery UFRA courses that received evaluations in the 2013 2014 and the 2014 2015 fiscal years. The table illustrates the percentage of courses evaluated and the percentage of students that completed evaluations. The results are compared against each of the two fiscal years. As you can see, the percentage of courses evaluated in FY 2014 2015 increased by 19% and the number of students that actually completed evaluations increased by 14% as well. Regardless of the increase, instructors and program managers should engage in strategies that encourages students to take time and provide clear, objective and honest feedback. By increasing feedback and particularly written comments, a much broader scope of problems, gaps or other issues can be identified. Next, in FY 2014 2015 the overall approval rating for course material scored the same as FY 2013 2014 but the instructor approval rating increased in FY 2014 2015 by 1%. 4
To increase the approval rating for course material, program managers should continually review and evaluate curriculum, presentations, and other materials used in the UFRA training courses. Course Evaluation Comparative Table (FY 2013 2015) Percentage of courses evaluated in fiscal year 2013 2014 71% Percentage of courses evaluated in fiscal year 2014 2015 90% Increase / Decrease +19% Percentage of student evaluations completed in fiscal year 2013 2014 68% Percentage of student evaluations completed in fiscal year 2014-2015 82% Increase / Decrease +14% Course material approval rating for fiscal year 2013 2014 95.0% Course material approval rating for fiscal year 2014 2015 95.0% Increase / Decrease 0.0% Instructor approval rating for fiscal year 2013 2014 96.0% Instructor approval rating for fiscal year 2014 2015 97.0% Increase / Decrease 1.0% To understand the next table, a baseline-scoring threshold score was developed from data collected between FY 2009 2012. As illustrated in the below chart, the baseline for those years was set at 90% (blue) for course material and 93% (purple) for instructor approval. Next, FY 2013 2014 is shown by the 95% (red) for course material, 96% (light blue) for Instructor approval and, finally, the scoring value for the FY 2014 2015 is represented by the 95% (green) for course approval and 97% (orange). 100 Baseline Chart Course Material Approval Baseline - 2009-2012 90 80 90 95 95 93 96 97 Course Material Approval 2013-2014 70 60 50 40 30 Course Material Approval 2014-2015 Instructor Approval Baseline 2009-2012 Instructor Approval 2013-2014 20 10 Instructor Approval 2014-2015 5
Summary UFRA continues to provide quality training to the firefighters of Utah by constantly reviewing, evaluating and revising all products and services as needed. By remaining proactive and staying ahead of the wants and needs and ever changing firefighting culture, UFRA will remain a top rated educational and training institution. 6
Utah Fire and Rescue Academy 2014-2015 Evaluation Summary Core Direct Deliveries, Wildland, Command Training Center, Emergency Apparatus Driving Simulator, Winter Fire School Type of Affiliation / Employment: Part-time / Paid 324 10.89% Volunteer 1549 52.05% Full-time / Career 1000 33.60% Seasonal / Non-Affiliated 103 3.46% No Response (these responses aren't included when % of total is computed) 628 Was the length of the course appropriate for the topic? Select the choice below that best describes the length of the course. Just the right length 3074 85.67% Too Short 331 9.23% Too Long 183 5.10% No Response (these responses aren't included when % of total is computed) 16 Was the curriculum easy to follow? Yes 3344 97.15% No 98 2.85% No Response (these responses aren't included when % of total is computed) 162 UFRA QA Office
Report 1: Utah Fire and Rescue Academy 2014-2015 Evaluation Summary Core Direct Deliveries, Wildland, Command Training Center, Emergency Apparatus Driving Simulator, Winter Fire School Was the curriculum and training material relevant? Yes 2783 98.24% No 50 1.76% No Response (these responses aren't included when % of total is computed) 771 Was this training applicable to your current job or the position you are seeking? Yes 2546 97.29% No 71 2.71% No Response (these responses aren't included when % of total is computed) 987 Would you recommend this course? Yes 3466 97.55% No 87 2.45% No Response (these responses aren't included when % of total is computed) 51 Were safety messages and safety plans reinforced during this class? Yes 2615 99.66% No 9 0.34% Not Applicable 0 0.00% No Response (these responses aren't included when % of total is computed) 980 UFRA QA Office
Report 1: Utah Fire and Rescue Academy 2014-2015 Evaluation Summary Core Direct Deliveries, Wildland, Command Training Center, Emergency Apparatus Driving Simulator, Winter Fire School Were all instructors organized? Yes 3349 98.18% No 62 1.82% No Response (these responses aren't included when % of total is computed) 193 Did all the instructors demonstrate the requisite knowlegde of this course subject? Yes 3517 99.18% No 29 0.82% No Response (these responses aren't included when % of total is computed) 58 Did all the instructors present themselves professionally (appropriately dressed, well groomed, positive attitude, professional behavior)? Yes 3395 99.56% No 15 0.44% No Response (these responses aren't included when % of total is computed) 194 UFRA QA Office
Report 1: Utah Fire and Rescue Academy 2014-2015 Evaluation Summary Core Direct Deliveries, Wildland, Command Training Center, Emergency Apparatus Driving Simulator, Winter Fire School Did all the instructors respect differing viewpoints and opinions? Yes 3523 99.02% No 35 0.98% No Response 46 Were all the instructors on time? Yes 3374 99.03% No 33 0.97% No Response (these responses aren't included when % of total is computed) 197 Were the presentations clear, understandable, and relevant? Yes, the presentations were clear and relevant 3029 89.32% Most parts of the presentations were clear, understandable and relevant 354 10.44% Not at all 8 0.24% No Response (these responses aren't included when % of total is computed) 213 UFRA QA Office