African Partnerships for Patient Safety Evaluation Handbook April 2012
WHO/IER/PSP/2012.8 World Health Organization 2012 The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. The mention of specific resources does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. All reasonable precautions have been taken by the World Health Organization to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall the World Health Organization be liable for damages arising from its us
PART A 1. Core APPS Evaluation Summary Page 5 1A. Core APPS Evaluation at a glance Page 7 2. Partnership Strength Evaluation Form Page 8 2A. Partnership Strength Evaluation Tracker Page 9 3. Patient Safety Improvement Evaluation Outline Page 10 3A. Patient Safety Improvement Evaluation Tracker Page 11 4. Patient Safety Spread Evaluation Form Page 12 4A. Patient Safety Spread Evaluation Tracker Page 13 PART B 1. Partnership Strength 1 page summary Page 15 2. Patient Safety Improvement 1 page summary Page 16 3. Patient Safety Spread 1 page summary Page 17
APPS EVALUATION HANDBOOK Part A
1. Core APPS Evaluation Summary Step 1: Partnership Development Outline of the approach to evaluation: The APPS approach to evaluation is designed to provide simple information capable of being used to demonstrate impact. It is based around the 6-step APPS improvement cycle, starting with the collection of baseline data through the Situational Analysis that takes place during Step 2. During the development of the Partnership Plan in Step 4, partners tailor their evaluation to the priorities and actions described within the plan. Each partnership provides core evaluation information to enable a uniform approach and data that is capable of being used to demonstrate impact and advocate for continued and expanded action on patient safety across the entire region of Africa. The APPS programme also encourages individual partnerships to utilize specific evaluation mechanisms for specific partnership work, in addition to the core uniform evaluation information mentioned above. Step 6: Evaluation and Review Step 5: Action Target action to: Strengthen health systems to support patient safety; Build patient safety capacity; Advocate and communicate for patient safety Step 4: Action Planning Step 2: Needs Assessment Step 3: Gap Analysis The 5 parts of APPS evaluation: 1. The Situational Analysis: Provides over 100 pieces of quantitative data on patient safety. Repeated annually to track progress; 2. Evaluation of Partnership Strength: The APPS team will undertake a telephone (or Skype) or face-to-face interview survey with partners every 6 months. Ten questions will be asked on different aspects of the partnership (responses use a modified Likert scale). 3. Evaluation of Patient Safety Improvements: Two core evaluations are required relating to Action Area 5 (health care-associated infection). First, the WHO Hand Hygiene Self Assessment Framework is repeated annually. Second, hand hygiene compliance monitoring takes place every 6 months (minimum). In addition, partnerships will provide a comprehensive outline of their local approach to evaluation as described in the partnership plan, and this data will be shared with the programme; 4. Evaluation of Patient Safety Spread: The APPS team will undertake a telephone (or Skype) or face-to-face interview survey with partners every 6 months. Ten questions will be asked, requiring yes/no answers. There will be an opportunity to capture qualitative data. 5. Annual APPS Evaluation Report: African partners are required to provide a 2 monthly activity report related to the APW. This is summarized into an annual APPS Evaluation Report. 5
Core APPS Evaluation at a glance On the next page is a 1-page summary of the APPS approach to evaluation. The icons within the table indicate whether action is required by you the partners, whether the detail provided requires partners to familiarize themselves with an aspect of evaluation, and whether there will be any input from the APPS Core Team (see legend below). LEGEND Familiarization required Data collection action required Participation in APPS Core Team evaluation calls/skype/face-to-face Report writing and submission required 6
1A. Core APPS Evaluation at a glance When to take action Facility-wide evaluation Pre-workshop Situational Analysis Go Live period Month 6 Month 12 Month 18 Date Month 24 Date Partnership call focused on evaluation Evaluation Webinar Repeat Situational Analysis (short form) Annual evaluation report (summary of 2 monthly APW Activity reports) Repeat Situational Analysis Annual evaluation report (summary of 2 monthly APW Activity reports) Evaluating the strength of the partnership Familiarize yourself with Partnership Strength Evaluation Form Partnership strength evaluation telephone survey (or face to face) Partnership strength evaluation telephone survey (or face to face) Partnership strength evaluation telephone survey (or face to face) Partnership strength evaluation telephone survey (or face to face) Evaluation activity to complete Evaluating the patient safety improvements Hand Hygiene Self Assessment Framework Hand hygiene compliance monitoring Evaluation baseline data, based on priority action areas listed in Action Plan Hand hygiene compliance monitoring Hand Hygiene Self Assessment Framework Hand hygiene compliance monitoring Evaluation based on priority action areas listed in Action Plan Hand hygiene compliance monitoring Hand Hygiene Self Assessment Framework Hand hygiene compliance monitoring Evaluation based on priority action areas listed in Action Plan Evaluating patient safety spread Familiarize yourself with Spread Pack Spread activity telephone survey (or face to face) Spread activity telephone survey (or face to face) Spread activity telephone survey (or face to face) Spread activity telephone survey (or face to face) 7
Strongly Agree Agree Disagree Strongly Disagree 2. Partnership Strength Evaluation Form: For more information on the partnership strength evaluation see the 1 page summary in Part B of this Evaluation Handbook To what extent do you agree with each of the following statements? Comments (the interviewer will explore feedback on issues and barriers to partnership working) 1. Our partnership shares a common vision of what we want to achieve 2. We always have up to date information about our partner s activities 3. Key decisions are made jointly between both sides of the partnership 4. Communication between the two sides of the partnership is effective 5. Partnership activities are owned by the institution, not just by an individual 6. All stakeholders and engaged and involved 7. Partners respect one another and understand one another s local rules, culture and customs 8. Our partnership working makes effective use of the particular skills that individuals within the partnership have 9. The two sides of our partnership celebrate success together 10. What do you think are the 3 key benefits of partnership working for your hospital 8
2A. Partnership Strength evaluation tracker Timing Evaluating the strength of the partnership Date(s) Completed (yes/no) Go live period Familiarize yourself with partnership strength evaluation form Month 6 Month 12 Month 18 Month 24 Partnership strength evaluation telephone survey Partnership strength evaluation telephone survey Partnership strength evaluation telephone survey Partnership strength evaluation telephone survey 9
3. Patient Safety Improvement Evaluation Outline For more information on the patient safety evaluation see the 1 page summary in Part B of this Evaluation Handbook Action Area (in addition to Action Area 5, highlight priority action areas below) Core Evaluation method 1. Patient Safety and Health Systems; Tracked via repeat Situational 2. National Patient Safety Policy; Tracked via repeat Situational 3. Patient Safety Knowledge and Learning; Tracked via repeat Situational 4. Raising Patient Safety Awareness; Tracked via repeat Situational 5. Health Care-Associated Infections; Hand Hygiene Self Assessment Framework Hand hygiene compliance monitoring 6. Health-care Worker Protection; Tracked via repeat Situational 7. Health-care Waste Management; Tracked via repeat Situational 8. Safe Surgical Care; Tracked via repeat Situational 9. Medication Safety; Tracked via repeat Situational 10. Patient Safety Partnerships; Tracked via repeat Situational 11. Funding for Patient Safety; Tracked via repeat Situational 12. Surveillance and Research in Patient Tracked via repeat Situational Safety Sources of Evaluation Tools Additional evaluation (list evaluation according to partnership plan) 10
3A. Patient Safety Improvement evaluation tracker Timing Go live period Core (HAI) and other priority evaluation methods according to Partnership Plan (list evaluation methods in space provided) Hand Hygiene Self Assessment Framework Hand hygiene compliance monitoring Date(s) Completed (yes/no) Month 6 Hand hygiene compliance monitoring Month 12 Hand Hygiene Self Assessment Framework Hand hygiene compliance monitoring Month 18 Hand hygiene compliance monitoring Month 24 Hand Hygiene Self Assessment Framework Hand hygiene compliance monitoring 11
4. Patient Safety Spread Evaluation Form: For more information on the patient safety spread evaluation see the 1 page summary in Part B of this Evaluation Handbook Question: In the last 6 months have you: Yes/No Brief details 1. Organized or participated in an event e.g. workshop, meeting, seminar, or training course on patient safety in your hospital, other hospitals or nationally? 2. Identified patient safety champions across multiple disciplines and at senior levels? 3. Started or been involved in any patient safety campaigns e.g. working with local radio and/or other media to promote patient safety or securing celebrity endorsement for patient safety improvement? 4. Written up any patient safety publications and/or case studies? 5. Become aware of other hospitals in your country now improving patient safety? 6. Had any contact with the Ministry of Health or WHO Country Office to discuss patient safety? 7. Been involved in any events to launch patient safety improvement? 8. Become aware of any patient safety networks that have started in your area or country? 9. Secured support from the Diaspora in Europe or elsewhere for the work you are doing? 10. Secured any funding for any aspect of the patient safety activity in addition to APPSrelated budgets (now or in the future)? 12
4A. Patient safety spread evaluation tracker Timing Patient Safety Spread Date(s) Completed (yes/no) Month 1 Month 6 Familiarize yourself with the APPS Spread Pack & patient safety spread evaluation form Spread activity telephone survey (or face to face) Month 12 Month 18 Month 24 Spread activity telephone survey (or face to face) Spread activity telephone survey (or face to face) Spread activity telephone survey (or face to face) 13
APPS EVALUATION HANDBOOK Part B 14
1. Partnership Strength: 1 page summary The evaluation of partnership strength is based on the APPS principles of partnerships. These principals were co-developed by APPS partnerships participating in the inception phase of the programme. An APPS definition of partnership was also developed during this inception phase. The partnership strength evaluation form assesses how the work of each partnership is aligned with this foundation. What Why How When Partnership Strength Evaluation Form The form is designed to help APPS partnerships reflect on aspects of their partnership working well, and aspects needing attention and development. The Principles of Partnership outline how a respectful, energized and productive partnership would wish to operate. Through asking 10 questions you are invited to reflect on how well your partnership is currently meeting these principles. The results are intended to be used in celebrating what is going well, and in highlighting areas that need further attention. Partnership working is difficult and complex. You should expect to find some areas that need further work. Face to face or telephone interview 6 monthly A collaborative relationship between two or more parties based on trust, equality, and mutual understanding for the achievement of a specified goal. Partnerships involve risks as well as benefits, making shared accountability critical. APPS Definition of Partnership 15
2. Patient Safety Improvement: 1 page summary The APPS situational analysis is at the core of the APPS evaluation approach the use of the situational analysis can track progress in the 12 patient safety action areas over time. It can also stimulate patient safety team development. This is supplemented by core programme wide evaluation focused on hand hygiene. Further evaluation, focused on specific action areas, can supplement the situational analysis and can be defined by partnerships. What Why How When Hand Hygiene Self Assessment Annually Framework Hand Hygiene Compliance Monitoring The common focus of action is Action Area 5 (Health Care- Associated infection, HAI) The common focus of action is Action Area 5 (Health Care- Associated infection, HAI) One person or a team work together to complete the form Information should reflect t the situation of the entire hospital Trained observers spend 20 minute periods on a ward recording opportunities and compliance selected wards or entire hospital for a total of 200 recorded observed opportunity and actions performed. Twice a year 16
3. Patient Safety Spread: 1 page summary The APPS approach to patient safety spread is outline in the APPS Spread Pack. Three core concepts are worth considering throughout partnership implementation: Spread can be considered to be the process of disseminating methods of practice between institutions and individuals at the same or different hierarchical levels. Horizontal spread refers to spreading safety activities across people and organizations at the same level in the hierarchy of a health-care system. Vertical spread refers to spreading safety ideas up or down the hierarchy, for example influencing government ministries. The evaluation of patient safety spread utilizes a simple form that assesses how each APPS partnership is aligned with the principles outlined in the APPS Spread Pack. What Why How When Patient Safety Spread Evaluation 6 monthly Form The Patient Safety Spread Evaluation Form is designed to capture at 6 monthly intervals the spread activities within a partnership. Face to face or telephone interview between APPS Core Team and APPS Partnership Lead (or nominated individual in your partnership hospital) 17