Respondent # 70 Sector NHS or other health-care provider Evaluation Title Using the 15 Steps Challenge in Mental Health Evaluation The 15 Steps Challenge originated from acute care when a carer said she could Purpose tell what the quality of care was like on a ward within the first 15 steps walked onto the unit. The purpose of the evaluation was to see how well this model has adapted for use in mental health setting. Drivers It is considered best practice, Intrinsic value and general benefits of involving them The findings when owned and presented by service users and carers have a greater impact and result in increased change for the better, than an evaluation conducted only by staff. What Was Done Service users and carers were trained to act as "co-inspectors", to work on an evaluation of the quality of care of in patient services. PPI contributors received: Formal training, informal training, peer support and payment for out-of-pocket expenses and time. Comprehensive administration support was provided for the "co-inspectors" ensuring that things like payments were processed promptly. Co-inspectors were provided lifts in staff cars so that it was one less thing for them to have to worry about. An article written by the "co-inspectors" about their journey and experience in taking part in the 15 Steps project has been accepted by Mental Health Practice Journal (facilitated by the project lead). Elements of PPI Deciding what should be evaluated Evaluation approach Data collection Data analysis Interpreting data Writing up the evaluation report, Dissemination Service users and carers led presentations in conferences and showcases that were held to share the findings of the project/evaluation. Benefits identified The findings when owned and presented by service users and carers have a greater impact and result in increased change for the better, than an evaluation conducted only by staff. Because the findings were written and presented by our service user and carer "co-inspectors", they carried more weight and impact with the clinicians who received them (compared to findings written and presented by staff). A "fresh" set of eyes through which to see our services as they really are, from the perspective of the service users and carers. Following on from that, the findings formed a baseline from which to carry out focused quality improvement work. Whilst on the wards, we wanted to obtain the views of the current in patients and we found that they were far more willing to talk to our co-inspectors than they were to us, the Senior Nurses. The 15 Steps encourages the co-inspectors to use their five senses, therefore it is an approach that avoids examining data, and we found this to be refreshing and enlightening. It is better to use service users and carers NOT directly involved in the service you are trying to develop, as they can sometimes have their 1
Challenges or barriers identified Additional info own agenda and be too close to the subject otherwise. More time consuming than a staff only inspection (it uses much more staff time when involving service users than when we do staff only inspections and there is often insufficient capacity eg. time to recruit and train PPI contributors and also the logistics was time consuming, such as the co-inspectors were collected to and from home which incurred huge detours. However, they appreciated this service so much that we felt it was worth it). Another challenge is that the Senior Nurses (myself and my deputy) were responsible for debriefing and managing the feelings that came up for the co-inspectors as a result of the inspections but again, it was all worth it to capture their perspective. Preparation for conferences and events where we presented the findings - it was stressful at times for our co-inspectors and they required a lot of time and reassurance to prepare, but the way that the conference attendees received them was fantastic. Administration staff/time is vital. To co-ordinate the task, give a central point of contact for service users, process all the paper work, help the service users write up their findings - having good admin really makes a big difference to how the service users experience the project. In the future we will engage in more 15 Steps Challenges centring on/driven by PPI we will go back over the wards that we visited a year ago to see what has changed. Also we are broadening it out so that we can now inspect Out Patient treatment and waiting rooms. Below are links to two YouTube videos, the first is of the project lead and deputy and the second of two of the co-inspectors. https://www.youtube.com/watch?v=9w6n39-fi60 https://www.youtube.com/watch?v=-uizx6ui2ys 2
Respondent #77 Sector Evaluation Title Evaluation Purpose Drivers and supporting documents NHS or other health-care provider 1) In patient mental health services 2) Evaluation of North Somerset Primary Care Liaison Service To evaluate the inpatient mental health acute adult wards in Bristol Drivers: For ethical and moral reasons It is considered best practice Intrinsic value and general benefits of involving them Methodological, Organisational, local or national policy drive - It is a requirement of evaluations and quality assurance we do for patients to be involved in everything. National user network NSUN 4 PPIs. Requirements and/or expectations of the Commissioner or funder. All quality assurance and evaluation work we do in commissioning in principle will involve service users and patients at all levels. We have the same expectations of providers, in that we expect them also to carry out some patient led evaluations and quality assurance. We have set up with the providers a Service User, Carer Board to ensure this happens Evaluation and quality assurance are included in KPIs, contracts and providers are signed up to these To generate better quality of services Supporting Documents: What Was Done We use APCRC supporting documents as we are part of CCG and they are in all contracts as well as our own team ones and NSUN 400/ 1) The inpatient mental health acute adult wards in Bristol were evaluated with the service AWP commissioners and service users. This evaluation was led by service users and the project lead. Suggestions were made to improve ward practice and increase patient involvement on wards, in developing actions plans, monitoring and in new evaluation planning. PPI contributors received formal training, informal training, mentoring and supervision, peer support and payment for out-of-pocket expenses and time. 2) The evaluation of North Somerset Primary Care Liaison Service looked in detail at the quality of services provided. The evaluation was carried out in partnership with commissioners, AWP and service users. The evaluation was carried out by the Bristol and North Somerset CCG service user evaluation teams which included trained service users from North Somerset. The findings have resulted in the development of a number of recommendations. An action 3
plan will be developed by commissioners and service users which are to be monitored through the local commissioner performance meetings and service user forums. Elements of PPI Deciding what should be evaluated Evaluation approach Data collection Data analysis Interpreting data Writing up the evaluation report Dissemination On-going service implementing of improvement plan and monitoring of actions. Benefits identified Overall the involvement of patients has improved the work enormously at all levels. Challenges or barriers identified Some issues around others recognising the validity of work due to fact it is patient led (despite being done in the same ways as other service initiative evaluations and quality assurance). Sometimes it has been difficult to get findings acted on (but that isn't just due to it being patient led). It does need resources and support to do - without it can be difficult to do properly. Evaluation frameworks - whilst needed - can be too complicated and put people off. There can be very different understandings of what evaluations are / should involve. More work around promoting different ways of doing evaluations and encouraging support to do them is way forward. General barriers identified: Insufficient funding for PPI activity, insufficient capacity. 4
Respondent # 84 Sector Charity, Voluntary, Community or Social Enterprise (VCSE) organisation Evaluation Evaluating A&E Services for People who Self Harm Title Evaluation To see whether A&E services in Bristol are meeting the needs of people who selfharm through an experience-led review of A&E Services for people who self-harm. Purpose Drivers For ethical and moral reasons It is considered best practice Intrinsic value and general benefits of involving patients and carers Organisational, local or national policy drive - The organisation we were working with explicitly states that service user involvement is central to all of its work What Was Done The survey was carried out between August and October 2013 by members of Self Injury Self Help Bristol (SISH), was commissioned by Public Health at Bristol City Council and supported the work of the Improving Pathways for Self Harm (STITCH) Health Improvement Team in Bristol. The experience-led review of A&E services was commissioned and carried out following health improvement work aimed at reducing suicide and self-harm across Bristol. Treatment in A&E for self-harm is known to be a difficult area for patients and staff; therefore it was decided to focus on this area of the treatment pathway. The review of the patient experience complements other work done by STITCH to find out about staff experiences. The review followed the principles of experience based design and involved people who had used A&E service for self-harm from the beginning. The review covered both Bristol Royal Infirmary and Frenchay Hospital and was conducted over 3 months in 2013. Participants could respond online, on paper or in person and could take part completely anonymously. Anyone who chose to leave contact details was contacted to let them know about the next stages of the project and to invite them to identify any support they would need to take part in the project on an ongoing basis. Following on from the survey stage, six participants attended a focused event with A&E staff and STITCH members to make an action plan for short and long term changes in the A&E department. People who used self-harm were asked which topics/questions should be asked and all subsequent evaluation material stemmed from these ideas. Involvement of people with personal experience throughout the process: Applied user-led evaluation good practice principles, as verified by BRI Patient Involvement Team and described in Experience Based Design, NHS guide Initially sought views of people with lived experience of self-harm, what questions should be asked? Online survey and face-to-face interviews based on these suggestions, questions deliberately left open ended to capture personal perspectives Thematic Analysis: Data obtained from: 40 online surveys and 4 face-to- face interviews (3 personal experience; 1 worker); time spent in A&E observing PPI contributors received: Informal training, Peer support and Payment: out-ofpocket expenses, Examples of ways to better facilitate more widespread involvement of serviceusers, patients and other members the public in evaluation/s? Invite them to write the questions 5
Elements of PPI Benefits identified Challenges or barriers identified Additional info Set up meetings in public venues not health or research settings Ask if you can talk to relevant support groups - go to them, don't wait for them to come to you Be clear about what the outcomes will be/will be used for Value all contributions Compensate people for their time/effort Ask people what would encourage them to become involved Deciding what should be evaluated and the evaluation approach - People who used self-harm were asked which topics/questions should be asked and all subsequent evaluation material stemmed from these ideas. Data collection Data analysis Interpreting data Writing up the evaluation report Dissemination. I wouldn't ever consider doing this kind of work if it wasn't being done in complete collaboration with people using the service otherwise key points can be missed and vital information ignored. Better insight into the issues and needs being evaluated; More focused evaluation More practical outcomes and changes that can be made; It can take longer than just sitting down and doing it yourself and needs significant time investment the problem is when other partners don't realise/understand this and are unable to be patient or don't see the value in it Assumptions that one token person on a committee is PPI. General barriers: Insufficient funding for PPI activity; Insufficient capacity; Lack of recognition, adequate funding and resourcing from public sector partners. This was a very small project so we used in-house experience of service users already available within the organisation http://www.sishbristol.org.uk/files/a&e%20review%20final%20report%202013.pdf 6
Respondent # 91 Sector Charity or other Voluntary, Community or Social Enterprise (VCSE) organisation Title Eating Disorder services for children and young people Evaluation To evaluate the experiences of the care pathway for children/young people (and Purpose their parents) in Bristol and South Gloucestershire who are being treated for eating disorders To reveal service user perspective/experiences of Eating disorder services for children and young people. (Project purpose - In partnership with an NHS Trust to promote and value the voice and experience of children and young people who are service users). Drivers It is considered best practice, Intrinsic value and general benefits of involving Methodological, Organisational, local or national policy drive, Requirements and/or expectations of the commissioner or funder (including of the evaluation itself) What Was Done Elements of PPI Benefits identified Challenges or barriers identified This is a commissioned service and the commissioner wanted a clinical care pathway model approach so that referrers and families could understand their likely journey. Individual families were directly how they wanted to be involved and the project team responded on that basis to tailor involvement as per each families wishes. It was very informal, working with people individually who were part of the evaluation itself and then took their lead on how they wanted to stay involved with the process thereafter. Young people were involved in verifying services to be 'Young People Friendly' and in training etc. All involvement was supported. Deciding how they wanted to be involved Deciding what should be evaluated Evaluation approach (e.g. design, methods) Dissemination Implementation of findings PPI contributors were more involved in the implementation of the findings as direct recipients of these services and the contribution they made by sharing their stories. This resulted in a much more powerful and effective outcome which included online resources. More parents have come forward through this process wanting to get involved in improving services. Mainly time (change always takes longer than you want it to). Logistics and managing many individuals commitments and needs over time. Insufficient funding for PPI activity. A general lack of understanding by partner organisations of the time this takes and this not being compatible with deadlines. *Much of the challenges and barriers could be overcome by making PPI a requirement with time and funding built in 7
Respondent # 66 Sector NHS/Other health-care provider Evaluation Title Rehabilitation, Reablement and Recovery (3R s) Evaluation To evaluate whether the model of care for the 3 R s fits the requirements of Purpose the service users and carers. Drivers and Drivers: supporting Ethical and moral reasons documents It is considered best practice Intrinsic value and general benefits of involving them Organisational, local or national policy drive; Requirements and/or expectations of the commissioner or funder Supporting Documents: Policies/documents utilised: South Gloucestershire CCG and Council both have a very clear policy and approach to the involvement of service users, carers and the public in their work What Was Done Feedback from service users and carer has informed the development of the 3Rs model from the outset and will continue to be central to the process for implementation of the model. As part of this on-going process of engagement, prior to the evaluation panel a workshop was held with service users and carers, and with other stakeholders to ascertain whether the proposed model of care reflects the feedback gathered from patients, public and carers throughout the consultation. The detailed comments collected from that workshop were received by the evaluation panel and were used to inform the scoring process in the evaluation. PPI contributors were provided with informal training, mentoring and supervision and expenses were reimbursed (time was not paid for). Elements of PPI Being part of an Evaluation Panel Deciding on what should be evaluated Data collection. Benefits identified PPI contributors brought a different perspective to the evaluation, which was identified as vital as they had first-hand experience of using the services and a clear understanding of what was required and in turn provide a level of assurance that the changes brought about will improve services in the future. Facilitators (general) identified good polices, key stakeholders seeing PPI as important, information from others who have done it in other areas Challenges or barriers identified Additional info The process can take longer and be time consuming. It can be hard to engage some service users, particularly carers as they have limited time available away from their caring responsibilities. General barriers further identified include insufficient funding for PPI activity and insufficient capacity (time to recruit/train PPI contributors) 1 member of the patient panel was recruited through Sirona, which worked well, but it was acknowledged that there could be a conflict of interest from with this person. 8