Longer Term Impact of NHS QIS Peer Review Visits: Continuous Quality Improvement of Blood Transfusion Services in NHS Boards

Transcription

1 Longer Term Impact of NHS QIS Peer Review Visits: Continuous Quality Improvement of Blood Transfusion Services in NHS Boards Dr Steve Cross Human Reliability Dr Guro Huby University of Edinburgh Commissioned by NHS QIS and SNBTS Published October 2008

2 NHS Quality Improvement Scotland 2008 First published October 2008 You can copy or reproduce the information in this document for use within NHSScotland and for educational purposes. You must not make a profit using information in this document. Commercial organisations must get our written permission before reproducing this document.

3 Contents Executive summary Introduction Background Overview of the standards and peer review process Scope and objectives Methodology Findings Pre-visit activity Self-assessment phase in boards Review of evidence by the review team Direct observation of evidence review The review visit The perspective of board participants The perspective of the review teams Direct observation of review meetings Actions after the review Board participant responses Review team responses Discussion and recommendations Impact evaluation Receptivity Changes in knowledge Changes in policy and practice Changes in outcomes for patients Impact evaluation - conclusion The peer review process The overall process: striking a balance between peer review and inspection Quality assurance approach Self-assessment Evidence review The review visit Follow-up The peer review process: conclusion... 44

4 - 2 -

5 Executive summary NHS Quality Improvement Scotland (NHS QIS) and the Scottish Blood Transfusion Service (SNBTS) commissioned this report. We reviewed the impact and process of the external peer review of blood transfusion services in four health boards of NHS Scotland. In each case, the review was associated with positive changes in systems and levels of care, and in most cases these changes were attributed directly to the review itself. As a process, the review was effective in winning co-operation from boards, was generally welcomed by board personnel despite carrying a relatively high administrative burden. The review mechanism was able to: involve staff at all levels and recognise their contribution probe and question where evidence was weak or inconclusive highlight good practice and challenges for the board achieve consensus on the status of standards share experience. Reviews were carried out by external teams. We observed their work and concluded that the reviews were: thorough, in that each standard and the evidence supporting it were examined in detail expert, in that each team was multidisciplinary and had experienced representatives for each standard fair, in that the team sought consistency with other reviews balanced, in that team leaders would seek consensus and discuss issues openly positive in that teams adopted the role of peers aiming to ensure and encourage high standards of performance and to share experience. Most respondents endorsed the facilitative and constructive nature of the review visit. We concluded that the combination of two perceived elements of inspection and peer review contributed to the success in facilitating changes to boards policy and practice in relation to blood transfusion services. The review process has not yet included a specific plan for re-visiting boards or for maintaining the momentum generated by the initial self-assessment and review. We have recommended a number of changes to the process to facilitate this, in particular a further round of risk-assessed, targeted return visits to boards. Overall, the process was carried out in a largely receptive environment, has contributed to the establishment of an evidence-based consensus, and has resulted in significant changes in policy and practice. There is a general belief among practitioners that patient care will improve as a result. On the basis of these findings, we conclude that the external peer review process of blood transfusion has had a significant impact on quality improvement

6 1 Introduction 1.1 Background 1 NHS QIS has a number of functions, all designed to bring about an improvement in the quality of the health services delivered to patients. Of these, the most well-recognised are the setting of standards of care and their peer review in NHS boards. 2 During 2006 and 2007 a wide-ranging impact evaluation study was carried out 1 which included responses from senior board management, frontline staff, government representatives and members of the Academies of the Royal Colleges and Faculties in Scotland. Overall, the peer review process was seen as valuable and necessary, but the administrative burden of reviews was highlighted. In addition, a number of senior respondents were able to offer specific comments on the process and suggestions for improvements in both quality and efficiency. A clear understanding of the strengths and weaknesses of the review process, the load placed on boards and the long-term effects of reviews will contribute to the development of strategies for enhancing quality improvement. 3 The SNBTS, NHS QIS and University of Edinburgh s School of Health in Social Science and Centre for Integrated Healthcare Research are now taking the opportunity to explore issues around organisational development to enhance the longer term impact of reviews in terms of continuous quality improvement of a specific service, namely the Blood Transfusion Service in NHS boards. This scoping study of peer review by NHS QIS of blood transfusion services was carried out jointly by Dr Guro Huby of the University of Edinburgh s School of Health in Social Science and Dr Steve Cross of Human Reliability Associates. 4 The study is also an opportunity for the SNBTS to consider its role in supporting continuous quality improvement of the service in NHS boards in view of the complex governance structures for the Service. 1.2 Overview of the standards and peer review process Key stakeholders 5 NHS QIS was established in 2003 and leads the use of knowledge to promote improvement in the quality of healthcare for the people of Scotland. It performs three key functions: providing advice and guidance on effective clinical practice, including setting standards; driving and supporting implementation of improvements in quality; and assessing the performance of the NHS, reporting and publishing the findings. 1 An Independent Evaluation of the Impact of NHS Quality Improvement Scotland; Steve Cross, Clare Blackett and Lorna McKee, NHS QIS

7 In addition, NHS QIS also has central responsibility for patient safety and clinical governance across NHSScotland. 6 The SNBTS is responsible for collecting, processing, storing and supplying all blood and blood components in Scotland and NHS boards are responsible for ordering and managing their supplies in a safe and effective manner. The Scottish Government introduced a programme of work to improve and support transfusion practice in Scotland and, as a consequence, NHS QIS appointed a project group to develop clinical standards for blood transfusion practices. The project group developed four standards, covering core principles, clinical management pre-transfusion, clinical management hospital transfusion laboratory and clinical management blood and blood component collection, administration and monitoring. The Clinical Standards for Blood Transfusion were published in September Standards and review 7 NHS QIS operates to an established work programme in setting and reviewing standards of care. Project groups working on standards are convened by NHS QIS and comprise clinical and non-clinical members drawn from a range of backgrounds. These groups have the responsibility to oversee the development of standards, consult on the standards and recommend an external peer review process. The project groups are expected to adopt an inclusive development process where public and voluntary organisations play a role in addition to healthcare professionals. 8 Standards are developed to a set format and are based on an assessment of the evidence. Each standard states the level of performance that a board is expected to achieve and the rationale or reasons as to why the standard is considered to be important. It is then broken down into a number of criteria which state explicitly what must be achieved for the draft standard to be reached. In some cases, these criteria are categorised as essential or desirable; beyond this, all standards are seen as of equal importance. The project group has the further responsibility of testing the measurability of the standards through undertaking pilot reviews. 9 Once the standards have been developed and piloted, each board or service in NHSScotland is asked by NHS QIS to undertake a self-assessment of its services. This process is facilitated by a documentary submission from NHS QIS listing the standards, the criteria required within each one, and examples of evidence which the board or service might provide to demonstrate compliance. 10 The review process has three key phases: preparation (self-assessment) prior to the visit; the visit by a review team of relevant professionals and public partners; and the report production and publication following the visit. During the visit, the review team assesses performance using the NHS QIS continuing quality improvement scale (development/implementation/monitoring/review) or the categories met, not met and not met (insufficient evidence); in this review, the latter categories were used: Met applies where the evidence demonstrates the standard and/or criterion is being attained

8 Not met applies where the evidence demonstrates the standard and/or criterion is not being attained. Not met (insufficient evidence) applies where no evidence is available for the review team, or where the evidence available is insufficient to allow an assessment to be made. 11 In practice, the board or service submits evidence of its internal review to NHS QIS. This is subsequently distributed to the external peer review team who are required to review the self-assessment and evidence, independently and collectively, and subsequently meet with board or service representatives to discuss and question the evidence. They then form a consensus view as to the achievement against each standard. Findings are reported back to the board or service and are available in the public domain through NHS QIS. In order to provide a cross-scotland perspective, NHS QIS also produces and publishes a National Overview of the area reviewed. A schematic overview of the process is provided in Figure 1. Figure 1: overview of review process 2 2 NHS QIS, Local reports on Blood Transfusion, 2008, at -

9 1.3 Scope and objectives 12 The overall aim of the study may be summarised as the identification of options for improving the review structures and processes in place within NHS boards to better facilitate long-term service improvement. Within this, the specific objectives of the scoping study are to: investigate and map the impact NHS QIS reviews have on the organisation, delivery and quality control of blood transfusion services in selected contrasting NHS board areas build future capacity among the NHS board blood transfusion service and NHS QIS staff for taking forward continuous quality improvement disseminate the learning from the facilitation work to all NHS board areas and NHS QIS inform future NHS QIS peer review methodologies to enhance ongoing quality improvement. 13 To achieve these objectives, we examined the peer review process in four NHS boards. We focused our work on outputs from NHS QIS prior to the reviews, the self-assessment phase of the reviews, the review visit itself and the actions which took place in boards following the review. The methodology we employed for the scoping study is outlined in the next section

10 2 Methodology Board sampling 14 In conducting the scoping study, we focused on four Scottish NHS boards. These were chosen to provide a range of environments Board 1, serving a large mixed urban and rural population Board 2, serving a large urban population Board 3, serving a medium urban population Board 4, serving a small mixed rural and urban population. The boards also represented a mix of time reviewed, from early in the review process to later boards. Observation phase 15 After meeting with key stakeholders at NHS QIS and SNBTS, we conducted direct observations of two board reviews (boards 3 and 4). We attended two days of pre-review meetings and two review visits. During the formal review meetings we adopted a strict observational approach, and had further discussions with review team members and NHS QIS representatives outside of the meetings. Our aims during this phase were to: understand and describe the process of evidence review carried out by the review team understand and describe the review visit identify issues of process and effect which would inform the development of semistructured interviews develop a model for the impact evaluation. Interview phase 16 Following the observation phase, we conducted a round of semi-structured interviews with board representatives, NHS QIS/SNBTS stakeholders and review team members. NHS QIS/SNBTS stakeholders provided a view of the strengths and challenges identified during the entire review process. Interviews took place at the conclusion of the reviews. 17 Board interviews focused on the three phases of review: actions prior to the review (self-assessment) the review visit actions after the review. We interviewed 17 board representatives, comprising four biomedical scientists, four senior clinicians, five transfusion practitioners, three risk managers and one quality manager

11 18 Review team interviews focused on: training and preparation for the review pre-visit review of evidence review team meeting prior to the visit the review visit. We interviewed two review team leaders, two leads for Standard 3, one lead for Standard 2 and one public partner. 19 For board representatives and review team members, we developed interview instruments covering each area. In all interviews, as well as seeking responses to practical issues such as process, workload, changes made as a result of the process, we employed open questioning to elicit respondents views, feelings and responses to the review s format, organisation and conduct. Interviews took place face-to-face on eight occasions, covering all four boards; other interviews were conducted on the telephone. All interviews were held in confidence. Impact evaluation 20 We employed a model of direct impact evaluation based on a four-step model of behavioural change which has been applied previously in NHSScotland in assessing the impact of NHS QIS interventions 3. These steps are: perceived need for the intervention, or receptivity changed knowledge, belief or attitude in the stakeholder group changed policy or practice in the stakeholder group changed outcomes for the members of the stakeholder group. Throughout this report, we present quantification of our findings where possible, and illustrate the range of views through direct quotes from interviews. 3 An Independent Evaluation of the Impact of NHS Quality Improvement Scotland; loc. cit

12 3 Findings 3.1 Pre-visit activity 21 In this section, we review the activity that took place during the self-assessment period at the four boards studied, and also the work of the review team carried out prior to the meeting. In terms of our model of impact evaluation based on the four steps of receptivity, change in knowledge, change in policy and practice, and change in patient outcomes, this part of the evaluation focuses most closely on the receptivity and policy/practice stages. 22 The inception of the peer review was carried out in boards through the NHS QIS liaison co-ordinators in each NHS board. These staff had the task of disseminating self-assessment material to appropriate departments and personnel and co-ordinating review visits. In some cases, liaison co-ordinators also took an active role in the self-assessment. (Liaison co-ordinator may be an additional role that a staff member adopts within a board; the contribution of the co-ordinator to a review will vary.) Self-assessment phase in boards Perceived purpose and significance of the peer review 23 All but four board respondents perceived the review as an assessment against the standards. Only one respondent described it as an inspection, although the term inspection was used on many occasions in discussions by board respondents. On the other hand, only one respondent described the process as a peer review, with one other emphasising the sharing of experience. The NHS QIS reviews are, in fact, explicitly not inspections, in that NHS QIS has no statutory powers to enforce change or sanction non-compliance. Nevertheless, we formed the view that they are seen as serious inspections by board staff perhaps more as inspections than as peer reviews. Though peer review may sometimes be taken to include the concepts of support in improvement or the sharing of experience, this was not an explicit element of the process we evaluated. There were, however, several examples of sharing of practice and experience during the review visits. 24 The overall perception of the importance of the review is illustrated in Figure 2, followed by quotes illustrating contrasting views

13 Perceived importance of the review (n=17) How important do you believe your colleagues felt the review to be? Not important Quite important Very important How important did you personally f eel the review was? Figure 2: perceived importance of the review It was a genuine peer review. In the laboratories, it was understood as a peer review and was generally seen as unthreatening. Biomedical scientist We saw this as an inspection, though the stressful nature of the inspection dissipated somewhat on the day. It was definitely a challenge. Consultant haematologist 25 NHS QIS review visits are one of three review and inspection regimes for the production and use of blood and the use of blood components or products and focuses most closely on clinical practice. The Clinical Pathology Accreditation (CPA) inspects and accredits laboratories, and the Medicines and Healthcare products Regulatory Agency (MHRA) inspects and accredits blood establishments (establishments which collect, process and test human blood and blood components) and hospital blood banks (hospital units which store, distribute, and perform compatibility tests on, blood and blood components for use in hospitals). The NHS QIS review visit partly overlaps with these, but concentrates on the way blood components/ products are distributed and used within a clinical service. CPA and MHRA accreditation are part of the requirements for an NHS QIS review. As a result of this, five respondents in laboratories viewed the NHS QIS review as yet another inspection. However, where laboratories have the accreditation of CPA and MHRA, demonstrating compliance with the relevant NHS QIS standards (mostly within Standard 3a, Clinical Management hospital transfusion laboratory) can be relatively straightforward due to the overlap of requirements (three respondents). 26 Respondents opinions as to the value of the inclusion of Standard 3 were variable. Five felt that this element could be dispensed with entirely if CPA and MHRA accreditation were upto-date, others took a different view:

14 CPA and MRHA completely cover standard three. Consultant haematologist I don t think there s too much overlap with MHRA and CPA when considering the standards overall. QIS does recognise the existing accreditation of labs. The MHRA assessment is becoming a more boxticking exercise. Labs do not have a set visit schedule from the MHRA and only receive visits where non-compliance has been registered. A tick-box does not mean the standard is achieved and the sustainability of this method is in question. Because the MHRA won t be able to achieve high coverage, QIS is needed and, in fact, QIS may not go far enough. Biomedical scientist All the assessments are linked but QIS has a more clinical focus. Standard three, much of which is covered by MHRA or CPA, also links the laboratory to wards via samples and transfusions. Senior biomedical scientist Agreement with the standards 27 Overall, there was strong endorsement of the NHS QIS standards from board respondents. The standards were also seen as clearly expressed, as covering the right areas, and as comprehensive. These findings are illustrated in Figures 3, 4, and 5. Were you in general agreement with the content of the standards for blood transfusion? (n=17) Yes No Partly Figure 3: agreement with standards

15 How clear was the expression of the standards? (n=17) Very clear Quite clear Not clear Figure 4: clear expression of standards Respondent views of self-assessment dataset (n=17) Were there any important areas not covered by the evidence request? Were there any areas, which, in your professional opinion, were over emphasized? No Yes Were there any areas, which, in your professional opinion, were not covered? Figure 5: comprehensiveness of standards data set 28 Despite general agreement with the standards, a number of respondents raised concerns about specific issues. These centred on two areas: the requirement to demonstrate competency-based training for staff, and the data set required for patient identification. Despite the strong imperatives for training (see the 2007 SHOT recommendations, for example 4 ), achieving compliance with the requirement to train all staff was seen as an impractical target by four senior respondents, due to the turnover of staff or the difficulty in winning the co-operation of senior medical staff. These issues were noted in all boards studied, and are illustrated in the following quotes: 4 Serious Hazards of Transfusion, Annual Report 2007, Summary;

16 The main issue with regard to the standards themselves was the issue of competency-based training for senior staff. Some senior staff feel that they do not need to prove competence and can be prickly and hypersensitive; it is usually the same people who react in this way. I believe that they do need to prove their competence and try to explain that this is a national standard, not a personal issue. This matter received a lot of focus from the team but there was still resistance from the more mature generation. Biomedical scientist [I am in] general agreement, but some standards will be aspirational. For example, staff training is problematic because we can t train everyone and do not believe that any board would be able to do so this is an unavoidable, moving conveyor effect as a result of staff turnover. Training packages are good but do not reliably assess competence. It might be possible to do an observational study on porters, for example, but this becomes difficult for nurses and doctors. Training packages need to address the competence needed at different levels of the organization or treatment pathway. For example, senior consultants employ a different skill set and have different decision-making needs when compared to junior doctors or nurses. Though this should be an inclusive hierarchy of skills, where senior consultants skill sets need to include junior doctors skill sets, the competence assessments should be relevant to the daily practice of each grouping. It might be possible to create a grid of competence assessments, for example. At present, in this board, some consultants refuse to demonstrate their competence through non-participation and feel affronted by the requirement to demonstrate competence at a basic level. Senior clinician Patient ID is a poorly drafted standard and is not consistent with other standards. The British Committee for Standards in Haematology sets a minimum data set of four elements: surname, forename, date of birth and unique identification number. CPA and MHRA adopt the same standard. The gender requirement in patient identification included in the NHS QIS standard is at odds with other standards and has no clinical significance 5. It may be that the introduction of CHI will cover gender and location. It is unlikely, however, that boards will change their policies on patient identification to meet QIS standards, in my opinion. Consultant haematologist The self-assessment process in boards 29 The approach to carrying out the self-assessment and providing the evidence portfolio to NHS QIS varied across boards. In all boards, the key staff focusing on the review were transfusion practitioners and senior clinicians. Risk management and clinical governance staff had only a peripheral involvement in the practical aspects of self-assessment in the boards we studied. 5 This view was not shared by other respondents, who pointed out the advantages of gender identification in neonatology

17 30 Board 1 (serving a large mixed urban and rural population) was able to set up a working group and involve a relatively large number of staff over the 3 months available to them. In this board, staffing was seen as adequate, and the perception of the workload was medium or high by the key staff involved. We set up a QIS working group, which consisted of two transfusion practitioners, a consultant anaesthetist, a clinical director, representatives from two laboratories, a representative from the independent sector, a representative from community facilities and satellites and a clinical governance representative. We met every 4 6 weeks until we submitted our evidence. We also organised prep sessions for staff in advance of the review, especially for those at a more junior level, including staff nurses, medical laboratory staff, porters and so on. Junior staff were very apprehensive about attending the review meeting, so we included presentations on QIS, the blood transfusion standards and the board s response to them. We emphasised the importance of openness and honesty in responding. Transfusion practitioner 31 Board 2 (serving a large urban population) carried out the self-assessment over a period of 4 5 weeks. They set up a working group, but in practice the transfusion practitioners took the central role. These staff assessed the workload as high, and reported that almost all of their time was devoted to the self-assessment during this period. We had very little time to conduct this a matter of a few weeks in the end. A QIS liaison officer coordinated a group, which included transfusion practitioners, the liaison co-ordinator, the chair of the hospital transfusion committee, a consultant haematologist, and a transfusion lead from another site. The liaison co-ordinator proposed a timetable for completion of the self-assessment We had about four five weeks altogether. This was influenced by the fact that [a senior manager] wouldn t let us begin work on the self-assessment documentation until the actual document was received despite the fact that we had a fair idea what was included anyway. At the first meeting of the group, we assigned leads for each section of the standards. For example, a consultant haematologist led for standard three; transfusion practitioners led for standards one and four, with advice from lead clinicians. Generally speaking, however, transfusion practitioners ended up being central to all of this and being the conduit for all communications. Transfusion practitioner 32 Board 3 (serving a medium-sized urban population) carried out the self-assessment over a 6- month period, in a group involving 6 10 people in all. Most of the practical work, however, was conducted by two transfusion practitioners, who again devoted most of their time to the self-assessment and assessed the administrative burden as high. 33 Board 4 (serving a small mixed rural and urban population) had a smaller number of sites to deal with and a smaller team. Two main people were involved throughout the process the consultant haematologist and the transfusion practitioner, with smaller contributions from other staff. The workload was assessed as high by both respondents

18 Obtaining the evidence 34 In general, respondents found the evidence relatively easy to obtain, as illustrated in Figure 6. Was the evidence requested easy to obtain? (n=17) Yes No Possibly Don't know Figure 6: obtaining evidence This is not to say, however, that the work involved was not time-consuming, especially for standards outside of those already under assessment by CPA or MHRA. Three transfusion practitioners, who felt that the bulk of the workload in this review had fallen to them, described difficulties in chasing evidence requests, collating evidence from several sites and checking for consistency. This was sometimes work that, while it may have been part of routine or annual work programmes, was demanded by the time-frame of the review - highlighting the general finding that reviews can accelerate change. 35 Because boards each have developed different practices and policies in some areas, it can be difficult to specify exactly what evidence to submit in support of what standard. This is complicated further where there are sites within the same board with varying policies or audit practices, and this underlines two potential weaknesses in the review system. 36 The first potential weakness in the review of evidence is that evidence may be unrepresentative. Where site policies, practices or audit results vary, it is possible to cherrypick evidence in order to show the board in the most favourable light. We emphasise that we did not encounter this practice in our review of the four boards in this study; two respondents, however, described their knowledge of selective presentation in other situations

19 There was a difficulty in providing evidence for all [board] sites. We finally decided to provide evidence for every single site, rather than cherry-picking the best evidence from one of them and providing that to QIS; we believe that other boards have cherry-picked in this way, and that therefore there is a lack of consistency. As a result of this, we heard that the evidence we had provided was a nightmare. Having worked so hard to demonstrate compliance over all sites, we found this demoralising. Transfusion practitioner 37 The second potential weakness is one of comparability. Where audit evidence is provided, the context and methodology of the audit may vary from site to site or between boards, rendering the results difficult to compare. This can occur when audits are carried out by different teams with different contextual knowledge, or when the methodology of the audit varies: Some audits were carried out specifically for QIS compliance. The people carrying out the audit, though, can affect the outcome. For example, the clinical effectiveness team, in preparing for the standard review, audited notes in [hospital]. This was to do with not including observations in patients notes. This [hospital] only achieved 50% compliance on the audit. However, when clinicians examined this, knowing that observations were filed in a slightly different place, their compliance level estimate was considerably higher. Transfusion practitioner Changes made in policy and practice during self-assessment 38 Blood transfusion is an area that receives a great deal of attention, both locally, through programmes managed by SNBTS and local boards, and nationally, through CPA and MHRA accreditation and two overarching reporting systems, Serious Hazards of Transfusion (SHOT) and Serious Adverse Blood Reactions and Events (SABRE). As such, it is an area that is under pressure for continuous review and improvement, and the precise causal attribution of changes made in the field can be difficult. In this study, we attempted to identify what changes were made, when they were made and why they were made, in order to separate the effect of the NHS QIS review from that of other influencing factors. In the following paragraphs, we describe changes made only during the period prior to, and related to, the review. 39 Board 1 made four key changes in blood transfusion during this period in the areas of patient identification, training and membership/activity of the hospital transfusion committee. As evidenced in the quote below, three of these improvements were in train prior to the review, but were receiving insufficient attention

20 Transfusion policy was updated during this period. There was a lack of clarity beforehand and the policy was stuck in committee until the review. Training has been an ongoing battle, competing with everything else in the board; during the run-up to the review there was a sudden improvement plan for training and it was drafted with unequivocal support from the nursing director and medical director. This was an excellent response, though there are still some problems with other senior staff. I believe that the QIS review representing an external group looking at [the board], whose results will be in the public domain, makes a real difference. Patient identification policies had also limped along before the review and were finalised during the run-up. This document, which had been initially stimulated by a blood transfusion incident, had been in development and consultation for two years previously. The hospital transfusion committee became more active, with a re-structured membership, and was attended better. Transfusion practitioner 40 Practitioners in board 2 were aware of changes needed prior to the self-assessment, and began to carry out a number of necessary improvements directly as a result of the NHS QIS review: documentation of patient discussions further provision of information leaflets and documentation unification of minimum data set policy for patient identification across sites inclusion of gender on patient ID training of key staff. 41 Board 3 introduced three new policies during self-assessment: major haemorrhage policy emergency blood management policy informed consent policy and documentation. In addition, the transfusion manual and the patient identification policy were updated. 42 Board 4 had recently undergone a change in staffing at the time of the review, with a newly appointed clinician leading change in the area. Though many changes were made in policy and practice in this board during the self-assessment period, respondents felt that each of these changes were already firmly in train. However, the lead clinician was convinced that the timing of the review had been materially helpful to the progress of improvement he had begun:

21 Overall, the process makes a difference, in that the board and senior management take notice of the QIS review. It enables those with responsibility to kick backsides in order to get resource. In particular, it has been useful in getting the CFO to take notice. Although the review didn t identify priorities that we didn t have anyway, in part through CPA/MHRA assessments, it has been helpful in gaining support. QIS is clearly of more significance to management than other assessments in the area. The COO here is particularly keen and uses QIS reviews to improve operations generally. This is accomplished through peer pressure at management level. Consultant haematologist 43 In three of these boards, a small number of new audits were carried out in order to demonstrate the degree of compliance with standards. It was notable that in none of these boards did respondents report changes in policy or practice in laboratory work, highlighting a general view that the added value of the NHS QIS review is chiefly in clinical areas. 44 Overall, we have formed the view that during the period of self-assessment, significant improvements were made in all boards studied. In two cases, major improvements were already in train, but the process of change was given significant acceleration by the forthcoming review. In two cases, we would attribute the changes made more directly to the review

22 Review of evidence by the review team 45 Review teams were supplied with evidence submissions for the board reviewed in advance of the review visit. Not all members had been able to thoroughly review all evidence provided, however, due to time constraints. Team members views of the evidence are shown in Figure 7. I reviewed as much as I could - but there was an outrageous excess of evidence provided. MHRA need much less, for example. Lead for Standard 3 [I reviewed all the evidence] for the most part. But we must judge the relationship between written work and actual practice. Note that the board does not have to present all evidence gathered in an area: bad audits or unfavourable evidence may not be presented at all. Team leader Review Teams' pre-visit work (n=6) Did you have a clear understanding of the standards and their significance? Had you identified strong, weak or missing evidence? Had you identified where standards were met or not met before the visit? No Yes Were you able to review all the evidence provided? Figure 7: team members views of the pre-visit work 46 In general, team members believed that they functioned well in the evidence review, and were able to sift through the evidence thoroughly. Continuity of assistance from NHS QIS was cited as an important factor, as was the existence of established relationships between team members. There are pros and cons with regard to the same people being present in the team. When established relationships exist, the team can function more smoothly. Against this is the possibility of having someone in the team that you don t work easily with. Team member

23 Direct observation of evidence review 47 We present here a series of observations from the study team, in no particular order of priority. We emphasise that they are based on observations of two reviews that were used to orient the research team and inform the development of the instruments used in the more extensive interview phase of the research. 48 Prior to the review visit, the review team meet as a group to review the evidence submitted by the board and distributed after some editing by NHS QIS for review. Generally, at this meeting there will be a team leader, review leads for each section of the standards, a public partner and staff from NHS QIS, including the project officer with direct responsibility for the board s review. These meetings were observed to be intense sessions where the entire standard set was evaluated against the evidence, a consensus view formed, and areas of weakness or missing evidence identified for further review with board representatives on the following day. When the volume of evidence was high, as it was in the boards we observed, this would clearly become a significant challenge. In one meeting, we observed that new or additional evidence was delivered to the team during the course of the meeting. This may reflect the board s commitment to achieving the standards, but made a balanced review of the evidence extremely difficult. 49 The role of the team leader was critical. It was apparent that the team leaders had achieved a high degree of familiarity with the evidence under consideration without which it would have been difficult to guide the team through the high volume of information. We observed team leaders to be authoritative and coherent, and to actively seek consensus from the team when arriving at decisions. 50 Though there was some variability between the two teams, both approached the task, under the guidance of the team leader, from the perspective of helping the board, rather than an attempt to catch them out. This was evidenced by the team leaders encouraging of members to consider their own professional experience and expertise and to evaluate the evidence in this context. Team leaders sometimes urged the team members to share their own knowledge and experience of good practice with the board under review. 51 With the exception of the public partner, whose role in the team is to represent the patient, all team members had expertise and experience in the area of blood transfusion. We observed that this enabled a degree of empathy with the staff in the board under review. Difficult areas in the standards were recognised by the team from their own experience, and this had the effect of underlining the peer element of the peer review, though not to the detriment of the objective nature of the assessment, which was always rigorous. In our view, this empathy, born of the recognition of shared difficulties and experience, is a major factor in the intended supportive nature of the review, as opposed to any policing role. 52 NHS QIS staff had the difficult role of collating and sourcing the evidence for the team, many of whom might be unfamiliar with areas outside of their own expertise. We observed that NHS QIS project officers would act as advisors on factual issues relating to the standards, to the evidence submitted, and to the consistent interpretation of evidence across boards

24 They were thus able to inform the team s discussions without actively guiding them. Only in very rare cases would the NHS QIS team offer opinions as to the attainment of standards. 53 In both of the meetings we observed, the key outcomes were: a provisional view of where the standards and key criteria had been met or not met a set of areas where evidence was weak and further information would be sought from the board a schedule of which member of the team would take responsibility for questioning in these areas. Effectively, and since the format for the review meeting followed the numerical order of the standards, this resulted in an agenda for the review of the following day. 54 Overall, we formed the view that the review teams were: thorough, in that each standard and the evidence supporting it were examined in detail expert, in that each team was multidisciplinary and had experienced representatives for each standard fair, in that the team sought consistency with other reviews balanced, in that team leaders would seek consensus and discuss issues openly positive in that teams adopted the role of peers aiming to ensure and encourage high standards of performance and to share experience

25 3.2 The review visit 55 In this section, we consider the review visits themselves, from the perspective of board participants and the review teams, and, in two cases, our own direct observations. In terms of our model of impact evaluation based on the four steps of receptivity, change in knowledge, change in policy and practice, and change in patient outcomes, this part of the evaluation focuses most closely on the receptivity and policy/practice stages. The perspective of board participants 56 As we have described earlier, the review visits are taken seriously by board staff, to the point where the distinction between policing and review becomes blurred. In some cases, however, the sense of inspection changed during the visit itself. The visit was OK, but nerve-wracking. The board team was very nervous up-front. Transfusion co-ordinator The notion of an inspection by NHS QIS, using the word advisedly, is threatening in itself. The review team s actual visit, however, was non-threatening and very much a peer review. Senior clinician When the review begins, the ambience can be threatening. The team should tell the board how they all fit in and how the day will go: break the ice. It could be friendlier. It is stressful, feels like an inspection and you can see the adrenaline and stress levels rise. Risk manager Very good, as expected. Clinical effectiveness organised the visit, which was held in the main lecture theatre due to the number of people who attended from all three sites. Inevitably, this created an usand-them situation due to the layout of the room. Consultant haematologist I felt that the visit began as an inspection and developed into a genuine peer review. One important point was that the review team would sometimes ask Is there anything else you want to say? This helps the peer review aspect of the process by allowing participants to bring out good practice and discuss general problem areas openly. It helps in sharing experience and in openness and encourages the organisation to take the action plan seriously. Transfusion practitioner 57 When asked for their overall impression of the visit, all respondents were positive and judged it good or very good

26 QIS put together a balanced team of people like us who were there to learn and share experience as well as to review. They seem well-trained and to work hard there is a lot to get through. The main benefit from my perspective is the networking we will go back to those people. Great potential. Risk manager It was genuinely a peer review and I did not perceive it as threatening or as an inspection. The areas that were questioned by the review team were those that needed attention and there was no impression of point-scoring. Transfusion practitioner The team related well all the way through the hierarchy of the board. They were aware of nervousness with junior staff but showed strong empathy and handled porters and MLAs very well. Senior biomedical scientist 58 In general, respondents felt that the meetings were well-attended by the appropriate personnel, were focused on the right areas, and had relevance to the daily practice of blood transfusion (Figure 8). Respondent views of review visit (n-17) Was the visit relevant to daily practice? Was the visit focused on the right areas? Partly No Yes Were the best or most appropriate participants f rom your Board present at the visit? Figure 8: board participants view of the visit 59 The NHS QIS review does not involve direct observation of practice, nor can it be as thorough in direct examination of laboratory practice, for example, as CPA or MHRA. The review methodology assesses compliance with standards through a review of documentation, followed by questioning and probing of staff by the team on the day of the visit, with the additional benefit of sharing practice and experience between peers. Most respondents felt that practice was assessed and there were many useful views expressed in the interviews:

27 I felt comfortable with the review and the methodology used. However, policies alone are not enough and that the way policies are implemented is critical. Can a review address practice? I am not convinced. Biomedical scientist As a tabletop exercise, this review was more far-reaching than I had expected, and I was viewing this from the perspective of a person used to thorough inspections. I felt that the review visit was a good opportunity for cross-department learning, especially for my clinical colleagues, and that the review overall was very comprehensive. Consultant haematologist It was apparent that the team were able to learn a lot about practice in the board very quickly at this meeting. I believe the methodology is effective. Senior clinician The team were very probing and MLA/BMS were all asked questions. This involved the correct people and gave them feedback, which is essential in maintaining involvement of staff at all levels. The questions were about daily practice and included even porters. This has the effect of binding all levels of the organisation into a team and confirming the success of our training. Senior biomedical scientist The team probed and questioned as to what we actually do in the board. For example, on the CBT issue, they asked people what they felt about it. There is some frustration from the board, in that if evidence is presented in the form of minutes from meetings, this may not reflect the thoroughness of practice or may be inaccurate. Transfusion practitioner 60 The involvement of senior management demonstrated that in some boards, the review is taken seriously throughout the organisation, and that the work of staff in preparing for the review was valued. In some cases, however, respondents were disappointed that senior management or risk management were not there throughout the review. The head of the medical unit was there and it was good to see the involvement, which emphasised the importance of the standards to a wider audience. It was also important for senior management to see the effort put into achieving the standards. Senior biomedical scientist We put a lot of effort into it and were maybe over-represented. It was notable that risk management were only present in the afternoon sessions so had no view of our achievements in the core principles section. Senior management was not present, except at the opening session: they were more interested in speaking than listening. Transfusion practitioner

28 61 Board participants were also very positive in their views of the review team s expertise and approach to the review visits, as illustrated in Figure 9. We heard no negative comments on review team skills or levels of expertise during these interviews. Respondent views of review team characteristics (n=17) Level of empathy Level of authority Presentational and interpersonal Attention to detail Level of organizational knowledge Level of technical knowledge Average Very good/good Figure 9: Board participant views of review team. 62 Overall, we formed the view that board participants felt that the review visits: were well conducted were conducted by reviewers with a high level of technical and organisational knowledge were non-threatening and co-operative were focused on relevant areas and probed actual practice