SIGNIFICANT ADVERSE EVENT REVIEW REPORT WEB MALWARE INCIDENT

Transcription

1 Report Author(s) Commissioned By SIGNIFICANT ADVERSE EVENT REVIEW REPORT Kerri Todd, AHPM Lesley Anne Smith, DoQ Calum Campbell, Chief Executive, NHS Lanarkshire Incident Date 12/05/2017 Date of notification 12/05/2017 Datix Reference WEB Report Date 04/10/2017 SAER Team Lead SAER Team SAER Outcome (see codes below) Dr Iain Wallace, Medical Director, NHS Lanarkshire Dr Lesley Anne Smith, Director of Quality, NHS Lanarkshire Carol McGhee, Corporate Risk Manager, NHS Lanarkshire Denise Brown, Patient Administration Transformation, ehealth Directorate, NHS Greater Glasgow and Clyde Kerri Todd, Assistant Health Promotion Manager, NHS Lanarkshire (secretariat) CLINICAL : NON CLINICAL: CLINICAL 1. Appropriate care well planned and delivered, unavoidable outcome. 2. Indirect system of care issues lessons can be learned although it did not affect the final outcome. 3. Minor system of care issues different plan and /or delivery may have resulted in a different outcome i.e. systemic factors identified though uncertainty regarding impact on outcome. 4. Major system of care issues different plan and/or delivery would on the balance of probability have been expected to result in a more favourable outcome i.e.: systemic factors considered to have an adverse and causal influence on outcome. NON CLINICAL 1. Appropriate services everything was performed correctly. 2. Indirect service issues lessons can be learned although it did not affect the final outcome. 3. Minor service issues aspects of the process were not performed correctly but it is difficult to say how much they contributed to the end result but they might have played a part in it - inconclusive. 4. Major service issues the process was not performed correctly and caused the end result. CHANGE RECORD Date Author Change Version No. 29/08/17 Kerri Todd 3 rd draft following conclusion of interviews v.3 Kerri Todd/ Carol Amendments to introduction and 31/08/17 McGhee background sections v.4 31/08/17 Lesley Anne Smith Formatting changes v.5 31/08/17 Amendment to wording of findings and Iain Wallace/Lesley fishbone diagrams. Outstanding queries Anne Smith followed up v.6 Inclusion of responses to outstanding 03/09/17 Lesley Anne Smith queries v.7 Final draft to Team Members and 04/09/17 Lesley Anne Smith Commissioner for review v.8 05/09/17 Lesley Anne Smith Updates following review by Commissioner v.9 06/09/17 Iain Wallace Amendment to rec. 11 and final review v.10 04/10/17 Iain Wallace Amendment after discussion at CMT/PPRC Final Agreed by Commissioner Final 4 October 2017 Page 1

2 EXECUTIVE SUMMARY 1. Situation On the afternoon of Friday 12 May 2017, the computer malware virus WannaCry started to infect a range of NHS Lanarkshire (NHSL) computer systems in both acute services and primary care. A NHSL Strategic Group was quickly convened to assess the severity of the situation and agree a prioritised response and actions to be taken to protect patients and the ehealth estate. In response to the escalating situation in NHSL the methodology of a major emergency response was adopted, albeit, this event was not declared formally as a Major Emergency/Incident. In taking this approach, NHSL quickly moved into a Command & Control position directing all Business Continuity Plans to be operationalised. Frequent meeting of the Strategic Group were held over the weekend period to continuously risk assess the situation and oversee recovery of priority ehealth and operational systems, with wider recovery continuing during the following working week and beyond. 2. Background WannaCry is a self replicating ransomware virus/worm which encrypts files in the computer it infects and causes a message to appear, which states that the files will only be released on the payment of a bitcoin ransom. This particular malware only infected computers and devices which were running Microsoft windows systems. NHSL recognised the increasing risk from cyber-attack back in 2015, identifying it as a specific risk (1364) within the Corporate Risk Register linked to a series of mitigating controls (see figure 1 below). The current assessed level of risk was raised from medium to high in October 2016, in response to an escalation of phishing and resultant increase in the potential for a significant cyber-attack. 3. Assessment 494 weekend patient appointments/procedures were cancelled. By Monday 15 May 2017, the main patient care systems were operational again and patients were able to attend for operations and appointments. Due to the backlog in processing laboratory specimens, appointments for routine bloods in treatment rooms did not restart until Wednesday 17 May. Final 4 October 2017 Page 2

3 While the malware affected many NHS organisations across England and Scotland, it had a significant impact on NHSL, with 1338 PCs affected in both acute and primary care settings. Over the course of the week following the incident, the infected PCs were either cleansed of virus on site or were replaced while cleansing took place. This was done on a prioritised basis, with additional IT support being provided by NHS Greater Glasgow and Clyde and NHS Ayrshire and Arran. No data was stolen during the incident and the understanding to date is that no data was lost or unrecoverable. A Significant Adverse Event Review (SAER) was commissioned through the Corporate Management Team (CMT), to enable an understanding of the factors that may have contributed to the situation, and to review the NHSL response. The review team has concluded that there were major service issues that contributed to the event. There were no significant adverse patient outcomes although there was the potential for major service implications. These were mitigated by the actions taken from May. 4. Recommendations The review team have made a number of recommendations that will be considered by the Corporate Management Team and the Planning, Performance & Resource Management Committee of the Board. Final 4 October 2017 Page 3

4 MAIN REPORT SIGNIFICANT ADVERSE EVENT REVIEW REPORT 1. Introduction The Incident On the afternoon of Friday 12 May 2017, NHS Lanarkshire (NHSL) became aware of an emerging ransomware virus infecting a range of computers and software across the NHSL ehealth estate. A NHSL Strategic Group was quickly convened to assess the severity of the situation and agree a prioritised response and actions to be taken to protect patients and the ehealth estate. Concurrently, it was recognised that this ransomware attack was not isolated to NHSL but was affecting systems, including healthcare systems, on an international scale. In response to the escalating situation in NHSL the methodology of a major emergency response was adopted, albeit, this event was not declared formally as a Major Emergency/Incident. In taking this approach, NHSL quickly moved into a Command & Control position directing all Business Continuity Plans to be operationalised. Frequent meeting of the Strategic Group were held over the weekend period to continuously risk assess the situation and oversee recovery of priority ehealth and operational systems, with wider recovery continuing during the following working week and beyond. A Significant Adverse Event Review (SAER) was commissioned through the Corporate Management Team (CMT), to enable an understanding of the factors that may have contributed to the situation, and to review the NHSL response. Background and Context Healthcare organisations worldwide have become more dependent on e-health solutions to help transform how their businesses operate. However, commensurate with the growth in the use of, and dependence upon, electronic systems has come an increasing risk from disruptive and serious cyber-attacks with the potential to impact on service delivery and patient safety. Many UK wide healthcare organisations have been subject to cyber-attacks experiencing disruption that has required investment to protect their organisations. It is also recognised that the level of attacks, the tools and the software used are becoming more sophisticated. NHSL recognised the increasing risk from cyber-attack back in 2015, identifying it as a specific risk (1364) within the Corporate Risk Register linked to a number of mitigating controls (see figure 1 below). The current assessed level of risk was raised from medium to high in October 2016, in response to an escalation of phishing and resultant increase in the potential for a significant cyber-attack. Final 4 October 2017 Page 4

5 Risk ID Date Opened Corp. Objective Title Description Risk Level (initial) Mitigating Controls Risk Level (current) Risk Level (tolerance) /11/2015 Safe Risk of cyberattack in respect of stored NHSL data There is a risk of malicious intrusion into patient data stored on NHSL digital systems. This is a growing risk as "cyber hacking" becomes more sophisticated and there are regular high profile examples of such activity reported in the national media, with the potential to result in significant adverse publicity for NHSL. HIGH 1. Security provided as part of national data communications contracts, i.e. SWAN. Local Firewall and intrusion detection arrangements. Local system security arrangements, i.e. password protection, audit capability HIGH MED Figure 1 Successful cyber-attacks involve the development of specific malicious software designed to disrupt, damage and gain access to a computer system. Generally, this malicious software is referred to as malware. Ransomware is a serious type of malware that blocks access to a computer or its data by encrypting its content and demands money (often bitcoins) in return for restoring access. Spread can be through infected files sent by . It can also be spread through use of computers that are already infected by viruses that enable a back door for further attacks. Reliance on unsupported software, e.g. Windows XP, is known to increase the vulnerability to a malware attack. It is not thought that healthcare organisations were specifically targeted. Within the ehealth community, the term patch refers to a piece of software that is designed to fix security vulnerabilities and other bugs to improve the usability or performance whilst forming part of systems defence. Deploying patches is a process known as patch management, and it is a necessary part of any organisation s defence to cyberattack to have a patch management strategy aligned to the ehealth estate in support of an overall organisation security policy. In the context of this review, it is important to note that in 2014 Microsoft informed users that from 2015 onwards Windows XP would no longer be supported and as a result no patches would be released beyond that point. In the event of a ransomware/malware attack occurring, it is essential that organisations have appropriate emergency plans in place in the same way as for other scenarios, e.g. major road traffic accidents. NHS Scotland and Health Boards have established major emergency plans to respond to significant clinical incidents and public health issues. Final 4 October 2017 Page 5

6 2. Review team Dr Iain Wallace, Medical Director, NHS Lanarkshire (Lead) Dr Lesley Anne Smith, Director of Quality, NHS Lanarkshire Carol McGhee, Corporate Risk Manager, NHS Lanarkshire Denise Brown, General Manager, Patient Administration Transformation, ehealth Directorate, NHS Greater Glasgow and Clyde Kerri Todd, Assistant Health Promotion Manager, NHS Lanarkshire (Secretariat) 3. How the review was carried out A hot debrief took place within two weeks of the incident to document immediate actions and learning points (see appendix 1). This provided the panel with an overview of the incident and identified the key stakeholders to be interviewed as well as areas to be explored in more detail as part of the review. The SAER terms of reference (appendix 2) were signed off by the Chief Executive and tasked the panel with: investigating the preparedness of NHSL for cyber-attacks and understanding the roots causes of why the Board was affected investigating the response to the incident and the impact it had on patient care and services determining any ongoing vulnerabilities to cyber-attacks providing recommendations on measures which could reduce vulnerabilities and improve the response to future incidents To gather evidence, the panel interviewed the following individuals: Donald Wilson, General Manager, ehealth Calum Campbell, Chief Executive Colin Sloey, Director of Strategic Planning and Performance (executive director oncall) Gabe Docherty, Interim Director of Public Health Dr Femi Oshin, Consultant in Public Health Medicine Dr Philip McMenemy, Associate Medical Director and chair of Information Governance Committee Craig Cunningham, Head of Commissioning and Performance, South Lanarkshire Health and Social Care Partnership Christine Jack, Operational Manager, Health and Social Care North Lanarkshire Heather Knox, Director of Acute Services Calvin Brown, Acting Head of Communications Alan Robertson, Emergency Planning Officer Final 4 October 2017 Page 6

7 Alan Ashforth, Infrastructure Operations Manager (security), ehealth Stuart Graham, Head of Infrastructure, ehealth Interviews took place between June and August Interview questions were agreed in advance and shared with participants. A further, follow-up interview took place at the end of August with Donald Wilson to clarify some issues that arose during other interviews. All interviews were analysed and key contributory factors identified using a fishbone /cause and effect diagram (see appendix 3). 4. Detail of incident On the afternoon of Friday 12 May 2017, the computer malware virus WannaCry started to infect a range of NHSL computer systems in both acute services and primary care. WannaCry is a self replicating ransomware virus/worm which encrypts files in the computer it infects and causes a message to appear, which states that the files will only be released on the payment of a bitcoin ransom. This particular malware only infected computers and devices which were running Microsoft windows systems. At approximately 3.00pm on 12 May 2017, it became clear to the ehealth management team that NHSL systems were being affected by a malware attack. At 3.30pm the Head of ehealth met with senior members of his team and agreed to shut down some critical systems to prevent spread of the attack across the estate. At 4.10pm the Head of ehealth alerted the Executive Director on-call of the cyber-attack and the local implications. By 4.15pm it was clear that the virus was spreading across acute and primary care systems so strategic and senior clinical management teams agreed to take down core systems. At this point, the Chief Executive caveated this instruction with a clear message to say that systems required to maintain life/deliver essential clinical care should be maintained. The first strategic leadership team conference call took place at 4.20pm. At 4.28pm the main objective of maintaining patient safety in all care settings across acute and primary care was agreed, along with a move to business continuity. At this point all strategic and clinical team members agreed that non-essential systems should be switched off to avoid further propagation of the malware virus. A second conference call was arranged for 5.00pm and strategic leads identified. Each site provided an update and the situation was further assessed. There was agreement to follow the principles detailed in the Major Emergency Plan but that the event would not be declared a major emergency/incident. A command centre was established in NHSL Headquarters and a further meeting was arranged for 6.30pm where the strategic team Final 4 October 2017 Page 7

8 further reviewed the situation as it developed and more detailed feedback from sites came through to inform further decision-making. No patient safety issues were reported. Clear actions were agreed with a review scheduled for 10.00pm. The strategic team reviewed the situation during seven teleconferences on 13 and 14 May At each meeting, no patient safety issues were reported. Additional staff were rostered to maintain patient care using paper based systems. IT staff worked throughout the weekend to identify the scale of the problem and to install protective software patches and recover encrypted files. 494 weekend patient appointments/procedures were cancelled. By Monday 15 May 2017, the main patient care systems were operational again and patients were able to attend for operations and appointments. Due to the backlog in processing laboratory specimens, appointments for routine bloods in treatment rooms did not restart until Wednesday 17 May. While the malware affected many NHS organisations across England and Scotland, it had a significant impact on NHSL, with 1338 PCs affected in both acute and primary care settings. Over the course of the week following the incident, the infected PCs were either cleansed of virus on site or were replaced while cleansing took place. This was done on a prioritised basis, with additional IT support being provided by NHS Greater Glasgow and Clyde and NHS Ayrshire and Arran. No data was stolen during the incident and the understanding to date is that no data was lost or unrecoverable. 494 patients had their appointments/procedures postponed a result of the incident. As of 23 rd June 2017 all of these appointments/procedures had been rescheduled. No significant adverse events have been reported as a result of the incident. The incident was logged on the Datix system as Web and an SBAR briefing document was completed. This recommended that a SAER was undertaken to consider all aspects of the incident and to identify any vulnerabilities and measures required to reduce them. Final 4 October 2017 Page 8

9 5. Key findings of review Factors leading up to the event Technical 5.1 The review found that there were three main technical issues that lead to NHS Lanarkshire being affected by the malware WannaCry. A Microsoft patch was issued in March 2017 which blocked WannaCry. This was being tested by the ehealth Team at the time of the attack. It had been deployed on GP servers but had not been rolled out due to ongoing testing and limited resources to deploy the patch sooner. 395 PCs were still using the XP operating system for which there was no patch available at the time of the attack. Microsoft has subsequently made a WannaCry patch available for XP but in general XP remains unsupported. 190 of these PCs were required to run XP as they were supporting medical devices which could not operate on more up to date software. Therefore, these PCs were particularly vulnerable. Previously a software audit reported that there were no PCs/laptops with XP installed in NHS Lanarkshire; however, the software used to undertake the audit was not functioning correctly and therefore not reporting correctly. A configuration on desktop PCs called SMB version 1 was left on. The SMB configuration requires to be active in order for the Board s laboratory system to operate effectively and be accessible. It could have been switched off on PCs that did not need to access the laboratory system but this required additional staffing which was not readily available so the default was to leave it active on all PCs. The active SMB configuration was exploited by the malware to allow it to move across internal networks. An additional technical issue was identified which, although not significant in this incident, represented an important security vulnerability. A DNS firewall was in place which would have been expected to detect and block malware from accessing the network; however this was not functioning as required at the time of the attack. A fault had been logged with the firewall supplier under the existing support contract; however, the fault was interpreted as only affecting the reporting element of the firewall system and so was not dealt with urgently by the supplier. In fact, the system was not operating correctly and was not blocking malware across the network. Organisational 5.2 While there was an ehealth Strategy in place, decisions regarding additional funding were made mainly on a non-recurring basis. Final 4 October 2017 Page 9

10 5.3 A governance gap was identified between the role of the Information Governance Committee and the ehealth Executive Group regarding oversight of IT security. 5.4 Overall monitoring and management of some contracts and response times were found to be inadequate. 5.5 A module for the DNS firewall was implemented without staff having been trained in its use. Response to the event 5.6 All stakeholders highlighted the excellent team work that staff demonstrated in response to the incident with many going above and beyond what would be expected of them. This was particularly noted in the response provided by the ehealth team. 5.7 The Chief Executive provided excellent leadership at the strategic team teleconference meetings which staff found supportive and meant they were clear in their roles. However there were some instances where these meetings would have benefitted from improved teleconference discipline by participants. 5.8 The timing of the incident (Friday afternoon) helped to minimised impact as fewer scheduled procedures are planned over the weekend. This allowed the ehealth team to ensure most systems were operational again by the Monday. 5.9 The organisation did not have a dynamic register of all business continuity plans to provide assurance that regular rehearsals of BCPs had taken place. During the incident, the use of business continuity plans was variable. This did not appear to impact on the response although this could have become an issue had the incident taken longer to resolve There was no effective alternative for communicating with a wide range of staff when was unavailable Agreed communication channels with Scottish Government were not always adhered to. 6. Involvement and Support of the Patient and / or Families The communications strategy (including written and televised press and social media) effectively engaged patients and the general population throughout the incident. Due to the nature of the incident, the panel took the decision not to involve patients or the public in the review. Final 4 October 2017 Page 10

11 7. Involvement and Support of Staff participating in the Adverse Event / Incident As this was an organisation-wide incident, a large proportion of staff were involved to some degree. Overall, the willingness of staff to support the organisation through the incident was commendable. The ehealth team played a key role in terms of identifying the scale of the issue and implementing the solutions and their commitment has been universally praised by all involved. Similarly staff from operational and clinical services demonstrated commitment to resolving the incident and minimising the impact on patients. Staff from NHS Ayrshire and Arran and NHS Greater Glasgow and Clyde were involved in the recovery phase and the wider ehealth community offered support throughout. The review has identified that it would be helpful to review and strengthen the Standard Operating Procedure (SOP) for supporting staff in any similar incidents that may occur in the future where they are required to work excessive hours to address a critical/major incident. 8. Outcome/Conclusion As a result of these findings, the review team has concluded that there were major service issues that contributed to the event. There were no significant adverse patient outcomes although there was the potential for major service implications. These were mitigated by the actions taken from May. As a result, the review team have made the following recommendations. 9. Recommendations NHS Lanarkshire Board 1. Governance arrangements relating to ehealth, and specifically IT security, should be reviewed to ensure that there is a clear line of accountability through to the Board. In doing so the architecture of groups with their terms of reference should be reviewed including the role of information governance committee, e-health programme board and e-health executive group. In improving the governance arrangements the Board should be cognisant of the recommendations contained in DL(2015)17 and the need for an operational Information Security Management System. 2. Due to the business critical nature of our IT systems and the heightened risk of malware attacks, it is important that future strategic investment decisions made by the Board take due cognisance of this It is important that given the increasing dependence on Final 4 October 2017 Page 11

12 ehealth solutions that the Board takes a holistic approach to its investment decisions. 3. Where the Board has contracts with external suppliers it should review these to ensure that business continuity of critical systems (including, but not limited to, ehealth/it systems) can be maintained at all times or restored within the shortest possible timescale. The Board should ensure that all new contracts also have this in place. 4. A category of incident called a major business continuity incident should be considered. At a strategic level this would follow the same process as applied for major incidents, however, at a tactical and operational level the response would be based on business continuity plans. This should include a section on how staff should be deployed if the response to an incident is likely to last more than 24 hours. 5. The Board should ensure there is a system in place that maintains a dynamic register of all business continuity plans in the organisation. This register should include the following as a minimum: Red, yellow, green or grey designation. Author Responsible Lead Executive Director Name of Endorsing Body Name of Governance or Assurance Committee Implementation Date Version Number Review date Responsible Person for Review An electronic copy of the BCP A system similar to that employed in the management of corporate policies should be implemented to ensure all business continuity plans are kept up to date and are assessed as being fit for purpose. Performance should be reported by the Resilience Group to the Corporate Management Team. The frequency of reports should be determined by the CMT. ehealth 6. Patching and upgrading of systems should be prioritised within ehealth and consideration given to how this is appropriately resourced. 7. ehealth should work with other ehealth departments across Scotland to share information and revise risk based on the experience of other Boards in terms of upgrading or patching systems similar to NHS Lanarkshire s. Final 4 October 2017 Page 12

13 8. A patch management policy should be developed to include a robust KPI that can be reported through the appropriate governance arrangements. 9. A planned maintenance schedule, including planned system downtime, should be agreed to ensure that, where necessary, ongoing patches and updates to systems are carried out in a timely manner. 10. Prior to implementation of any new or updated ehealth support systems, appropriate training should be provided for relevant technical staff to ensure that such systems are used effectively. 11. The current hardware estate and software systems should be reviewed to ensure that they have the latest software updates installed and that there is a reliable system in place to detect any out of date software. 12. In exceptional circumstances it is recognised that some clinical systems continue to operate on out of date software. In order to minimise the risk that this presents: 12.1 These systems should be moved to a segregated area of the network System suppliers must provide assurance that appropriate anti-virus and other IT security measures are in place to manage the risk of malware and virus attacks Contract owners for such systems must put in place monitoring and supplier management procedures to ensure that agreed measures are in place and operating effectively. 13. A lifecycle reporting process should be developed that reports to and is monitored by the ehealth strategy group and incorporated into the annual Property Asset Management Survey return. 14. ehealth should explore how staff can be assured that the DNS firewall is functioning properly at all times. Resilience Group 15. The terms of reference of the Board s Resilience Group should be reviewed to ensure they include the relevant recommendations contained within this report and also: Endorsement of business continuity plans and oversight of the testing programme with an appropriate escalation process if BCPs are not updated on time. The development of strategic partnerships with external organisations. 16. A business continuity testing plan should be developed, implemented, approved and overseen by the Resilience Group. Final 4 October 2017 Page 13

14 17. Simulation exercises of major business continuity incidents should be undertaken on a frequency to be determined by the Resilience Group to ensure all staff likely to be involved are familiar with how such incidents are managed. Operational Services 18. It is important that all business critical IT systems are identified and an informed decision made based on risk assessment regarding potential shutdown should a malware incident occur again. Similarly, there should be agreement on the recovery process so that there is harmonisation between technical recovery and operation practice. 19. Business continuity planning, quality assurance, review of alignment with other relevant business continuity plans, and ensuring all business continuity plans are up to date, should be an explicit responsibility of Executive Directors and senior management teams. 20. All services, including corporate services, should review the robustness of their on-call arrangements and risk assess against the likelihood and impact of insufficient staff being available to manage a major business continuity incident. 21. All business continuity plans and relevant supporting documentation should be kept in hard copy format in local areas for easy access when a major business continuity incident occurs. Particular attention should be given to ensuring access to current phone numbers of relevant staff, including independent contractors. 22. All staff should adhere to conference call etiquette during a major business continuity incident. The command and control approach required should be a key element of major business continuity incident rehearsals. Communications 23. An evaluation of alternative communication channels when and wifi access is not available should be undertaken. This should include communication with staff and the public. 24. Communication arrangements with Scottish Government should be agreed for any similar level of incidents that may arise in the future. Final 4 October 2017 Page 14

15 Transferable Learning The panel also identified some wider issues that require to be addressed. 1. The Board should consider introducing a standard format for the Terms of Reference of all business groups. This should include details of the committee the group reports to and the frequency of reporting. 2. Credit card sized contact lists should be developed to support individuals who are oncall. Contact lists should be reviewed and updated quarterly. 3. The Board should review the purpose of its seminars and determine if these are for information purposes only or, additionally, for noting any learning/issues raised as part of discussions. If the latter, should any actions be agreed, a process should be put in place to ensure these are completed. Date final SAER Report signed off by Commissioner: / _/ Signature of Commissioner: Final 4 October 2017 Page 15

16 NHS Lanarkshire Response to the Ransomware Attack Hot Debrief Report - 05 June 2017 Appendix 1 The Incident and Response On Friday 12 May 2017, at approximately 3.00pm, it was becoming clear to the ehealth management team that NHS Lanarkshire (NHSL) systems were being affected by a malware attack. At 3.30pm the head of ehealth met with senior members of his team and agreed to shut down some critical systems to prevent spread of the attack across the estate. At 4.10pm the head of ehealth alerted the Executive Director on-call of the cyber-attack and the local implications. By 4.15pm it was clear that the virus was spreading across acute and primary care systems so strategic and senior clinical management teams agreed to take down core systems. At this point, the Chief Executive caveated this instruction with a clear message to say that systems required to maintain life/deliver essential clinical care should be maintained. The first strategic leadership team conference call took place at 4.20pm. At 4.28pm the main objective of maintaining patient safety in all care settings across acute and primary care was agreed, along with a move to business continuity. At this point all strategic and clinical team members agreed that non-essential systems should be switched off to avoid further propagation of the malware virus. A second conference call was arranged for 5.00pm and strategic leads identified. Each site provided an update and the situation was further assessed. There was agreement to follow the disciplines from the emergency plan but that this would not be declared a major incident. A command centre was established in NHSL Headquarters and a further meeting was arranged for 6.30pm where the strategic team further reviewed the situation as it developed and more detailed feedback from sites came through to inform further decisionmaking. No patient safety issues were reported. Clear actions were agreed with a review scheduled for 10.00pm. The strategic team reviewed the situation during seven teleconferences on 13 and 14 May At each meeting, no patient safety issues were reported. By early Monday morning it was confirmed that recovery had commenced. Debriefing Arrangements In keeping with good practice following a significant incident, it was agreed to hold a hot debrief on Thursday 18 May 2017 in the Boardroom of NHS Headquarters. A small group of key stakeholders were invited, comprising the strategic group members and those who had key roles in acute and primary care settings. The purpose of the debrief was to identify what went well, what could have been better and recommendations for changes or training. Specifically the aims of the debrief were to: Final 4 October 2017 Page 16

17 Identify major decisions made Reflect on decision making process Reflect on use of business continuity plans Identify immediate learning points - what went well and what could be improved Discussion took place as a whole group and then in smaller groups to facilitate information sharing. Key Themes from Debrief There were a number of themes identified, a summary of which is provided below. Command, Control and Co-ordination What Went Well? Early recognition of the significant event and quick response in all areas. Good senior leadership - immediate response was very quick and very well organised with clear chain of command which freed up ehealth to focus on the problem. Strategic team, primary care and ehealth made good use of business continuity plans. Clarity re the critical and important systems to be prioritised which allowed high risk patient areas to be supported and protected. Having local experts on site to support response. Staff teamwork, support and willingness to help. What could have been better? Access to, and systematic use of, BCPs across all sites and systematic use of these documents. Clarity re definition business continuity versus major incident. Discipline during teleconferences, including one spokesperson for each site. Establishment of a sixth hub lead for access issues (e.g. labs, diagnostics). Clarity re leads each day, who all needed and handover between managers on a daily basis DMT conference calls established quicker in process Recommendations Readily accessible (paper) BCPs that reflect recent organisational changes and standalone PCs. Definition for incidents (major versus business continuity). Develop etiquette for conference calls. Major incident training prioritised for all staff and review of grey pack. Review the number of operational centres calling in to strategic groups. Final 4 October 2017 Page 17

18 Communication SIGNIFICANT ADVERSE EVENT REVIEW REPORT What went well? Use of other means to disseminate information e.g. text messaging, clinical guidance disseminated using photographs, pharmacy printed guidance. Very effective communications to patients and staff using social media. DMT conference call and safety huddles on acute sites. What could be improved? Methods for managing high volume of calls at Director level and to Head of ehealth, in particular from Scottish Government. Access to contacts for staff and patients. Contact details for general practices and access to premises. A process for alerting all staff e.g. written guidance displayed in all wards. Listen to site concerns and not have blanket approach as there was variation across the system. Increase awareness of availability of teleconference accounts. Recommendations Need for alternative forms of communication for cascade, e.g. WhatsApp, Have standalone PCs and ipads for accessing BCPs, procedures, etc. Need for alternative methods for accessing patient contact lists. Implementation and Recovery What went well? Link between and within clinical and technical teams on priority actions. Total commitment of all staff. What could have been better? Assessing competing priorities on a corporate basis. In recovery phase there was confusion amongst staff between infection and routine IT issues. Recommendations Review risk register. Have regular BCP tests. Replicate good manual systems across all services. Review of arrangements for contact and support from third party/external suppliers (e.g. Siemens). Final 4 October 2017 Page 18

19 Action Points SIGNIFICANT ADVERSE EVENT REVIEW REPORT A number of immediate actions were identified: Planned Treatment 1. Contact numbers for patients attending for planned treatments/ investigations to be included in elective lists (which should be printed out) to enable patients to be quickly notified about any changes in attendance. Access to GP Practices 2. Access arrangements for GP premises to be confirmed. 3. Primary Care to obtain home/mobile contact number for managers of GP practices to allow IT staff to get access to GP premises/offices outwith normal working hours. Information to also be held in paper format or non-networked computer. Contact Directory 4. Operational centres/departments should maintain a paper based directory of contact numbers for the staff. Business Continuity Plans 5. A paper copy of the Business Continuity Plans, with action cards, should be printed out and held in each Site/Department, as appropriate. Communication Channels 6. We have also noted the need to establish effective communications channels for when our mainstream IT systems go down. This has been discussed by members of the strategic group who have identified issues that need further discussion and clarification before a recommendation can be made. This will be progressed as quickly as possible. Summary The hot debrief captured the feedback from key stakeholders and this process will be followed by significant adverse event review (SAER). There was general agreement that the response to the incident had been positive with particular reference made to staff support (especially ehealth) and public understanding. The key learning points were in relation to business continuity plans and acceptance that these require to be updated and made available in paper format. All participants would welcome training on resilience. Report authors Kerri Todd, Assistant Health Promotion Manager Gabe Docherty, Interim Director of Public Health Femi Oshin, Consultant in Public Health Medicine Alan Robertson, Emergency Planning Officer Elspeth Russell, Assistant Health Promotion Manager Final 4 October 2017 Page 19

20 Remit SIGNIFICANT ADVERSE EVENT REVIEW REPORT Terms of Reference Significant Adverse Event Review Malware Incident June 2017 Appendix 2 To investigate the preparedness of NHS Lanarkshire for cyber-attacks including in particular the Wannacry malware and understand the root causes of why the Board was affected To investigate the response to this serious adverse event and the impact it had on patient care and services To determine ongoing vulnerabilities to cyber-attacks To provide recommendations on measures which could reduce vulnerabilities and improve the response to future incidents Review Team Membership Dr Iain Wallace, Medical Director, NHS Lanarkshire Lead Dr Lesley Anne Smith, Director of Quality, NHS Lanarkshire Carol McGhee, Corporate Risk Manager, NHS Lanarkshire Denise Brown, Head of Patient Administration and Transformation, ehealth Directorate, NHS Greater Glasgow and Clyde Kerri Todd, Assistant Health Promotion Manager, NHS Lanarkshire - Secretariat Reporting Arrangements The report will go to CMT and PPRC. Timescales The SAER will be completed by 31 August Initial Staff to be interviewed in connection with the Cyber-Attack Incident Donald Wilson, Head of E-Health Calum Campbell, Chief Executive Colin Sloey, Director of Planning and Executive Director on Call Heather Knox, Director of Acute Services Craig Cunningham, Head of Commissioning and Performance, South H&SCP Christine Jack, Operational/Business Manager, HSCNL Dr Philip McMenemy, Chair IG Committee Gabe Docherty, Interim Director of Public Health Dr Femi Oshin, Consultant in Public Health Medicine Additional members of staff may be interviewed depending on the information gained from the initial interviews. A list of questions will be provided to the participants prior to the meeting with the Review Team. Participants who had lead roles in the response to the cyber-attack will be asked to submit a timeline of key decisions and actions. Final 4 October 2017 Page 20

21 Appendix 3a Final 4 October 2017 Page 21

22 Appendix 3b Final 4 October 2017 Page 22