Patient experience survey adult primary care: Methodology and procedures

Transcription

1 Patient experience survey adult primary care: Methodology and procedures February 2017

2 Health Quality & Safety Commission 2017 Health Quality & Safety Commission PO Box 25496, Wellington, New Zealand This document is available on the Health Quality & Safety Commission s website:

3 Contents 1. Introduction Background Purpose of the survey Patient experience survey project Pilot phase Project governance PHO service agreement process Phased rollout Evaluation Participation The survey tool Survey development Domains of patient experience Question classification and scoring methodology Domain performance rating Coordination sub-domains Overall GP and nurse rating Score calculation Patient contact and demographic information Long-term solution NES Patient- and practice-identifiable information Data access Informing patients of the use of their information Official Information Act requests National survey and report system National system flexibility Survey process Process chart interim Process chart final (NES) solution Data flows and system access Interim solution NES long-term solution In situ solution Error! Bookmark not defined.

4 3.4 Collection method Pilot analysis Sampling method notification and online collection SMS notification and online collection In situ collection Frequency of collection Eligible patients Sampling Sample size Reminders Sampling method Demographic weighting Non-response bias National system provider services Use of the survey results National accountability Calculation of system level measures Online reports Accessing the survey results The dashboard Opening the reports Viewing reports Printing reports Quantitative and qualitative analysis Continuous improvement Annual review Appendix 1: Primary care patient experience survey Appendix 2: Domain questions and picker scoring Coordination Physical and Emotional Needs Communication Partnership Appendix 3: Patient experience interim solution patient data extract Data extract rules File transfer... 64

5 Data fields DHB codes Appendix 4: Correspondence with patients correspondence Reminder SMS correspondence Survey introduction Survey conclusion Appendix 5: Bias in response Appendix 6: Survey data file Appendix 7: Patient experience report examples Appendix 8: Licensed software support... 94

6 1. Introduction 1.1 Background Patient experience is a vital but complex area. Growing evidence tells us patient experience is a good indicator of the quality of health services. Better experience, developing partnerships with consumers, and patient and family/whānau-centred care are linked to improved health, clinical, financial, service and satisfaction outcomes With this in mind, the Health Quality & Safety Commission (the Commission) would like to make patient experience part of our measurement of health care quality and safety. Patient experience is a component of our quality and safety indicators framework. Until August 2014, there was no consistent national approach to collection, measurement and use of patient experience information on a regular basis. The Commission designed a 20-item adult inpatient survey, which began in August 2014, as part of addressing this gap. Patient experience measures are now routinely in place for hospitals, providing both quantitative and qualitative results. The survey runs at least quarterly in all district health boards (DHBs) and covers four key domains of patient experience: communication, partnership, coordination, and physical and emotional needs. In December 2014, the Ministry of Health (the Ministry) and the Commission began work to introduce patient experience measures for primary care using online patient surveys. The System Level Measures concept began with the Integrated Performance and Incentive Framework (IPIF). The refresh of the Health Strategy provided an opportunity to extend and evolve the IPIF concept to the whole of health system. The Ministry worked closely with the sector to co-develop the new System Level Measures that provide a system-wide view of performance. The new measures engage the health sector more broadly (professions, settings and health conditions). IPIF is therefore transitioning to the System Level Measures Framework to reflect this broadening of approach and the increased focus on value and high performance. Patient experience of care is one of the four new System Level Measures to be implemented from 1 July For 2016/17, uptake of the primary care patient experience survey can be used as one of the contributory measures. 1 Picker Institute Policy Position no. 3: Why patients should share in decision-making? Oxford: Picker Institute Europe. URL: 2 Balik B, Conway J, Zipperer L, et al Achieving an Exceptional Patient and Family Experience of Inpatient Hospital Care. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement. URL: 3 CAHPS Consumer Assessment of Healthcare Providers and Systems The Clinical Case for Improving Patient Experience. URL: Experience.aspx 4 CAHPS Consumer Assessment of Healthcare Providers and Systems The Business Case for Improving Patient Experience. URL:

7 1.2 Purpose of the survey The primary care patient experience survey, or PES, is being developed by the Commission to find out what patients experience in primary care is like and how their overall care is managed between their general practice, diagnostic services, specialists, and or hospital staff. The survey looks at a patient s experience of the whole health care system using primary care as a window. It focuses on the coordination and integration of care, rather than just the last visit to a GP s surgery. Being able to capture, understand and act on patient experiences in a timely manner is a vital contributor to improving health service delivery and also in prioritising attention and resources. The online survey and real-time reporting enables patients to have a voice and health teams that care for them can hear it through a direct and timely link. The PES is a useful tool for practices, primary health organisations (PHOs) and DHBs to identify what is being done well, as well as areas for improvement. The survey results contain both quantitative and qualitative information. The comments from respondents provide a wealth of information that is not included in the scoring. 1.3 Patient experience survey project The PES project involves the following work to develop, test, and implement the survey: 1. Survey tool development, including cognitive testing and cultural appropriateness. 2. Sampling method. 3. Patient contact and demographic information. 4. Privacy impact assessments interim and final. 5. PHO service agreement process. 6. Survey and reporting system. 7. Communication and engagement with stakeholders. 8. Establishment of a governance structure. 9. Project and survey evaluation. While the project is a long way through its life span and the survey has begun, it is still in the implementation phase. The survey is yet to be rolled out across all PHOs and implemented via the National Enrolment Service (NES) system as the long-term solution to source patient s contact information for the survey Pilot phase Six PHOs agreed to work with the Commission through the pilot phase: Procare Networks, National Hauora Coalition, Midland Health Network (cognitive testing process only), Whanganui Regional Health Network, Compass Health and Pegasus Health. Sixteen practices participated in the pilot, which consisted of eight survey rounds conducted between July 2015 and October Two of the Whanganui practices conducted in situ surveys during this time. In situ surveying, where patients can complete the online survey on a tablet while at the practice, is being piloted to see if this method captures the experience of patients that or text surveying is unlikely to reach.

8 With the exception of in situ surveying, the pilot phase is now complete and has been evaluated. The evaluation of the pilot survey can be found on our website Project governance Originally governance for the PES sat with the IPIF s Joint Project Steering Group. With this group moving into abeyance in mid-2015, there was a governance gap. Ultimately governance of the PES should link to the governance structures for System Level Measures, but while this is being properly established, an interim solution was put in place. The interim PES governance group acts as the decision-making body for the implementation of PES across PHOs and practices, and over the information that the survey collects. The group comprised people appointed by the PHO Services Agreement Amendment Protocol Group (PSAAP) parties, and covering the following skill sets/constituencies: PHO chief executives Commission senior manager general practitioner(s) Ministry of Health Deputy Director-General DHB senior manager Royal New Zealand College of General Practitioners representative consumer representative PHO service agreement process PHOs successful use and implementation of the patient experience tools, once those tools are developed by the Health Quality and Safety Commission was agreed to be added to part G IPIF of the PHO services agreement in The PES will be adopted by all practices as part of the PHO services agreement. However, there will be a phased rollout. During the course of the PES project, the PSAAP has been briefed and the project team has worked with the Ministry and General Practice New Zealand (GPNZ) to identify where changes to the agreement are required to facilitate the survey process. The interim PES governance group reports to PSAAP and the Ministry. At this point in time, aside from reference in part G, only changes to the referenced document PHO enrolment requirements have been recommended to accommodate the collection of enrolled patients contact details ( and cell phone contact), or preference to opt out, for PES purposes Phased rollout The first online PES, using the NES as a partial data source, began in November Initially only the five pilot PHOs participated in the survey using the interim process. The interim solution required each PHO to provide a data file containing contact details for eligible patients, through a secure transfer, to the survey system provider (Cemplicity). As NES is implemented and patient contact information is captured in the NES system, enrolled patients from all PHOs will be invited to participate in the survey. Patients will have given their explicit permission for their contact information to be used for the survey purposes when it is captured in the NES at the practice and the Ministry will provide a single file to Cemplicity.

9 The survey process is described in detail in the survey process chart below Evaluation Both the survey and project need to be evaluated at appropriate points in time. The following process and timing will be followed for this: Pilot evaluation completed by 31 March 2016 and published on the Commission website once reviewed. Evaluation of the survey tool and reporting system. This is to be reviewed after a year s worth of surveying (four survey cycles). An independent party will be contracted in early 2017 to complete the evaluation with all stakeholders. The scope of the evaluation will be agreed with the PES governance group and is likely to include: o user feedback via a survey developed by the independent party, facilitated by Cemplicity o interviewing a selection of key stakeholders o the independent party analysing the results o a sub-group possibly being formed to recommend changes; this may be done by workshop o question changes possibly requiring cognitive testing with patients (a longer process). o the PES governance group approving any changes to be made. The timeframe for completing changes will depend on whether cognitive testing is required or not. Ideally we would have the majority of PHOs and DHBs using the survey before it is reviewed, however this may not be possible. The review process will need to be timely. 1.4 Participation The PES will be adopted by all practices as part of the PHO services agreement. However, there will be a phased rollout beginning with practices that are members of the five pilot PHOs. This is due to the method of collecting patient contact details via NES currently not being available. The five pilot PHOs can determine their own internal process to roll out the national survey across their contracted network of practices. A key constraint is the PHOs ability to easily access the required patient contact data in practice management systems, as permission needs to be sought from general practices to extract the information. Once the NES is in widespread use and practices are populating their patient preferences field, more PHOs will be able to participate in the survey. NES will supply the patient contact details, replacing the need for a PHO to send data directly to Cemplicity. For details see section A patient will not be sent a survey until their practice and PHO have informed Cemplicity they wish to participate in the PES.

10 2. The survey tool 2.1 Survey development The primary care survey builds on the design, system and processes developed for the adult inpatient survey. As the survey looks at a patient s experience of the whole health care system using primary care as a window, the survey is tailored for the out of hospital environment. It focuses on the coordination and integration of care, rather than just the last visit to a GP s surgery. The survey is modular, with questions about medicines, diagnostics, specialists (other than GPs), allied health and support care, emergency department, hospital care and chronic conditions. Patients only answer questions relevant to their experiences. For example, questions on medication and chronic conditions will be answered only by patients for whom this is relevant. The survey has been under development since 2014, with multiple iterations and review steps. In 2014, the Commission partnered with the Australian National Health Performance Authority for the survey development. The initial survey received from Australia in March 2015 underwent a preliminary feedback process with the pilot PHOs, the Ministry and the Royal New Zealand College of General Practitioners. It was revised to reflect the New Zealand health care environment. This work built on a number of tested and used tools that were evaluated by an expert advisory group. While each country is now developing its own survey, we are continuing to collaborate with Australia and keep track of survey alignment. Patient feedback is voluntary and anonymous. Point Research was commissioned to cognitively test, refine and evaluate the draft survey tool. The testing occurred between April and June Cognitive testing was used to understand how patients understand and interpret questions and instructions. The aim was to find out from patients if: the survey instructions are easy to understand the questions are easy to understand the questions are relevant the survey enables them to talk about what they think is important any important questions are missing. The assessment included cultural appropriateness and tests for suitability in New Zealand s primary health care context. Patient views on survey length, use of personal or mobile contact information and completing the survey online were captured. The survey underwent three phases of cognitive testing with patients from the enrolled populations. The survey was refined between each round, and refinement of the tool

11 between those rounds. The first phase involved cognitively testing the questions with patients in 15 focus groups from the six PHOs, covering the following population groups: Adults aged years (2) Māori (2) Pacific peoples (2) Asian (1) Refugee new migrant (1) Older adults aged 65+ years (2) Younger adults aged years (2) Women (2) People with disabilities (1). The second phase tested the draft survey questions online with a wider set of patients from PHOs and health professionals. The third phase tested the revised survey with patients in individual interviews. The survey changed significantly as a result of this process and the final outcome is the survey in Appendix 1. The rigor of this stage of survey development significantly informed question wording and designed the structure of the survey to be both clear to the consumer and informative for the provider. An online dummy version of the survey as it appears to patients can be viewed and completed here Domains of patient experience The Commission s approach uses four domains (coordination, partnership, physical and emotional needs, and communication) to provide a consistent structure to measure patient experience in different care settings. These four domains communicate that a high-quality experience for patients depends upon high-quality and effective communication, a real partnership, excellent coordination of care and meeting both physical and emotional needs Question classification and scoring methodology The four domains are comprised of various survey questions. Coordination consists of 14 questions; physical and emotional needs,15; communication, 12; and partnership, 6. All answers are assigned a value based on the Picker scoring methodology 5 (eg, 10 = Excellent, 0 = Poor; 10 = Yes/Completely, 5 = Yes/To some extent, 0 = No). A complete list of these questions and their answer values is shown in Appendix Domain performance rating The national average performance ratings for each domain are displayed at the top of the online reporting dashboard. The national average performance ratings are broken into Māori national average and non-māori national average to enable a quick overview of any equity gaps. Practice, PHO or DHB domain results are displayed beneath this, depending on which user is logged on. 5

12 2.2.3 Coordination sub-domains Effective coordination of care means the patient experiences the care pathway and the way information is conveyed as seamless. As coordination is an integral part to a patient s experience of care, there are two sub-domains within this category. Excellent coordination reduces barriers to care and improves continuity of care. Within the survey, there are four questions that relate to barriers to care and 10 questions that relate to continuity of care. The sub-domain scores are calculated in the same way as the domain scores. Overall GP and nurse rating The national average shown in the overall GP/nurse rating is calculated by averaging all patient responses to the statement Overall, was your experience with your GP or nurse clinic rank from very poor (0) to excellent (10). Within the reporting portal, users have the option of applying filters, eg, your PHO, against the national average.

13 2.2.3 Score calculation The question scores are calculated by adding the score calculation of all responses and dividing by the total number of responses. Below are examples of how the performance ratings are calculated for each question (Table 1 and Table 2). 6 Table 1: Calculating question score example domain: coordination Survey question: Has cost stopped you from seeing a specialist doctor? Response option Number of responses Percentage of respondents Score assigned Score calculation No % Yes % 0 0 TOTAL (3500/400) Table 2: Calculating question score example domain: coordination Survey question: Were you involved as much as you wanted to be in decisions about the best medication for you? Response option Number of responses Percentage of respondents Score assigned Score calculation Yes, definitely % Yes, to some % extent No % 0 0 TOTAL (3375/400) At the domain level, the aggregation works in the same way. Table 3: Calculating domain score example domain: coordination Survey question: Has cost stopped you from seeing a specialist doctor? Response option Number of responses Percentage of respondents Score assigned Score calculation No % Yes % 0 0 Survey question: Were you involved as much as you wanted to be in decisions about the best medication for you? Yes, definitely % Yes, to some % extent No % 0 0 TOTAL (6875/8 00) 2.3 Patient contact and demographic information 6 Both tables have been adapted from: Care Quality Commission NHS Patient Survey Programme: Survey scoring method. Retrieved from

14 In an effort to minimise costs and administrative burden for PHOs and practices, patient data will be obtained through the NES. The data collected includes patient contact information and demographic information such as age, gender and ethnicity. Only information that is directly needed for the purpose of this survey is received. The national system for obtaining patient information from practices and PHOs is described in Appendix Long-term solution NES The NES will be used to provide patient information from a single source. When practices update their practice management systems software for NES, it contains a patient preferences field. This will enable practice staff to ask patients for their and mobile contact information and whether they wish to opt out of the national survey. It is envisioned that practices will populate the patient preferences field during survey weeks, gradually building up their number of enrolled patients with saved preferences. Patient preferences only need to be captured for patients aged 15 years and over who have a consultation during each survey sample week (see timetable). The method for obtaining patient contact details used in the interim solution by the pilot PHOs automatically opts all eligible patients into the survey each quarter. It has been agreed that pilot PHOs will continue to use this method until the patient preferences field is changed to populate automatically. Auto-population of patient preference fields from existing mobile and contacts in practice management systems will reduce administration time and is likely to increase survey response rates. Options for auto-population are currently being explored. 2.4 Patient- and practice-identifiable information An interim privacy impact report has been prepared for the programme and reviewed by the Office of the Privacy Commissioner. Once the survey is using patient contact data from the NES, the final privacy impact report will be prepared, reviewed and published. All responses to the survey are voluntary and anonymous unless responders choose to provide their contact details because they wish to talk to someone at their general practice. All notices and correspondence relating to the survey make this clear. Each survey has a unique identification which enables line-by-line analysis of responses. When the patient data extract (whether from a PHO or NES) is imported to the national system, a number is assigned to each line of information. Neither the national survey nor the reporting process requires patient-identifiable information to be held in the database. Patient contact information is needed only initially to allow and text correspondence to be addressed individually. Once each survey is closed, all identifiable information is deleted from the system. Demographic information is retained only to enable a comparison from time to time of who is not responding to the survey. Cemplicity is required to host the database within New Zealand and strict privacy and security protocols are maintained. Routine system penetration tests are run to maintain security. The reporting system uses filters to allow PHOs, practices and DHBs to tailor reports. The Commission has incorporated features to mitigate the risk of identification through these filters, such as using age bands and not showing data or comments where there are fewer than five responses.

15 During the pilot surveys the Commission noted responders have a tendency, more so than during the adult inpatient survey, to mention the name of their practice/doctor/nurse. There is an option to review and moderate patient comments and remove names prior to reporting. A process for reviewing patient comments has been agreed with the PES governance group and is outlined in this document Data access Data access rules decided by the PES governance group are in place. Different system users have access to different levels of information. The data access matrix is shown below. Table 4: Data access matrix What can be seen Who can see it Information Data level Patient Practice PHO DHB National Public Patient data file Individual identifiable (ex PHO/NES) Survey Individual identifiable only if patient requests contact & approves access to responses survey Online reports Practice their own, identifiable Practice others in their PHO, Online patient comments anonymously All PHOs, identifiable All DHBs, identifiable Individual (anon) practice level, their own Individual (anon) PHO level, their own Individual (anon) DHB level, their own Individual (anon) national level Survey data file Anonymous responses Published reports High level, national aggregate information Table 5: Identifiable data access matrix Organisation Role Can view Cannot view Practice General manager Practice manager General practitioner Nurse Administration team PHO Quality manager/lead Clinical director Primary care manager DHB Planning and funding Quality and risk managers DHB alliance representative Their own practice s results and comments. Other practices results within their PHO (unidentifiable). Results by other PHOs. All practices within their PHO. Patient comments by practice (identifiable). Results by all PHOs. Results for practices and PHOs in their area. Results for the PHOs for whom they are the Lead DHB (eg ADHB can only see Procare). Comments for practices in their area (unidentifiable). Results grouped by DHB (in a way similar to national users). Results by all DHBs. Results for all PHOs. Results for all DHBs.

16 National Health Quality & Safety Commission Ministry of Health Filter by lead DHB and DHB of domicile. Patient comments by PHO and DHB (identifiable). Patient comments by practice. The quarterly survey data files will be limited in accordance with the standard report access. At a national level, the Commission receives this data file for three purposes: 1. Calculating the weightings and the weighted reports. 2. Long-term analysis of trends. 3. Placing a qualitative lens on the survey results as done for the adult inpatient publication here. There are currently no published reports as the survey is currently implemented by only a small number of practices and PHOs. The governance group will agree the content of these reports and when it is appropriate to begin publishing. 2.5 Informing patients of the use of their information PHOs and general practices are well aware of the Health Information Privacy Code 1994 (HIPC) and the need to inform patients of the use of the information they provide. When patients enrol with a practice they sign an enrolment form agreeing to the enrolment process and are informed how their information will be used. Given this process will have occurred some time ago for many patients, the Commission needs people to be informed specifically about the survey. During the pilot phase the Commission specifically tested a range of ways to achieve this, although it is mindful no single method will ensure all patients are fully informed. The privacy impact assessment discusses this in detail. A summary follows. Each quarter, participating PHOs and practices are provided a Getting started pack by the Commission. The pack reminds them of key dates and provides the following process information: o consent to extract data from the practice (applicable for interim solution) o frontline staff guidance (one page) o a display poster o a survey slide that can be added to a TV slideshow if applicable for the practice o a flyer to be handed out to all eligible patients during the survey sample week. Practices can choose to text patients seen during the sample week to remind them they may receive a survey invitation. This is optional due to cost. Practices are encouraged to ask for patients addresses during the sample week (individual rather than family). ed survey invitations contain more information for the patients, are referable at a more convenient time, and have a significantly higher response rate than SMS requests. Practices can use a classification code to record in their practice management system if a patient advises they do not want to receive a survey, eg, NOPES. This applies to the interim solution only.

17 Once a practice has adopted the NES software, their practice management system will be updated to allow practice staff to ask patients for their survey contact information and whether they wish to opt out of the survey. The Office of the Privacy Commissioner has advised that this proposed process is well inside the privacy rules: patients are notified, give their consent to receive the survey and have the ability to say no. All s sent by Cemplicity have a clear unsubscribe option and if a patient clicks the unsubscribe button, no further s will be sent to that address. Once the survey is more commonly used, the Commission will publish national and local media releases to inform patients and encourage participation. The Commission also has information publicly available on its website here. 2.6 Official Information Act requests If Cemplicity receives an Official Information Act (OIA) request, it is required under its agreement to refer the request to the Commission. If the Commission receives an OIA request, the governance group will be informed. If multiple PHOs data is sought, the governance group will be referred to for decision. Where an individual PHO s data is sought, the PHO concerned will be referred to for decision. 2.7 National survey and report system Cemplicity has been contracted by the Commission to provide the national survey and reporting system that supports the survey. The system describes the process to import patient information, send survey invitations, receive and store patient responses (anonymously), and provide real-time reporting to authorised people. Cemplicity has worked with public health sector clients for a number of years and understands the critical importance of data security when managing patient contact details. Its approach to security touches every part of the company, from personnel management to hosting arrangements. Cemplicity s protocols have been developed in close consultation with government agencies responsible for the protection of patient privacy and the data security of public health records. It is part of the Connected Health network. In order to participate in the national survey, PHOs, practices and DHBs will be provided with licences to access this funded patient experience system. No additional IT investment is required for PHOs, practices and DHBs to participate in the core national survey. Only access and an internet browser are required. The survey uses the same system currently being used for the national inpatient survey in DHBs. This means consistent reporting and the potential to develop an integrated dashboard of patient experience results. The system electronically reports patient feedback to enable real-time updates to the dashboard reports. This means minimal administration for PHOs and practices, no postal delay and higher quality, more timely data with minimal intervention. Patients requesting contact can be attended to as soon as possible by an appropriate staff member. This approach also allows the one survey to be presented to each patient under the brand of the practice they attended so there is trust and relevance in the invitation to give feedback.

18 A key requirement from participating practices, via PHOs, is the provision of patient addresses and mobile phone contact information. The national system will support in situ surveying (via tablets), subject to the completion of the in situ pilot still in progress National system flexibility The national survey is the core question set that all PHOs must administer for a defined group of patients (see below) once a quarter. This allows derivation of results for domains, modules or specific questions at a national, DHB or PHO level each quarter. Provided the methodology and procedures are followed, the findings should be comparable and statistically robust. The system is able to present the survey in different languages. The need to introduce language options will be assessed by the governance group in consultation with PHOs once the national programme is implemented. The survey and reporting system is flexible enough to enable PHOs, practices and DHBs to invest in the programme and add their own questions and/or reports. They may also run a more continuous survey process or one-off surveys if they choose. This would involve negotiating an additional local service agreement with Cemplicity. Should PHOs, practices or DHBs opt into a more continuous approach (eg, weekly or fortnightly survey invitations) they would also get access to an integrated case management functionality that is designed to embed a process of continuous improvement. PHOs, practices and DHBs considering adding new questions or incorporating the national survey in existing local surveys should be mindful that the size of the survey and time required to complete it directly affect the response rate. For this reason, the governance group will be notified if Cemplicity receives a request to amend the quarterly survey. Any such adaption is likely to be timed for after the programme is implemented nationally. This will avoid conflict or confusion with the national requirements.

19 3. Survey process 3.1 Process chart interim PHOs, practices, DHBs and the Commission communicate the new survey through a range of methods Enrolled patient has a consultation during a set sample week each quarter See survey annual timetable Five pilot PHOs extract patient data from participating practices, in accordance with the survey rules Extract uploaded via secure FTP to national survey provider (as close to day extracted from practices as possible) Survey ed or texted, using practice logo/contact details, to the sample of patients with a mobile or contact in the extract, the same day as received from the PHO During this time, responses can be viewed updating live in the online system Reminder sent to patients 7 days later Survey closes after 21 days Unweighted reports available immediately Weighted reports available 14 days after the survey closes

20 3.2 Process chart final (NES) solution PHOs, practices, DHBs and the Commission communicate the new survey through a range of methods Enrolled patient has a qualifying event during the sample week each quarter See survey annual timetable Practice captures survey preference for eligible patients in the practice management system Patient data extracted in accordance with survey rules from NES on set date after sample period end Extract uploaded via secure FTP to national survey provider (as close to day extracted from practices as possible) Survey ed or texted, using practice logo/contact details, to a sample of patients with a mobile or contact in the extract, the same day as received from the NES During this time, responses can be viewed updating live on the dashboard Reminder sent to patients 7 days later Survey closes after 21 days Unweighted reports available immediately Weighted reports available 14 days after the survey closes

21 3.3 Data flows and system access Interim solution Step 1: PHOs send extract information to Cemplicity PHOs each connect via an SFTP client to the SFTP server housed within its own dedicated virtual machine (VM) within the Datacom application server environment. This SFTP server is secured via individual IP addresses within each of the PHOs and is only open through the firewall on the SFTP port. All other traffic is denied. On connection, each of the PHOs uploads its patient data extract (csv file) for the period. During the period the surveys are run, an integration application takes this csv file, converts the data to the appropriate format and stores it inside MS SQL server. The csv file is then deleted. Once this data is transformed, invitations can be sent to the patients via and SMS. Step 2: Invitations are sent from the application servers to patients On sending of invitations, a 36-character GUID is created and used to link the patients responses to the non-clinical background data provided in the file. For SMS invitations, a code is given which links to this same 36-character GUID. Patients receive the invitation via or SMS then complete the survey online via any web based browser, mobile phone or tablet. invitations are all sent from feedback@myexperience.health.nz with each participating practice s name or logo (if available) and contact information. Eligibility for a.health.nz domain name is restricted to organisations that deliver health services through registered practitioners. The myexperience.health.nz address is used as a trusted source of correspondence for recipients. SMS invitations are sent from the number 2333 with the practice name. This mode is more challenging given character limitations for each SMS sent. Clicking the link in the invitation, URL takes patients through the firewall over port 80, and into the application server environment. Patients answers are stored in the database as they complete each question page (eg, click next ). Patients who receive a SMS invitation via a non-smartphone can go to and enter their unique eight-digit code to complete the survey. Note: There is no external access directly to the database servers; these are routed through internal IP addresses. The SFTP server and the application VMs are separate machines.

22 Step 3: Report users access data In the third step, users at individual PHOs, practices, DHBs, the Commission and the Ministry of Health access the reporting portal via a 256-bit encrypted https link. These users are authenticated via a log in that utilises a unique key/salt algorithm. On gaining access to the portal they are routed to the report application server, which displays aggregated data pulled from the SQL server. Cookies are used sparingly within the reporting application (but not the survey application accessed by patients responding) and are limited to standard settings like group by or sort order. Users preferences are remembered via the cookie. If these preferences do not exist, the application simply defaults to the global setting. Cemplicity also uses a cookie for authentication purposes when users log on. Cemplicity does not record behaviour-specific information outside of these standard preferences. Step 4: Tidy up Cemplicity runs scheduled clean-up tasks that delete invitation information at agreed times after a period closes. Patient contact information is only retained in the system for as long as needed to send the survey invitation and reminders. All reportable data is therefore anonymous. The information flows are shown below in Figure NES long-term solution Step 1: The Ministry of Health sends extract information to Cemplicity Patient information to be used in the survey is captured within the Ministry s NES database. Practices can update the NES database in real time through their patient management system. In order to send patient information to Cemplicity, the Ministry connects via an SFTP client to the SFTP server that is housed within its own dedicated VM within the Datacom application server environment. This SFTP server is secured via individual IP addresses within each of the PHOs and is only open through the firewall on the SFTP port. All other traffic is denied. On connection, the Ministry uploads the patient data extract (csv file) for the period. During the period the surveys are run an integration application takes this csv file, converts the data to the appropriate format, and stores it inside MS SQL server. The csv file is then deleted. In the event a PHO wishes to mail surveys to patients, a new file is put back on the SFTP server for the PHOs to download, which contains the mail sample contact details. Once this data is transformed, invitations can then be sent to the patients via and SMS. Steps 2 4 are the same as described in The information flows are shown below in Figure 2.

23 3.3.3 In situ solution The in situ process is available to all practices but may especially be valuable to practices who service communities with lower access to , internet or affordable data plans on their phones. The tablet can be connected via wifi, SIM card or a hotspot device (meaning one SIM card can provide access for a number of tablets). Tablets need to be set up with a survey link by Cemplicity. The survey link is specific to the practice with the tablet. The practice logo is displayed on the survey. This allows for the practice name to be recorded against survey responses as well as mapping the responder s PHO and DHB. Each practice is assigned a 36-character GUID link. Tablets placed inside practices then have the 36-character GUID link added to the tablet. A patient taking the patient experience survey clicks the link and is directed to the survey. On clicking on the in situ link, the URL will take patients through the firewall over port 80 and into the application server environment. Patients answers are stored in the database as they complete each question page (eg, click next ). No survey response data is saved on the tablet. As a patient clicks from page to page, the survey item is saved to the Cemplicity server. In situ surveying is somewhat simpler because no contact details have to be captured. To prevent patients who complete the survey in situ from also completing the survey if they receive a SMS or invite, it is recommended reception staff alert patients that this might happen and recommend they ignore the survey invite. Steps 3 4 are the same as described in

24 Figure 1: Data flows interim solution

25 Figure 2: Data flows NES solution

26 3.4 Collection method The primary care survey information is collected online. This differs to the adult inpatient survey where some paper collection is involved; however, the sample size per DHB for the inpatient survey is much smaller, being limited to 400 patients per quarter. Online collection is the cheapest, quickest method and enables real-time updates to online reports for practices and PHOs. Patients selected for surveying will be contacted by (as a preference due to no cost and higher response rate), then SMS (text) and provided with a unique online survey link to click on or type into an internet browser to submit their responses. and SMS invitation processes can be augmented in some practices with tablets (in situ) as outlined above. Once the patient s unique link closes, there is no patient-identifiable information connected with it so responders are anonymous unless they choose to provide their contact details. This option is provided in case the patient wishes to discuss an issue with the practice (the practice will receive an alert). Once the survey has been completed, the link expires so there is only one response per patient. All online links expire three weeks from the time the survey is sent. According to Statistics New Zealand, 80 percent of New Zealanders 7 have residential internet access, with 1.6 million connections, while mobile phone internet connections accounted for 3.9 million in Internet access continues to rise nationwide. Absence of internet access is concentrated among poorer populations, rural communities and people aged 75 and older, some of whom may be high users of hospital services. In situ surveying is the most promising method to address these audiences Pilot analysis Our pilot analysis of over 1000 respondents has shown survey response online is straightforward and about two-thirds of people complete the survey within 20 minutes. Some people complete the survey in multiple sessions, which takes longer. Some people, albeit with no free text responses, completed the survey in less than five minutes. The more modules of the survey completed, the longer the response time. The survey contact method ( , SMS, in situ) had little effect on the response time. There was no relationship between time spent answering the survey and the positivity of responses. Nor was there a relationship between survey contact method and positivity of responses. The process of completing the survey did not appear to affect the content of responses. Female and middle-aged people were more likely to respond. Asian and European peoples were overrepresented while Māori and Pacific peoples were under-represented. Providing in situ sampling in practices with high Māori and Pacific populations is one way to counteract this. 7

27 3.5 Sampling method notification and online collection The PHO and NES run the patient data extract for the one-week period in accordance with the data extract and file format rules. The file is uploaded through a secure portal to Cemplicity on the same day. Cemplicity will then select all patients with an address and send the survey invitations on the same day. If the patient has not completed the survey within seven days, they will receive a reminder. After 21 days from the original send date, the survey link will expire SMS notification and online collection Any patient records without an address but with a mobile phone number will be sent an SMS invitation to complete the survey. If the patient has a smartphone, they can click on the link to complete the survey on their phone. If they have a standard mobile phone they can go to the web page in any device s internet browser, enter a short code (that is in the SMS message) and complete the survey online. If the patient has not completed the survey within seven days, they will receive a SMS reminder. After 21 days, the survey link will expire In situ collection In situ collection is currently being piloted and shows promising results for under-represented populations. In areas where there is low access to the internet, online survey delivery presents a substantial risk of unrepresentative and/or very low response rates. In these areas, practices or PHOs should consider an in situ alternative for the collection of data. Potential negatives of in situ reporting may include the following. The power/mana imbalance between patient and professional may inspire the patient to give unrealistically positive responses. Patients seeking assistance with either the device or completing the survey. Cost of tablets. Interruption preventing completion. The process for in situ survey collection is as follows. 1. Discussion between each PHO and Cemplicity to set up the in situ scenario. This tends to be bespoke for each practice/pho in light of IT policy considerations. 2. Agreement of technical requirements and method for transmitting the data to the national system needs to be made with Cemplicity before undertaking this method. 3. Frontline staff guidance from the Commission. 4. Tablets are set up by Cemplicity and sent to the practice or PHO. 5. Data must be collected in a consistent manner for all patients surveyed so that: any eligible patient is offered a tablet to complete the survey (not just friendly, positive patients) the member of staff administering the survey does not stay with, advise or otherwise prompt the person taking the survey

28 the survey is physically completed by the person taking it, or a carer with them. Members of staff administering the survey cannot complete the survey for patients. 3.6 Frequency of collection For national reporting purposes and consistency, the survey will be sent to patients seen within the same one-week period in each quarter in each participating practice. The oneweek period chosen avoids public holidays, but it is not intended as a representation of the quarter in which it sits. Rather, it will be one of a series of snapshots recorded over the course of a year. This fits with the Commission s recommendation that PHOs collect patient feedback constantly to monitor changes and stimulate improvement in patient experience. The timetable for the survey is shown in Table 3. Table 6: Indicative survey timetable Process Patients with a date of last consultation at the practice they are enrolled with, in a set one week sample period each quarter PHO extracts patient data Survey ed or text to all patient with an or cell contact Real time, unweighted reports available for the quarter Reminder or text sent seven days later and text survey links close have twentyone days Survey response data files provided to PHOs Draft weighted reports provided to the Ministry, DHBs and PHOs by the Commission Final weighted reports provided by the Commission Q3 Jan-Mar February / /17 Q4 Q1 Q2 Apr-Jun16 Jul-Sep16 Oct-Dec May August October 6 November 2016 Q3 Jan-Mar February 2017 Q4 Apr-Jun May Feb May Aug Nov Feb May Feb May Aug Nov Feb May Feb May Aug Nov Feb May Mar May Aug Nov Mar May Mar Jun Sep Dec Mar Jun Mar Jun Sep Dec Mar Jun Mar Jun Sep Dec Mar Jun Mar Jun Sep Dec Mar Jun Eligible patients The survey is designed to be answered by enrolled patients aged 15 and older who have had a consultation (defined per the PHO services agreement) with the practice they are enrolled with in the one-week sample period. Full details of the patient data extract, including rules for inclusions and exclusions, are set out in Appendix 3. In order to use the online system effectively, addresses and mobile phone numbers need to be collected. The Commission recommends that addresses and phone

29 numbers are captured in a consistent format to help make administration of the survey as simple and cost effective as possible. 3.8 Sampling Adequate sample size is essential so the recorded results represent a population as a whole. At least three risks need to be addressed: too small a sample of responders, leading to excessively wide confidence intervals and unstable results, which make changes hard to interpret a sample unrepresentative of the population structure response bias (ie, responders with unrepresentative opinions more or less content with their experiences than the average being more likely to respond). Well-designed samples can mitigate these risks Sample size The Commission s approach for the primary care survey is essentially a census seeking to get as many responses as possible among all eligible contactable patients. National reports that use weighted statistics will be restricted to PHOs. The Commission aims deliberately to over-sample high users of health services, which is why this survey is given to all eligible 8 adults receiving services, rather than simply the enrolled population. The survey asks questions about all interactions with health services in the last year, rather than just the most recent GP or nurse appointment Reminders Evidence shows that a reminder will generate one-third to a half of the responses the original survey contact achieves. For this reason, the Commission includes a reminder seven days after the initial contact of the patient. During the pilot phase, the Commission tested sending a second reminder but this generated negative feedback from patients who were sick of receiving texts. Patients are captured in the survey invitation extract if they have attended a practice in the past week. Invitations are then sent promptly on receipt by Cemplicity. This aims to minimise the risk of the patient s circumstances having changed, and this may include their death. It can take up to three months for systems to be updated following a patient s death. While there is no way to entirely remove the risk of sending a survey or reminder to the family of someone who has died, it can be mitigated by minimising the timeframes Sampling method Regardless of sample size, if the sample is systematically unrepresentative of the population the results will be misleading. 9 There are two ways to mitigate this: either stratify the sample d Patients enrolled with and seen by participating practices in the survey sample week each quarter will receive a survey invitation via or SMS. Children under 15 will not be surveyed. 9 Famously, the Literary Digest miscalled the 1936 US election as a victory for the Republican challenger Albert Landon, despite a sample size of over 2 million, at least in part because of using a telephone directory as the basis of its sample at a time when telephones were a luxury item.

30 so the sample looks like the overall population, or use a random sample and apply weights in line with the local population to the results post hoc. The Commission has found the second approach more practical in this instance. 10 Weighting according to demographic characteristic (age, gender and ethnicity are captured in the survey) allows results to be adjusted to reflect recorded differences. This is an approach used widely by opinion polling organisations Demographic weighting Weighting is a relatively straightforward process which takes either a local or national population structure, compares this with the sample structure, and creates a co-efficient which is applied to the results of the survey. This then increases or decreases a particular score and provides a weighted result. This approach is distinct from standardisation. We are not seeking to compare PHOs or DHBs with each other using this method. Rather we are seeking to weight so that results accurately reflect the views of a representative local population inside a specific PHO or DHB. The national system will show un-weighted results online for local purposes. Once a sufficient number of practices are participating in the survey, the Commission will prepare quarterly patient experience indicator reports that will show weighted results in accordance with the method described in this document. This will be published on the Commission s website in the same manner as the adult inpatient experience survey. The issue that needs to be considered is what variables should be used in the weighting. The primary care survey collects data on age, ethnicity and gender. The Commission has analysed results from the early rounds of surveying by these different groupings to identify whether there are systematically difference results for different populations. The results are quite complex. Age is associated with different responses for all domains with the three age groups aged 65 and over being consistently more positive than working age adults. However, for both gender and ethnicity only the coordination domain and coordination barriers to care sub domain show differences with men and Europeans being more positive. There are no significant differences between other ethnic groups. In order to have a manageable weighting method across the three groups we have aggregated the groups as follows. Gender is aggregated to male, female and other; age is aggregated to and 65+ and ethnicity to European and other. The reason for these groupings is to avoid distortion of the results caused by a large array of cells in the weighting calculations with very small numbers of respondents in each one. To reiterate, the national report does not report results for sub populations, the local reports provide this information in an entirely accurate, unweighted manner. The notional response of a particular PHO will demonstrate how this base population can be used to create weightings. Table 6 shows PHO A has 1500 respondents to the survey and are divided into groups as follows 10 For stratified approach to completely address this risk, the Commission would have to assume that each strata had a similar response rate and, if they did not, a further exercise in weighting would be required.

31 Table 7: Demographic make-up of PHO: Survey respondents by age/gender/ethnicity Gender Age Ethnicity Respondents % Female European % Other % 65+ European % Other % Male European % Other % 65+ European % Other % Other European 4 0.3% Other 5 0.3% 65+ European 1 0.1% Other 0 0.0% The proportion in each group can be compared with the sample population inside the PHO (Table 8) which shows the variance between the survey response and the notional population. Some differences become immediately apparent. There is an overrepresentation of younger non-european women and older European women, while younger non-european men and older European men under-representation of men in general. This is addressed through weighting. The population proportions are divided by the responder proportions to create a weighting factor for each sub-group, as shown in Table 9 (to avoid individual sub-groups with very small numbers having too much weight in the final weighting, weighting factors are limited to a maximum of five). Table 8: Demographic make-up of PHO A survey respondents and overall population by age/gender/ethnicity Gender Age Ethnicity % responders % population Female European 32.7% 33.6% Other 15.7% 13.9% 65+ European 14.3% 13.4% Other 1.9% 2.6% Male European 13.3% 12.7% Other 5.9% 6.9% 65+ European 13.5% 14.3% Other 1.9% 2.0% Other European 0.3% 0.3% Other 0.3% 0.2% 65+ European 0.1% 0.0% Other 0.0% 0.0% Table 9: Weighting factors by age/gender for PHO A Gender Age Ethnicity Weighting coefficient

32 Female European 1.03 Other European 0.94 Other 1.36 Male European 0.95 Other European 1.05 Other 1.08 Other European 1.08 Other European 0.61 Other 0.00 These factors can then be applied to individual responses received to provide weighted scores for individual questions at each DHB. The effect of this is to give different values to responses effectively reflecting how many patients of a different age, gender and ethnicity each respondent is representing. The more over-represented a particular group among the responders, the fewer total patients each responder represents and thus the response is down-weighted and vice versa Non-response bias The inpatient survey is collected primarily online with sampled patients contacted by , letter and text. Typical completion rates are around 40 per cent for letter, 35 per cent for and 15 per cent for text with a consistent response rate of approximately 27 per cent. This low response rate has led some to question whether the survey responses can be considered valid and reliable. To address this concern, the Commission worked in collaboration with Cemplicity and the Health Economics Department of Victoria University to investigate further. The issue of low response rate involves three related but distinct issues. First, the number of responses can be so low that results are unreliable. In practical terms this means having confidence intervals so wide that it is nearly impossible to compare one question with another (ie, identifying priority areas for improvement) or the same question over time (ie, identifying improvement over time). Strictly, this is not an issue about response rate. If the sampled population is large enough in the first place, there will be sufficient respondents to avoid this even if the response rate itself is low. Second, the low response could be because some demographic groups are underrepresented, leading to a non-response bias. If these groups hold substantially different opinions and have substantially different experiences, then results will be invalid as they do not represent the whole population. In contrast to the first issue, this pitfall is not addressed by increasing sample size. The risk of unrepresentative results can, to some extent, be mitigated by weighting of responses against the population make up (up-weighting responses from respondents belonging to underrepresented groups and vice versa). This process has its limitations: if numerous demographic characteristics are used to weight for a small sample, individual responses are able to dramatically distort results.

33 Finally, the low response rate could lead to invalid results if responders systemically held different opinions or had different experiences from non-responders, independent of their demographic makeup. Theoretically, responders could be systematically more positive than non-responders because the latter wish to forget their negative experience. Conversely, responders could be more negative as people are more likely to respond if they have something to complain about. Ironically, both these conflicting explanations have been put forward to suggest that the inpatient survey may be invalidated by its response rate. The study concluded that the non-responders to the original survey gave responses to the follow-up survey that are similar to the responders to the original survey. This suggests putting more effort into increasing the response rate of the Patient Experience Survey is unlikely to change its conclusions. The Commission will publish the final results of this study on its website by June National system provider services The services that the national survey and reporting system provider, Cemplicity, has agreed with the Commission are as follows: Development work a. Provide a draft survey online for the online cognitive testing process, working with the appointed cognitive testing provider. b. Become part of the Connected Health network. c. Import the patient data extracts to the data warehouse initially for up to 6 PHOs and create a survey invitation list based on sampling criteria specified by the Commission notwithstanding that the sample criteria will be able to be applied using existing Supplier functionality and will not require significant data manipulation. d. Create real time dashboards to allow reporting of the results, in accordance with the methodology prescribed. Users will be able to filter results and download reports in multiple formats. e. Create log in access for up to two users at each PHO, two users at each DHB and twenty national users which they will then manage themselves to reflect changes in required staff access. f. Create the initial log in access for each general practice and provide these details to the practice s PHO to be distributed to the practices by the PHO. Provide instructions and functionality to each PHO so they can add and delete users at their PHO and practices. g. Set up the in situ process for piloting. Ongoing work a. Import the patient data extract(s) to the data warehouse and apply any stipulated sampling rules. b. Contact the sample of patients using, in order of priority: then SMS, and invite participation in the survey using the invitations that are shown in Appendix 4. c. Ensure that the branding, signature and survey conclusion signature viewed by each respondent to the electronic survey match the branding of the general practice at which they are enrolled, as informed by the data extract.

34 d. Provide PHOs that opt to administer mail surveys (this will not be a national requirement) a data file via the secure file transfer process, in accordance with the adult inpatient process. For clarity, the Commission would be responsible for establishing the paper survey process with any PHO wishing to do this. e. Send electronic reminders to complete the survey seven days following initial contact using the same method as initial contact. f. Surveys will remain open for 21 days. After each survey is closed, all personally identifiable information in the data extracts that were provided by the PHOs or NES will be permanently deleted from Cemplicity s system. (Cemplicity will not be able to reactivate survey links once this information is deleted if respondents wish to provide responses after the survey links are closed.) g. Send an alert to a nominated practice address when a respondent indicates they wish to be contacted by the practice. Provide contact details and reason for contact to the practice for those patients who wish to discuss some aspect of their response. h. Provide a survey data extract to PHOs and DHBs (as deemed appropriate by the governance group) for local analysis on request up to once per quarter in the format prescribed in the methodology document, in accordance with agreed data access rules. i. Provide a national extract (or a simple user interface to enable the Commission to extract data as required) at least quarterly, to allow calculation of the national indicators. The Commission will be able to access and download national data at any time, in accordance with agreed data access rules. j. Conduct the above processes in accordance with the national timetable. k. Data received from PHOs or from NES should be validated (batch files and records) with error reports sent back to the data provider. All data will be hosted and stored in New Zealand unless agreed in writing by the Commission and the Health IT Board that an alternative hosting arrangement is acceptable. Further information on the Services agreement can be sought from the Commission, however this document is commercially confidential and any information provided needs to be done so with Cemplicity s consent.

35 4. Use of the survey results 4.1 National accountability The survey is designed to encourage local improvement and provide national indicators by DHB or PHO for each of the domains. This will consist of the weighted average of the 0-10 rating. The weighting will apply the factors set out in section above. To allow comparability, this will be based on the national population of eligible patients (this will be tested as the survey develops. This isn t in the contract and needs more thought to be sure it is a good idea) Calculation of system level measures Each responder is given a weighting factor based on the method set out at The 0-10 rating given by the individual is multiplied by the weighting factor to give a weighted rating. These are then summed and divided by the total number of responders to give a weighted average. The following shows a worked example using the notional PHO A survey responders set out in section Step 1: find ratings for each individual Respondent Gender Age group Communication rating (0-10) 1 Female Female Male Male Average 7.45 Step 2: apply appropriate sub group weighting for each individual Respondent Communication rating (0-10) Weighting factor 1 9 x x x x Average 7.45 Step 3: calculated weighted average Respondent Communication rating (0-10) Weighting factor Weighted rating 1 9 X X

36 390 9 X X Average 7.45 Weighted Average 7.54 In this worked example, an arithmetic average rating of 7.45 increases to 7.54 once the weighting factors are applied. This calculation will be applied to each rating for each DHB for each national quarterly snapshot in order to provide a weighted indicator. 4.2 Online reports All practices, PHOs and DHBs have the same national standard report view (subject to the data access matrix) when they log in to the national survey and reporting system. Appendix 7 shows each of the main report types Accessing the survey results Type in an internet browser. Enter and password and click the green log in button The dashboard Once logged in, the user will arrive at the dashboard. This landing page provides a snapshot of key metrics at a national level Opening the reports Below is the basic report layout showing the three main elements of the screen.

37 Report menu The report menu allows users to navigate from the main dashboard to more detailed reporting views. The report menu aligns closely with the structure of the patient experience survey to give the user a quick reference point for evaluating patient response. Graphs and tables Across the reporting portal, different types of graphs are carefully selected and used to present the data in the clearest format possible. For example, bar charts are used to give a clear view of trends from quarter to quarter. Filters The filters allow users to isolate responses by age, gender or ethnicity for in-depth comparison between general practices, PHOs, DHBs and the national average. For example, using the ethnicity filter the user can compare how Māori responses vary from non- Māori responses in relation to GP access between PHOs Viewing reports Several types of reporting in the patient experience survey can be found by clicking on each heading within the report menu. Note: the view you see will depend on whether you are from a practice, a PHO or DHB. Trend reports These reports are designed to show how the user s ratings are changing over time. By PHO These reports allow you to compare patient feedback across PHOs. By lead DHB These reports allow you to compare patient feedback across DHBs.

38 Comments These reports present the comments patients have given in the survey. The comments may be moderated for the recommended process. (For more information on how to do this read the full portal user guide link below.) Printing reports You can print reports by clicking on the Print button to the top right of each graph. This will generate an image of the report, along with any filters you currently have applied. Alternatively, you can click on the three horizontal lines, on each graph just below the response count section. The options in this list will allow you to export the graph by itself, without any other information. The full system manual is available by going to the reporting portal ( and clicking on View the Reporting Guide under Useful Links. 4.3 Quantitative and qualitative analysis Cemplicity will provide a survey data file in accordance with Appendix 6 to PHOs and the Commission so that further analysis can be performed. PHOs are able to further analyse or report the patient survey information as they choose for internal purpose. In the patient comments reports users are able to filter and view word clouds, and search on particular terms. Further analysis can be undertaken by PHOs using the data extract provided. The Commission will take a longer term view of the information and perform analysis as required to look at differential response, response rates and identify national trends that may suggest changes to the survey tool. 4.4 Continuous improvement The patient experience survey is about improving the quality of health services in New Zealand by enabling patients to provide feedback that can be used to monitor and improve the quality and safety of health services. Being able to capture, understand and act on patient experiences in a timely manner is a vital contributor to the improvement of health service delivery and the prioritisation of attention and resources. A critical element of this programme is the delivery of value to practices, PHOs and DHBs, while meeting the national reporting needs. Practices, PHOs and DHBs need a programme that is simple for them to participate in, one which provides actionable information for improvement activities and enables prompt follow up with patients requesting contact.

39 Key to enabling this continuous improvement is the decision to develop an electronic system that captures and reports patient feedback on a timely basis. Enabling patients to support their ratings with comments and examples lets practices and PHOs know what is being done well and what needs to change. The online methodology is effective at capturing rich patient stories as people can complete the survey when and where it suits them and without the presence of an interviewer.

40 5. Annual review Changes to the primary care patient experience survey, the reporting system and the methodology and procedures document will be made on an annual basis. This is to minimise the impacts on trending reports, process changes and costs. Following the first evaluation process described in 1.3.5, an annual evaluation of the programme will be conducted by the Commission to ensure continuous improvement. As part of this annual evaluation Cemplicity will survey all system users to understand the user experience with the tools, how the system is being used and outcomes at practice, PHO, DHB and national levels. The evaluators would provide the questions to be asked of users through the survey to the governance group. At this time we would agree how the programme could be enhanced to lead to better patient safety and quality of health service delivery.

41 Appendix 1: Primary care patient experience survey Survey overview 1. This version of the survey does not show the logic that skips people to appropriate questions based on their answers. Not all people see all the questions. 2. Questions in blue are the main logic drivers of the survey. Respondents will only be asked about things they have indicated are applicable to them. This is represented by the flower image to the right, whereby all patients see questions about their regular GP and place of care but only see other questions (the petals/modules) if they are relevant to the patient s experience. 3. All open comment questions are optional for respondents. Introduction Could you tell us if you are answering this survey on behalf of yourself or someone else? Single selection (radio buttons) 1. Myself 2. Someone else (Please tell us why) GP clinic or community health clinic This section has a few questions about your GP clinic or community health clinic. Is there one GP clinic or community health clinic you usually go to? Single selection (radio buttons) 1. Yes 2. No, I do not have a place that I usually go Next question only applicable to people that answer No to the previous question You are enrolled at [name of GP clinic or community health clinic]. Is there a reason you don t usually go there? Large text area Does the place you usually go to have a phone service, doctor or nurse available during evenings, nights or weekends? (Either there or another place) Single selection (radio buttons) 1. Yes 2. No 3. Don't know

42 Is there one GP or nurse you usually see? Single selection (radio buttons) 1. Yes 2. No How long have you been going to your current GP or nurse clinic? Single selection (radio buttons) 1. Less than one year 2. One to five years 3. More than five years Please answer this next section based on your experiences over the last 12 months When you ring to make an appointment how quickly do you usually get to see... Matrix (radio buttons) Top options 1. Same day 2. Next working day 3. Within a week 4. Over a week 5. Not applicable Side options 1. Your current GP? 2. Any other GP at the clinic you usually go to? 3. A nurse at the clinic you usually go to? Is this acceptable? Single selection (radio buttons) 1. Yes 2. No (Please tell us why): How long do you usually have to wait for your consultation to begin with... Matrix (Radio Buttons) Top options 1. 5 minutes or less minutes minutes 4. More than 30 minutes 5. Not applicable Side options 1. Your current GP? 2. Any other GP at the clinic you usually go to? 3. A nurse at the clinic you usually go to? Is this acceptable? Single selection (radio buttons)

43 1. Yes 2. No (Please tell us why): Did the reception and admin staff treat you with respect? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No In the last 12 months was there a time when you did not visit a GP or nurse because of cost? Single selection (radio buttons) 1. No 2. Yes (Please tell us why): Could you tell us why cost stopped you from seeing a GP or nurse? Multiple selection (checkboxes) 1. The appointment was too expensive 2. The cost to travel was too expensive 3. I couldn't afford to take the time off work 4. Other (Please tell us): Was there ever a time when you wanted health care from a GP or nurse but you couldn t get it? Single selection (radio buttons) 1. No 2. Yes (Please tell us why): When you contact your usual GP clinic about something important, do you get an answer the same day? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Does your GP or nurse explain things in a way that is easy to understand? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Are you confident that your GP or nurse is aware of your medical history? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes

44 3. No 4. Don t know Have you been involved in decisions about your care and treatment as much as you wanted to be? Single selection (radio buttons) 1. Yes 2. Yes, to some extent 3. No Does your GP or nurse... Matrix (radio buttons) Top options 1. Yes, always 2. Yes, sometimes 3. No Side options 1.Treat you with respect? 2. Treat you with kindness and understanding? 3. Listen to what you have to say? 4. Spend enough time with you? Is there anything you would like to tell us about your experience with your GP or nurse? Large text area Overall, was your experience with your GP or nurse clinic? (Please select a number) Very poor Excellent Now we ask you some general questions so we know which topics to ask about later in the survey Do you take any medication regularly? This includes vitamins, painkillers, supplements and any prescribed medication. Single selection (radio buttons) 1. Yes 2. No In the last 12 months... Matrix (radio buttons)

45 Did you have any tests such as x-rays, scans, blood tests or other tests? Have you seen any health care professionals other than a GP or nurse? Some examples are a midwife, physiotherapist, psychologist, social worker, counsellor, pharmacist, or dietitian. Have you seen a specialist doctor, other than a GP? Have you been to the emergency department at the public hospital? Have you stayed in hospital overnight? Do you have a health condition that will last more than six months? Yes No Don't know Yes No Don't know Yes No Don't know Yes No Don't know Yes No Don't know Yes No Don't know Medication The next questions are about medication, including vitamins, painkillers, supplements and any prescribed medication you have used during the last 12 months. From now on we ll refer to these as medication. Were you involved as much as you wanted to be in decisions about the best medication for you? Single selection (radio buttons) 1. Yes, definitely 2. Yes, to some extent 3. No 4. I did not want to be involved Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). Matrix (radio buttons) Top Options 1. Yes definitely 2. Yes to some extent 3. No 4. N/A Side options 1. Was the purpose of the medication properly explained to you? 2. Were the possible side effects of the medication explained in a way you could understand? 3. Were you told what could happen if you didn t take the medication, in a way you could understand? 4. Were you told what to do if you experienced side effects?

46 Did you follow the instructions when you took the medication? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No You said that you did not always follow the instructions when you took the medication. Please tell us why. Multiple selection (checkboxes) 1. Cost 2. Side effects 3. I forgot 4. I felt better 5. Other (Please tell us more): Has cost stopped you from picking up a prescription? Single selection (radio buttons) 1. No 2. Yes In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist (outside of hospital)? Single selection (radio buttons) 1. No 2. Yes 3. Don't know Because of the wrong medication or dose, did you... Matrix (radio buttons) Top options 1. Yes 2. No Side options 1. Stop taking it? 2. Get medical advice? 3. Get medical care? 4. Get admitted to hospital? Is there anything else you d like to tell us about being given the wrong medication or dose? Large text area Is there anything you would like to tell us about your experiences with your medication? Large text area Medical tests

47 Now a few questions about medical tests and scans you may have had in the past 12 months. This includes x-rays, scans, blood tests and other tests. Was the need for the x-ray, test or scan(s) explained in a way you could understand? Single selection (radio buttons) 1. Yes, completely 2. Yes, to some extent 3. No 4. I did not need an explanation Were you told how you could find out the results of your x-ray, scan(s) or tests? Single selection (radio buttons) 1. Yes 2. No 3. Not sure 4. I did not need an explanation Were the results of the x-ray, test or scan(s) explained in a way you could understand? Single selection (radio buttons) 1. Yes, completely 2. Yes, to some extent 3. No 4. Not sure 5. I was told I would get the results at a later date 6. I was never told the results of the tests Is there anything you would like to tell us about your experiences with x-rays, scan(s) or tests? Large text area Other health care professionals Now a few questions about health care professionals (other than a doctor or nurse) you may have seen or talked to. Some examples are midwife, physiotherapist, psychologist, social worker, counsellor, pharmacist, and dietitian. Was there a time when test results or information was not available at the time of your appointment with the health care professional? Single selection (radio buttons) 1. No 2. Yes 3. Don't know 4. N/A Were you given conflicting information by different doctors or health care professionals, e.g. one would you tell you one thing and then another would tell you

48 something different? Single selection (radio buttons) 1. No 2. Yes, sometimes 3. Yes, always How quickly do you usually get to see a health care professional? Single selection (radio buttons) 1. Less than a week weeks months 4. Longer Has cost stopped you from seeing a health care professional? Single selection (radio buttons) 1. No 2. Yes Could you tell us why cost stopped you from seeing a health care professional? Multiple selection (checkboxes) 1. The appointment was too expensive 2. The cost to travel was too expensive 3. I couldn't afford to take the time off work 4. Other (Please specify): Is there anything you would like to tell us about your experiences with health care professionals (other than a doctor or nurse)? Large text area Specialist doctors (other than GP) Now a few questions about specialist doctors (other than GPs) that you may have seen in the past 12 months. When you were referred to a specialist did you have any difficulties getting an appointment? Single selection (radio buttons) 1. No 2. Yes Any comments? Large text area In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor? Single selection (radio buttons) 1. Less than a week weeks

49 months 4. Longer Any comments? Large text area Has cost stopped you from seeing a specialist doctor? Single selection (radio buttons) 1. No 2. Yes Could you tell us why cost stopped you from seeing a specialist doctor? Multiple selection (checkboxes) 1. The appointment was too expensive 2. The cost to travel was too expensive 3. I couldn't afford to take the time off work 4. Other (please tell us): When you received care or treatment from specialist doctors, did they do the following? Matrix (radio buttons) Top options 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Side options 1. Ask what is important to you? 2. Tell you about treatment choices in ways you could understand? 3. Involve you in decisions about your care or treatment as much as you wanted to be? Any comments? Large text area Do the specialist doctors know your medical history and the reason for your visit? Single selection (radio buttons) 1. Yes, fully aware 2. Yes, aware in part 3. No 4. Don't know Has a doctor ordered a test (e.g. blood test, x-ray, etc) that you felt you didn t need because the test had already been done? Single selection (radio buttons) 1. No 2. Yes 3. Don't know 4. N/A

50 Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. Don't know 5. N/A Is there anything you would like us to know about how well your GP and specialist doctors are working together? Large text area Emergency departments The next questions are about accessing health care from a public hospital emergency department. The last time you went to the public hospital emergency department, why did you go there? Multiple selection (checkboxes) 1. It was clearly an emergency 2. I was told to go to the emergency department by a health care professional 3. I can t afford to go anywhere else 4. Other (Please tell us why): Is there anything you would like us to know about how well your GP and the emergency department are working together? Large text area

51 Hospital stays The next questions are about your most recent stay in hospital. Could you please tell us which hospital you received care from? Single selection (dropdown menu) Dropdown list of hospitals. Did the hospital arrange follow-up care with a doctor or other health care professional? Single selection (radio buttons) 1. Yes 2. No 3. N/A 4. Don't know Did your current GP seem informed and up-to-date about the plan for follow-up? Single selection (radio buttons) 1. Yes 2. No 3. N/A 4. Don't know Did you have to go back to hospital or get emergency care because of complications within a month after being discharged from hospital? Single selection (radio buttons) 1. No 2. Yes Is there anything you would like to tell us about your experience of your GP and the hospital working together? Large text area Long-term conditions The next questions are about health conditions that are expected to last 6 months or more. These are referred to as 'long-term' conditions. How long ago were you first diagnosed for the condition(s)? Matrix (radio buttons) Top options 1. Less than 6 months ago months ago years ago years ago years ago 6. Over 10 years ago 7. Don't know 8. N/A

52 Side options 1. Anxiety 2. Arthritis 3. Asthma 4. Cancer 5. Chronic obstructive pulmonary disease (COPD) 6. Depression 7. Diabetes 8. Heart disease 9. High blood pressure 10. Long-term pain 11. Other mental health conditions 12. Stroke 13. Other Which other long-term condition(s) have you been diagnosed with? Large text area Please answer each of the following questions using the column headings as a guide. Matrix (radio buttons) Top options 1. Yes, always 2. Yes, sometimes 3. No 4. N/A Side options 1. Were you given information you could understand about things you should do to improve your health? 2. Did you get help to make a treatment or care plan for your long-term condition that would work in your daily life? 3. After a treatment or care plan was made were you contacted to see how things were going? Is there anything you would like to tell us about your experience being treated for a long-term condition? Large text area We have just a few quick questions about you to help us better understand your answers Are you... Single selection (radio buttons) 1. Female 2. Male 3. Gender diverse Please tell us the year of your birth. Single selection (dropdown menu)

53 Which ethnic group or groups do you belong to? Multiple selection (checkboxes) 1. New Zealand European 2. Māori 3. Samoan 4. Cook Island Māori 5. Tongan 6. Niuean 7. Chinese 8. Indian 9. Other (such as Dutch, Japanese, Tokelauan) or Prefer not to answer You selected 'other' as an option for your ethnic group. Which of these ethnic groups do you belong to? Multiple selection (checkboxes) 1. Other European 2. Tokelauan 3. Fijian 4. Other Pacific Peoples 5. Southeast Asian 6. Other Asian 7. Middle Eastern 8. Latin American / Hispanic 9. African (or cultural group of African origin) 10. Other ethnicity 11. Don't know 12. Prefer not to answer In which language(s) could you have a conversation about a lot of everyday things? Multiple selection (checkboxes) 1. English 2. Māori 3. Samoan 4. New Zealand Sign Language 5. Other language(s), eg, Gujarati, Cantonese, Greek (Please tell us): 6. Would rather not say Did you need an interpreter to communicate with a health care professional? Single selection (radio buttons) 1. No 2. Yes, I had an interpreter 3. Yes, I used a family member as an interpreter 4. Yes, but I did not have an interpreter Any comments? Large text area Did the interpreter help you clearly communicate with the health care professional? Single selection (radio buttons) 1. Yes

54 2. Yes to some extent 3. No Any comments? Large text area Was cultural support available when you needed it? Single selection (radio buttons) 1. Yes, always 2. Yes, sometimes 3. No 4. I did not need cultural support Contact request Would you like someone to contact you to discuss your feedback in this survey? This means you will no longer be anonymous. Single selection (radio buttons) 1. No thanks 2. Yes, I would like someone from my current GP clinic to contact me to discuss my feedback or health experience Please phone your GP clinic as usual for any medical matters that require a consultation. Are you happy for the person contacting you to see a copy of your survey response? This means that your response will no longer be anonymous. Single selection (radio buttons) 1. Yes, I am happy for them to see a copy of my survey response 2. No, I do not want them to see my survey response I would like it to remain anonymous Please tell us your contact details. Vertical text box list 1. Your name: 2. address: 3. Phone number during the daytime: Please provide some information on what you would like to talk to us about. We can then ensure the right person contacts you. Large text area Survey copy request Would you like to be ed a copy of your survey response? Please note that as this survey is anonymous, if you do not request a copy of your survey response now and you have not requested contact from us, we will be unable to obtain a copy for you in future.

55 Single selection (radio buttons) 1. No thanks 2. Yes, please a copy of my survey response to [text box] Thank you for your time and feedback. You have now finished this survey. We have recorded all your answers so you can now close this window. Thanks again, [Practice Signatory]

56 Appendix 2: Domain questions and picker scoring Coordination Sub-domain Question Text QOptionText Value RC, IC, Seamless Is there one GP or nurse you usually see? Yes 10 No 0 Barriers to access In the last 12 months was there a time when you did not visit a GP or nurse because of cost? No 10 Barriers to access In the last 12 months was there a time when you did not visit a GP or nurse because of cost? Yes (Please tell us why): 0 RC, IC, Seamless Was there ever a time when you wanted health care from a GP or nurse but you couldn t get it? No 10 RC, IC, Seamless Was there ever a time when you wanted health care from a GP or nurse but you couldn t get it? Yes (Please tell us why): 0 RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? Yes, always 10 RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? Yes, sometimes 5 RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? No 0 RC, IC, Seamless When you contact your usual GP clinic about something important, do you get an answer the same day? N/A Not scored RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? Yes, always 10 RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? Yes, sometimes 5 RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? No 0 RC, IC, Seamless Are you confident that your GP or nurse is aware of your medical history? Don t know Not scored RC, IC, Seamless In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist No 10 (outside of hospital)? RC, IC, Seamless In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist Yes 0 (outside of hospital)? RC, IC, Seamless In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist Don't know 5 (outside of hospital)? Barriers to access Has cost stopped you from picking up a prescription? No 10 Barriers to access Has cost stopped you from picking up a prescription? Yes 0 RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare No 10 professional? RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare Yes 0 professional? RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare Don't know 5 professional? RC, IC, Seamless Was there a time when test results or information was not available at the time of your appointment with the healthcare N/A Not scored professional? RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn t need because the test had already been Yes 0 done? RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn t need because the test had already been No 10 done? RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn t need because the test had already been Don't know 5 done? RC, IC, Seamless Has a doctor ordered a test (e.g. blood test, x-ray etc.) that you felt you didn t need because the test had already been done? N/A Not scored

57 Barriers to access Has cost stopped you from seeing a healthcare professional? No 10 Barriers to access Has cost stopped you from seeing a healthcare professional? Yes 0 Barriers to access Has cost stopped you from seeing a specialist doctor? No 10 Barriers to access Has cost stopped you from seeing a specialist doctor? Yes 0 RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? Yes, fully aware 10 RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? Yes, aware in part 5 RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? No 0 RC, IC, Seamless Do the specialist doctors know your medical history and the reason for your visit? Don't know Not scored RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Yes, always 10 RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Yes, sometimes 5 RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? No 0 RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Don't know Not scored RC, IC, Seamless Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? N/A Not scored RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? Yes 10 RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? Don't know 5 RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? No 0 RC, IC, Seamless Did the hospital arrange follow-up care with a doctor or other healthcare professional? N/A Not scored

58 Physical and Emotional Needs Question Text Question Sub-part QOptionText Value Is this acceptable? Yes 10 Is this acceptable? No (Please tell us why): 0 Did the reception and admin staff treat you with respect? Yes, always 10 Did the reception and admin staff treat you with respect? Yes, sometimes 5 Did the reception and admin staff treat you with respect? No 0 How quickly do you usually get to see a healthcare professional? Less than a week 10 How quickly do you usually get to see a healthcare professional? 1-4 weeks 7 How quickly do you usually get to see a healthcare professional? 1-3 months 3 How quickly do you usually get to see a healthcare professional? Longer 0 In general, how long did you wait from the time you were first told you needed an appointment to the time Less than a week 10 you went to the specialist doctor? In general, how long did you wait from the time you were first told you needed an appointment to the time 1-4 weeks 7 you went to the specialist doctor? In general, how long did you wait from the time you were first told you needed an appointment to the time 1-3 months 3 you went to the specialist doctor? In general, how long did you wait from the time you were first told you needed an appointment to the time Longer 0 you went to the specialist doctor? Was cultural support available when you needed it? Yes, always 10 Was cultural support available when you needed it? Yes, sometimes 5 Was cultural support available when you needed it? No 0 Was cultural support available when you needed it? I did not need cultural support Not scored Does your GP or nurse... Treat you with respect? Yes, always 10 Does your GP or nurse... Treat you with respect? Yes, sometimes 5 Does your GP or nurse... Treat you with respect? No 0 Does your GP or nurse... Treat you with kindness and understanding? Yes, always 10 Does your GP or nurse... Treat you with kindness and understanding? Yes, sometimes 5 Does your GP or nurse... Treat you with kindness and understanding? No 0 Does your GP or nurse... Spend enough time with you? Yes, always 10 Does your GP or nurse... Spend enough time with you? Yes, sometimes 5 Does your GP or nurse... Spend enough time with you? No 0 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Same day 10 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Next working day 7 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Within a week 3 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Over a week 0 When you ring to make an appointment how quickly do you usually get to see... Your current GP? Not applicable Not scored When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Same day 10 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Next working day 7 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Within a week 3 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Over a week 0 When you ring to make an appointment how quickly do you usually get to see... Any other GP at the clinic you usually go to? Not applicable Not scored When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Same day 10 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Next working day 7 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Within a week 3 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Over a week 0 When you ring to make an appointment how quickly do you usually get to see... A nurse at the clinic you usually go to? Not applicable Not scored

59 Is this acceptable? Yes 10 Is this acceptable? No (Please tell us why): 0 How long do you usually have to wait for your consultation to begin with... Your current GP? 5 minutes or less 10 How long do you usually have to wait for your consultation to begin with... Your current GP? 6-15 minutes 7 How long do you usually have to wait for your consultation to begin with... Your current GP? minutes 3 How long do you usually have to wait for your consultation to begin with... Your current GP? More than 30 minutes 0 How long do you usually have to wait for your consultation to begin with... Your current GP? Not applicable Not scored How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? 5 minutes or less 10 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? 6-15 minutes 7 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? minutes 3 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? More than 30 minutes 0 How long do you usually have to wait for your consultation to begin with... Any other GP at the clinic you usually go to? Not applicable Not scored How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? 5 minutes or less 10 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? 6-15 minutes 7 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? minutes 3 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? More than 30 minutes 0 How long do you usually have to wait for your consultation to begin with... A nurse at the clinic you usually go to? Not applicable Not scored

60 Communication Question Text QOptionText Value Does your GP or nurse explain things in a way that is easy to understand? Yes, always 10 Does your GP or nurse explain things in a way that is easy to understand? Yes, sometimes 5 Does your GP or nurse explain things in a way that is easy to understand? No 0 Does your GP or nurse explain things in a way that is easy to understand? N/A Not scored Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). Yes, definitely 10 Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). Yes, to some extent 5 Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). No 0 Here are some questions about your medications prescribed or recommended by a doctor, nurse or pharmacist (outside of hospital). N/A Not scored Was the need for the x-ray, test or scan(s) explained in a way you could understand? Yes, completely 10 Was the need for the x-ray, test or scan(s) explained in a way you could understand? Yes, to some extent 5 Was the need for the x-ray, test or scan(s) explained in a way you could understand? No 0 Was the need for the x-ray, test or scan(s) explained in a way you could understand? I did not need an explanation Not scored Were you told how you could find out the results of your x-ray, scan(s) or tests? Yes 10 Were you told how you could find out the results of your x-ray, scan(s) or tests? No 0 Were you told how you could find out the results of your x-ray, scan(s) or tests? Not sure 5 Were you told how you could find out the results of your x-ray, scan(s) or tests? I did not need an explanation Not scored Were the results of the x-ray, test or scan(s) explained in a way you could understand? Yes, completely 10 Were the results of the x-ray, test or scan(s) explained in a way you could understand? Yes, to some extent 7 Were the results of the x-ray, test or scan(s) explained in a way you could understand? No 3 Were the results of the x-ray, test or scan(s) explained in a way you could understand? Not sure Not scored Were the results of the x-ray, test or scan(s) explained in a way you could understand? I was told I would get the results at a later date Not scored Were the results of the x-ray, test or scan(s) explained in a way you could understand? I was never told the results of the tests 0 Were you given conflicting information by different doctors or healthcare professionals e.g. one would you tell you one thing and then another would tell you something different? No 10 Were you given conflicting information by different doctors or healthcare professionals e.g. one would you tell you one thing and then another would tell you something different? Yes, sometimes 5 Were you given conflicting information by different doctors or healthcare professionals e.g. one would you tell you one thing and then another would tell you something different? Yes, always 0

61 Question Text Question Subpart QOptionText Value Does your GP or nurse Listen to what you have to say? Yes, always 10 Does your GP or nurse Listen to what you have to say? Yes, sometimes 5 Does your GP or nurse Listen to what you have to say? No 0 Does your GP or nurse Listen to what you have to say? N/A Not scored When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could Yes, definitely 10 understand? When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could Yes, to some extent 5 understand? When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could No 0 understand? When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could N/A Not scored understand? Please answer each of the following questions using the column headings as a guide. Were you given information you could understand Yes, completely 10 about things you should do to improve your health? Please answer each of the following questions using the column headings as a guide. Were you given information you could understand Yes, to some extent 5 about things you should do to improve your health? Please answer each of the following questions using the column headings as a guide. Were you given information you could understand No 0 about things you should do to improve your health? Please answer each of the following questions using the column headings as a guide. Were you given information you could understand about things you should do to improve your health? I did not need an explanation Not scored

62 Partnership Question Text Question Sub-part QOptionText Value Have you been involved in decisions about your care and treatment as much as you wanted to be? Yes 10 Have you been involved in decisions about your care and treatment as much as you wanted to be? Yes, to some extent 5 Have you been involved in decisions about your care and treatment as much as you wanted to be? No 0 Were you involved as much as you wanted to be in decisions about the best medication for you? Yes, definitely 10 Were you involved as much as you wanted to be in decisions about the best medication for you? Yes, to some extent 5 Were you involved as much as you wanted to be in decisions about the best medication for you? No 0 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? Yes, always 10 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? Yes, sometimes 5 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? No 0 When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? N/A Not scored When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or Yes, always 10 treatment as much as you wanted to be? When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or Yes, sometimes 5 treatment as much as you wanted to be? When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or No 0 treatment as much as you wanted to be? When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions about your care or N/A Not scored treatment as much as you wanted to be? Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for Yes, always 10 your long-term condition that would work in your daily life? Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for Yes, sometimes 5 your long-term condition that would work in your daily life? Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for No 0 your long-term condition that would work in your daily life? Please answer each of the following questions using the column headings as a guide. Did you get help to make a treatment or care plan for N/A Not scored your long-term condition that would work in your daily life? Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you Yes, always 10 contacted to see how things were going? Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you Yes, sometimes 5 contacted to see how things were going? Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you No 0 contacted to see how things were going? Please answer each of the following questions using the column headings as a guide. After a treatment or care plan was made were you contacted to see how things were going? N/A Not scored

63 Appendix 3: Patient experience interim solution patient data extract This document describes the rules surrounding the data extract and resulting file format required for importing patients information from PHOs into the national system s data warehouse. Data extract rules Each PHO extracts the patient data for importing into the PES data warehouse (for sending survey invites, reminders and generating dashboard reports), in accordance with the survey timetable. We highly recommend the process is automated as it means the file format and rules are consistently applied over time and the process can run irrespective of staff availability. The patient extract should include all patient consults that satisfy the following rules: 1. Frequency one week each quarter according to the survey timetable. 2. Date Range: for the first survey February 2016, patients with a qualifying event that falls within the 7 day period from Wednesday 10 February 00:01 to Tuesday 16 February 23: The extract file should exclude any patients if they have been previously included in an extract file supplied in the last 12 weeks. Exclude these patients using NHI prior to FTP upload. (This rule needs to be turned off if an extract is being regenerated for some reason) 4. Only include people aged 15 and over [Consultation date] [Date of birth] >= 15 years. 5. Include all records even if no address or mobile number. 6. All deceased patients as at the date the extract is run should be excluded from the extract (do not exclude based on the extract date range). 7. Each patient should only appear once in the extract file PHO to check for and remove duplicates. 8. Only the patient s most recent qualifying event within the extraction period should be included (eg, if seen 10 Feb, 12 Feb and 14 Feb, only include 14 Feb). File format The file received from the PHO needs to meet the following criteria: The file must be in comma delimited format (conforming to standard). All column headings must be provided as the first line and must match the Field name specified in the table below.

64 Where required, use 0d0a as a record terminator. 0 is a zero. The extract file must be named using the following convention: e.g. PROCARE_PE_From_yyyymmdd_To_yyyymmdd.csv Codes as opposed to descriptions will be used in columns wherever possible. DateTime values should be provided using yyyymmdd hh:mm:ss When fields have embedded commas, commas need to be placed inside double quotes as per the following example: Nick, 1 Story Road, Otahuhu, 21, 33, This is a comment, with a comma In the example above, the fields with embedded commas are enclosed in double quotes as per normal CSV rules. This then renders 5 columns for the comma delimited row. If there is an embedded double quote in the field, then it can be escaped by preceding it with another double quote as per the specification. File transfer We are able to accept secure file transfers using the FTPS or SFTP protocols. While both protocols are secure, we recommend using SFTP to avoid firewall setup issues. The details for connecting to Cemplicity s server will be given directly to the PHO s nominated IT contact person. SFTP (Secure Shell FTP) - recommended Requires port 22 to be open on your firewall. FTPS (FTP over SSL) FTPS may be connected using explicit or implicit modes; however explicit is recommended. Explicit mode requires port 21, and passive port range to be open on your firewall. Implicit mode requires port 990, and passive port range to be open on your firewall.

65 Data fields Field Data type Mandatory value Allowed options (if restricted) Example data NHI Number Alphanumeric - CHB2702 Title / Prefix Text - - Mrs First Given Name Alphanumeric - Jennifer First Preferred Alphanumeric - - Jenny Name Family Name Alphanumeric - Smith Mobile phone Alphanumeric address Alphanumeric - david@gmail.com Gender Alphanumeric F F I M U Comment Allowed codes from: Date of birth Date The patient s date of birth Date of qualifying Date Otherwise known as Date of last event consultation and often the Date of Ethnicity 1 Integer Only those codes in the Level 2 code table. Ethnicity 2 Integer - Only those codes in the Level 2 code table. invoice field is used 21 Allowed codes from: Allowed codes from:

66 Field Data type Mandatory value Allowed options (if restricted) Ethnicity 3 Integer - Only those codes in the Level 2 code table. Example data HPI-O (practice) Alphanumeric F2N084-H HPI-O (PHO) Alphanumeric F2N084-H PHO Org ID Alphanumeric Comment Allowed codes from: DHB of domicile (patient) Integer DHB Area codes 123 or 011 DHB that the patient is domiciled in. See DHB codes below Practice DHB Integer DHB Area codes 123 or 011 DHB the practice is physically located in. See DHB codes below. Lead/PHO DHB Integer DHB Area codes 123 or 011 DHB the PHO holds a contract with. See DHB codes below. DHB codes 011 Northland 021 Waitemata 022 Auckland 023 Counties Manukau 031 Waikato 042 Lakes 047 Bay of Plenty 051 Tairāwhiti 071 Taranaki 061 Hawke's Bay 081 Midcentral 082 Whanganui 091 Capital & Coast 092 Hutt 093 Wairarapa 101 Nelson Marlborough 111 West Coast 121 Canterbury 123 South Canterbury 160 Southern

67 Appendix 4: Correspondence with patients correspondence From: Health Experience To: patients supplied address Subject: Feedback on your health care experience [Practice name or logo] Dear Mary Smith We invite you to complete this survey about your experience of your own health care over the past 12 months. Taking part is voluntary but we would appreciate your feedback. Click here to begin The time it takes to complete the survey depends on your answers. Generally it takes 10 to 15 minutes, but may take longer if you have a lot you d like to say. If you need some help to complete this survey, please ask a relative or friend. You have three weeks from the date of this to respond before the survey is closed. We take your privacy very seriously. Your response is anonymous. Unless you choose to provide your contact details in this survey, no one will be able see your name or any other contact details. The feedback you give here is a very valuable way for us to understand how we can improve our service, so thank you in advance for your participation. Yours sincerely [Practice signatory] [Role title] [Practice name] [ ] [phone number] If you see your GP or Nurse more than three months from now you may receive this survey again. Please unsubscribe from this list if you do not wish to receive another .

68 Reminder Subject: Reminder - Feedback on your health care experience [Practice name or logo] Dear Mary Smith Recently we sent you an invitation to complete a survey about your experience of your own health care over the past 12 months. Taking part is voluntary, but we would appreciate your feedback. Click here to begin The time it takes to complete the survey depends on your answers. Generally it takes 10 to 15 minutes, but may take longer if you have a lot you d like to say. If you need some help to complete this survey, please ask a relative or friend. You have two weeks from the date of this to respond before the survey is closed. We take your privacy very seriously. Your response is anonymous. Unless you choose to provide your contact details in this survey, no one will be able see your name or any other contact details. The feedback you give here is a very valuable way for us to understand how we can improve our service, so thank you in advance for your participation. Yours sincerely [Practice signatory] [Role title] [Practice name] [ ] [phone number] If you see your GP or Nurse more than three months from now you may receive this survey again. Please unsubscribe from this list if you do not wish to receive another .

69 SMS correspondence Note that SMS correspondence is constrained by the maximum number of characters per SMS (160) and associated costs. Both the initial and reminder SMS are the same. Please give us feedback on your health care.go to Taking part is voluntary & anonymous.thanks [Practice Name 27 characters allowed] Survey introduction Once respondents have clicked on the link to complete the online survey, they are taken to the following introduction page. [Practice name or logo] Thank you for taking the time to complete this survey. This survey is about your experience of your own health care over the past 12 months. Your response is anonymous. Unless you choose to provide your contact details in this survey, no one will be able see your name or any other contact details, so please be open and frank in your feedback. There are several places in this survey where you can explain things in your own words. It is important not to mention any names or specific details if you wish to remain anonymous. The time it takes to complete the survey depends on your answers. Generally it takes 10 to 15 minutes, but may take longer if you have a lot you d like to say. If you need some help to complete this survey, please ask a relative or friend. The survey closes three weeks from the date you received the or text invitation. Warm regards [Practice contact details] Click here to begin

70 Survey conclusion Thank you for your time and feedback. You have now finished this survey. We have recorded all your answers so you can now close this window. Thanks again, [Practice Signatory]

71 Appendix 5: Bias in response The purpose of looking at results between different sub-groups of responders is to consider whether there are systematic patterns of different responses between different groups. This is important for the purposes of weighting responses. (Over-representation of a group which is likely to respond to questions differently will skew the overall result.) Typically age, gender, method of admission, method of response and more controversially ethnicity are included as variables on which surveys are weighted. Each is considered in turn below. Age older people are happier Communication Q1 Qs answered Q2 Explanations Q3a Dr listened Q3b Nurse listened Q3c Other listened Q4 Side effects explained Participation Q6 Involved Q7 Family involved Coordination Q9 Different things said Q10 Discharge info Physical and Q12 Toilet emotional Q13 Pain needs Q14 Respect Q15 Kindness Q16 Culture Overall Q20a Confidence Dr Q20b Confidence Nurse Q20c Confidence Other Questions with Sig Low Aggregate Scores Questions with Average Aggregate Scores Questions with Sig High Aggregate Scores

72 Gender Men are happier Female Male Communication Q1 Qs answered Q2 Explanations Q3a Dr listened Q3b Nurse listened Q3c Other listened Q4 Side effects explained Participation Q6 Involved Q7 Family involved Coordination Q9 Different things said Q10 Discharge info Physical and emotional Q12 Toilet needs Q13 Pain Q14 Respect Q15 Kindness Q16 Culture Overall Q20a Confidence Dr Q20b Confidence Nurse Q20c Confidence Other Questions with Sig Low Aggregate Scores 0 0 Questions with Average Aggregate Scores Questions with Sig High Aggregate Scores 0 0 (note: none of these reach α=0.95 but several reach α=0.9, I anticipate that this will change as we get more data)

73 Ethnicity - no obvious variation between European and Māori responses (not enough data for other ethnic groups) European Māori Communication Q1 Qs answered Q2 Explanations Q3a Dr listened Q3b Nurse listened Q3c Other listened Q4 Side effects explained Participation Q6 Involved Q7 Family involved Coordination Q9 Different things said Q10 Discharge info Physical and emotional needs Q12 Toilet Q13 Pain Q14 Respect Q15 Kindness Q16 Culture Overall Q20a Confidence Dr Q20b Confidence Nurse Q20c Confidence Other Questions with Sig Low Aggregate Scores Low 0 0 Questions with Average Aggregate Scores Ave Questions with Sig High Aggregate Scores High 0 0

74 Admission method - no obvious variation between methods Arranged Acute Waiting list Communication Q1 Qs answered Q2 Explanations Q3a Dr listened Q3b Nurse listened Q3c Other listened Q4 Side effects explained Participation Q6 Involved Q7 Family involved Coordination Q9 Different things said Q10 Discharge info Physical and emotional needs Q12 Toilet Q13 Pain Q14 Respect Q15 Kindness Q16 Culture Q20a Confidence Dr Q20b Confidence Nurse Q20c Confidence Other Questions with Sig Low Aggregate Scores Questions with Average Aggregate Scores Questions with Sig High Aggregate Scores 0 0 0

75 Distribution method and post seem to have some more positive responses (not clear if there is an age effect going on here) cell post Communication Q1 Qs answered Q2 Explanations Q3a Dr listened Q3b Nurse listened Q3c Other listened Q4 Side effects explained Participation Q6 Involved Q7 Family involved Coordination Q9 Different things said Q10 Discharge info Physical and emotional Q12 Toilet needs Q13 Pain Q14 Respect Q15 Kindness Q16 Culture Q20a Confidence Dr Q20b Confidence Nurse Q20c Confidence Other Questions with Sig Low Aggregate Scores Questions with Average Aggregate Scores Questions with Sig High Aggregate Scores 0 3 2

76 Appendix 6: Survey data file COLUMN HEADING DEFINITION COMMENT Q1 Q1_QO2_OtherText Q2 Q3 Q4 Q5 Q6 Q7_SP1 Q7_SP2 Q7_SP3 Could you tell us if you are answering this survey on behalf of yourself or someone else? Open comment field Is there one GP clinic or community health clinic you usually go to? You are enrolled at [PracticeName]. Is there a reason you don t usually go there? Does the place you usually go to have a phone service, doctor or nurse available during evenings, nights or weekends? (Either there or another place) Is there one GP or nurse you usually see? How long have you been going to your current GP or nurse clinic? When you ring to make an appointment how quickly do you usually get to see...your current GP? When you ring to make an appointment how quickly do you usually get to see Any other GP at the clinic you usually go to? When you ring to make an appointment how quickly do you usually get to Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

77 see A nurse at the clinic you usually go to? Q8 Is this acceptable? Free text field question answers are Q8_QO2_OtherText Q9_SP1 Open comment field How long do you usually have to wait for your consultation to begin with Your current GP? How long do you usually have to wait for your consultation to begin with Any Did the reception and admin staff treat you with respect? In the last 12 months was there a time when you did not visit a GP or nurse because of cost? exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Q9_SP2 other GP at the clinic you usually go to? How long do you usually have to wait for your consultation to begin with A Q9_SP3 nurse at the clinic you usually go to? Q10 Is this acceptable? Free text field question answers are Q10_QO2_OtherText Open comment field exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, Q11 etc. Q12 Q13 Q13_OtherText Could you tell us why cost stopped you from seeing a GP or nurse? Open comment field Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell. Free text field question answers are exported with the actual verbatim given.

78 Q14 Q14_QO2_OtherText Q15 Q16 Q17 Q18 Q19_SP1 Q19_SP2 Q19_SP3 Q19_SP4 Q20 Q21 Q22 Was there ever a time when you wanted health care from a GP or nurse but you couldn t get it? Open comment field When you contact your usual GP clinic about something important, do you get an answer the same day? Does your GP or nurse explain things in a way that is easy to understand? Are you confident that your GP or nurse is aware of your medical history? Have you been involved in decisions about your care and treatment as much as you wanted to be? Does your GP or nurse...treat you with respect? Does your GP or nurse...treat you with kindness and understanding? Does your GP or nurse Listen to what you have to say? Does your GP or nurse Spend enough time with you? Is there anything you would like to tell us about your experience with your GP or nurse? Overall, was your experience with your GP or nurse clinic? Do you take any medication regularly? This includes vitamins, pain killers, supplements and any prescribed medication. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given rating given by the patient. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

79 Q23_SP1 Q23_SP2 Q23_SP3 Q23_SP4 Q23_SP5 Q23_SP6 Q24 Q25_SP1 Q25_SP2 Q25_SP3 Q25_SP4 Q26 In the last 12 months...did you have any tests such as x-rays, scans, blood tests or other tests? In the last 12 months...have you seen any health care professionals other than a GP or nurse? Some examples are a midwife, physiotherapist, psychologist, social worker, counsellor, pharmacist, or dietitian. In the last 12 months...have you seen a specialist doctor, other than a GP? In the last 12 months...have you been to the emergency department at the public hospital? In the last 12 months...have you stayed in hospital overnight? In the last 12 months...do you have a health condition that will last more than 6 months? Were you involved as much as you wanted to be in decisions about the best medication for you? Was the purpose of the medication properly explained to you? Were the possible side effects of the medication explained in a way you could understand? Were you told what could happen if you didn t take the medication, in a way you could understand? Were you told what to do if you experienced side effects? Did you follow the instructions when you took the medication?

80 Q27 Q27_OtherText Q28 Q29 Q30_SP1 Q30_SP2 Q30_SP3 Q30_SP4 Q31 Q32 Q33 You said that you did not always follow the instructions when you took the medication. Please tell us why. Open comment field. Has cost stopped you from picking up a prescription? In the last 12 months have you been given the wrong medication or wrong dose by a doctor, nurse or pharmacist (outside of hospital)? Because of the wrong medication or dose, did you Stop taking it? Because of the wrong medication or dose, did you Get medical advice? Because of the wrong medication or dose, did you Get medical care? Because of the wrong medication or dose, did you Get admitted to hospital? Is there anything else you d like to tell us about being given the wrong medication or dose? Is there anything you would like to tell us about your experiences with your medication? Was the need for the x-ray, test or scan(s) explained in a way you could understand? Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

81 Q34 Q35 Q36 Q37 Q38 Q39 Q40 Q41 Q41_OtherText Q42 Were you told how you could find out the results of your x-ray, scan(s) or tests? Were the results of the x-ray, test or scan(s) explained in a way you could understand? Is there anything you would like to tell us about your experiences with x-rays, scan(s) or tests? Was there a time when test results or information was not available at the time of your appointment with the health care professional? Were you given conflicting information by different doctors or health care professionals e.g. one would you tell you one thing and then another would tell you something different? How quickly do you usually get to see a health care professional? Has cost stopped you from seeing a health care professional? Could you tell us why cost stopped you from seeing a health care professional? Open comment field Is there anything you would like to tell us about your experiences with health care professionals (other than a doctor or nurse)? Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and seperated by a pipe within the one cell. Free text field question answers are exported with the actual verbatim given. Free text field question answers are exported with the actual verbatim given.

82 Q43 Q44 Q45 Q46 Q47 Q48 Q48_OtherText Q49_SP1 Q49_SP2 Q49_SP3 When you were referred to a specialist did you have any difficulties getting an appointment? Any comments? In general, how long did you wait from the time you were first told you needed an appointment to the time you went to the specialist doctor? Any comments? Has cost stopped you from seeing a specialist doctor? Could you tell us why cost stopped you from seeing a specialist doctor? Open comment field When you received care or treatment from specialist doctors, did they do the following? Ask what is important to you? When you received care or treatment from specialist doctors, did they do the following? Tell you about treatment choices in ways you could understand? When you received care or treatment from specialist doctors, did they do the following? Involve you in decisions Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

83 Q50 Q51 Q52 Q53 Q54 Q55 Q55_OtherText Q56 Q57 about your care or treatment as much as you wanted to be? Any comments? Do the specialist doctors know your medical history and the reason for your visit? Has a doctor ordered a test (e.g. blood test, x-ray etc) that you felt you didn t need because the test had already been done? Does your current GP or nurse seem informed and up-to-date about the care you get from specialist doctors? Is there anything you would like us to know about how well your GP and specialist doctors are working together? The last time you went to the public hospital emergency department, why did you go there? Any comments? Is there anything you would like us to know about how well your GP and the emergency department are working together? Could you please tell us which hospital you received care from? Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell. Free text field question answers are exported with the actual verbatim given Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

84 Q58 Q59 Q60 Q61 Q62_SP1 Q62_SP2 Q62_SP3 Q62_SP4 Q62_SP5 Q62_SP6 Q62_SP7 Q62_SP8 Q62_SP9 Did the hospital arrange follow-up care with a doctor or other health care professional? Did your current GP seem informed and up-to-date about the plan for follow-up? Did you have to go back to hospital or get emergency care because of complications within a month after being discharged from hospital? Is there anything you would like to tell us about your experience of your GP and the hospital working together? How long ago were you first diagnosed for the condition(s)? Anxiety How long ago were you first diagnosed for the condition(s)? Arthritis How long ago were you first diagnosed for the condition(s)? Asthma How long ago were you first diagnosed for the condition(s)? Cancer How long ago were you first diagnosed for the condition(s)? Chronic Obstructive Pulmonary Disease (COPD) How long ago were you first diagnosed for the condition(s)? Depression How long ago were you first diagnosed for the condition(s)? Diabetes How long ago were you first diagnosed for the condition(s)? Heart disease How long ago were you first diagnosed for the condition(s)? High blood pressure Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc.

85 Q62_SP10 Q62_SP11 Q62_SP12 Q62_SP13 Q63 Q64_SP1 Q64_SP2 Q64_SP3 Q65 Gender (respondent) Ethnicity How long ago were you first diagnosed for the condition(s)? Long term pain How long ago were you first diagnosed for the condition(s)? Other mental health conditions How long ago were you first diagnosed for the condition(s)? Stroke How long ago were you first diagnosed for the condition(s)? Other Which other long-term condition(s) have you been diagnosed with? Were you given information you could understand about things you should do to improve your health? Did you get help to make a treatment or care plan for your long-term condition that would work in your daily life? After a treatment or care plan was made were you contacted to see how things were going? Is there anything you would like to tell us about your experience being treated for a long term condition? Are you... Which ethnic group or groups do you belong to? Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Ethnicity is sourced from the patient s answer to the survey question(s) and mapped according to the Ethnicity protocol codeset. Note that multiple selections are allowed and separated by a pipe within the one cell.

86 Ethnicity Other Q70 Q70_OtherText Q71 Q72 Q73 Q74 Q75 Age Group Gender Ethnicity 1 You selected 'other' as an option for your ethnic group. Which of these ethnic groups do you belong to? In which language(s) could you have a conversation about a lot of everyday things? Any comments? Did you need an interpreter to communicate with a health care professional? Any comments? Did the interpreter help you clearly communicate with the health care professional? Any comments? Was cultural support available when you needed it? Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Note that multiple selections are allowed and separated by a pipe within the one cell. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Free text field question answers are exported with the actual verbatim given. Survey question answers are exported as integers, where 1 = the first option in the list, 2 = the second option in the list, etc. Summarised into age groups from the year of birth the patient provides in the survey. Data is sourced from the patient extract file. Data is sourced from the patient extract file.

87 Ethnicity 2 Ethnicity 3 HPI-O (practice) DHB of Domicile (patient) Practice DHB PHO DHB Response ID StartedDate LastActivityAt Data is sourced from the patient extract file. Data is sourced from the patient extract file. HPI Facility ID for the practice. Random unique ID automatically generated by the system. Date and time the survey was started. Date and time the survey was completed.

88 Appendix 7: Patient experience report examples Dashboard view the first page users land on is the dashboard. This gives an instant snapshot of key results, comparisons and work-on areas. Results menu

89 The left hand side menu closely matches the modules that patients are asked about in the survey. Each module is a section in the menu, all of which concertina open to reveal results about that area of care.

90 Trend reports - see trends over time with trend reporting on practices (subject to user access limitations), PHOs, DHBs or nationally.

91 Practice comparisons practice level reports have limited access to protect patient anonymity and practice privacy. Compare the results of practices within your PHO.

92 Filters report filters enable you to place a lens on date ranges, age groups, gender and ethnicity.

93 Patient contact requests Patients are given the opportunity to request contact from their practice in the survey. If they do so, a notification will be sent to the nominated practice s address, along with an attachment of the patient s full survey response (if they have given permission for their survey answers to be attributed to their contact details).