The costs and benefits of asking patients for their opinions about general practice

Transcription

1 Family Practice Oxford University Press 1996 Vol. 13, No. 1 Printed in Great Britain The costs and benefits of asking patients for their opinions about general practice Hilary Hearnshaw, Richard Baker, Alison Cooper, Martin Eccles* and Jean Soper** Hearnshaw H, Baker R, Cooper A, Eccles M and Soper J. The costs and benefits of asking patients for their opinions about general practice. Family Practice 1996; 13: Background. Patient views are important in the evaluation of the quality of health care. The use of surveys needs to be evaluated to determine their cost-effectiveness and benefits. Objectives. To determine the costs of conducting patient opinion surveys in general practice and to find out how effective patient surveys are in stimulating changes which are beneficial for patient care. Method. Postal questionnaire to all 102 medical audit advisory groups (MAAGs) and 98 family health services authorities (FHSAs) in England and Wales, followed by postal questionnaire to 302 general practices reported to have conducted surveys, sampled by the type of questionnaire used. Numbers of MAAGs and FHSAs reporting surveys in general practice; types of questionnaire used; estimated costs; changes made; and benefits identified were measured. Results. Eighty-five (83%) MAAGs and 75 (77%) FHSAs responded. One hundred and fiftyfour (96%) of MAAGs or FHSAs reported survey activity. Types of questionnaire used were 1) designed by the practice, 2) designed by the MAAG or FHSA, possibly in collaboration with a practice, or 3) standard 'off-the-shelf. One hundred and thirty-three (44%) practices responded. Total costs to a practice of conducting a survey ranged from nothing to over Questionnaires designed by the practice are likely to be more costly than other designs. Some practices had surveys provided free of charge by MAAG or FHSA. Sixtyone per cent of practices said changes had been implemented and a further 22% of practices said changes were planned. The most common change was to appointment systems. Benefits were identified for patients, staff, the practice, the MAAG or FHSA and the NHS. Surveys also brought benefits in relationships and understanding. Only 8.2% of practices felt the costs of surveys outweighed the benefits. Conclusions. Many practices are surveying patients' opinions. Surveys can be costly but MAAGs and FHSAs can provide expertise and resources. Surveys using any of the types of questionnaire are likely to lead to changes and identifiable benefits. Benefits of surveys are perceived by the majority of practices to outweigh the costs. Keywords. General practice, patient opinion, costs, benefits, questionnaire. Introduction It is widely recognized that the views of patients are important in the evaluation of the quality of health care. 1-2 The government white paper 'Promoting Better Health' called for a greater emphasis on the views of consumers of primary health care. 3 Family health Eli Lilly National Clinical Audit Centre, Department of General Practice and Primary Health Care, University of Leicester, Leicester General Hospital, Leicester LE4 5PW, Centre for Health Services Research, University of Newcastle, Newcastle upon Tyne and ** Department of Economics, University of Leicester, Leicester, UK. service authorities (FHSAs) are encouraged to carry out patient surveys 4 and the directive for the creation of medical audit advisory groups (MAAGs) in 1991 acknowledged the importance of gathering patient opinion. 3 Policy-makers need information about the perceptions and requirements of patients 4 and patient surveys are actively encouraged by the Patients' Charter. 7 Suitable methods are available for patient opinion surveys in general practice. 8-9 Whilst those who wish to see general practice become more sensitive to the wishes of patients may be encouraged by these developments, a number of questions must be answered before the use of patient surveys is 52

2 accepted as an essential adjunct of practice management. First, it is important to establish that the findings are helpful to practices in identifying aspects of services that patients wish improved and that can be improved. If surveys only rarely lead to changes and corresponding benefits, then their routine use cannot be justified. Second, the planning and conduct of surveys use time and resources which might be better devoted to other activities. If the costs of surveys are high, it would be reasonable to question the extension of their use. However, there has been little study of the effectiveness of surveys in leading to improvements in the quality of care or to benefits for health care providers. Costs of patient opinion surveys In estimating the true cost of conducting a survey, economic appraisal measures the value of each item in terms of the price it could command in its best alternative use. 10 The appropriate way of valuing time is by considering what else that time could have been used for, and what that time would have been worth. This is the concept of opportunity cost. Opportunity costs should take account of the time spent by those developing, administering and analysing a survey, the time spent by patients in completing a questionnaire and the time spent by those planning, implementing and evaluating changes made, stimulated by the survey. In addition, costs incurred in conducting the survey such as postage, stationery and computer time should be considered. Although there are difficulties in estimating these costs accurately, they should not be ignored. Benefits of patient opinion surveys It is possible to identify the benefits of conducting patient opinion surveys, and the people to whom they are beneficial, but it can be difficult to put monetary values on these benefits. Benefits for patients may include not only improved care but also the feeling that staff want to know their opinion and hence care about them as people, not just as patients. Such benefits are even greater if changes made can be seen to reflect patients' requests. Benefits for practice staff may include affirmation of their professional skills, satisfaction in providing improved care, development of new skills in running a survey and opportunity to introduce changes that have been seen as necessary, but which previously had not had sufficient justification. Very few reports of patient opinion surveys mention an assessment of the true costs or true benefits, yet these are important factors in the value of surveys. The collaboration of researchers and practitioners in evaluating various aspects of health care is necessary if the best care for patients is to be achieved within the limited budget of the NHS. U Critical review and evaluation of methods of surveying patient opinion will contribute Costs and benefits of patient surveys 53 to this search for value for money. The aim of this study was to determine the costs of methods which have been used to survey patients' opinions in general practice and to find out how effective patient surveys are in stimulating changes which are beneficial for patient care. Method In order first to find out what patient opinion survey activity was taking place in general practices and what methods were in use, a postal questionnaire was sent to all 102 MAAGs and 98 FHSAs in England in The three categories of survey questionnaire used in practices were: A) a questionnaire designed by members of the practice staff; B) a questionnaire designed by the MAAG or FHSA either with or without the help of practice staff; or C) a standard questionnaire, defined in this study as one which has been used by practices in more than one FHSA area and is known to be easily and widely available, for example 'Ask the Patient', published by the College of Health. 16 A second questionnaire to collect information from practices about costs of surveys and resulting benefits of surveys was developed and piloted for each of the three categories. Topics covered are shown in Table 1. In order to estimate costs incurred, respondents were asked to identify the time used, by each member of staff involved, in planning the survey, administering the survey, analysing the data, planning and implementing changes. These were converted to opportunity costs by allocating a monetary value using Whitley NHS pay scales, standard nursing and general practitioner locum pay scales as at October u Costs of other resources were estimated directly by the respondent. The opportunity costs, consumables costs and costs of help purchased from outside the practice, for example, for a temporary data clerk, were summed to give an, overall estimated cost of conducting the survey. Respondents were asked to identify any changes, planned or made, to care or services, as a result of their TABLE 1 Topics covered in the questionnaire on cost-effectiveness of patient opinion surveys Topic area Design and use Implementing changes Overall evaluation Items Staff time used Resources used Help received Changes identified, planned or introduced Costs of changes made Benefits from changes made Indirect changes Did gains outweigh costs Best and worst aspects

3 54 Family Practice an international journal survey. Since changes are not always successful in leading to benefits, respondents were also asked to indicate whether the changes they had made had led to any benefits for patients, staff, the practice, the FHSA or MAAG or for the wider NHS. They were also asked to evaluate the balance between gains and costs of their survey. In order to send this questionnaire to practices who had conducted surveys, a random sample of MAAGs and FHSAs were telephoned by the research team. They were asked to forward cost-effectiveness questionnaires to practices on behalf of the research team. This ensured confidentiality for practices who posted their responses, anonymously, direct to the research team. The recruitment of MAAGs and FHSAs continued until 100 practices who had used a questionnaire they had designed themselves (category A) and 100 practices who had used one designed by the MAAG or FHSA (category B) were identified. MAAGs and FHSAs were asked to recruit all practices whom they had identified as having used a standard design questionnaire (category C). The MAAGs and FHSAs were also asked to complete a cost-effectiveness questionnaire on their own survey activities. A follow-up of non-responding practices was made after 8 weeks by providing the MAAG or FHSA with reminders to send to those practices who had not responded. Results Eighty-five (83 %) of the 102 MAAGs responded to the first questionnaire, of whom 83 (98%) reported some patient opinion survey activity taking place. Seventyfive (77%) of the 98 FHSAs responded of whom 71 (95 %) reported surveys being undertaken by practices. Three hundred and two questionnaires were recorded as having been sent to practices. One hundred and thirty-three (44%) responses were received (50 (47 %) for category A, 57 (42%) for category B and 26 (43 %) for category C). The standard questionnaires identified in use were: two questionnaires from the Centre for Health Services Research at the University of Newcastle, one concerned with the accessibility of care and the other with aspects of the consultation; 15 'Ask the Patient', a set of questionnaires available from the College of Health; 16 Measurement of Patient Satisfaction (MOPS); 17 Dialogue, comprising the Surgery Satisfaction Questionnaire and the Consultation Satisfaction Questionnaire; 18 and a set of questionnaires developed by the Centre for Primary Care Research, University of Manchester. 1 ' Costs Total costs to practices of conducting a survey are shown in Table 2. These can be broken down as follows: the mean percentage of the total attributed to staff costs was 81.7%, to resources 10% and to the costs of outside help 2.4%. The staff time costs were the highest cost category for each type of questionnaire. These were highest for practice-own-design surveys (median , 95% confidence interval , n = 50), next highest for practices using standard questionnaires (median , CI , n = 26), and lowest for MAAG/FHSA design questionnaires (median , CI , n=57). Those practices who identified staff costs as zero were those for whom an external agency, for example the FHSA, had provided the survey. In working in collaboration with a practice, costs may be incurred by the MAAG or FHSA. Estimates of these costs were made from the responses given to costeffectiveness questionnaires by MAAGs and FHSAs, as shown in Table 3. Help for the practice came from a number of different sources including MAAGs, FHSAs, university academic departments, commercial organizations and other health organizations. The types of help received included conducting and analysing the whole survey for a few practices, or, more commonly, help with one or TABLE 2 The number (3h) of practices reporting estimated costs of patient opinion surveys where practices bore the cost Estimated costs Type of questionnaire n < > 1500 Max Median 95% CI for median Practice own 50 14(28) 24(48) 10(20) 2(4) MAAG or FHSA designs 57 6(11) 25(44) 14(25) 9(16) 3(5) Standard designs 26 2(8) 13(50) 7(27) 2(8) 2(8)

4 Costs and benefits of patient surveys 55 TABLE 3 The number (%) ofmaags or FHSAs reporting levels of costs of practice surveys which were borne by MAAGs or FHSAs Costs Type of questionnaire used n 0 < O O- 15OO Over 1500 MAAG or FHSA 13 4(31) 5(38) 2(23) 1(8) Standard design 11 1(9) 2(18) 5(46) 2(18) 1(9) more of the stages of planning, design, data collection or data anlysis. Often the MAAG provided help at no cost to the practice, whereas one commercial organization charged a practice over 1500 to conduct the whole survey. Changes reported Sixty-one per cent of respondents reported that changes suggested as a result of the patient survey had been implemented, and 22% of respondents reported that changes had been planned. Over 90% of all respondents reported that the need for changes had at least been identified. The changes made by practices are shown in Table 4. Benefits reported Table 5 lists the benefits identified and shows for whom they were beneficial. Seventy per cent of respondents reported that benefits could be identified resulting from the changes. Respondents were also asked whether there were less tangible changes such as improved understanding of the value of patients' views or relations with the MAAG. Eighty-eight per cent of respondents reported that there had been benefits just from the activity of running a survey. Table 6 shows the percentage of respondents who said things improved, remained unchanged or got worse, for each item. Very few items got worse. Three practices reported their relationship with the FHSA got worse. Two of these practices reported they felt the questionnaire had been 'imposed' on them by the FHSA. Perceived value of surveys Respondents were asked for their agreement with the statement "In my opinion, the gains from using our questionnaire outweighed the costs". Of the 127 responses, 29 (23%) strongly agreed, 59 (46%) agreed, 28 (22%) neither agreed nor disagreed, 7 (6%) disagreed and 4 (3%) strongly disagreed. Only 9% of users, of all three types of questionnaire, felt that the costs outweighed the gains. The four who strongly disagreed with the statement included the three shown in Table 6 as feeling the practice's relationship with the FHSA had worsened as a result of the survey. Another one of the four, which reported incurring no costs since the MAAG had given the practice a grant, also reported that no changes were made and felt it had all been a waste of time. TABLE 4 The number (%) of practices reporting aspects of care which were changed as a result of a patient opinion survey Aspect of care which was changed Practice own designs n = 50 MAAG/FHSA designs n = 57 Standard designs n = 26 All questionnaires n = 133 Appointment system/surgery hours 11(22.0) 10(17.5) 8(30.8) 29(21.8) Alterations to facilities 7(14.0) 12(21.1) 6(23.1) 25(18.8) Moved premises 2(4.0) 2(3.5). 4(3.0) Patient confidentiality 6(10.5) 1(3.8) 7(5.3) Telephone system 3(6.0) 3(5.3) 1(3.9) 7(5.3) Others 11(22.0) 13(22.8) 2(7-7) 26(19.5) Total iv is the number of practices using each type of questionnaire. Some practices reported more than one change.

5 56 Family Practice an international journal TABLE 5 Reported benefits from surveys, showing the number of times each benefit was identified (a) For patients: shorter waiting times for appointments (30) improved premises and comfort (19) awareness that patients views are valued and acted upon (IS) improved telephone system (10) increased patient confidentiality (5) expansion of services offered (3) other (21) Total 103 For staff: more aware of patients' views (17) smoother running of practice (16) improved staff morale and job satisfaction (15) reassurance that work is appreciated by patients (13) improved work environment (8) happier patients (7) less stress and fatigue (6) improved telephone system (3) improved communication between staff and patients (2) other (11) Total 98 For the practice: improved image of practice (21) information for planning and budgeting (4) reduced complacency (1) Total 26 For FHSA or MAAG: questionnaire for use at other practices (8) gained picture of patients' views at each practice (7) more efficient practices (2) other (13) Total 30 For the wider NHS: experience and knowledge gained from doing a survey (4) better targeting of services and resources (4) happier patients (3) patient knowledge the NHS is working for them (2) Total 13 Discussion This study has found that many surveys of patient opinion have been conducted. Costs of surveys have been quantified and evidence that surveys lead to beneficial changes has been collected. However, before putting these findings into context, the fact that the response rates of the cost-effectiveness questionnaire used in this study were not high should be considered. It is recognized that postal survey response rates can be low. 21 Having no direct access to practices known to have conducted surveys meant sending questionnaires to practices via MAAGs and FHSAs as intermediaries with a loss of direct control of the recruitment process. However, the low response rate does not invalidate the conclusion, drawn from most of the respondents, that patient opinion surveys can lead to beneficial changes. Taking the worst case, if all the non-respondents had said they gained no benefit at all from their survey, that still suggests that 39% (88% of 44%) of practices who conducted surveys did gain benefit. Ninety-five per cent of FHSAs report patient opinion surveys happening in their areas, which translates to a high number of practices generating benefits for patients throughout the country. There is no reason to suppose that the costs reported in this study are any different from costs of most surveys, even though the response rate was low. The study has shown that questionnaire surveys are effective in stimulating change in many aspects of the service provided with reported changes to areas such as appointment systems and surgery hours, both key aspects of the Patients' Charter. 7 It is notable that the aspects of care where changes had been identified by patients as desirable did not include any of clinical prac-

6 Costs and benefits of patient surveys TABLE 6 Number (%) of practices reporting indirect changes from carrying out all types of questionnaire survey 57 Possible areas for change Total Improved No change Worse Don't know The practice's understanding of patient views (77.6) 29(22.3) Team members' understanding of audit (62.6) 44(35.8) The practice's relationship with patients (57.5) 47(37.0) 2(1.6) 5(3.9) Methods of practice management (37.1) 76(61.3) 2(1.6) The practice's relationship with the MAAG (35.0) 69(57.5) 9(7.5) Team members' understanding of each other's work (30.7) 85(68.5) Team members' understanding of practice management (29.8) 85(68.5) Individual skills (24.2) 86(71-7) 5(4.2) The practice's relationship with the FHSA (16.0) 87(73.1) 3(2.5) 10(8.4) tice. Perhaps patients are not yet being asked, nor volunteering, their opinions on clinical aspects of general practice care. Changes made were reported as being perceived as beneficial by the practices, though few practices would introduce changes they did not feel to be beneficial. There are also benefits, especially for practice staff, from the survey activity itself. The results suggest (Table 6) that working relationships can improve. It could be expected that a survey of patient opinion would improve the relationship with patients and improve understanding of patients' views. However it may not have been predicted that surveys would so frequently improve the team members' understanding of audit (62.6%) or improve methods of practice management (37.1 %) or improve the relationship with the MAAG (34.2%). These benefits are not trivial. Good working relationships are fundamental to the development of a positive approach to the management of quality in general practice. The difficult task of attempting to assign financial values to the benefits identified was not undertaken in this study. However, most of the respondents perceived their surveys as providing benefits which outweighed their costs, even though the costs may be several hundred pounds. The estimated costs of surveying patient opinion in a general practice range from nothing to over Those surveys which cost the practice nothing had their costs borne by the FHSA or MAAG. These costs can be as high as those for practices (Table 3), but may be spread over a number of practices. This is probably a good use of resources since the expertise of FHSAs and MAAGs may be likely to lead to better designed questionnaires and result in findings which provide usable and valid data to guide the practice in its decisions. However, the imposition of a survey by an FHSA could be seen by a practice as unwelcome. The cost of a survey using a questionnaire designed by the practice is likely to be more expensive than using a MAAG, FHSA or standard design. It is possible that many questionnaires designed by practices, but neither tested nor validated, not only are costly but may not be providing good data upon which to base decisions. 22 It is all too easy to design questions which produce answers to confirm prejudices, without realizing it is happening. 23 The greatest component of costs is that of staff time. These costs may not be obvious to a practice. A practice manager or a general practitioner may spend 'free' time working on the survey design or entering the data. One of the clerical staff may undertake data entry and analysis during office hours. The costs in these cases may not seem like 'extras', and rightly so, but someone has to pay them and they should be recognized when deciding whether to conduct a survey. It is difficult to reduce these costs, but using the services offered by the MAAG or FHSA may be one way. Generally, the responses to our questionnaire showed that the costs

7 58 Family Practice an international journal of consumables used on conducting a survey are low. It is only time which is expensive, especially that of a general practitioner. Many practices said their understanding of audit had improved by doing the survey and many practices have received help from MAAG in running a survey for audit purposes. The FHSAs, on the other hand, tend to provide help in the form of a complete survey instrument which they administer. This has sometimes been seen as an imposition, but mostly as a welcome source of information for the practice. It is acknowledged that the benefits reported in this study are those perceived by practices and not necessarily by patients. It was not feasible in this study to survey directly the opinions of the patients of our responding practices. Our purpose is to inform practices of the likely costs and benefits to them of surveying their patients' opinions, in order that informed decisions on surveying may be made by practices. Conclusions Surveys of patient opinion can be costly but they do lead to changes and benefits. MAAGs and FHSAs should continue to encourage practices to conduct patient opinion surveys and offer their own expertise in designing and providing questionnaires. Surveys are likely to stimulate improvement in the service provided by the primary health care team and benefit both the patients and the team itself. Standard questionnaires, MAAG/FHSA designed questionnaires and practice own design questionnaires can all be costly to use in surveys, but there is considerable variation. There is therefore the potential to avoid incurring high costs, possibly by sharing the MAAG or FHSA expertise. Practices should continue to carry out patient opinion surveys and should take advantage of the help which MAAGs and FHSAs can offer. This will not only reduce costs for the practice, but may provide more valid data than using one designed by themselves. Standard questionnaires are no more costly than other types and may provide more valid and reliable data. Thus changes made based on the data collected will be more likely to be appropriate and a good use of resources. They will also allow comparison of results between practices who use the same standard instrument. Benefits of surveys are perceived by the majority of practices to outweigh the costs. Acknowledgements This study was supported by the Trent Regional Health Authority under their Patients' Charter Initiative 1993/4. The research team is grateful to the staff at MAAGs and FHSAs who helped with this study and to all respondents to the cost-effectiveness questionnaire. References 1 Donabedian A. The Lichfield Lecture quality assurance in health care: Consumers' role. Quality in Health Care 1992; 1: William SJ, Calnan M. Key determinants of consumer satisfaction with general practice. Fam Pract 1991; 8: Promoting better health. The government's programme for improving primary health care. London: HMSO, * Department of Health and Welsh Office. General practice in the National Health Service. A new contract. London: Department of Health and Welsh Office, Department of Health. Medical audit in the family practitioner services. HC(FP)(90)8. London: Department of Health, Van Crampen C, Friele RD, Kerssens JJ. Methods for assessing patient satisfaction with primary care: Review and annotated bibliography. NTVEL bibliography No. 35. Utrecht: Netherlands Institute of Primary Care (NIVEL), NHS Management Executive, The Patient's Charter and Primary Health Care. EL(92)88. London: Department of Health, Cartwright A. Patients and their doctors. London: Routledge Kegan Paul, Cartwright A. Health surveys in practice and in potential: a critical review of their scope and methods. London: King Edwards Hospital Fund, Robinson R. Costs and cost minimisation analysis. Br Med J 1993; 307: Walshe K, Coles J. Evaluating audit: Developing a framework. London: CASPE Research, n Jones R, Spencer J. Making changes? Audit and research in general practice. Br J Gen Pract 1993; 43: Hutton J. How providers should respond to purchasers' needs. In Drummond MF, Maynard A (eds). Purchasing and providing cost-effective health care. Cambridge: Churchill Livingstone, 1993: Medeconomics October 1993: Patient satisfaction questionnaires. Newcastle MAAG, St Nicholas Hospital, Jubilee Road, Newcastle Upon Tyne NE3 3XT. 16 Ask The Patient. The College of Health, St Margaret's House, 21 Old Ford Road, London E2 9PL. 17 MOPS. 75 Sheen Lane, East Sheen, London SW14 8AD. 18 Dialogue. Eli LiDy National Clinical Audit Centre, Department of General Practice, University of Leicester, Gwendolen Road, Leicester LE5 4PW. 19 Centre for Primary Health Care Research, The Department of General Practice, University of Manchester, Rusholme Health Centre, Walmer Street, Manchester M14 5NP. 20 Altaian DG. Practical statistics for medical research. London: Chapman & Hall, Bailey KD. Methods of social research. 2nd edition. The Free Press, Whitfield M, Baker R. Measuring patient satisfaction for audit in general practice. Quality in Health Care 1992:1: Wilson N, McClean S. Questionnaire design. A practical introduction. University of Ulster, 1994.