SPECIAL ISSUE. Evaluating Health Plan Quality 2: Survey Design Principles for Measuring Health Plan Quality

Transcription

1 Evaluating Health Plan Quality 2: Survey Design Principles for Measuring Health Plan Quality Douglas R. Wholey, PhD; Jon B. Christianson, PhD; Michael Finch, PhD; David Knutson, MA; Todd Rockwood, PhD; and Louise Warrick, DrPH Objective: To develop principles for measuring the quality of specific health plans from a physician s perspective. Study Design: Literature review, expert review, cognitive interviews. Methods: We did a literature review on the use of physician surveys about managed care to determine the contributions and weaknesses of those surveys. Then, an expert review of prior survey efforts to measure health plan quality from the physician s perspective was performed. Results: A survey instrument based on a conceptual model of health plan quality was developed. Its purpose was to measure health plan quality from the physician s perspective. Principles for surveying physicians guided the structure of the survey. Conclusion: Survey instruments can be designed to take into account a physician s unique perspective on health plan quality and can include measures that control for potential biases such as anti-managed care bias. (Am J Manag Care 2003;9:SP65-SP75) Physicians offer an important viewpoint on health plan quality. 1 The physician viewpoint is important not only because physicians can provide unique and timely information but also because the physician viewpoint can be used to corroborate other measures. 1-4 Physicians observe the point where health plan processes and structures (eg, utilization review, profiling, formularies, guidelines, provider networks) meet patients. In addition to this unique vantage point, physicians have a distinct blend of diagnostic and therapeutic knowledge, training, skills, and normative commitments. 4,5 Consequently, physicians can provide an important perspective on how healthcare is managed within a health plan, an assessment of quality, and a timely, early warning about possible problems relating to utilization management. Our goal was to develop a survey instrument that obtains the physician s perspective on health plan quality and that results in information that is informative and useful to consumers, health plans, provider organizations, and purchasers. Earlier, we developed a theory about the relationship of health plans to physicians that guided the survey development. 1 We differentiated between health plan quality and quality of care, arguing that high-quality health plans cause high-quality care. 6 We followed the Institute of Medicine by defining quality of care as the degree to which health services for individuals and populations increase the likelihood of positive health outcomes and are consistent with current professional knowledge. 6 We defined health plan quality as the degree to which organizational structures and processes increase the quality of care for individuals and populations. We expect highquality health plans (1) to ensure that patient care management processes shown to be effective (eg, reminders, guidelines) are being used by plan physicians and (2) to ensure that care management processes that provide feedback or training to physicians (eg, profiling, quality reports) also are in use. Our theory for understanding health plan quality from a physician s perspective focused on the aspects of health plans that physicians would be best able to observe. Health plans can use a wide variety of structures and processes to influence cost and quality. Through credentialing processes, plans can vary the accessibility and quality of physicians and laboratories. Through benefit design features (eg, copayments, deductibles, coverage) and education, From the Division of Health Services Research and Policy (DRW, TR) and the Department of Health Care Management (JBC), University of Minnesota, Minneapolis, Minn; UnitedHealth Group Center for Health Care Policy & Evaluation, Eden Prairie, Minn (MF); and the Park Nicollet Institute for Research and Education, Minneapolis, Minn (DK). Preparation of this article was assisted by grant from The Robert Wood Johnson Foundation, Princeton, NJ. Correspondence to: Douglas R. Wholey, PhD, Division of Health Services Research and Policy, University of Minnesota, 420 Delaware St SE, MMC 729, Minneapolis, MN whole001@umn.edu. VOL. 9, SPECIAL ISSUE 2 THE AMERICAN JOURNAL OF MANAGED CARE SP65

2 plans can influence consumer behavior. Through incentive arrangements, plans can encourage physicians to meet standards. Health plans manage physicians indirectly through credentialing processes, network selection, and incentive arrangements. 7 Health plans manage physicians directly through processes such as utilization review; preauthorization; clinical practice guidelines; reminder systems; clinical decision-support systems; diagnostic test order management; educational outreach visits; audit, feedback, and profiling; and continuing medical education. 1 Physicians are likely to be better observers of these latter management processes rather than plan structures such as copayments and deductible levels, because physicians are likely to be directly involved with these management processes and the processes are likely to be salient to them. In this paper, we develop principles for asking physicians about these activities. In the first section, we review the literature on use of physician surveys about managed care and discuss the contributions and weaknesses of those surveys. We then develop survey design principles for questions that will measure the concepts discussed in our previous work. 1 In the third section, we describe the process of developing a survey instrument to measure health plan quality from the physician s perspective. In the final section, we discuss the survey instrument. SURVEYING PHYSICIANS ABOUT HEALTH PLANS: THE EXISTING LITERATURE Surveys are an important source of information for quality assessment. Respondents include consumers, 8,9 corporate health benefits officers, 10 health plans, 11 physicians in specific medical groups, 12 physicians in health plans, 2-4 physicians in communities, 13,14 and physicians in academic settings To guide our research on physician surveys about the quality of particular health plans, we conducted a literature search for the period from 1984 to 2001 for articles that asked physicians about managed care. The search revealed 66 published articles that surveyed physicians regarding managed care. The vast majority of these articles asked physicians about their general views on managed care, their views on the effects of managed care structures, or their views on managed care outcomes. Fewer articles surveyed physicians on what managed care organizations do to manage care or on their perceptions of quality in a particular health plan. Settings varied from large national surveys to small local surveys and intra-organizational surveys The next section briefly reviews this literature. Asking Physicians About Managed Care in General We found a large number of surveys asking physicians about their overall views of managed care. These surveys addressed knowledge, attitudes, and beliefs about managed care in general 13-15,17,22,30-36 ; ethical and legal concerns 37,38 ; capitation 29,39 ; and clinical guidelines Because these surveys tapped general beliefs across all forms of managed care, rather than experience with a specific health plan, they are not useful for making inferences about the quality of specific health plans. They typically used broad sampling frames, assuming that all physicians were equally reliable informants about managed care, independent of their level of experience with managed care. Sampling in this manner may introduce a halo effect bias, in which general beliefs about managed care could bias physicians reports about their experiences with a particular plan, limiting the usefulness of these types of surveys. Asking Physicians About the Effects of Managed Care We found a second cluster of surveys that asked physicians about the effect of managed care, or some feature of managed care, on a specified outcome such as Medicaid participation, 43 communication patterns, 16 costs of common primary care problems, 44 medical school and residency training, 18,45 addiction treatment, 24 drug prescribing, 46 and cardiovascular practice. 47 Physicians also have been asked about the effect of capitation on referrals, 48 on their incentive to reduce services, 49 and on authorization delays and access to specific services. 23 An important assumption underlying these physician surveys is that physicians are experts who can observe the effect of managed care structures. However, physician views may be problematic and biased because of their normative beliefs, 4 the nonrandom sample of patients they observe, and their inability to directly observe all outcomes. Because of these difficulties, physician views may not always represent an unbiased measure of the effect of managed care. A significant number of studies have focused on the effect of financial incentives, 12,20,50-52 yet the results of these surveys may be of limited value in assessing health plan quality for a number of reasons. For instance, research suggests that the effects of incentive structures are nonspecific, decreasing both SP66 THE AMERICAN JOURNAL OF MANAGED CARE JUNE 2003

3 Survey Design Principles for Measuring Health Plan Quality appropriate and inappropriate services. 53 Systematic reviews of randomized controlled trials of the effect of incentives find few studies and few effects, consistent with physician reports of the small, insignificant influence of incentives on quality of care. 4 However, there are reasons to doubt the accuracy of physician evaluations of the effect of incentives on healthcare delivery. Physicians in group practices may be reimbursed in salary arrangements, with the group administrator managing the contractual relationships with health plans and the remaining physicians not being directly exposed to those arrangements. Also, physicians perceptions of incentive and organizational structures may be biased to the degree that they are influenced by qualityof work-life issues such as satisfaction with pay. Our review found that surveys addressing the effect of managed care on physician quality of work life are relatively common. These surveys asked physicians for their opinion on how managed care had affected them or had influenced their practice. Measures of physician quality of work life included job satisfaction, career satisfaction, work life satisfaction, physician patient relationships, and professional autonomy ,26,43,57-71 These surveys are less problematic than surveys asking physicians about the effect of managed care structures because physicians are reliable respondents concerning their own satisfaction. But studies that assess the influence of managed care on physician work life do not directly address health plan quality. We have argued that physician quality of work life is an outcome of health plan quality. 1 Just as quality of care is an outcome experienced by patients and is conceptually distinct from health plan quality, quality of work life, autonomy, and satisfaction 72 are outcomes experienced by physicians and conceptually distinct from health plan quality. Health plan practices do affect physician satisfaction, with high denial rates, long waits for authorization, and time pressure for patient visits resulting in less satisfied physicians, whereas guidelines and education result in more satisfied physicians. 4,21,71 These effects are likely to be a joint function of health plan actions and physician expectations that were acquired through socialization. Although physician satisfaction and quality of work life may affect patient satisfaction, 73 they are distinct from health plan quality. Asking Physicians About How Healthcare Is Managed Of particular interest to the survey development process was research on physician perceptions of care management tools (eg, utilization review, profiling) that can be used by health plans, provider organizations, and medical groups to manage care processes and physicians. 1 We found 4 surveys in the literature that asked physicians about the way healthcare is managed. Given the growth of physician groups contracting with HMOs to deliver care, where the locus of control remained with the groups, Kerr and colleagues 74 chose to identify the care management techniques used by 94 medical groups. They asked medical directors, utilization management directors, and administrators about group use of 5 healthcare management strategies: gatekeeping; referral and testing preauthorization; pharmacy, referral, and utilization profiling; guideline use; and education. Second, physicians in a single mixed-model HMO an HMO in which no specific type (group, staff, network, independent practice association [IPA]) dominates were queried about their receptiveness to managed care drug cost-containment strategies. 75 Third, physicians in a Medicare HMO were asked about their experience with 7 HMO management strategies: a computer reminder system for preventive care, a geriatric education seminar, profiling, office record review, prior authorization for tests and referrals, case management services, and case review with colleagues; survey questions addressed whether the physicians perceived these strategies to be burdensome, whether the strategies were beneficial to patients, and whether the physicians were willing to implement them. 76 Finally, in a comprehensive national survey, 22 physicians were asked about their experience with 3 health plan tools intended to improve care. Physicians were asked about the frequency with which health plans (1) provided guidelines or innovative tools to manage patient illnesses, (2) helped encourage a patient to practice better health, and (3) provided clinical data to help physicians provide more effective care. These surveys begin to address health plan quality and are a valuable addition to our knowledge. But a potential problem occurs when the respondent is a medical director; what a medical director reports as occurring may not be what physicians are experiencing. Physicians as the end users of these tools are arguably in the best position to report on whether plan management strategies are available to them and to evaluate their application. Asking Physicians About Quality of Care in Specific Health Plans The use of consumer surveys to report on particular health plans has fostered an interest in obtaining physician views on particular health plans. 2-4 VOL. 9, SPECIAL ISSUE 2 THE AMERICAN JOURNAL OF MANAGED CARE SP67

4 However, we found only 2 surveys that asked physicians about specific health plan practices. Borowsky and colleagues surveyed physicians in 3 health plans on a number of quality-management strategies. 2 Physicians were asked to evaluate health plan processes such as reminder systems, clinical guidelines, individual performance indicators, physician education, and access to specialty care. Similarly, Williams and colleagues surveyed physicians in 5 health plans, representing 8 distinctive organizational types, to rate practices based on model and reimbursement. 4 Their survey included measures of provider quality, administrative support, enrollee support, perceived autonomy, authorization procedures, clinical guidance, general evaluations, and the influence of plan management strategies on clinical practice and quality of care. Both surveys found that physicians were able to provide plan-specific information and that ratings of the health plans varied substantially, suggesting that physician surveys are useful for assessing health plan quality. Physicians have strengths and weaknesses as respondents. There is some evidence that physicians are less reliable than patients in providing information on patient symptoms 77 and on physician counseling of patients regarding lifestyle and health habit issues. 78 Physicians also may have difficulty reporting the financial elements of contract arrangements negotiated by their medical groups and foundations. 50 Physicians are not likely to be the best respondents about activities in which they have peripheral involvement, such as the provisions by plans of information to consumers or the benefit arrangements offered by plans to enrollees. Although physicians as respondents about health plans have some limitations, their strength is their unique position in the treatment process physicians are the clinical face of a health plan to patients. Asking Physicians About Health Plans Our survey development, which extends the reviewed literature on surveying physicians, is based on work by Borowsky and his colleagues, 2,3 who developed and refined a telephone survey to measure physicians assessment of the quality of care in selected health plans in the Minneapolis St. Paul metropolitan area. In 1996, the Robert Wood Johnson Foundation provided support for further survey instrument refinement involving physician focus groups, national testing of the instrument to include Medicaid and Medicare risk plans, and the involvement of national stakeholders. (This development was carried out under the direction of Nicole Lurie, MD, RAND Corporation, Arlington, VA.) The result was a 43-item pilot instrument that was used to survey 5050 generalist and specialist physicians in 16 health plans in 5 areas nationwide, representing Medicaid, Medicare, and commercial plans. The sample for each plan was stratified to include 50% generalists and 50% specialists. Each physician reported on 1 health plan. The vast majority of surveys were administered by telephone; however, a small number of physicians (fewer than 1%) requested that the survey be faxed to them. The survey contained items measuring 3 domains of health plan quality: barriers to high-quality care, clinical capability of health plan physicians, and facilitators of high-quality care. It also contained 1 measure of overall quality the physician s willingness to recommend the health plan to a family member. The response rate was 70% for generalists and 67% for specialists. Response rates, by plan, ranged from 44% to 95% for generalists and from 15% to 94% for specialists. We convened an expert survey research panel consisting of purchaser groups; representatives from the Health Care Financing Administration (now the Centers for Medicare and Medicaid Services), RAND, and the Robert Wood Johnson Foundation; state Medicaid administrators; health plan partners; and the research team. This research group reviewed the survey implementation, experience, results pertaining to the properties of the survey instrument, and the uses and value of the survey data. Based on this review, we concluded that it was feasible to collect data about health plan quality from physicians and that these data could be used for comparisons among health plans and to assist health plans in identifying quality-improvement opportunities. Following the suggestions of the expert panel and our own analysis of implementation issues and the survey data, we extensively revised the conceptual model of health plan physician relationships, 1 resulting in significant changes in survey design and questions. We viewed the testing of the redesigned survey instrument not only as an opportunity to assess the performance of the instrument, but also as an opportunity to evaluate the potential for mail versus telephone survey administration. We believed that an experiment to test the relative acceptability, feasibility, and cost of different modes of survey administration was needed for both theoretical and practical reasons. From a theoretical perspective, there are strong arguments that a mail survey can yield more accurate, thoughtful responses from physicians for the types of questions addressed in the survey. A SP68 THE AMERICAN JOURNAL OF MANAGED CARE JUNE 2003

5 Survey Design Principles for Measuring Health Plan Quality mail survey is less likely to result in social desirability, acquiescence, and satisfying behavior than is a phone survey in which a respondent is interacting with the interviewer. Further, in a mail survey, respondents have a greater chance for more cognitive processing. Among survey modes, face-to-face is best, mail is next best, and phone is worst From a practical point of view, face-to-face surveys are prohibitively expensive, and it is difficult and costly for organizations interested in using the survey to carry out a telephone effort. New developments in survey research suggest that mail surveys, administered appropriately and with aggressive telephone follow-up, can yield acceptable response rates. 93,95,96 Knowing the relative response rates, response values, and costs of these various survey methods and determining the most cost-effective survey administration method are significant for future efforts to survey physicians. We used cognitive interviews with physicians in April and May 2000 to review preliminary surveys. The physicians included 10 from the Minneapolis St. Paul, Minnesota, area and 10 from the Carbondale, Illinois, area. They were chosen to reflect the survey s target population (physicians who had health plan contracts) and included both generalists and specialists. Their responses were used to revise items to make them easier to understand and easier to respond to, and to better fit the survey measures to theoretical concepts. A Conceptual Model for Physician Measurement of Health Plan Quality The Figure in the first paper in this Special Issue shows the revised conceptual model that guided the new version of the survey instrument we used to measure health plan quality. 1 The columns in the Figure delineate the 3 major areas of health plan management: physicians, drugs, and hospitals, and the rows are arrayed in 4 major blocks. The first block is based on the PRECEDE model for changing physician behavior. 97 Enabling, immediate, patientspecific structures and processes (eg, the provision of reminders) affect care delivery in real time. Reinforcing, timely, and practice-specific structures and processes (eg, profiling) provide timely data that allow physicians to adjust practice policies and organization. Predisposing, profession-specific structures and processes are health plan activities (eg, continuing medical education) aimed at changing physician beliefs. The second block addresses physician evaluations of the quality of structures or processes in each area. The third block is physician evaluations of the quality of outcomes, which measures physicians perceptions of the quality of patient care, environments of trust, continuity of care, and ability to customize care. The fourth block consists of physician overall evaluations of a health plan and physician recommendation of a health plan. GUIDING PRINCIPLES FOR SURVEYING PHYSICIANS Based on our literature review, the previous pilot survey, and the comments from the expert advisory group about the results from the prior survey, we developed a set of guiding principles for surveying physicians in the context of the conceptual framework provided by the Figure. We followed standard questionnaire design principles. 98 These standards include designing the survey to minimize respondent burden and avoiding double-barreled questions such as Plan s efforts to implement clinical guidelines have helped my patients to get better care. (This question requires respondents to assess the effort the plan is making to implement clinical guidelines, the effect of the guidelines on care, and whether the respondent s patients have benefited from the intervention). Filters also were used to guide response. (An example would be asking only respondents who say they receive utilization and profiling reports to answer questions on the usefulness of those reports.) Other principles were followed to support the assessment of internal reliability and validity in the context of surveying physicians about specific health plans. Principles to Support Survey Reliability and Validity Both Behavioral and Evaluative Measures Are Needed. The survey should ask for evaluative ratings of health plans (eg, overall quality ratings, health plan recommendation) and reports of the behaviors that theory suggests affect those evaluations (eg, receiving profiling supports, availability of disease management programs). This strategy has been used effectively for measuring hospital performance from a patient s perspective 99 and HMO quality from a consumer s perspective. 100 Capturing both behavioral and evaluative measures is advantageous for a number of reasons. First, behavioral items, which focus on whether something occurs, are subject to less bias than items measuring expectations. Second, behavioral items tend to have VOL. 9, SPECIAL ISSUE 2 THE AMERICAN JOURNAL OF MANAGED CARE SP69

6 strong face validity and often are actionable by physicians, medical groups, and health plans. Third, having both behavioral and evaluative items supports the assessment of internal validity by relating evaluations to behaviors. A high correlation between the behavioral and evaluative items suggests that the evaluative items are relatively unbiased. A further advantage of using behavioral and evaluative items for health plans is to enable exploration of the oft-noted difference in physician evaluations between group/staff models. Researchers have noted that physicians in group- or staff-model HMOs rate their health plans higher than do physicians in IPA or network HMOs in terms of quality. 2,4 One explanation is that staff and group HMOs are actually higher in quality. If this is true, both behavioral and evaluative items will indicate high quality. If this is not true, the measures will not be corroborative. Items Should Be Designed to Control for General Anti And Pro Managed Care Bias. Physician training, experience, and practice setting incorporate extensive socialization that can influence physician evaluations of quality of care and health plans. 4 It is reasonable to expect that a physician who sees managed care negatively in general may see many of the managerial tools in a negative light. However, it is reasonable to expect the opposite as well. It is well known that physicians seek practice environments that fit their preferences. If the distribution of these physicians varies across health plans, variations in measures of health plan quality could be caused by variations in beliefs about managed care. This suggests that questions about general managed care beliefs and satisfaction with the work environment (eg, satisfaction with pay) should be included so that these self-selection effects can be controlled for in analyses. Items Should Address What Physicians Can Reasonably Be Expected to Observe and Allow Physicians to Opt Out of Answering. Although managerial tools may be difficult for physicians to observe directly, the consequences of the tool may not be. Each management tool has consequences that physicians should be able to directly observe if the structure or process is implemented. It is reasonable to expect that physicians observe guidelines, disease management programs, authorization processes, and profiling if these structures or processes are in place. Allowing physicians to opt out when they do not feel qualified to answer a question should result in more reliable responses. (The risk is that physicians may choose to opt out to complete the survey faster. Making the cognitive burden of opting-out and that of answering as similar as possible can minimize this risk.) An important advantage of allowing physicians to opt out is that it addresses some of the issues related to physicians contracting with multiple health plans. If contracted physicians do not have a high volume of patients with the health plan about which they are being surveyed, an opt-out option provides an appropriate response. Without the opt-out option, the response may be simply missing, which has a variety of interpretations, only one of which is lack of expertise. The pattern of opting out can be analyzed to see whether it is consistent with an interpretation of physician expertise. If the volume of a physician s patients from the health plan or the length of time the physician has been associated with a health plan is an indicator of a physician s expertise about the health plan, then these variables should be related to a physician opting out of responding. Another example would be that physicians are more likely to opt out of evaluating the overall quality of profiling reports if they did not report receiving these reports. The survey is structured to observe opting out both for all items concerning a health plan and for specific items. The survey that was developed begins with general items about managed care beliefs and then moves to health plan specific items. Physicians who do not feel knowledgeable about a health plan can stop responding after completing the general items. The opt-out options at the item level provide a finer-grained measurement of expertise. In summary, a survey designed with opt-out options lets the pattern of physician expertise, as perceived by physicians, be empirically analyzed rather than merely debated. A physician opting out of responding where appropriate is another indicator of internal validity. Passive Questions Should Be Used Where Multiple Sources of Accountability Might Exist. 1 Health plans manage care in a wide variety of ways, including delegating functions to a physician group or directly managing care. Measuring what is being done using a passive question can measure the occurrence of an event. Questions like How often have [PLAN s] preauthorization procedures delayed the admission of a patient into a hospital? can be worded instead as For your patients in [PLAN], how often have preauthorization procedures delayed the admission of a patient into a hospital? Although the latter wording does not allow determination of the cause of the delays whether they result from a SP70 THE AMERICAN JOURNAL OF MANAGED CARE JUNE 2003

7 Survey Design Principles for Measuring Health Plan Quality health plan action or the action of a medical group that contracts with a health plan it does have the important advantage of determining whether delays are occurring for patients in a given plan. In fact, this is what some purchasers (eg, Medicare) are most interested in. 101 Items Should Be Used That Discriminate Among Health Plans. If no plan is using a managerial tool, or if every plan is, the survey will provide no information about plan differences. Ideally, the survey should include a small number of questions about routine tools that most plans use and a small number of questions about state-of-the-art tools that advanced health plans use. However, most of the questions should focus on the middle range of plans. This strategy provides a quality metric that measures floors and ceilings and increases measurement precision in the realm where the average health plan is operating. Another benefit of this approach is that the survey can evolve with a changing industry. Health plan management activities that have become routine can be moved off the survey, and new stateof-the-art questions can be moved onto the survey. THE PHYSICIAN SURVEY INSTRUMENT The survey instrument we developed based on these principles (available at edu/phps/using%20the%20survey.htm) consists of 48 items; 31 items address managed care beliefs and quality of health plan management strategies and 17 items ask about physician demographics and physician practice characteristics. Questions on health plan quality, utilization structures, and managerial processes are asked in the areas of clinical practice, pharmacy management, and inpatient care. First, behavioral (report) items ask about the physician s experience with specific, individual health plan managerial tools. Second, the physician is asked to evaluate the usefulness and/or accuracy of the tools. Each sequence of questions is followed by a summary evaluation of that area. There are global items to evaluate primary, specialty, pharmacy, and inpatient care as well as physician willingness to recommend the health plan to others. Items on managed care beliefs and on physician characteristics and practice are included so that they can be used in statistical models for controlling bias and in identifying variation within and among health plans. The structure of the questions, particularly the passive voice, is also a strength because it supports extension of the instrument beyond HMOs to other organizational forms such as preferred provider organizations (PPOs) 102,103 and medical groups. 104 It is likely that responses will differ systematically across organizational forms. Although PPOs may not do as much profiling and guideline implementation as HMOs, which may reflect negatively on PPOs, they probably do less preauthorization for specialist referral, which may be seen as positive by physicians. Given the weight that may be attached to the physician survey responses, particularly for accountability and report card purposes, it is important that the observations are accurate. A key assumption of the survey development process was that physicians are accurate respondents about some aspects of health plan organization. Physicians are unlikely to be accurate in observing structures (eg, incentive contracts) and interactions in which they are not directly engaged (eg, health plan communications to consumers about benefits) because these are of low salience or are indirectly observed. Physicians are most likely to be accurate in observing everyday care processes because these are salient. However, this assumption about accuracy may be challenged for 3 reasons. First, physicians in other than staff HMOs may deal with multiple health plans. In many clinics, patients check in with a staff assistant who verifies insurance coverage before the patient sees the physician. Because of this process, a physician may be unable to accurately connect health plan with the patient. Second, dealing with multiple health plans may make it difficult for physicians to discriminate among health plans. (On the other hand, dealing with multiple health plans may provide the physician with a metric on which to judge quality. A physician in a staff HMO who deals only with patients from 1 health plan arguably has little knowledge of quality across health plans because he or she has not observed operations in other health plans.) Third, the physician sample may be incorrect, or may not consist of physicians who reflect the health plan s physician panel. In particular, this may be a problem in markets that are experiencing significant turmoil. Finally, low patient volume may make it difficult for physicians to observe patients in specific health plans. If these problems are severe, then physicians may not be able to provide useful information about health plans. The presence of opt-out options will enable physicians to respond only to the questions about which they feel knowledgeable. VOL. 9, SPECIAL ISSUE 2 THE AMERICAN JOURNAL OF MANAGED CARE SP71

8 DISCUSSION In this paper, we described the development and refinement of an instrument that can be used to survey physicians about the usefulness and value of quality and utilization management policies and procedures associated with specific health plans in which they participate. A conceptual framework and a set of design principles for surveying physicians guided the survey development. In the resulting survey, physicians are asked about the presence or absence of specific health plan management strategies and/or support systems, their use (if present), and their effectiveness. They also are asked to provide an overall evaluation of specific health plans. Our survey differs from other physician surveys in several important ways. It is significantly different from surveys that focus on physician quality of work life (eg, satisfaction with managed care, autonomy) and surveys that ask physicians about their general beliefs on the impact of managed care and how health plans (generically) have influenced their practice of medicine. 22,105 The difficulty with these latter surveys is that they may measure predominately a global belief about managed care, rather than actual effects. Although physicians report that managed care has decreased the amount of time they spend with patients, 22,105 evidence from national surveys about the amount of time spent with patients does not reflect this belief. 106 The emphasis in our survey on behavioral measures associated with specific health plans should provide more reliable information on what physicians see that health plans actually do, rather than what physicians believe health plans do. When we initiated our work, we speculated that there would be 3 possible uses for physician assessments of health plans. We expected that physician assessments might (1) help purchasers in choosing which health plans to offer employees or public program participants; (2) be incorporated in consumer report cards or other frameworks that assisted consumers in selecting from among the health plan options made available by Medicaid, Medicare, or their employers; and (3) be used by health plans to benchmark performance from a physician perspective, and possibly to target areas for quality-improvement initiatives. From November 2000 to early 2001, the Physician Evaluation of Health Plan instrument was used to survey physicians, representing 23 health plans in 5 geographic regions. The surveyed physicians represented 3 types of plans: commercial, Medicare, and Medicaid. The survey design provides 2 major ways to evaluate the survey instrument with this national sample of physicians. First, if physician evaluations of health plans are based on global views about managed care more than on observations of a particular plan, we expect that physicians will be able to answer the questions about their global beliefs but will have difficulty in answering the questions about behavioral (report) measures for specific care processes for their patients in a plan. Second, we presented expectations about internal relationships 1 between measures that would be present if physicians are reasonable respondents regarding health plans. The presence of these relationships would suggest that the survey instrument is internally consistent, supporting its accuracy. If instrument analyses support its internal consistency and validity, the survey will be useful to consumers, health plans, provider organizations, and purchasers. Alternatively, if the survey instrument fails these tests, then its use to measure health plan quality is not warranted. Acknowledgments We appreciate the assistance of the other members of Physicians Evaluating Health Plans research team: Andy Bindman, MD; Steven Borowsky, MD, MPH; Bruce Center, PhD; Paula Henning, MA; Margaret King-Davis, MS; and Mary Jo O Brien, MS. We also appreciate the comments of the anonymous reviewers. REFERENCES 1. Wholey DR, Christianson JB, Finch M, et al. Evaluating health plan quality 1: a conceptual model. Am J Manag Care. 2003; 9(suppl):SP53-SP Borowsky SJ, Davis MK, Goertz C, et al. Are all health plans created equal? The physician s view. JAMA. 1997;278: Borowsky SJ, Goertz C, Lurie N. Can physicians diagnose strengths and weaknesses in health plans? Ann Intern Med. 1996;125: Williams TV, Zaslavsky AM, Cleary PD. Physician experiences with, and ratings of, managed care organizations in Massachusetts. Med Care. 1999;37: Blumenthal D, Epstein AM. Quality of health care, part 6: the role of physicians in the future of quality management. N Engl J Med. 1996;335: Campbell SM, Roland MO, Buetow SA. Defining quality of care. Soc Sci Med. 2000;51: Landon BE, Wilson IB, Cleary PD. A conceptual model of the effects of health care organizations on the quality of medical care. JAMA. 1998;279: Carman KL, Short PF, Farley DO, et al. Epilogue: early lessons from CAHPS Demonstrations and Evaluations. Consumer Assessment of Health Plans Study. JAMA. 1999;37:MS SP72 THE AMERICAN JOURNAL OF MANAGED CARE JUNE 2003

9 Survey Design Principles for Measuring Health Plan Quality 9. Crofton C, Lubalin JS, Darby C. Consumer Assessment of Health Plans Study (CAHPS) [Foreword]. Med Care. 1999;37:MS Berwick DM, Baker MW, Kramer E. The state of quality management in HMOs. HMO Pract. 1992;6: Landon BE, Epstein AM. Quality management practices in Medicaid managed care: a national survey of Medicaid and commercial health plans participating in the Medicaid program. JAMA. 1999;282: Kerr EA, Mittman BS, Hays RD, et al. Quality assurance in capitated physician groups. Where is the emphasis? JAMA. 1996;276: Bell RA, Kravitz RL, Siefkin AD, et al. Physicians attitudes toward managed care: assessment and potential effects on practice behaviors. Am J Manag Care. 1997;3: Friedenberg RM. The primary care physician s view of managed care in Radiology. 1999;210: Ellsbury KE, Schneeweiss R. Physician knowledge and attitudes about health insurance after the introduction of capitated health care plans. J Fam Pract. 1988;26: Roulidis ZC, Schulman KA. Physician communication in managed care organizations: opinions of primary care physicians. J Fam Pract. 1994;39: Simon SR, Pan RJ, Sullivan AM, et al. Views of managed care a survey of students, residents, faculty, and deans at medical schools in the United States. N Engl J Med. 1999;340: Stiffman MN, LeFevre ML. Are resident physicians serving as primary care providers for managed care patients? Fam Med. 1997;29: Nadler ES, Sims S, Tyrance PH Jr, et al. Does a year make a difference? Changes in physician satisfaction and perception in an increasingly capitated environment. Am J Med. 1999;107: Remler DK, Donelan K, Blendon RJ, et al. What do managed care plans do to affect care? Results from a survey of physicians. Inquiry. 1997;34: Linzer M, Konrad TR, Douglas J, et al. Managed care, time pressure, and physician job satisfaction: results from the physician worklife study. J Gen Intern Med. 2000;15: Kaiser Family Foundation/Harvard School of Public Health. Survey of Physicians And Nurses. Menlo Park, Calif: Kaiser Family Foundation/Harvard School of Public Health; Collins KS, Schoen C, Sandman DR. The Commonwealth Fund Survey of Physician Experiences with Managed Care. Available at: Accessed March 23, Galanter M. The impact of managed care on addiction treatment: evaluating physicians views and the value of health plan benefits. J Addict Dis. 1999;18: Scheckler WE, Schulz R, Moberg P. Physician satisfaction with the development of HMOs in Dane County: Wis Med J. 1994;93: Warren MG, Weitz R, Kulis S. The impact of managed care on physicians. Health Care Manage Rev. 1999;24: Johnson RE, Freeborn DK, Azevedo DJ. Physicians attitudes toward an expanded role for pharmacists in an HMO. Group Health J. 1984;5: Allison JJ, Kiefe CI, Cook EF, et al. The association of physician attitudes about uncertainty and risk taking with resource use in a Medicare HMO. Med Decis Making. 1998;18: Wynia MK, Picken HA, Selker HP. Physicians views on capitated payment for medical care: does familiarity foster acceptance? Am J Manag Care. 1997;3: Halm EA, Causino N, Blumenthal D. Is gatekeeping better than traditional care? A survey of physicians attitudes JAMA. 1997;278: Bailey JE, Bush AJ, Bertram PD, et al. Physicians experiences with Medicaid managed care: the Tennessee American College of physicians survey on TennCare. Tenn Med. 1998;91: Foulke GE, Bell RA, Siefkin AD, et al. Attitudes and behavioral intentions regarding managed care: a comparison of academic and community physicians. Am J Manag Care. 1998;4: Taslimi MM, Miller PE, Hicks WH. Hamilton county physicians experience with managed care. Tenn Med. 2000;93: Dreher T, Whetten-Goldstein K. A survey of beliefs about managed care. North Carolina Med J. 1999;60: Skootsky SA, Slavin S, Wilkes MS. Attitudes toward managed care and cost containment among primary care trainees at 3 training sites. Am J Manag Care. 1999;5: Hojat M, Gonnella JS, Erdmann JB, et al. Physicians perceptions of the changing health care system: comparisons by gender and specialties. J Community Health. 2000;25: Levine RA, Lieberson A. Physicians perceptions of managed care. Am J Manag Care. 1998;4: Eastman JK, Eastman KL, Tolson MA. The ethics of managed care. An initial look at physicians perspectives. Mark Health Serv. 1997;17: Ellsbury K, Montano D, Krafft K. A survey of the attitudes of physician specialists toward capitation-based health plans with primary care gatekeepers. QRB Qual Rev Bull. 1990;16: Shye D, Brown JB. Primary care HMO clinicians opinions about clinical practice guidelines. HMO Pract. 1995;9: Salem-Schatz SR, Gottlieb LK, Karp MA, et al. Attitudes about clinical practice guidelines in a mixed model HMO: the influence of physician and organizational characteristics. HMO Pract. 1997; 11: Inouye J, Kristopatis R, Stone E, et al. Physicians changing attitudes toward guidelines. J Gen Intern Med. 1998;13: Silverstein G, Kirkman-Liff B. Physician participation in Medicaid managed care. Soc Sci Med. 1995;41: Meyer CM, Ladenson PW, Scharfstein JA, et al. Evaluation of common problems in primary care: effects of physician, practice, and financial characteristics. Am J Manag Care. 2000;6: Bucci RV. Has managed care changed the skills required for physicians to practice medicine? Manag Care Q. 1999;7: Hasty M, Schrager J, Wrenn K. Physicians perceptions about managed care restrictions on antibiotic prescribing. J Gen Intern Med. 1999;14: DeMaria AN, Lee TH, Leon DF, et al. Effect of managed care on cardiovascular specialists: involvement, attitudes and practice adaptations. J Am Coll Cardiol. 1996;28: Grumbach K, Osmond D, Vranizan K, et al. Primary care physicians experience of financial incentives in managed-care systems. N Engl J Med. 1998;339: Sulmasy DP, Bloche MG, Mitchell JM, et al. Physicians ethical beliefs about cost-control arrangements. Arch Intern Med. 2000; 160: Kao AC, Green DC, Zaslavsky AM, et al. The relationship between method of physician payment and patient trust. JAMA. 1998;280: Leider PJ, Solberg R, Nesbitt T. Family physician perception of economic incentives for the provision of office procedures. Fam Med. 1997;29: Tyrance PH Jr, Sims S, Ma luf N, et al. Capitation and its effects on physician satisfaction. Cost Qual Q J. 1999;5: Lohr KN, Brook RH, Kamberg CJ, et al. Use of medical care in the RAND Health Insurance Experiment. Diagnosis- and servicespecific analyses in a randomized controlled trial, chapter 5: effect of cost-sharing on use of medically effective and less effective care. Med Care. 1986;24:S Hellinger FJ. The impact of financial incentives on physician behavior in managed care plans: a review of the evidence. Med Care Res Rev. 1996;53: VOL. 9, SPECIAL ISSUE 2 THE AMERICAN JOURNAL OF MANAGED CARE SP73

10 55. Giuffrida A, Gosden T, Forland F, et al. Target payments in primary care: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2000: Gosden T, Forland F, Kristiansen IS, et al. Capitation, salary, fee-for-service and mixed systems of payment: effects on the behaviour of primary care physicians. Cochrane Database Syst Rev. 2000: Freeborn DK. Physician satisfaction in a prepaid group practice HMO. Group Health J. 1985;6: Baker LC, Cantor JC. Physician satisfaction under managed care. Health Aff (Millwood). 1993;12(suppl): Ahern M. Survey of Florida physicians. Characteristics and satisfaction. J Fla Med Assoc. 1993;80: Deckard GJ. A comparison of concurrent general and HMOaffiliated medical practice experience. J Health Soc Policy. 1995;7: Weinberg DJ, Engasser PG. Dermatologists in Kaiser Permanente northern California. Satisfaction, perceived constraints, and policy options. Arch Dermatol. 1996;132: Collins KS, Schoen CA, Khoransanizadeh F. Practice satisfaction and experiences of women physicians in an era of managed care. J Am Med Womens Assoc. 1997;52: Schulz R, Scheckler WE, Moberg DP, et al. Changing nature of physician satisfaction with health maintenance organization and fee-for-service practices. J Fam Pract. 1997;45: Kerr EA, Hays RD, Mittman BS, et al. Primary care physicians satisfaction with quality of care in California capitated medical groups. JAMA. 1997;278: Conway T, Hu TC, Daugherty SR. Physicians perceptions of managed care: a structural equation model assessment of key dimensions. Med Care. 1998;36: Warren MG, Weitz R, Kulis S. Physician satisfaction in a changing health care environment: the impact of challenges to professional autonomy, authority, and dominance. J Health Soc Behav. 1998;39: Bates AS, Harris LE, Tierney WM, et al. Dimensions and correlates of physician work satisfaction in a midwestern city. Med Care. 1998;36: Hadley J, Mitchell JM, Sulmasy DP, et al. Perceived financial incentives, HMO market penetration, and physicians practice styles and satisfaction. Health Serv Res. 1999;34: Murray A, Montgomery JE, Chang H, et al. Doctor discontent a comparison of physician satisfaction in different delivery system settings, 1986 and J Gen Intern Med. 2001;16: Gazewood JD, Longo DR, Madsen R. Physician satisfaction with Medicaid managed care: the Missouri experience. J Fam Pract. 2000;49: Kerr EA, Mittman BS, Hays RD, et al. Associations between primary care physician satisfaction and self-reported aspects of utilization management. Health Serv Res. 2000;35: Konrad TR, Williams ES, Linzer M, et al. Measuring physician job satisfaction in a changing workplace and a challenging environment. SGIM Career Satisfaction Study Group. Society of General Internal Medicine. Med Care. 1999;37: Haas JS, Cook EF, Puopolo AL, et al. Is the professional satisfaction of general internists associated with patient satisfaction? J Gen Intern Med. 2000;15: Kerr EA, Mittman BS, Hays RD, et al. Managed care and capitation in California: how do physicians at financial risk control their own utilization? Ann Intern Med. 1995;123: Schectman JM, Elinsky EG, Kanwal NK, et al. HMO physician attitudes toward drug cost containment strategies. HMO Pract. 1995;9: Dixon MK, Kirschner PB, Edelberg HK, Aykanian, JZ, Wei JY. Physician perceptions of HMO care for older persons. Health Maintenance Organization. J Am Geriatr Soc. 2000;48(6): Justice AC, Rabeneck L, Hays RD, et al. Sensitivity, specificity, reliability, and clinical validity of provider-reported symptoms: a comparison with self-reported symptoms. Outcomes Committee of the AIDS Clinical Trials Group. J Acquir Immune Defic Syndr. 1999;21: Stange KC, Zyzanski SJ, Smith TF, et al. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care. 1998;36: Groves RM, Beimer PP, Lyberg LE, et al. Telephone Survey Methodology. New York, NY: Wiley; Aquilino WS, LoScuito LA. Effects of interview mode on selfreported drug use. Public Opin Q. 1990;54: Biemer PP. Measurement Errors in Surveys. New York, NY: Wiley; Schwarz N, Sudman S, Zentrum für Umfragen Methoden und Analysen (Germany), et al. Context Effects in Social and Psychological Research. New York, NY: Springer-Verlag; Aquilino WS. Interview mode effects in surveys of drug and alcohol use: a field experiment. Public Opin Q. 1994;58: Salant P, Dillman DA. How to Conduct Your Own Survey. New York, NY: Wiley; Dillman DA, Sangster RL, Tarnai J, et al. Understanding differences in people s answers to telephone and mail surveys. In: Braverman MT, Slater JK, eds. Advances in Survey Research. Vol 70. San Francisco, Calif: Jossey-Bass, Inc; 1996: Sudman S, Bradburn NM, Schwarz N. Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. 1st ed. San Francisco, Calif: Jossey-Bass Inc; Tourangeau R, Smith TW. Asking sensitive questions: the impact of data collection mode, question format, and question context. Public Opin Q. 1996;60: de Leeuw E, Collins M. Data collection methods and survey quality: an overview. In: Lyberg L, Biemer P, Collins M, et al, eds. Survey Measurement and Process Quality. New York, NY: Wiley; 1997: Krosnick JA. Thinking about answers: the application of cognitive processes to survey methodology. Public Opin Q. 1997;61: Krosnick JA. Answering questions: methodology for determining cognitive and communicative processes in survey research. Public Opin Q. 1997;61: Rockwood TH, Sangster RL, Dillman DA. The effect of response categories on questionnaire answers: context and mode effects. Sociol Methods Res. 1997;26: Fowler FJ Jr, Roman AM, Di ZX. Mode effects in a survey of Medicare prostate surgery patients. Public Opin Q. 1998;62: Dillman DA. Mail and Electronic Surveys: The Tailored Design Method. 2nd ed. New York, NY: Wiley; Tourangeau R, Rips LJ, Rasinski K. The Psychology of Survey Response. Cambridge, UK: Cambridge University Press; Dillman DA, Tarnai J. Administrative issues in mixed mode surveys. In: Groves RM, Beimer PP, Lyberg LE, Massey JT, Nichols WLI, Waksberg J, eds. Telephone Survey Methodology. New York, NY: Wiley; 1988: Dillman DA. The design and administration of mail surveys. Annu Rev Sociol. 1991;17: Solomon DH, Hashimoto H, Daltroy L, et al. Techniques to improve physicians use of diagnostic tests: a new conceptual framework. JAMA. 1998;280: Trochim WMK. The Research Methods Knowledge Base. Cincinnati, Ohio: Atomic Dog Publishing; Homer CJ, Marino B, Cleary PD, et al. Quality of care at a children s hospital: the parent s perspective. Arch Pediatr Adolesc Med. 1999;153: SP74 THE AMERICAN JOURNAL OF MANAGED CARE JUNE 2003