August 2008 SUPPORT Summary of a systematic review Does pay-for-performance improve the quality of health care? Explicit financial incentives have been proposed as a strategy to change physician and healthcare system behaviour. Linking payments at different levels in the health system to performance on quality measures is currently being used by a number of organisations in the USA and other high-income countries. The incentives have been implemented at different levels, including the payment system; the provider group; and the individual physician. Key messages Compared to no incentives, the use of explicit financial incentives at different levels in the health system might: Decrease hospital admission rates or death of nursing home patients; Produce adverse selection of users; Improve access to community mental health care; Improve some processes of care such as influenza immunisation rates and diabetes care. However, the evidence is of low or very low quality and in most of the studies there was a positive effect on some outcomes and no effect on others. Little or no evidence is available regarding the specific design elements of effective pay-for-performance schemes, including: the optimum magnitude, frequency, and duration of financial incentives; and the performance measures and standards to be used. Little or no evidence is available regarding the cost-effectiveness of pay-forperformance schemes. Factors that need to be considered in assessing whether the intervention effects are likely to be transferable to other settings include the availability of: Resources to finance the incentives beyond restructuring existing payment systems; Routine data on quality of care. Who is this summary for? People making decisions concerning the use of financial incentives to improve the quality of care. This summary includes: Key findings from research based on a systematic review Considerations about the relevance of this research for low and middleincome countries Not included: Recommendations Additional evidence not included in the systematic review Detailed descriptions of interventions or their implementation This summary is based on the following systematic review: Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does pay-forperformance improve the quality of health care. Ann Intern Med 2006; 145:265-72. What is a systematic review? A summary of studies addressing a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise the relevant research, and to collect and analyse data from the included studies. SUPPORT an international collaboration funded by the EU 6th Framework Programme to support the use of policy relevant reviews and trials to inform decisions about maternal and child health in low and middle-income countries. www.support-collaboration.org Glossary of terms used in this report: www.supportcollaboration.org/summaries/explanat ions.htm Background references on this topic: See back page
Background Linking payments at different levels in the health system to performance on quality measures may be a strategy for achieving high-quality health care. This summary is based on a systematic review published in 2006 by Petersen and colleagues. The summary focuses on the effects of explicit financial incentives on different measures of quality of care at different levels of the health system. How this summary was prepared After searching widely for systematic reviews that can help inform decisions about health systems, we have selected ones that provide information that is relevant to low and middle-income countries. The methods used to assess the quality of the review and to make judgements about its relevance are described here: www.supportcollaboration.org/methods Knowing what s not known is important A good quality review might not find any studies from low and middleincome countries or might not find any well-designed studies. Although that is disappointing, it is important to know what is not known as well as what is known. About the systematic review underlying this summary Review objective: To assess the effects on measures of health care quality of explicit financial incentives for improved performance Interventions Participants Settings Outcomes What the review authors searched for Empirical studies of the relationship between explicit financial incentives designed to improve health care quality and a quantitative measures of health care quality The studies were categorised according to the level of the financial incentive: individual physician, provider group, or healthcare payment system. Any setting where explicit financial incentives have been used to improve quality of care Quality of care domains: access to care, structure of care, process of care, outcomes of care, and patient experience of care What the review authors found 17 studies: 9 randomised trials, 4 controlled before-after studies, and 4 cross-sectional surveys Considerable differences in numbers of physicians and organisations included in each study because of the different scope of the intervention at each level. Settings were not described in detail. At least 4 studies from the USA. Most of the other 13 studies also appear to be from US settings Most studies reported multiple effect measures that were grouped into one or more quality domains. Date of most recent search: November 2005 Limitations: This is a systematic review with moderate limitations. Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does pay-for-performance improve the quality of health care. Ann Intern Med 2006; 145:265-72. Background 2
Summary of findings The review included 17 studies evaluating explicit financial incentives on quality of care measures. Four of them were conducted in North America while the other 13 studies appear to have been carried out in high-income settings (mostly the USA). 1) Explicit financial incentives at the payment-system level compared with no incentives There was no explicit definition of payment-system level, but it seems to be a nonprovider organisation commissioning specific services on behalf of its clients from a provider organisation. The two studies identified employed different designs (1 randomised trial and 1 controlled before-after study), and focused on very different populations (nursing home patients versus Medicaid Office of Substance Abuse clients) in the US health system. The overall effect of financial incentives at the payment-system level on access to care is inconclusive, showing improved access in one study and adverse selection 1 in the other. There is low quality evidence of improved outcomes for patients in nursing homes that received financial incentives. About quality of evidence High: Further research is very unlikely to change our confidence in the estimate of effect. Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. : We are very uncertain about the estimate. For more information, see last page. Explicit financial incentives at the payment-system level Patients or population: Nursing home patients with debilitating acute and chronic conditions and Medicaid Office of Substance Abuse clients Settings: US healthcare system Intervention: Explicit financial incentives Comparison: No incentives Outcomes Impact Number of participants (studies) Access to care In one study, intervention nursing homes admitted more ill patients than control homes, but in the other study there was a significant decrease in the likelihood that a patient in the intervention group was a most severe user. Quality of the evidence - (2 studies) Outcome of care Patients in intervention nursing homes had less likelihood of hospital admission or death (p<0.001). 36 facilities (1 study) Low Cost-effectiveness The authors model estimated an average cost savings of US $3,000 per nursing home stay. 36 facilities (1 study) p: p-value GRADE: GRADE Working Group grades of evidence (see above and last page) 1 Adverse selection refers to a selection process where high-risk people are excluded from care in order to obtain better performance. Summary of findings 3
2) Explicit financial incentives targeted at provider groups compared with no financial incentives Nine studies (5 randomised trials, 2 controlled before-after studies and 2 crosssectional surveys) were identified, mostly from the USA. Eight of these evaluated the effects of financial incentives on process measures of quality, mainly related to preventive care. The other study aimed to improve access to services offered by community mental health centers. There is very low quality evidence that financial incentives at the provider level improve access to care, compared with no incentives. There is very low quality evidence that financial incentives at the provider level improve process of care measures for preventive care, compared with no incentives. Explicit financial incentives at the provider group level Patients or population: Population served by a number of providers (primary care or physician organisations) and pharmacies Settings: US healthcare system Intervention: Explicit financial incentives targeted at provider groups Comparison: No incentives Outcomes Impact Number of participants (studies) Quality of the evidence Access to care (community mental health care) The average time spent in community treatment per client increased (from 31 to 39 minutes) compared with a decrease in weekly time per client in office-based case management (from 33 to 23 minutes). 185 clients (1 study) Process of care Four studies showed positive effects on some outcomes (receiving a smoking cessation intervention, rates of tobacco use status identification, cervial cancer screening) but not in others (support of smoking cessation interventions, provision of quitting advice, tobacco quitting rates, mammography screening rates and HbA1c testing). 2499 providers + 200 pharmacies (8 studies) p: p-value GRADE: GRADE Working Group grades of evidence (see above and last page) Summary of findings 4
3) Explicit financial incentives at the individual physician level compared with no intervention Six studies (3 randomised trials, 1 controlled before-after study and 2 cross-sectional surveys) evaluated financial incentives at the individual physician level. In the four experimental studies the incentive was a bonus or an enhanced fee-for-service payment given at different intervals. The magnitude of the incentive was variable and depended on the type of behaviour targeted (range: US $50 to US $7500). Financial incentives targeted at physicians might improve patients experience of primary care services in a US setting, but the evidence is of very low quality and based on a cross-sectional survey. There is very low quality evidence that financial incentives targeted at physicians might improve documentation of specific processes of care (e.g. immunisation status). Explicit financial incentives at the physician level Patients or population: Population attended by a number of providers (primare care or physician organisations) and pharmacies. Settings: US healthcare system Intervention: Explicit financial incentives Comparison: No incentives Outcomes Impact Number of studies Process of care (mammography, pediatric immunisations, diabetes care, STD screening) The impacts ranged from no effect (mammography referral rates) to improvements in documented up-to-date immunisation status (absolute change of 5.9% to 7.4% compared with controls) and in annual adherence to screening for Chlamydia trachomatis in women age 20-25 years (observational data). The other two studies showed positive effects on some outcomes (documented up-to-date immunisation status, some diabetes care indicators) and no change in others (performing Hb A1c test). 5 studies Quality of the evidence Patient experience Improvements in two of the four aspects of primary care evaluated by patients: access to care (p<0.01) and dimensions of comprehensiveness of care (p<0.05). 1 study p: p-value GRADE: GRADE Working Group grades of evidence (see above and last page) Summary of findings 5
Relevance of the review for low and middle-income countries Findings APPLICABILITY Most of the studies were carried out in the US health system environment. Interpretation* This restriction, and the inconsistent pattern of findings, make difficult the translation of this evidence to low and middle-income countries. Additional factors that should be considered to assess the potential effects of pay for performance in other settings include: the availability and reliability of routine data on quality of care; the availability of resources to finance the incentives beyond restructuring existing payment systems; existing remuneration systems for individual healthcare providers and groups of providers; the impact of the intervention on other types of healthcare providers (e.g. nurses). EQUITY The included studies provided little data regarding differential effects of the interventions for disadvantaged populations. However, there was evidence of adverse selection of most severe users in one study. Depending on which quality indicators are rewarded by the financial incentives, there may be differential effects on disadvantaged populations. Because of uncertainty about the differential effects of financial incentives on high versus low-performing providers, it is possible that financial incentives could have differential effects on disadvantaged populations served by low-performers. COST-EFFECTIVENESS Only one study estimated the costeffectiveness of the intervention, estimating a savings of US $3,000 per nursing home stay. The assumptions used in this study were not explicit and it is not clear how the savings and costs would be distributed across the organisation paying the incentives and the organisations receiving the incentives. Because of uncertainty about the magnitude, frequency and duration of the financial incentives for improving quality, the additional resources needed to scale-up pay for performance at different levels are not clear and must be estimated for a specific pay-for-performance scheme in a specific setting. MONITORING & EVALUATION The evidence summarised in this review is inconclusive and suggests that pay for performance can have unintended effects. Monitoring is an inherent component of pay for performance. However, it is also important to monitor for unintended adverse effects, including adverse selection of patients, adverse effects on processes that are not rewarded with financial incentives, and effects on documentation as opposed to actual improvements in practice. Because there is substantial uncertainty about the effects of pay for performance and potential adverse effects, these schemes should be carefully designed and rigorously evaluated before being implemented on a large scale in low and middle income countries. *Judgements made by the authors of this summary, not necessarily those of the review authors, based on the findings of the review and consultation with researchers and policymakers in low and middle-income countries. For additional details about how these judgements were made see: http://www.support-collaboration.org/summaries/methods.htm Relevance of the review for low and middle-income countries 6
Additional information Related literature This report provides an overview of the evidence for both supply and demand side results-based financing (pay for performance) in the health sector with the primary focus on low and middle-income countries: Oxman AD, Fretheim A. An overview of research on the effects of results-based financing. Report Nr 16-2008. Oslo: Nasjonalt kunnskapssenter for helsetjenesten, 2008. http://www.kunnskapssenteret.no/publikasjoner/3219.cms?threepage=1 This paper provides a discussion of both supply and demand side pay for performance and case studies from low and middle-income countries: Eichler R. Can Pay for Performance Increase Utilization by the Poor and Improve the Quality of Health Services? Discussion paper for the first meeting of the Working Group on Performance-Based Incentives. Washington DC: Center for Global Development, 2006; 5. http://www.cgdev.org/doc/ghprn/pbi%20background%20paper.pdf A general analysis about payment for performance in health care with useful reference is presented in: Mannion R, Davies HTO. Payment for performance in health care. BMJ 2008;336:306-308. This study reports findings of a cross-sectional survey for the first-year of the national pay-forperformance program in family practices in the UK: Doran T, Fullwood C, Gravelle H, Reeves D, Konropantelis E, Hiroeh U, Roland M. Pay-for-Performance Programs in Family Practices in the United Kingdom. N Engl J Med 2006;355:375-84. This summary was prepared by Tomás Pantoja, Escuela de Medicina, Pontificia Universidad Católica de Chile. Conflict of interest None declared. For details, see: http://www.support-collaboration.org/summaries/coi.htm Acknowledgements This summary has been peer reviewed by: Laura Petersen, USA; Paul Smithson, Tanzania; Atle Fretheim, Norway; Tracey Perez Koehlmoos, Bangladesh; Maylene Beltran, Philippines. This summary should be cited as Pantoja T. Does pay-for-performance improve the quality of health care? A SUPPORT Summary of a systematic review. August 2008. http://www.support-collaboration.org/summaries.htm About quality of evidence The quality of the evidence is a judgement about the extent to which we can be confident that the estimates of effect are correct. These judgements are made using the GRADE system, and are provided for each outcome. The judgements are based on the type of study design (randomised trials versus observational studies), the risk of bias, the consistency of the results across studies, and the precision of the overall estimate across studies. For each outcome, the quality of the evidence is rated as high, moderate, low or very low using the definitions on page 3. For more information about GRADE: www.supportcollaboration.org/summaries/grade.pdf SUPPORT collaborators: The Alliance for Health Policy and Systems Research (HPSR) is an international collaboration aiming to promote the generation and use of health policy and systems research as a means to improve the health systems of developing countries. www.who.int/alliance-hpsr The Cochrane Effective Practice and Organisation of Care Group (EPOC) is a Collaborative Review Group of the Cochrane Collaboration: an international organisation that aims to help people make well informed decisions about health care by preparing, maintaining and ensuring the accessibility of systematic reviews of the effects of health care interventions. www.epoc.cochrane.org The Evidence-Informed Policy Netowrk (EVIPNet) is is an initiative to promote the use of health research in policymaking. Focusing on low and middleincome countries, EVIPNet promotes partnerships at the country level between policy-makers, researchers and civil society in order to facilitate both policy development and policy implementation through the use of the best scientific evidence available. www.who.int/rpc/evipnet/en/ For more information, see: www.support-collaboration.org To receive e-mail notices of new SUPPORT summaries, go to: www.supportcollaboration.org/summaries/newsl etter/ To provide feedback on this summary, go to: http://www.supportcollaboration.org/feedback/ Additional information 7