Towards Performance Indicators for the Health Care Sector

Transcription

1 21 th Bled econference ecollaboration: Overcoming Boundaries Through Multi-Channel Interaction June 15-18, 2008; Bled, Slovenia Towards Performance Indicators for the Health Care Sector Jan Luijsterburg 1, Marcel van Ham 1, Evelien Brouwers 1, Martin Smits 1,2 (1) Tranzo Research Institute for Transformation of Health Care (2) School of Economics and Business Administration, Tilburg University, PO Box 90153, 5000LE Tilburg, Netherlands Abstract The health care sector is a huge industry in many Western countries going through fundamental changes, with increasing needs for new monitoring systems and performance indicators. The aim of this paper is to identify factors that influence the success of external reporting systems, which will ultimately affect the transparency and performance of the sector. We review theory on performance indicators, national care systems, and inter-organizational reporting systems, resulting in formulating several hypotheses. We use a data set and 12 interviews in one case study (the mental health care sector in the Netherlands) to evaluate the hypotheses. Our findings show that the new system is more successful for integrated care organizations than specialized care organizations, and more successful if care organizations have better internal information systems. Keywords: performance measurement, inter-organizational systems, health care, quality of care, external reporting system, quality indicators. 1 Introduction The health care sector is a huge industry in many Western countries. For instance, health care expenditures in the US economy were nearly two trillion dollars per year in 2004 (Smith et al., 2006; CMMS, 2007). The health care sector in many countries represents more then 10% of the gross national product, is knowledge and information intensive and employs over 10% of the total national workforce (OECD, 2004). Performance of health care organizations is an important issue in many countries (OECD, 2004). A number of factors have converged to establish a national agenda to increase transparency and to monitor and improve the quality of health care. Rapid changes in the organization and financing of care have put unprecedented pressure on health care delivery organizations to reduce utilization and costs, leading to a need to ensure that quality is not adversely affected (Hermann et al, 2000).

2 Monitoring systems for health care delivery and health care status on the national level have been developed by many governments, government agencies, insurance companies, and other health care related agencies (Arah et al, 2006; Westert and Verkleij, 2006). These national or regional monitoring systems are expected to provide adequate indicators for different aspects (such as effectiveness, quality, and efficiency) of the various sections of the health care system to one or more supervising agencies. The success of a monitoring system is determined by various factors, for instance, (i) the output of the system must fit the requirements of the supervising agencies, and (ii) the input requirements of the system must fit the capabilities of the health care organizations involved, meaning that these organizations must be able to provide the data input for the monitoring systems. Because of the many different organizations and supervising agencies involved, many different national and regional monitoring systems have emerged over the past decades. To fulfill all reporting requirements, health care organizations have developed many different external reporting systems. This paper focuses on the development of a national monitoring system for mental health care in the Netherlands in , as part of the national reporting system on health care. Eight parties in the mental health care sector have developed one new monitoring system for mental health, replacing eight existing national reporting systems, meaning that mental health care organizations can reduce the number of external reporting systems and ultimately reduce administrative burden. The aim of this research is to identify factors that influence the success of external reporting of health care organizations. We define success of external reporting as the degree to which an organization is able to answer a set of standardized questions focusing on several aspects of their organizational performance, in particular focusing on the quality of care. We further focus on how the instrument (the set of questions) to assess the performance of health care organizations can be improved. Improving the success of external reporting will in the end lead to more transparency of the health care sector. We first summarize theory on health care performance measurement and measurement systems (section 2), then we describe our research method and give an overview of the mental health care sector (section 3), provide our findings regarding the implementation of a national performance measurement system in the Netherlands (section 4), ending with discussion and conclusions. 2 Theory on health care performance measurement To identify factors that influence the success of health care performance measurement we review literature on performance indicators (2.1), identifying the boundaries of the health care system to be assessed (2.2), and the design of the inter-organizational reporting system (2.3). We conclude with the formulation of hypotheses (2.4). 2.1 Health care performance indicators Health care performance assessment includes a large variety of indicators. Hermann et al (2000) evaluated the American (USA) National Inventory of Mental Health Quality Measures and identified 86 measures in the following categories: (1) treatment appropriateness (65% of all indicators); (2) treatment

3 continuity; (3) accessibility of care; (4) coordination of care; (5) detection; and (6) prevention. Few measures were identified for reliability and validity of care. Many authors discuss the validity of performance indicators and argue the relevance of indicators for predicting main outcome measures such as quality of life (Stiles et al, 2002; Schmidt et al, 2005) and time to heal, to recover, and patient satisfaction (Nieuwenhuijsen at al, 2005). Careful selection and validation of indicators is needed to realize successful reporting systems. Another issue is the reliability of indicators. Adherence to guidelines (or compliance ) is known to be a key factor determining reliability. Rebergen et al (2006) showed that guideline adherence by physicians lags behind guideline acceptance, indicating that even if guidelines for data entry and information delivery are accepted by a group of professionals- the quality of data entered into a system lags behind. Similar findings were reported for compliance and the adoption of rules for external accounting by institutions (Carpenter and Feroz, 2001). Careful implementations of reporting rules and information systems are needed to ensure compliance of actors and reliability of indicators. A third issue influencing the success of performance measurement is the relevance of the performance indicators (Valenstein et al., 2004). Indicators are relevant if they fit the information needs of the managers and health professionals involved. Such indicators can be developed by using a Balanced Scorecard, i.e. a set of indicators for different aspects of the system (including indicators for finance, customer focus, process logistics, and innovative responsiveness). To avoid sub-optimization of the performance of inter-organizational processes, a limited set of performance indicators should be shared among managers and health professionals of different organizations in a network (Kleijnen and Smits, 2003). 2.2 Performance of National Health Care Systems Some studies address the performance and comparison of national health care systems. Arah et al (2006) distinguish between health care performance and health performance of national health care systems. Health care performance refers to the maintenance of an efficient and equitable system of health care without emphasizing an assessment of other determinants of health (like the physical environment, the social environment, and lifestyle). Health performance is a much broader conceptual approach to measuring performance by explicitly using non-health care determinants, health care, and contextual information to give a clearer picture of population health. The main policy goals for health performance may be efficiency and equity, but a much wider view of the determinants of health and their costs are adopted. Given that a health performance framework is largely concerned with all the interrelationships among health, health care, and non-health care factors, health performance includes health care performance (Arah et al, 2006). National health care systems typically include three sub-systems: somatic medical care, mental medical care, and public health (health protection, disease prevention, and health promotion) (Westert and Verkleij, 2005). Assessing the performance of a national health (care) system varies among countries, depending on the conceptual frameworks that a national government uses to assess health care performance (Tawfik-Shukor et al, 2006).

4 2.3 Inter-Organizational Reporting Systems Performance data of national health care systems are gathered by means of interorganizational systems (IOS) that connect health care organizations to a regional or national platform organization. IOS are IT based systems providing linkages between organizations in a business network and vary from relatively simple trade data exchange applications to complex cash management systems (Holland, 1995), extended Enterprise Resource Planning systems and external reporting systems (Clark and Lee, 2000). More precise, IOS are IT applications that transcend organizational boundaries and require specific IT applications and business process changes at both sides of business relations in a business network. An IT application in one organization (an internal information system) might be adapted for inter-organizational communications, but without requiring changes at the other side of the SC relation (Holland, 1995). IOS are closely linked to the structure of business networks and interorganizational partnerships. The structure of a network influences the diffusion of information through the network. Gibbons (2007) found that fully connected structures (all organizations are linked to one another) outperform hierarchical structures and group to group chains. Fairchild et al (2004) distinguish between seven success factors for business networking. Four indicators relate to the business network context and three to inter-organizational business processes. Network context success indicators can be summarized as (i) a high number, high volume, high variability, and high frequency of the transactions, (ii) low complexity, low specificity, and high value of the product, (iii) convergence of stakeholder motives, and (iv) the presence of government regulations. Business process success indicators can be summarized as (i) low learning costs and low entry barriers, (ii) availability of multiple transaction mechanisms, (iii) trust, based on neutrality of the market, partnership with domain experts, high quality of product- and trading partner information, security of information, and a local focus. 2.4 Research model and hypotheses Following the previous sections, we hypothesize that the following nine factors influence the success of external reporting of health care organizations: 1. Validity of the performance indicators ( high validity of the indicators leads to more success ). 2. Reliability of the performance indicators ( high reliability of the indicators leads to more success ). 3. Compliance of the health care providers and organizations to provide data ( more compliance leads to more success ). 4. Relevance of the indicators, being the fit between the size (boundaries) of the health care domain on which reporting takes place and the focus of the health care organization ( good fit leads to more success ). 5. Quality of the inter-organizational systems (or applications) through which the data gathering takes place ( good IOS leads to more success ). 6. Reliability of the data stored in the internal information systems of the organizations ( reliability leads to more success ) 7. Availability of the data, i.e. the degree to which required data are stored in the internal information systems of the organizations ( availability leads to more success ).

5 8. Accessibility of the data, i.e. the degree to which internal information systems in the health care organizations provide input for the external reporting systems ( good internal IS lead to more success ). 9. Definitions and specifications of the data (good definitions and specifications of the data that need to be entered lead to more success ). 3 Methods To evaluate the hypotheses we analyzed one (large) case, being the development and implementation of a new national dataset of performance indicators for mental health care in the Netherlands. Our research method consists of retrospective analysis using a combination of quantitative analysis (of a dataset) and in depth qualitative analysis of one case (semi-structured interviews with multiple actors involved in the use of the performance measurement system), as suggested by Klein and Myers (1999) and Provan et al (2007). In depth knowledge of one case can be used to evaluate hypotheses and generalize findings. Our case study research can be regarded as positivist but critical (since we bring to light restrictive and alienating conditions of the status quo) (Mingers, 2001). Our research method and data acquisition are described in 3.1 and an overview of the mental health care sector is given in Data acquisition and analysis We used part of the dataset JMV-2006, semi-structured interviews with 12 operational managers from mental health care organizations, and additional information on JMV from several publicly available reports. JMV-2006 is the Yearly Document Societal Accountability 1 and is the first dataset in the Netherlands that combines the scores of all mental health care organizations in the Netherlands on 35 questions representing 17 performance indicators in three categories: effectiveness, safety and client-focus of care. This so-called basic set of performance indicators and the 35 questions have been designed and developed since 2005 by eight large organizations: the national platform of health care insurers (ZN); the national platform of mental health care patients (LPGGZ); the national institute of mental health care organizations (GGZ-Nl); the associations of psychiatrists (NVvP), psychotherapists (NVP), psychologists (NIP), the national platform of care and cure (V&V), the association of nurses in mental health care (FVidGGZ), the Ministry of Health (VWS), and the Netherlands Health Care Inspectorate (IGZ). Individual interviews were done (by telephone) in 2007 with 12 managers from 12 organizations, randomly selected out of the 64 mental health care organizations. Managers were selected who had been responsible for (part of) the input to JMV The interviews took about one hour each and focused on the difficulties in providing the right data and answers to the 35 questions. Analysis of the interviews was done by clustering the answers into the following categories (related to factors F4 and F6 to F9, as distinguished in our research model above): 1. Questions are not relevant for the mental health care organization (F4) 2. Data to answer the question are not reliable (F6) 3. Data are not available (not stored in any organizational system) (F7) 1 In Dutch: Jaardocument Maatschappelijke Verantwoording

6 4. Data to answer the question are not easily accessible in the organization (F8) 5. Difficulties in interpreting the definitions in the questions (F9) For additional information on the basic set of performance indicators for mental health care we used the IGZ publication on this topic ( October 2006) and the Tranzo research report (Ham et al, 2007). 3.2 Mental Health Care in the Netherlands Mental health care spending in the Netherlands was about 4 billion Euro (2003) and accounts for about 7% of the total health care spending. The numbers of patients treated for mental health have risen from (2003) to (2005) ( The mental health care sector consists of about 25 large, integrated institutions that provide many types of care, and around 50 smaller organizations for specialized care, including care for addictions, institutes for guided living (RIBW), child and adolescent care. The Ministry of Health in the Netherlands has indicated several core themes for mental health care in 2008, such as prevention, chronic mental health care, and forensic psychiatry ( Financing of mental health care organizations has changed substantially over the past years. All financial revenues used to come from one (public) national source (AWBZ), based on a system of yearly budgets and input-financing. The mental health care sector is now changing into output financing (like the USA diagnosis related payments) by (private) insurance companies. These changes have resulted in the introduction of various new (internal) information systems in mental health care organizations to support the new financial billing rules and aiming to assess treatment effectiveness, efficiency, and quality. Additionally, several interorganizational information systems have emerged to enable transactions between care providers, insurance companies, government agencies, and platform organizations and associations. The national Act on Quality of Health Care Organizations in the Netherlands states care providers must systematically gather and store data on the quality of care.must evaluate how care processes lead to good care. In order to reduce the administrative burden for health care organizations, the health care sector has implemented the Yearly Document Societal Accountability (JMV), replacing the variety of separate inter-organizational reporting systems. JMV will ultimately include performance indicators for the total health care sector in the Netherlands. Since 2006, the basic set of performance indicators for mental health care is included in JMV. Since 2006, mental health care organizations enter their performance data by answering an on-line questionnaire in the web-based application digimv. DigiMV was developed in 2004 by the Ministry of Health and is intended to be used by all health care organizations. 4 Results Statistical analysis of the basic dataset was done by comparing descriptive statistics only, since the data represent almost the total population (more then 80% of all mental health care organizations in the Netherlands participate in the national reporting system). In total population scores all differences between descriptive statistics are significant. Below we report on the relevant differences in our results.

7 We define success of external reporting as the degree (percentage) to which an organization is able to answer a set of standardized questions focusing on several aspects of their organizational performance, in particular focusing on the quality of care. We count a question as being answered only if an answer has been entered (independent of the correctness). Figure 1 shows the success rates (vertical axis) of two types of institutions: integrated mental health care organizations versus specialized institutions for mental health care. Each curve represents the set of care institutions, ranked on the horizontal axis from lowest performance (left) to highest performance (right) 2. Integrated institutions score better since they answer on average 19% (seven) more questions than specialized institutions. 120% 100% 80% 60% integrated institutions specialized institutions 40% 20% 0% Figure 1. Success of external reporting (as percentage of 35 questions answered) of integrated care organizations (N=24) and specialized care institutions (N=25). See text. Table 1 shows the success rates of all 64 organizations for questions on specific performance categories. Questions on safety were answered more often than questions on customer orientation and effectiveness. Integrated institutions score well on all performance categories, specialized institutions don t score well on effectiveness questions. Table 1. Success of external reporting on all questions (N=35) and per performance category (effectiveness, safety, and customer orientation) of integrated and specialized care institutions Type of organization Questions answered (N = 35) Effectiveness questions (N = 11) Safety questions (N = 4) Customer orient (N = 20) N Average (sd) Average (sd) Average (sd) Average (sd) All % (27) 48.9% (31) 69.9% (28) 56.8% (31) Integrated mental health % (23) 70.8% (24) 75.0% (29) 70.4% (27) Specialized % (28) 38.9% (30) 68.0% (30) 55.4% (32) Others (1) % (23) 30.3% (21) 65.0% (25) 37.3% (21) (1) Organizations that could not be categorized as integrated or specialized. 2 The same type of comparison is used by the Netherlands Health Care Inspectorate (IGZ).

8 Table 2 shows the success rates of all 64 organizations for questions related to different information sources. Three questions relate to patient records (paper based or electronic), 12 come from patient questionnaires, 17 from activity registration systems, one from incident registration systems, and two from internal processing systems. Highest success is found for questions relating to internal processing systems Table 2. Success of external reporting per information source category of of five types of care institutions. Type of care institution Patient records (3 questions) Patient quest (12 questions) Information source Activity Incident registr (17 registr (1 questions) question) Internal proc (2 questions) N average (sd) average (sd) average (sd) average (sd) average (sd) all % (9) 64.3% (15) 47.9% (17) 46.9% 85.9% (9) Integrated mental health % (9) 77.1% (17) 66.2% (16) 54.2% 93.8% (3) Addiction % (17) 54.2% (23) 48.0% (23) 33.3% 91.67% (12) Supported housing % (6) 84.2% (7) 40.6% (28) 50.0% 70.0% (14) Youth % (6) 49.1% (19) 36.6% (22) 44.4% 77.8% (0) Others % (21) 43.9% (18) 29.0% (21) 40.0% 86.7% (19) The 12 interviews focused on the difficulties in answering the 35 questions grouped according to the 17 performance indicators (each indicator is assessed by one to four questions). Table 3 shows an overview of the findings per performance category. Most difficulties (43) were reported on effectiveness questions: on average 3.9 difficulties per question. Fewest difficulties were reported on customer orientation questions (1.6 difficulties per question). Most difficulties reported relate to question is not relevant (0.5 difficulties reported per question), fewest difficulties relate to interpreting the data definitions provided by the digimv application (0.19 difficulties reported per question). Table 3. Interview findings indicating the difficulties in answering the questions per performance category. See text. Table 4. Interview findings indicating the difficulties in answering the questions per information source category. See text. Performance Questions Difficulties Not Availablable Access- Reliable Category (N) per question relevant Effectiveness Safety Customer or Average Definition Information source category Questions (N) Difficulties per question Not relevant Reliable Available Accessable Definition Patient records Patient quest Activity reg Incident reg Internal proc Average

9 Table 4 shows an overview of the findings per information source category. Most difficulties (54) were reported on activity registration systems (3.2 difficulties per question) and on the (single) question related to incident registration systems (6 difficulties). Fewest difficulties relate to data definitions. 5 Discussion We now summarize the findings and relate these to the nine factors (F1-F9) and hypotheses formulated above. Our research does not provide insight in the effects of validity and reliability of the performance indicators on the success of external reporting (F1, F2). We have not assessed the degrees of validity and reliability of the indicators, so we cannot test the hypothesis high validity of the indicators leads to more success. We found high compliance of the health care organizations (F3): all organizations have been actively involved to answer all 35 questions, providing on average 56% of the required data into the new inter-organizational system. However, we have not assessed the relations between the levels of compliance in the organizations, so we cannot test whether more compliance leads to more success. The mental health care organizations and in particular the quality managers in our study report a positive attitude towards the new reporting system and willingness to improve internal processes to enable more questions to be answered. This suggests some support for the hypothesis compliance leads to more success. We found that higher relevance of the indicators leads to more success of external reporting (F4). Integrated care organizations that cover a larger part (boundaries) of the health care domain on which reporting takes place are more successful than specialized (focused) care organizations, supporting the hypothesis good fit leads to more success. All institutions found one or multiple questions not relevant for the institution. For instance, questions on treatment of patients are not relevant for institutes for supported housing. Success of the reporting system would increase if the system would allow organizations to skip irrelevant questions. Integrated care organizations were more successful in answering the questions than specialized, focused organizations. Quality of the inter-organizational systems (F5). The current IOS (DigiMV with its set of 35 questions) is not (yet) of a high quality, since the majority of the organizations were not able to answer more than 50% of all questions. We found the emergence of new information processing services offered by several intermediary organizations in the health sector ( Successful implementation of these intermediary services might result in more automation and increased transparency of the interorganizational reporting processes and in vertical disintegration of the reporting chain. This suggests support for the hypothesis good IOS lead to more success. Reliability and validity of the performance indicators (F6). Many organizations have a positive attitude towards the reporting system, indicating system success. However, the organizations also report that they doubt the reliability of the indicators because of the low quality of the input data. Hypothesis higher reliability and validity of the indicators lead to more success is not supported. Availability and accessibility of the data (F7 and F8).Organizations that have more data available and accessible in the internal systems are more successful. Organizations report that they are planning to improve their internal systems in order to improve the linkages with the reporting applications. Hypotheses more

10 availability leads to more success and good internal IS lead to more success are supported. Definitions and specifications of the data (F9). Questions based on poor data definitions and specifications were answered less successfully. This findings support the hypothesis good data definitions improve success. 6 Conclusions The aim of this research was to identify factors that influence the success of external reporting of health care organizations. We analyzed the literature on reporting systems in health care and literature on inter-organizational reporting systems and identified factors that influence the success of reporting systems, including (i) reliability and validity of the performance indicators, (ii) accessibility and availability of internal data, (iii) compliance of the health care providers, (iv) relevance of the indicators, and (v) quality of the interorganizational systems. We evaluated the influence of these factors on the success of a new external reporting system for mental health care institutions. Success is defined as the degree to which the required questions on organizational performance are answered. Our findings show that the new system is more successful for integrated care organizations than specialized care organizations, and more successful in organizations with better internal information systems. More research is needed on the relations between the design of the (DigiMV) reporting system, the variety of care organizations and supporting agencies in the sector, the objectives and strategies of the national health care authorities (such as the Ministry of Health), and the success of external reporting systems. Acknowledgement: we thank dr. Bert Bettonvil for his statistical advice. References Arah O.A., Westert G.P., Hurst J., Klazinga N. (2006): A conceptual framework for the OECD Health Care Quality Indicators Project. Int. J. for Quality in Health Care. September: pp Carpenter VL, Feroz EH (2001): Institutional theory and accounting rule choice: an analysis of four US state governments decisions to adopt generally accepted accounting principles. Accounting, Organizations, and Society. 26: Clark T.H., H.G. Lee (2000): Performance, inter-dependence, and coordination in business to business electronic commerce and supply chain management, IT and Management, CMMS (2007): Centers for medicare & medicaid services, Office of the Actuary, National Health Statistics Group, National Health Accounts ( Fairchild AM, Ribbers PMA, Nooteboom B (2004): A success factor model for electronic markets: Defining outcomes on stakeholder context and business process. Business Management Process J (10) 1: Ham M van, Luijsterburg J, Brouwers E (2007): Evaluatiestudie basisset prestatieindicatoren. Tranzo, Tilburg University, NL.

11 Hermann R.C, Leff H.S., Palmer R.H., Yang D., Teller T., Provost S., Jakubiak C., Chan J. (2000): Quality Measures for Mental Health Care: Results from a National Inventory. Medical Care Research and Review, 57 (Suppl 2), Holland CP (1995): Cooperative supply chain management: the impact of interorganizational systems. J of Strategic IS. 4 (2) Klein H.K. and Myers M.D. (1999): A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly 23 (1) Kleijnen, J. and Smits, M.T. (2003). Performance metrics in Supply Chain Management. Journal of the Operation Research Society, 54 (5), Mingers, J. (2001): Combining IS Research Methods: Towards a Pluralist Methodology. Information Systems Research 12 (3) Nieuwenhuijsen K, Verbeek JHAM, Boer AGEM de, Blonk RWB, Dijk FJH van (2005): Validation of Performance Indicators for Rehabilitation of workers with mental health problems. Medical Care. 43 (10) OECD (2004): Towards High-Performing Health Systems. Organization for Economic Cooperation and Development, Paris ( Provan KG, Fish A, Sydow J (2007): Inter-organizational networks at the network level: a review of the empirical literature on whole networks. J of Management 33: RIVM (2006): Costs of Diseases (in Dutch). National Institute for Public Health and the Environment, Bilthoven, Netherlands. Rebergen D, Hoenen J, Heinemans A, Bruinvels D, Bakker A, Mechelen W van (2006): Adherence to mental health guidelines by Dutch occupational physicians. Occupational Medicine. 56: Schmidt A, Power M, Bullinger M, Nosikov A (2005): The conceptual relationship between health indicators and the Quality of Life: results from the cross-cultural analysis of the EUROHIS field study. Clinical Psychology and Psychotherapy 12: Smith, C., Cowan, C., Heffler, S., and Catlin, A. (2006). National health spending in 2004: Recent slowdown led by prescription drug spending. Health Affairs, 25 (1) Stiles PG, Boothroyd RA, Snyder K, Zong X (2002): Service penetration by persons with severe mental illness: how should it be measured? J of Behavioral Health Services & Research. 29 (2) Tawfik-Shukor AR, Klazinga NS, Arah AO (2007): Comparing health system performance assessment and management approaches in the Netherlands and Ontario, Canada. BMC Health Services Research. 7: 25 (12 pages) Valenstein M, Mitchinson A, Ronis DL, Alexander J, Duffy SA, Craig TJ, Lawton Barry K (2004): Quality indicators and monitoring of menatl health services: what do front line providers think? Am J Psychiatry. 161:

12 Westert G.P, Verkleij H (2006): Zorgbalans, de prestaties van de Nederlanse gezondheidszorg in Bohn Stafleu Van Loghum, Bilthoven. ISBN