Evaluation of The Health Council of Canada (HCC)

Transcription

1 KPMG LLP Bay Adelaide Centre 333 Bay Street, Suite 4600 Toronto ON M5H 2S5 Canada Telephone (416) Fax (416) Internet Evaluation of The Health Council of Canada (HCC) Final Report September 24, 2013 kpmg.ca A Proposal to Serve the Health Council of Canada

2 Contents Executive Summary Introduction and Context Study objectives History and conduct of the evaluation Evaluation questions Description of the Health Council of Canada About the Health Council of Canada The rationale for the Council Alignment with government priorities Target beneficiaries Stakeholders Governance Operations Resources Logic model Methodologies Methodologies and Work Plan Study Limitations Findings Evaluation Question 1: What methods of operation, reporting, or activities differentiate the Health Council from other Canadian health care knowledge intermediaries? Evaluation Question 2: How effective have each of the following been in the Health Council s reporting on health system performance (including health status and outcomes): Evaluation Question 3: How effective have each of the following been in the Health Council s reporting on innovative practices in health care: Evaluation Question 4: What have been the extent of awareness and understanding of Health Council information among its government and health agency organization target audiences regarding: Evaluation Question 5: What have been the extent of awareness and understanding of Health Council information among the general public? Evaluation question 6: What has been the extent of use and usefulness of Health Council information among its government and health agency target audiences regarding: Evaluation question 7: What have been the impacts of the Health Council s activities on Canadian health policy, programs, and services i.e., have organizations or staff started doing things differently as a result of the Health Council s work? Final Report i

3 4.8 Evaluation question 8: What are the best practices and lessons learned (both strengths and weaknesses) from the Health Council s model and operations that could inform the future effective delivery of health-related information? Conclusions Final Report ii

4 Executive Summary (HCC) undertook this evaluation to satisfy its funding agreement with Health Canada. The study objectives included the evaluation of: the effectiveness, impact, unique contributions and added value of the activities and initiatives of the organization as outlined by its funding agreement with Health Canada from 2010 to date, that fulfills its mandate to monitor and publicly report on the implementation of the Accords as well as on health status and health outcomes (Funding Agreement, page 1) and best practices. 1 The evaluation covers the timeframe from 2010 through March 2013, which reflects the timing of the current funding agreement. The evaluation focused on addressing eight research questions, called evaluation questions. The overall findings of the study are very positive, given that (1) the Health Council is a relatively low budget (average $5.2 million per year over the past five years) organization, and (2) it is focused almost exclusively on the dissemination of knowledge (i.e., as opposed to, for example, the implementation of knowledge). The evaluation questions and the main findings related to each question are as follows: 1. What methods of operation, reporting, or activities differentiate the Health Council from other Canadian health care knowledge intermediaries? The Health Council occupies a unique position among Canadian health care knowledge intermediaries, with only minimal overlap with other organizations. The Council responds to important needs of its key audiences, and its national scope and perspective are important differentiating characteristics of the organization. 2. How effective have each of the following been in the Health Council s reporting on health system performance (including health status and outcomes): (a) The quality of the Health Council s reporting (b) The methods used by the Health Council to disseminate information? The quality of Health Council reports and activities is comparable to, or better than, other organizations that are active in reporting on health system performance. Most of the dissemination methods used by the Health Council for reporting on health system performance have been moderately to highly effective. 3. How effective have each of the following been in the Health Council s reporting on innovative practices in health care: (a) The selection of topics the Health Council has reported on (b) The quality of the Health Council s reporting (c) The methods used by the Health Council to disseminate information? 1 Health Council of Canada, Request for Proposal, September 2012, p. 5. Final Report 1

5 The quality of the Health Council s reports and activities in both of its areas of activity (reporting on health system performance and reporting on innovative practices in health care) is comparable to or better than other organizations that are active in one or both of these areas. Most of the dissemination methods used by the Health Council for reporting in each of these areas have been moderately to highly effective. With specific reference to its work on innovative practices, the Health Council s key audiences generally believe that the topics covered by the reports and activities of the Council are important to their organizations. 4. What have been the extent of awareness and understanding of Health Council information among its government and health agency organization target audiences regarding: (a) health system performance (b) innovative practices in health care The Health Council is the top organization that comes to mind among its key audiences when it comes to seeking information on both health system performance and innovative practices in health care. The Health Council s information has influenced understanding of both health system performance and innovative practices among its key audiences to a moderate degree. 5. What have been the extent of awareness and understanding of Health Council information among the general public? 2 The Health Council is satisfying the interests of Canadians with its information on health system performance and innovative practices in health care to a moderate degree. The Health Council s approach and media draw is comparable with other organizations when it comes to reaching the general public. An interested community has been steadily growing; however it is difficult to identify the population as general public. 6. What has been the extent of use and usefulness of Health Council information among its government and health agency target audiences regarding: (a) health system performance (b) innovative practices in health care? The Health Council s reports and activities related to both health system performance and innovative practices have been used to at least a moderate degree by the Council s key audiences. The majority of the Health Council s client base considers the Council s reports in both these areas to be useful. 7. What have been the impacts of the Health Council s activities on Canadian health policy, programs, and services i.e., have organizations or staff started doing things differently as a result of the Health council s work? 2 This is a simplification of the original evaluation question, which split out awareness and understanding regarding (a) health system performance and (b) innovative practices separately. The reasons is that, except for the (very few) media interviews, it was not possible to analyze these separately. Final Report 2

6 There is evidence that the Health Council s activities have had a moderate influence on policy changes within its client organizations, but the findings in this area (impacts of the Council s work) are not clear cut. 8. What are the best practices and lessons learned (both strengths and weaknesses) from the Health Council s model and operations that could inform the future effective delivery of health-related information? The Health Council has been a national voice for the pan-canadian health care system, and it has been an excellent source of consolidated national and international information in a number of areas. There are some issues related to the structure of the Council and the Canadian health care system (e.g., health care is mainly a provincial-territorial responsibility) that may have limited the extent of impacts of the Council s work. Although the evidence is somewhat limited, it appears that there are no other approaches besides the Health Council model that could have achieved the same impacts more efficiently. Final Report 3

7 1.0 Introduction and Context 1.1 Study objectives (HCC) undertook this evaluation to satisfy its funding agreement with Health Canada. The study objectives included the evaluation of: the effectiveness, impact, unique contributions and added value of the activities and initiatives of the organization as outlined by its funding agreement with Health Canada from 2010 to date, that fulfills its mandate to monitor and publicly report on the implementation of the Accords as well as on health status and health outcomes (Funding Agreement, page 1) and best practices (Future Directions letter February 2010). 3 The evaluation covers the timeframe from 2010 through March 2013, which reflects the timing of the current funding agreement, but some data, to allow for comprehensive reporting may date to History and conduct of the evaluation This is summarized below: KPMG LLP was retained by HCC in October, 2012, to carry out the evaluation, including the preparation of a detailed plan for the evaluation. The evaluation was overseen by a Steering Committee, which consisted of representatives of HCC and Health Canada. The draft evaluation framework (evaluation plan) was submitted in January, 2013, and finalized in February, In early April, 2013, the federal government announced that it would not be re-funding HCC and that the organization would be sun-setting by As a result, the study objectives were re-scoped to focus on retrospective performance and lessons learned. It was decided that evaluation questions related to continued relevance and future positioning would not be addressed. The detailed evaluation design was completed in April, The full evaluation study was launched in May, This document is the draft report on the evaluation. The purpose of this report is to present the study team s analysis of the data and findings to the Steering Committee prior to the preparation of the Final Report. 3 Health Council of Canada, Request for Proposal, September 2012, p. 5. Final Report 4

8 1.3 Evaluation questions The initial formulation of the research questions to be addressed during the evaluation followed a formal process which included interviews with people who are knowledgeable about the Health Council and who have an interest in the evaluation results. This included Health Council senior staff, Council members, federal government representatives, and broader health care policy oriented stakeholders. The evaluation was structured around eight questions approved by the Steering Committee, which are listed below. Note that these questions have changed somewhat from the initial set of questions developed through the scoping exercise. This was due to federal government decision to not re-fund HCC, which is discussed above. Question 1: What methods of operation, reporting or activities differentiate the Health Council from other Canadian health care knowledge intermediaries? Question 2: How effective have each of the following been in the Health Council s reporting on health system performance (including health status and outcomes): a. The quality of the Health Council s reporting? b. The methods used by the Health Council to disseminate information? Question 3: How effective have each of the following been in the Health Council s reporting on innovative practices in health care: a. The selection of topics the Health Council has reported on? b. The quality of the Health Council s reporting? c. The methods used by the Health Council to disseminate information? Question 4: What has been the extent of awareness and understanding of Health Council information among its government and health agency organization target audiences regarding: a. health system performance? b. innovative practices in health care? Question 5: What has been the extent of awareness and understanding of Health Council information among the general public regarding: a. health system performance? b. innovative practices in health care? Question 6: What has been the extent of use and usefulness of Health Council information among its government and health agency organization target audiences regarding: a. health system performance? b. innovative practices in health care? Question 7: What have been the impacts of the Health Council s activities on Canadian health policy, programs, and services i.e., have organizations or staff started doing things differently as a result of the Health Council s work? Question 8: What are the best practices and lessons learned (both strengths and weaknesses) from the Health Council s model and operations that could inform the future effective delivery of healthrelated information? Final Report 5

9 2.0 Description of the Health Council of Canada 2.1 About the Health Council of Canada was created by Canada s First Ministers in 2003 as a vehicle to effectively collaborate with federal/provincial/territorial (F/P/T) governments on the shared priority of accountability in health care in accordance with the implementation of the 2003 First Ministers Accord on Health Care Renewal. The Accord sets out a plan for Canadian health care reform reflecting renewed commitment by governments to collaborate amongst themselves, with providers and with Canadians to shape the future of the Canadian public health care system. The intent of the Accord was three-fold: Ensure that all Canadians have timely access to health services on the basis of need, not ability to pay, regardless of where they live or move in Canada; Ensure that the health care services available to Canadians are of high quality, effective, patientcentred and safe; and Ensure that the Canadian health care system is sustainable and affordable and will exist for Canadians and their children in the future.4 As identified in the Accord, the objective of the Council was to monitor and produce annual public reports on its implementation. It would publicly report through federal/provincial/territorial (F/P/T) Ministers of Health. Budget Plan 2004 affirmed the Council s role as It will monitor and make annual public reports on the implementation of the Accord, with an emphasis on its accountability and transparency provisions. The work of the Council will enable Canadians to assess the performance of the health system and the pace of implementation of the various commitments made in the Accord. In 2004 the 10-Year Plan to Strengthen Health Care was developed by the First Ministers. This Plan laid out the commitments of the F/P/T governments with regard to numerous activities in 10 areas as follows: 1. Wait times and improved access; 2. Strategic Health Human Resource (HHR) Action Plans; 3. Home care; 4. Primary care reform; 5. Access to care in the North; 6. National Pharmaceuticals Strategy; 7. Prevention, promotion and public health; 8. Health innovation; 9. Accountability and reporting to citizens; and 10. Dispute avoidance and resolution. 4 Health Canada First Ministers Accord on Health Care Renewal. Final Report 6

10 Moreover, under this Plan, the Council s mandate was expanded to include reporting on health status and health outcomes in Canada. Following a Corporate Members review of the scope, mandate, role, objectives, effectiveness and the relevance of the Council that concluded in 2010, a new set of directions were confirmed. The Council was mandated to: Continue reporting on progress related to the Accord elements based on a multi-year cycle (only half of the Accord elements are reported on in any given year); Place greater emphasis on identifying, reporting and disseminating information on best practices and innovation in its public reports; and Increase government engagement in the planning and development of its public reports to ensure greater value-add to the jurisdictions. 5 The most recent five year strategic plan of the Council outlines the following four strategic priorities: Informing: the Council will monitor and report on the health accords as a trusted source of information and evidence on Canada s health system renewal. Communicating: The Council s work will be easily accessed by health organizations, stakeholders, governments and the public, and it will seek ongoing input and feedback. Collaborating: The Council will maintain strong working relationships with governments, stakeholders and other health organizations to add value to one another s work to improve Canada s health system. Having impact: the Council strives to measure the contribution of its work in strengthening Canada s health system. 2.2 The rationale for the Council Findings and recommendations from the Kirby Report, The Health of Canadians - the Federal Role, (October 2002) and the Romanow Commission on the Future of Health Care in Canada (November 2002) both identified the need and value of an independent council informing Canadians on health care matters while promoting accountability and transparency. To further elaborate on the rationale for such a body, the Kirby Report found that: It is essential to improve the governance of Canada s health care system. The question of governance (which is to say leadership) brings together a number of issues that the Committee has raised in previous volumes and that witnesses have addressed from a number of perspectives. One thing is very clear. Canadians are tired of the endless finger-pointing and blame-shifting that have been recurring features of intergovernmental relations in the health care field. As the Honourable Monique Bégin has accurately pointed out, the current state of federal-provincial relations is dysfunctional.[7] On far too many occasions, each side seems more interested in attributing blame for the system s apparent deterioration to the other, rather than taking the lead to ensure that the health services Canadians need and deserve are there when they need them. 5 Health Council of Canada. Strategic Directions 2011: Five-Year Strategic Plan. Final Report 7

11 Fundamentally, the underlying issue is one of accountability. In order to establish who is to be held accountable for the deficiencies (and also the strengths) of the health care system, the Committee has repeatedly pointed out that detailed and reliable information on the performance of the system and on health outcomes is essential. 6 The Kirby report identified four fundamental elements necessary to create capacity to evaluate the performance of the health care system and the health status of the Canadian population: 1. The creation of an evaluative body that must be independent of government in order to obtain independent assessment facilitating the avoidance of conflicts of interest and assurance of credibility. 2. The evaluative body must perform the evaluation at the national level as Canada s health care system is a joint responsibility of the provincial/territorial and federal governments. 3. The evaluative body must be funded by the federal government, despite the national (as opposed to federal) character of the evaluation organization. 4. The evaluative body build on the successes of existing organizations, such as the Canadian Institute for Health Information (CIHI). The Romanow Commission reported similar findings and recommendations. 2.3 Alignment with government priorities Federal priorities Issues around health care and health care reform have been a priority for the Government of Canada (GoC), over the past decade. In the Budget, the GoC committed $70.1 billion over eight years to health care and health care reform activities. This included the funding for the Council. 7 The Health Council monitors and reports on progress made towards fulfilling the Accord s commitments in the following health care themes, as refined by the Council: 1. Access and Wait Times 2. Health Human Resources 3. Home and Community Care 4. Access to Care in the North 5. Pharmaceuticals Management 6. Health Promotion 7. Health Status and Health Outcomes 8. Aboriginal Health 9. Electronic Health Records 10. Primary Health Care 6 The Standing Senate Committee on Social Affairs, Science and Technology, The Health of Canadians The Federal Role Final Report, Volume Six: Recommendations for Reform, Chapter 1, Section 1.2, October Health Canada. Health Care System: Federal Health Investments. February 5, Final Report 8

12 Cross-cutting themes in Council reporting further include: Health Innovation (through Innovative Practices), Health System Integration, Sustainability, Accountability, Quality, Performance Measurement, and Patient Engagement. These issues remain a priority of the GoC today and are all encompassed in Health Canada s first strategic outcome: A health system responsive to the needs of Canadians. Provincial and territorial priorities Evidence shows that the goals of the Health Council are also aligned well with the various priorities of P/T governments. A recently completed environmental scan of provincial and territorial health system priorities completed by the Canadian Foundation for Healthcare Improvement (CFHI) (formerly the Canadian Health Services Research Foundation) found that P/T governments have numerous priorities related to health and health reform. In particular, many of the P/T priorities identified by CFHI appear to be aligned with the activities and objectives of the Health Council. For instance, P/Ts are working to make health system improvements that enrich the health and well-being of Canadians through improving the accessibility, quality and responsiveness of healthcare through initiatives that will improve health system sustainability. 8 They are also committed to and focused on demonstrating improved performance through formal performance measurement. Finally, P/Ts have taken a highly collaborative approach to overcome challenges and make the most of opportunities through shared learning, cross-fertilization of ideas and innovation, 9 all of which are of inherent interest to the Health Council. 2.4 Target beneficiaries The Council s target population is comprised of a number of groups from across the health field and beyond. The primary focus of the work undertaken by the Council is aimed at health organizations, F/P/T governments, and the public at large via the media. Target beneficiaries have been defined by the Council as those individuals, groups and decision makers that play a role or can play a role in shaping and influencing policy in health care. The Council is unique in that it is mandated to inform Canadians directly and thereby encourage their participation in the health care renewal agenda. This information is intended to enable the general public to understand what has been promised and what has been accomplished. 2.5 Stakeholders The target populations discussed in Section 2.4 also comprise many of the key stakeholders of the Health Council -- i.e., people and organizations that have a direct interest in the work of the Council. There are numerous entities that partner with the Council in its analysis and reporting roles, including the following organizations. Canada Health Infoway Canadian Partnership Against Cancer Accreditation Canada Provincial health quality councils Canadian Foundation for Health Care Improvement Canadian Institutes of Health Research 8 Canadian Health Services Research Foundation. Provincial and Territorial Health System Priorities: An Environmental Scan. December EN.sflb.ashx 9 Ibid. Final Report 9

13 Canadian Patient Safety Institute University of British Columbia Centre for Health Services and Policy Research Canadian Home Care Association Canadian Association for Health Services and Policy Research Statistics Canada Canadian Patient Coalition Canadian Agency for Drugs and Technology Canadian Health Care Association Canadian Institute for Health Information Canadian Blood Services 2.6 Governance The corporate members of the Health Council consist of the federal Minister of Health as well as the Ministers of Health of 12 of the 13 Canadian provinces and territories (Quebec is not a member). The corporate members elect a Council consisting of councillors and Chair. The member elected Council (effectively the Board of Directors) manages the work of the Health Council. The Council is comprised of 13 councillors and one ex-officio councillor. It is made up of six government councillors appointed by the corporate members representing six jurisdictional groupings: (1) British Columbia/Yukon, (2) Alberta, (3) Manitoba/ Saskatchewan/ Nunavut/ Northwest Territories, (4) Ontario, (5) Nova Scotia/New Brunswick/Prince Edward Island/Newfoundland and Labrador, and (6) the Government of Canada (collectively referred to as the government councillors ) and seven non-government councillors elected by the corporate members through a nominating committee. The government councillors act as independent experts rather than as representatives of the governments that appointed them. The non-government councillors are respected leaders in their field of expertise and do not act as representatives of any particular organization, nor can they be employed by any federal, provincial or territorial government or elected to an office within any federal, provincial or territorial government. The Health Council is governed by a by-law and supported by a secretariat with expertise in health policy, research, stakeholder and government relations, and communications. The Council oversees the work of the secretariat through an executive committee, finance and audit committee, and a health system reporting committee Operations The Health Council undertakes a series of activities in support of its mandate. There are three core organizational groups that deliver: Research, analysis and reporting: Conducts research and system level analysis to monitor and report on health care renewal, including progress on the elements in the federal/provincial/territorial agreements, barriers and health outcomes and highlighting innovative practices in delivery and policy environments. Stakeholder & Government Relations: Facilitates contact between the Health Council and the federal, provincial and territorial health ministers of participating jurisdictions and Canada's key health agencies, professional organizations, and advocacy groups. 10 Health Council of Canada Annual Report. Final Report 10

14 Communications: Activities in support of disseminating publicly accessible information, education and dialogue about health care renewal, including public engagements, media relations and social media, stakeholder briefings/meetings, and corporate annual reporting. 2.8 Resources The Council is funded directly through a grant funding agreement with Health Canada. The allocation of funds is identified for all years following its inception year (2003/04)11 in the table below Compensation $ $ 1,873.7 $ 2,409.1 $ 2,346.7 $ 2,250.7 $ 2,074.2 $ 2,956.2 $ 3,300.4 O&M $ 2,185.0 $ 4,168.3 $ 3,861.4 $ 3,372.6 $ 2,840.5 $ 2,548.5 $ 2,173.9 $ 2,365.6 Capital Costs $ 95.5 $ 87.0 $ 42.6 $ 54.4 $ 27.5 $ - $ $ 51.4 Total $ 3,230.6 $ 6,129.0 $ 6,313.1 $ 5,773.7 $ 5,118.7 $ 4,622.7 $ 5,288.9 $ 5,717.4 **All data is from the Health Council s audited financial statements attached to annual reports 2.9 Logic model The logic model for the Health Council is shown below. A logic model illustrates the main activities, outputs, and outcomes of a program and how these are logically linked. Activities are the tasks that are carried out by the Health Council. Outputs are the goods and services that are produced as a result of the activities for example, information on health outcomes and health status reporting. Outcomes are things that are done or experienced by others (people other than those delivering the program) as a result of the outputs for example, beneficiaries are more informed and aware of health system performance. The Health Council s priorities are reflected within the outcomes category. 11 The first year of funding for the Council in 2003/04 was a start up budget of approximately $1M that included funds for facilities and furnishing, incorporation, the search for an Executive Director, information technology and telecommunications charges, etc. Final Report 11

15 Logic Model for the Health Council of Canada Health Council of Canada Situation Statement: To report on the progress of renewal of Canada s health care system, focusing on best practices and innovation Outcomes - Impact Results Chain Activities Reach Outputs Immediate Outcomes Intermediate Outcomes Ultimate Outcomes Situation Priorities Informing Communicating Collaborating Having impact Research, Analysis and Reporting Stakeholder relations and collaborations Communication and Public Outreach Health Care Policy and Decision Influencers, Government Decision Makers Canadian public via the media Health Accords Reporting (outcomes and status) Reporting on innovative health care practices, policies, programs and services New/ongoing relationships, partnerships and networks Support materials (media, social media, multimedia, web), Symposia, E-newsletters Target beneficiaries have access to relevant information on innovative practices in health care Target beneficiaries are more informed and aware of the progress of health system renewal Target beneficiaries collaborate with Council and participate in Council activities (symposia, workshops) Information is used to support policy, program and service change Improved sharing and collaboration among F/P/Ts in progressing on health care renewal Improved accountability of F/P//T governments Strengthened and renewed Canadian Health System Theory of Change A=Assumptions, R=Risks Including external influencing factors A: The Council has a niche area and some unique reporting lines. R: The Council functions within a crowded health care information landscape. A: Evidence is available to support reporting activities. R: Necessary evidence is not available A: HCC products are useful to target beneficiaries R: Supply and demand for products is not balanced Final Report 12

16 3.0 Methodologies 3.1 Methodologies and Work Plan Our work plan for the evaluation of the Health Council is illustrated in the exhibit below. The key features of each of the data collection methodologies are described in the following sections. Evaluation design and work plan report Conduct field work: Review of HCC documents and performance data Internal and external key informant interviews Survey of stakeholders Organization comparative analysis Mini impact studies Media analysis Preliminary analysis report Evaluation Steering Committee meeting Draft Evaluation Report Evaluation Steering Committee meeting Final Evaluation Report Document review The main documents reviewed in the evaluation were: Documentation on HCC quality processes Documentation on HCC s Innovative Practices Evaluation Framework Available media analysis (and underlying context) undertaken by HCC since 2006 (including social media measurements). Data from the HCC s impact evaluation framework. CEO reports to Council (in relation to the reporting of impacts). Final Report 13

17 Internal and external key informant interviews A series of key informant interviews were conducted by telephone with selected representatives of: Government; Knowledgeable health agencies (agencies that were thought to be most familiar with and/or have the most interaction with the Health Council); and Media contacts and knowledgeable public-facing health organizations. The Health Council was asked to provide a purposeful sample 12 of potential interviewees for each key informant group. The list of interviewees is contained in Appendix A and the interview guides are contained in Appendix B. The following interviews were completed: Interviews of Director and DG level representatives of 10 provinces and territories. (Representatives of Quebec, Newfoundland, and Nunavut were not interviewed Quebec is not a member of the Council, and the Newfoundland and Nunavut representatives declined to be interviewed.) Representatives of 12 knowledgeable health agencies. (Two of these interview responses were subsequently excluded from the analysis because the interviewees were not sufficiently knowledgeable about HCC.) Representatives of four media and public facing health organizations. (The sample size for these interviews was 12, but many potential interviewees did not participate because they perceived a conflict of interest.) Survey of stakeholders This was a web-based survey of HCC s database of 3,941 individuals who are thought to be potential clients for HCC s reports (representatives of regional, provincial/territorial, and national healthcare-related organizations; healthcare service delivery organizations; professional associations; and so on), as well as media contacts. The response rate was 17.9% 13, which is respectable for this type of sample. Our analysis reflects only respondents who indicated they had at least a moderate familiarity with HCC. A 90% confidence level was used in the analysis. Comparative Review The comparative review was an analysis of similar Canadian healthcare information organizations. The review included an analysis of the roles and mandates of these organizations and their methods of operation. The objective of the review was to determine the degree of overlap and the differences between the Health Council and these organizations. The review of each organization included the assembly of publicly available information and an in-depth interview with a senior organization representative. Initial research was conducted to identify possible comparator organizations which have a similar mandate and functional role within the Canadian health care information landscape. Five organizations were selected for the review: 12 Those representatives that are most knowledgeable about the Health Council and its work. 13 The survey sample consisted of 3,941 contacts (adjusted for invalid addresses, retirees, etc.) with 704 responses received. Final Report 14

18 Canadian Institute for Health Information (CIHI) A national not-for-profit organization Annual budget is approximately $104 million Mandate is To lead the development and maintenance of comprehensive and integrated health information Main activities are the collection of data on Canada s health system and carrying out analyses and preparing reports based on these data. Canadian Foundation for Healthcare Improvement (CFHI) A not-for-profit organization governed by a Board of Trustees Funded by endowment-type funding from the federal government. Revenue (and expenses) in 2011 and 2012 were $17 million and $11 million respectively. Mission is To accelerate healthcare improvement and transformation for Canadians Main area of activity is collaboration with healthcare professionals and regional and provincial/territorial healthcare authorities to improve healthcare systems and develop approaches to managing healthcare challenges. Health Quality Ontario (HQO) An arms length agency of the Ontario government, funded by the Ministry of Health and Long Term Care Budget for 2012 was $32 million Mandate is to evaluate the effectiveness of new health care technologies and services and support quality improvement activities Main roles are (a) monitor and report to the people of Ontario on health services, health human resources, consumer and population health status, and health system outcomes, (b) support continuous quality improvement, and (c) make recommendations to health care organizations on standards of care and recommendations to the government regarding funding for health care services and medical devices. New Brunswick Health Council (NBHC) A public agency created by the government of New Brunswick in 2008 that reports to the Department of Health Annual budget is about $1.5 million Mandate is to foster transparency and accountability of the health system and engagement of citizens in a meaningful dialogue Main activities involve citizen engagement activities, experience surveys (for primary health care, acute care, and supportive specialties), the preparation of report cards on health services and population health, and the preparation of special reports. Final Report 15

19 Institute for Health Services and Policy Research (IHSPR) One of the 13 institutes of the Canadian Institutes of Health Research (CIHR) Annual budget is $9 million $8 million for research grants and $1 million for Institute staff and overhead Like all CIHR institutes, IHSPR is primarily a funder of research. Mission is to foster excellence and innovation in health services and policy research and to catalyze the application of research findings to policies, practices, and programs Mini impact studies We carried out three mini impact studies as part of the evaluation. The main purpose of the mini impact studies was to provide descriptions of some of the impacts of the Health Council s reports. The HCC reports that were the focus of the mini impact studies were: Stepping It Up: Moving the Focus from Health Care in Canada to a Healthier Canada Seniors in Need, Caregivers in Distress: What are the Home Care Priorities for Seniors in Canada? Impacts in the Field of Urban Aboriginal Health Services: Indigenous Cultural Competency Training in Aboriginal Health. Each mini impact study involved a review of available documentation and interviews with one to two respondents (the users) for each report within each identified user organization. Each mini impact study describes: The information provided by the Health Council that was used by various organizations and how it was used; and What resulted from the organizations use of this information i.e., what were the impacts of the use of this information? The mini impact studies are contained in Appendix C. Media Analysis This involved the comparison of high level media analysis results regarding mentions of the Health Council with comparable data from two other public facing health organizations. A search strategy was defined to identify general media mentions of the Health Council and the two other organizations. The search parameters were as follows: One media monitoring database was used, namely Factiva business news and information services which is mainly English publications. Canadian press sources only Only headlines and lead paragraphs were searched for an exact phrase or acronym of the organization name in both English and French. Duplicate findings were excluded.. The search timeframe was from 2004 to Even though this evaluation covers the period of 2010 through March 2013, the search timeframe was chosen to begin in 2004 to allow for comprehensive reporting. It did not include patterns of mentions (e.g., mentions in passing, quotes from someone at HCC, specific report mentions, etc.) Final Report 16

20 The identification and selection of comparator organizations included the consideration of a number of factors including: differences in frequency of reporting and types of reports (newsworthiness); lobbying efforts undertaken by some organizations; extra public exposure through polling and direct patient engagement activities; etc. The organizations selected for comparison were The Canadian Institute of Health Information (CIHI) and the Canadian Institutes of Health Research (CIHR) and one of its institutes, the Institute for Health Services and Policy Research (IHSPR). 3.2 Study Limitations The following points should be kept in mind when reviewing our analyses and findings: The low response rate to the media interviews and the fact that the media analysis did not provide (and could not have provided) impact information did not enable us to draw definitive conclusions regarding the usefulness of HCC s outputs to the general public. All surveys of general populations such as the sample used for our survey of stakeholders are subject to potential response bias i.e., the possibility that members of the sample who have positive opinions regarding HCC may have been more likely to respond, or, alternatively, the possibility that people with negative opinions would have been more likely to respond. We have no reason to expect that there was response bias in this case, but it was not possible for us to check this. Final Report 17

21 4.0 Findings 4.1 Evaluation Question 1: What methods of operation, reporting, or activities differentiate the Health Council from other Canadian health care knowledge intermediaries? Key findings 1. HCC occupies a unique position, with only minimal overlap with other organizations. 2. With regard to reporting on the performance of the healthcare system, there is some overlap with Health Quality Ontario and the Canadian Institute for Health Information, but there are significant differences between HCC and these organizations. 3. Similarly, with regard to reporting on innovative healthcare practices, HCC overlaps somewhat with Health Quality Ontario and the Canadian Foundation for Healthcare Improvement, but again there are significant differences between HCC and these organizations. 4. The Health Council s key audiences believe that HCC responds to important needs and that HCC s national scope and perspective are significant Analysis and supporting evidence Results from the comparative review With regard to reporting on the performance of the healthcare system, Health Quality Ontario (HQO) does the same sort of thing as HCC, but it is focused on Ontario. CIHI overlaps with HCC, but CIHI puts more emphasis on data collection and dissemination and less on policy implications. The other three comparison organizations do not overlap significantly with HCC in this area. This situation is illustrated in the following diagram: HCC HQO CIHI With regard to reporting on innovative healthcare practices, HCC overlaps somewhat with HQO and CFHI. However, HQO focuses mainly on providing information to health service providers and the Final Report 18

22 Ontario Ministry, while CFHI focuses mainly on implementing innovative practices by working directly with delivery organizations. The other three comparison organizations do not overlap significantly with HCC in this area. This situation is illustrated in the following diagram: HQO HCC CFHI Following is the detailed analysis of the comparative review information, including the analysis of operational and reporting similarities and differences. Canadian Institute for Health Information (CIHI) Provision of information on the performance of the Canadian healthcare system This is CIHI s main area of activity. HCC is more policy driven than CIHI. In the continuum of data to knowledge to action, CIHI occupies more of the data to knowledge space, while HCC occupies more of the knowledge to action space. Other similarities and differences: Primary audiences are similar (the general public is a new audience for CIHI); CIHI probably does more consultation than HCC to identify information requirements of their primary audiences; CIHI puts more resources into information dissemination, because they have far more resources than HCC in particular, they put a lot of effort into media coverage; also, lots of face-to-face meetings with the provinces. Provision of information on innovative healthcare practices CIHI is not very active in this area (does some minor work in this area by way of identifying best practices they come across in other activities). Final Report 19

23 Canadian Foundation for Healthcare Improvement (CFHI) Provision of information on the performance of the Canadian healthcare system This is not a major area of activity for CFHI. They don t do systematic reporting like HCC. Provision of information on innovative healthcare practices This is CFHI s main area of activity not just providing information, but also supporting the adoption of innovative practices. CFHI is much more implementation focused than HCC; works directly with delivery organizations (all CFHI activities have an improvement coach ); CFHI concentrates on providing information on the how as well as the what. Other similarities and differences: The primary audiences are similar, except that CFHI does not target the public and places more emphasis on delivery organizations; HCC prints much more hard copy material than CFHI; With regard to dissemination, CFHI s work is targeted to working closely with managers and front-line professionals. Health Quality Ontario (HQO) Provision of information on the performance of the Canadian healthcare system HQO is very active in this area, but is focused on Ontario. Other similarities and differences: No major differences in the primary audiences (HCC has more emphasis on the public and less on health service providers); Methods used to identify the information requirements of primary audiences are similar (both are guided by framework documents the Accord and HQO s strategic plan); Information dissemination methods are similar lots of emphasis on print. Provision of information on innovative healthcare practices HQO is active in this area, but is primarily focused on health service providers (more so than HCC) and the Ontario Ministry of Health. To identify the information needs of primary audiences HQO puts lots of emphasis on the information needs of the Ontario Ministry of Health; Other similarities and differences: Information dissemination methods are similar. Final Report 20

24 New Brunswick Health Council (NBHC) Provision of information on the performance of the Canadian healthcare system NBHC is active in this area, but the activities are minimal by comparison with HCC (small budget, small organization) mainly a health system report card and some experience surveys; and their activities are limited to New Brunswick. Other similarities and differences: Primary audiences are very similar; For the identification of information requirements of primary audiences, NBHC probably relies more on stakeholder involvement than HCC; Dissemination methods are similar, although NBHC has more face-to-face interactions with their primary audiences. Provision of information on innovative healthcare practices NBHC is not active in this area. Institute for Health Services and Policy Research (IHSPR) Provision of information on the performance of the Canadian healthcare system IHSPR is not very active in this area, except that some individual researchers do research on how to measure the performance of the healthcare system. Other similarities and differences: IHSPR is a totally different model a research funding organization, and, therefore, not really comparable to HCC. Provision of information on innovative healthcare practices ISHPR does fund considerable research on innovative healthcare practices. Other similarities and differences: The primary audiences overlap, but IHSPR does not have the general public as a primary audience; The methods used to identify the information needs of the primary audiences are very different IHSPR relies on partnerships between researchers and users; Likewise, the dissemination methods are different IHSPR relies on the direct involvement of users in the research. Following are some quotes made by the comparison organizations during the interviews, one quote from each of the five organizations: The Health Council is the only organization that reports on health system performance in a systematic, pan-canadian, qualitative way [i.e., doesn t just report the data. [Also] some Health Council reports on innovative practices have been landmark documents. The Health Council s policy-oriented reports on pan-canadian health system performance are unique. Final Report 21

25 Results from the survey The Health Council is the only agency that reports on a pan-canadian level on the theme areas of the Accord. [Also] certain innovative practices reports of the Health Council would not have been done by other organizations. The Health Council provides a synthesized, independent, and objective national perspective on the healthcare system. The Health Council provides a national perspective on health system performance insights across Canada that nobody else has. The respondents were asked: Are there needs in the Canadian healthcare reporting and analysis landscape that the Health Council responds to that are not being addressed by other organizations? Their responses were as follows (n=372): 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 30.9% 35.2% 18.0% 5.9% 9.9% 84.1% of survey respondents believe the Health Council responds to moderately important, important or very important needs not being addressed by other organizations. No Yes, moderately important needs Yes, very important needs Yes, minimally important needs Yes, important needs Specific reports and analyses that were identified by survey respondents as important included: Reports related to Aboriginal health Progress reports on healthcare renewal Reports utilizing Commonwealth Fund data Cross-jurisdictional and pan-canadian focused reports Results from the interviews and case studies The interview respondents (government and knowledgeable health organizations) saw HCC s national scope and perspective as significant. 14 In addition, several of the case study interviewees identified instances where the HCC information would not have been available from other sources. 14 Four interviewees, out of 23, felt there was some overlap with other organizations in both areas reporting on healthcare system performance and innovative practices but we did not consider this small number of responses to be significant. Final Report 22