An Evaluation of the Yukon Research Centre FINAL REPORT

Transcription

1 An Evaluation of the Yukon Research Centre FINAL REPORT

2

3 Table of Contents Executive Summary 1 Introduction 3 Background 3 Scope of the Evaluation 4 Methodology 5 Context for the Evaluation 5 A Timeline for YRC 6 Charting Recent Growth 7 Current Organization 8 Evaluation Findings 9 Consistency with Funding Agreements 9 YRC Strategic Planning 9 A Research Policy Context 10 Reduced Reliance on YG Funding 11 Future Evaluation of the YRC 12 Program Design 13 Overview 13 Scope of the CCI Program 14 Governance and Structure 18 The Governance System 18 YRC Structure 20 Organizational Cohesion 21 Implementation of Strategic Plan 22 CCI Proposal Process 23 Project Monitoring 24 Project Objective Achievement 25

4 An Evaluation of the Yukon Research Centre - FINAL REPORT Long-term Goal Achievement 26 Stakeholders Views of YRC 28 Stakeholders Comments 28 Project Application Process - Revisited 29 Possible Improvements 30 An Effective Organization 30 Organization Sustainability 31 Conclusion 32 Appendices Stakeholder Survey Summary Report Third Party Funding Detail ReSDA Organization CCI Project Lifecycle Chart File Review Summary Report Case Studies: 69 Landscape Hazards 69 Hydro Security 70 PlastoVac 71 BioChar Towards an Evaluation System for YRC 75 Acronyms ARIF - Arctic Research Infrastructure Fund CA - Contribution Agreement CCI - Cold Climate Innovation C-CIARN - Climate Change Impacts and Adaptation Research Network DED - Department of Economic Development IRC - Industrial Research Chair NCE - Northern Climate Exchange NSERC - Natural Sciences and Engineering Research Council PEO Hub - Public Education and Outreach Hub ReSDA - Resources and Sustainable Development in the Arctic SERNNoCa - Social Sciences Research Network in the North SSHRC - Social Sciences and Humanities Research Council TI - Technology Innovation YC - Yukon College YG - Yukon Government YRC - Yukon Research Centre

5 Executive Summary The Yukon Government substantially increased its support for the former Northern Research Institute through two incremental core funding agreements: An agreement with the Department of Education that pays specified salaries and operating costs; and Contribution agreements with the Department of Economic Development (DED) to fund the Yukon Cold Climate Innovation Centre (YCCIC, now CCI). Beginning in 2010, the expanded operations included the appointment of a Vice-President, Research and other key staff organized under the banner of Yukon Research Centre. After some three years of experience with the new funding agreements, the YG requested a formal evaluation of the Centre to assess compliance and inform future funding processes. This project represents the first-ever external evaluation of the YRC or any of its programs. The overall picture that emerges from this start-up period is positive. YRC not only complies with but actively pursues the dual mandates provided by the funding agreements. Most informants expressed the view that YRC has met or exceeded most expectations for the short term, including start-up organization; launching an ambitious research agenda; and securing outside funding. While the process has not been totally free of difficulty, and there is room for improvement in certain areas, transition to the new YRC has been remarkably effective. YRC s organization and governance are still evolving, which is an appropriate state in light of Yukon s dynamic and growing research environment. The actions of YRC leaders to date indicate that the Centre is capable of adapting to and sustaining itself within this environment in the foreseeable future. This evaluation produced the following findings and conclusions with respect to the evaluation issues: YRC goals and strategy, as defined in its current draft strategic plan, are highly consistent with and supportive of the purposes of the core funding agreements. While the design of most YRC programs appears appropriate, the vision set out for CCI under the Economic Development Contribution Agreement is inconsistent with both YRC capacity and the logic of the innovation process. The governing system, primarily in the form of the YRC Oversight Committee established in 2010, is confusing and requires re-thinking. Although not directly involved in governance, the CCI Advisory Council operates substantially as intended. YRC has met or exceeded most short-term expectations in successfully implementing its strategic plan. Much remains to be done with regard to moving towards achievement of some longer-term goals. 1

6 An Evaluation of the Yukon Research Centre - FINAL REPORT The CCI proposal application and review process is well-documented and transparent. Some clients, however, reported problems with delays and conditions imposed by external funding agencies within the application process. Although most project proponents maintain that their projects are well-monitored, the majority of project files contain scant evidence of monitoring activity. Multiple sources of feedback confirm that YRC projects have been markedly successful in achieving their objectives. Although some progress can be seen in achievement of all goals, most require considerably more activity and attention to attain success over the longer term. The vast majority of feedback from stakeholders was positive, as documented throughout this report, although a number of areas were suggested for improvement. Summary of Recommendations 1. As part of its deliberations regarding disposition of the current YRC funding agreements, the Yukon Government may wish to consider establishing a broader policy context for research and innovation funding to complement the current work on a YG research agenda. [Page 17] 2. Re-assess the requirement for YRC to reduce reliance on these core funding agreements, and focus instead on the need to secure project or research-based funding from other sources. [Page 17] 3. Include in any future agreements a requirement as well as funding to prepare an evaluation plan during the first years of the agreement. [Page 17] 4. Consider revising both the DED Contribution Agreement and the YRC/CCI Strategic Plans to reflect a more discreet and specific role for the CCI program, namely that of (1) leading the cold climate research and development function; and (2) contributing to appropriate networks. Provide for a seamless transition from YRC support to other agency(s) or initiatives that have responsibility for the implementation/ commercialization dimension of innovation. [Page 23] 5. Consider carrying out formal evaluations of those YRC programs that have operated for more than a decade. [Page 23] 6. Re-assess the rationale for and operation of the YRC Oversight Committee with a view to re-constituting it to fulfill a defined purpose. An appropriate model would appear to be a President s advisory council charged with advising on research priorities, strategies, and coordination. [Page 27] 7. Identify and implement measures designed to overcome the organizational risks associated with the YRC dual mandate; and balance governance mechanisms between the two sides. [Page 27] 8. For application-driven programs: (1) prepare an information package for prospective applicants describing the process and success criteria; and (2) make this information available in website and paper form. [Page 31] 9. Implement procedures to ensure that appropriate documentation of project monitoring activities, such as status reports and relevant communications, appears in project files. [Page 31] 10. Identify ways to monitor implementation of the recent YC structural change that aligned the Extension and YRC Divisions, with the intent of assessing incremental achievement of the longer-term goals regarding community and First Nations engagement in research. [Page 35] 11. Include within a comprehensive YRC evaluation/monitoring plan mechanisms to regularly capture and report on stakeholder feedback. [Page 37] 2

7 Introduction Background Over the past few years, the Yukon Research Centre (YRC) has grown from the fledgling Northern Research Institute to become a nationally-recognized body in the field of northern research. The Centre a division within Yukon College has recently experienced significant expansion of its facilities, research activities, staff and budget. Among other developments, YRC built a new $2.7 million science laboratory with contributions from the Arctic Research Infrastructure Fund, part of Canada s Economic Action Plan; and successfully applied for funding from the Natural Sciences and Engineering Research Council of Canada (NSERC), Canada s federal funding agency for university- and college-based research. Until about 2009, the Northern Research Institute (NRI) carried out relatively small amounts of research. With a full-time staff of about eight people and annual budgets around $1 million, NRI activities consisted mainly of delivery or coordination of such programs/projects as: Northern Climate Exchange (NCE) Technology Innovation (TI) Social Sciences Research Network in the North (SERNNoCa) Science Adventures (formerly Innovation in the Schools) Prompted in part by its Climate Change Strategy, the Yukon Government (YG) in fiscal substantially increased its support through two incremental core funding agreements: An agreement with the Department of Education that pays specified salaries and operating costs; and Contribution agreements with the Department of Economic Development (DED) to fund the Yukon Cold Climate Innovation Centre (YCCIC, now CCI) in addition to TI, for which funding began in The expanded operations were launched in 2010 with the appointment of a Vice-President, Research and subsequent appointments of other key staff. Both the legacy and the new programs were organized under the banner of Yukon Research Centre (the term of Excellence was subsequently omitted from the title). After some three years of YRC operating under the new funding agreements, the YG requested a formal evaluation of the Centre. The YG will soon consider whether to enter into additional 3

8 An Evaluation of the Yukon Research Centre - FINAL REPORT agreements to fund the YRC and wanted an evaluation to inform this process. This project represents the first-ever external evaluation of the YRC or any of its programs. Through a public tendering process, Yukon College contracted Malloch Graham + Associates to assist in planning and conducting the evaluation. The project was launched with preparation of an Evaluation Framework 1 or plan that describes the scope of the evaluation, the main questions to be addressed, and methods to be used in researching the issues. Key portions of the Framework are repeated here to enhance understanding of the evaluative process. This document is the final report of findings relevant to each of the evaluation questions. Scope of the Evaluation The formal purpose of the evaluation which focuses on the last three years of activity is to: Satisfy the Yukon Government s directive to evaluate YRC programming in order to inform consideration of future funding arrangements Assess the implementation, progress and results of programs, and compliance with the funding arrangements with the Yukon Government, over the past several years Prepare an evaluation plan suitable to enable ongoing monitoring and periodic assessment of YRC performance Generate baseline data to be used in comparing and assessing future YRC performance Develop evidence to assist Yukon College/YRC decision-making related to ongoing resourcing, organization and delivery of research programming. The scope of the evaluation is established by the formal evaluation questions set by the project Steering Committee: 1. Are YRC goals and strategy consistent with core funding agreements? 2. Is program design appropriate? 3. Is the governing structure effective? 4. How effectively has YRC implemented its strategic plan? 5. Are CCI processes used to evaluate proposals effective? Transparent? 6. Are projects monitored effectively? 7. Were project objectives achieved? 8. Is YRC achieving its longer-term goals? 9. What do stakeholders think about the program? 10. How could the program be improved? These questions cover a wide range of issues, from documenting implementation of the program to assessing long-term outcomes. The scope reflects investigation and analyses relating to three levels of inquiry, namely at the: Individual project level (although not related to the technical content of projects) Thematic or major funding level, i.e. the two core funding agreements Overall YRC in its capacity as a division of Yukon College. 1. Yukon Research Centre Evaluation Framework - Final. May Malloch Graham + Associates. 4

9 An Evaluation of the Yukon Research Centre - FINAL REPORT The Steering Committee specified that, in addition to the findings for each evaluative question, the review should include recommendations for improvement emanating from the evidence. The final report was intended to represent an independent appraisal of issues relevant to design, delivery and results of YRC programming, with a focus on the past three years of activity. Methodology The selected research methods were designed to generate a variety of both quantitative and qualitative information, and enable triangulation of feedback, i.e. comparisons of findings from more than one source. The methods used in the evaluation include: 1. Document Review Compiled and assessed relevant background information and documents, including planning studies, periodic status reports, project monitoring data, financial and other internal reports, and so on 2. Project File Review Review of a directed, representative sample of 21 (a 33% sample) research project files to document project reporting and outcomes 3. Survey Online survey of staff and managers; researchers and project proponents; YC managers and faculty; advisory committee members; sponsors and partners; users of research; YG departments and agencies; municipalities; First Nations; and other informants familiar with the work of YRC. Invitations were sent to 165 candidates and 63 surveys were completed (a 38% response rate, which is high for a survey of this type). Results are shown in Appendix Interviews Individual or group interviews with 24 informants (identified by the Steering Committee) including senior managers, major funding agencies and partners, researchers and sponsors. Planned focus groups were not carried out because the logistics of gathering candidates during the peak field/summer holiday season proved too difficult. Instead, we conducted additional interviews with targeted informants in which issues planned for focus group were discussed. 5. Case Studies Four case studies of representative research projects were prepared as a means of illustrating how research is carried out and applied within the YRC; these are presented in Appendix 6. A matrix that relates research methods to each evaluation question is displayed in the Evaluation Framework. Context for the Evaluation As noted previously, the two current YG core funding agreements were launched in fiscal The increased funding provided for immediate additions to YRC operating budgets and staff, who subsequently secured access to national research funding sources such as NSERC. Most significantly, the increased resources have generated greater numbers, variety and scope of research and innovation projects, which form the heart of this or any such institution. The past three years essentially represent a start-up period of incremental growth in which YRC sought to establish itself as a credible, viable centre of northern research. This period is the focus of the evaluation, which is oriented to determining the extent to which YRC: Is complying with the terms and conditions of these two major funding agreements Has implemented the necessary strategies, systems and supports to facilitate effective management of its projects 5

10 An Evaluation of the Yukon Research Centre - FINAL REPORT Has established or is establishing the appropriate mix of atmosphere, supports, facilities and systems to enable achievement of its mandate and goals Is achieving its research and innovation goals. These issues largely reflect a formative evaluation, which is consistent with a first evaluation of a program that was for all intents and purposes launched three years ago. Investigation and analysis revolves around the achievements and challenges encountered during the start-up period. Consideration of summative issues primarily around the achievement of research or institutional goals is limited in that this relatively short period of activity is not capable of yielding definitive long-term results. Instead, the evaluation focuses on whether: YRC is providing the necessary inputs with appropriate reach and intensity to achieve the desired outcomes over the longer term; and The necessary measures and data collection systems are in place to enable timely assessment of those outcomes and impacts. In this context, the evaluation seeks to (1) document and assess the implementation of programming as envisioned in the two YG funding agreements; and (2) determine whether YRC outcomes are evaluable over the longer term. In the latter situation, the ultimate aim is to identify what data or systems need to be in place to conduct a summative evaluation in future, recognizing that the detailed work of planning such a system is beyond the scope of this evaluation. A Timeline for YRC As background to the evaluation, it is useful to be aware of the evolution of the YRC. The following timeline notes key events leading to its current status Establishment of the Whitehorse Vocational Training Centre (predecessor to Yukon College) 1988 Formal establishment of Yukon College by the College Act 1989 Establishment of Northern Research Institute at Yukon College 1992 Establishment of Innovators In the Schools at Yukon College 1998 Establishment of Biodiversity Monitoring program at Yukon College 1999 Establishment of Yukon Technology Innovation Centre at Yukon College 2000 Establishment of Northern Climate ExChange at Yukon College 2002 Establishment of Climate Change Impacts and Adaptation Research Network (C-CIARN) at YC 2002 Establishment of the Public Education and Outreach Hub (PEO Hub) at Yukon College 2007 Establishment of SERNNoCA Program (SSHRC funded) 2008 Establishment of Cold Climate Innovation Centre (CCI) at Yukon College 2009 Yukon College gains SSHRC eligibility 2009 Establishment of Yukon Research Centre (YRC) at Yukon College 2009 Successful Arctic Research Infrastructure Fund grant for new YRC lab and cold storage area 2011 Yukon College gains NSERC eligibility 2012 Establishment of ReSDA Program (SSHRC funded) 2012 Idea to Innovation (I2I) NSERC grant with University of Saskatchewan established 2012 Yukon College publishes Protocols and Principles for Conducting Research with Yukon First Nations 2012 Industrial Research Chair in Mining Life Cycles established (funded by NSERC) 6

11 An Evaluation of the Yukon Research Centre - FINAL REPORT Charting Recent Growth YRC has grown substantially over the past five years in virtually every category of measurement: budgets, staff, facilities, and projects under management. Revenues/operating funds have increased from around $1.2 million in to the current level of about $4.7 million annually: Table 1 YRC Third Party Revenues (Fiscal Year: 01 July 30 June) Fiscal Year Third Party Funding from: YG Education YG Ec Dev Other Sources* Total , ,295 1,860,444 2,499, ,221,225 1,335,660 4,906,238 7,463, ,033,439 1,237,641 1,262,034 3,533, ,091, ,189 2,627,375 4,692,282 *A detailed breakdown of Other Sources is presented in Appendix 2. Total $18,188,259 As of June 2013, about 41 people were working in the YRC office: 17 permanent positions; about 12 students, most doing seasonal work; and 12 researchers/contractors who are not staff. YRC offices are located in the lower level of the Yukon College student residence building and include 31 desks; an application is underway to convert adjacent vacant space to additional offices. Other facilities include the new laboratory, a cold storage building, researcher residence, and a fenced compound for equipment and several research facilities such as a greenhouse and a building insulation testing structure. YC is currently considering the acquisition and development of the former Experimental Farm site near Haines Junction, which could become some combination of research station and/or community campus. As documented in the Evaluation Framework, YRC completed 24 projects since 2011, and had 34 projects underway as of March 2013: Table 2 YRC Projects Summary Program Area Northern Discovery Current/ Ongoing Recently Completed Biodiversity Monitoring 5 - Northern Climate ExChange 5 5 ReSDA & Miscellaneous 4 3 Science Adventures 4 2 Northern Innovation 7

12 An Evaluation of the Yukon Research Centre - FINAL REPORT Program Area Current/ Ongoing Recently Completed Cold Climate Innovation 10 5 Technology Innovation 6 9 Total In summary, growth has occurred since 2010 in the following key areas of measurement: Measure Third party funding $3.5 mm $4.7 mm Staff (Person-Years) 6 17 Projects underway Projects recently completed N/A 24 Current Organization The organization of the YRC in early 2013 was documented in the Evaluation Framework. In July 2013, Yukon College re-organized to place its Extension Division (which includes all community campuses) under the purview of the VP, Research, as shown in Table 3 below. Table 3 YRC Organization Chart Vice President Research & Community Engagement Director Research Services Director Cold Climate Innovation Industrial Research Chair Dean Extension 8

13 Evaluation Findings EQ1. Are YRC goals and strategy consistent with core funding agreements? Conclusion: YRC goals and strategy, as defined in its current draft strategic plan, are highly consistent with and supportive of the purposes of the core funding agreements. YRC Strategic Planning Strategic planning for YRC got underway in late 2010 following appointment of its Vice-President and senior managers. It took about 18 months to work through the process and the initial strategy was approved in May Two stakeholder meetings were held, in December 2010 and February 2011, that served to articulate strategic directions and most of the goals. In addition, a communication plan and a branding exercise were carried out in In concert with Yukon College strategic planning, YRC in late 2012 embarked on an update (Strategic Plan II) of this plan; it was in final draft form as this report was written. Among other refinements, this version excludes the operational detail of V.1 and focuses on strategic initiatives and success indicators. The contents of YRC Strategic Plan II are highly compatible with the two funding agreements. Indeed, portions of the strategy appear to be formed with funding agreement provisions in mind; for example: Strong emphasis on collaboration with industry, First Nations, and other stakeholders A goal to develop and commercialize technology innovations and export them Several initiatives aimed at involving Yukon students (both College and K-12) in research A specific goal to increase and diversify funding beyond YG sources. Both the strategic plan and discussions with YRC leaders indicate that the Centre is strongly oriented to fulfilling the purposes defined by the funding agencies. In the case of the Education agreement, the purpose is general and fulfillment is straightforward. The Economic Development agreement, in contrast, contains a series of specific conditions and expectations, some of which may prove challenging to fulfill (a notable example is pursuit of the R-I-C process, discussed under EQ2). Nevertheless, the stated intentions and strategies of YRC are entirely consistent with Agreement purposes. 9

14 An Evaluation of the Yukon Research Centre - FINAL REPORT A Research Policy Context The two YG core funding agreements relevant to YRC have in common a focus on climate change research and cold climate innovation, but their mandates, orientation and content are quite different: Department of Education funding is by means of a Transfer Payment Agreement whose purpose is broadly defined: YRC will focus on research around climate change as well as climate adaptation in Yukon. The centre will serve to expand Yukon s research capability and knowledge on northern climate change and to focus on the multi-disciplinary issues related to climate change adaptation. Beyond this statement, the Agreement is essentially silent on such program content issues as research priorities, decision-making, or stakeholder involvement. Department of Economic Development (DED) funding is through a Contribution Agreement (CA) that sets out quite specific purposes for establishing and supporting the YCCIC (now CCI): The purpose of the YCCIC is to enable the development of Yukon technological innovations, particularly those related to Yukon experience towards the commercialization of those innovations to generate private sector wealth and thereby diversify and grow the Yukon economy. The CA goes on to state that the purpose of the YCCIC is to foster partnerships to achieve a number of stated objectives. It also sets out quite detailed terms and conditions regarding strategic planning; establishing an Advisory Council; and a plan for reducing the YCCIC s reliance on YG funding for core operational activities. The mandates and approaches of the two Agreements are markedly different. While Education approaches YRC in a hands-off manner similar to what it takes with other Yukon College funding agreements, DED takes a more prescriptive stance in that the CA: Establishes research priorities ( research-innovation-commercialization or R-I-C) Requires a program-specific strategic plan that includes identifying measurable deliverables and performance indicators Specifies industry involvement Establishes an Advisory Committee that undertakes a strategic role in providing guidance and oversight, co-chaired by an Economic Development ADM. CCI is an important component of Economic Development s broader R-I-C strategy. In contrast, the northern climate change research funded by Education has a relatively narrow connection to Education goals (primarily around the knowledge exchange between research and education), and YRC is seen as an entity distinctly separate and apart from the Department. Furthermore, some provisions of the two Agreements are at opposite ends of a spectrum: e.g., while dissemination of research is expected under Education programming, full technical disclosure may be delayed under the R-I-C process when proprietary products or technology are involved. Neither approach is necessarily superior, nor is the situation problematic in the short term. Indeed, this evaluation finds that implementation of the two Agreements despite their marked differences has met or exceeded most expectations. The situation does, however, raise questions around the longer-term implications of the funding approach. Since development of a formal Yukon Government research agenda is still underway, the Agreements become policy instruments that establish Yukon s current (and perhaps sole) research priorities as that of climate change research and cold climate innovation. In considering its longer-term funding support for YRC, the Yukon Government may also want to consider: Where cold climate research and innovation fits within the spectrum of Yukon research priorities including such fields as social, humanities, housing, or biological research, where possible research is of a multidisciplinary nature 10

15 An Evaluation of the Yukon Research Centre - FINAL REPORT What role YRC could or should play in the delivery of other fields of research (for example, is the current YRC strategic objective to Develop social science research capacity within Yukon College compatible with YG s research funding approach?) Alternate frameworks for supporting YRC and research and innovation in general, apart from the disparate approaches represented by the two existing funding agreements. It is not the purpose of this evaluation to suggest what this broader research and innovation agenda might look like (although we received numerous comments and questions from stakeholders on this topic through interviews and the online survey see Appendix 1). There is no indication that the YG s purpose is to direct YRC research and innovation activities, although a clear YG research agenda would inform the work of YRC and contribute to its strategies. The recent experience of these Agreements while largely positive raises the above and other questions about the efficacy of the YG funding approach over the longer term. Reducing Reliance on YG Funding Both funding agreements provide monies for projects as well as non-project costs. Both also express the need for YRC to reduce its reliance on Yukon Government funding for core operational activities. Although core operational activities are not defined, the term is understood to mean the administrative or non-project costs such as salaries, space, supplies and equipment that are noted within each agreement. While desirable on the surface, this move to reduce YG core funding may be counterproductive, for several reasons. The measure implies that YRC (and by extension, Yukon College) could and should generate new sources of revenue that will cover core operational activities. These are distinct from funding for research/innovation projects and infrastructure; YRC has successfully pursued such funding and should of course continue to do so. Virtually all funding sources available to YRC, however, operate on a cost recovery basis, which usually covers only a small portion of core operating costs. This approach typically leaves a gap in funding that is difficult to cover. The current YG funding arrangement that covers most core operating costs is an important factor in YRC s ability to attract a wide variety of research funding and partners. Current policy is that only 5-15% of research project funds are allocated to overhead and support services; the balance is spent directly on research. In some institutions, that figure is as high as 50%. This is an attractive feature to many private sector partners and research funding agencies. It has helped YRC lever additional research funding in recent years, and there is considerable potential to lever even more. In other words, the existence of core funding enables fund-raising; removing core funding would inhibit YRC s efforts. A final factor is simply that of resource allocation: time spent by staff on such core fundraising is time not spent on pursuing YRC goals. The fact that core operational activities is not a particularly appealing target for fund-raising makes the proposition challenging; it would no doubt be a time-consuming exercise. 11

16 An Evaluation of the Yukon Research Centre - FINAL REPORT Future Evaluation of the YRC Both YG funding agreements contain clauses calling for their evaluation, which in turn prompted this evaluation study. As noted previously, the first phase of this study was preparation of an evaluation plan that was completed in conjunction with funding agreement representatives and YRC leaders. This plan identifies measures and data sources that contribute a great deal towards the goal of a comprehensive performance measurement system, while focusing on the shortmedium term of the past three years. It would be more effective in any subsequent funding agreements if this planning was carried out early, during the first years of the agreement. This would enable the parties to build on this initial Framework by: Developing more specific measures and indicators of longer-term performance Identifying what additional data should be collected within YRC to inform outcome evaluation Preparing a monitoring program to inform ongoing YRC performance and support internal decision-making. To help guide and inform further development of evaluation mechanisms within YRC, Appendix 7 offers additional detail and suggestions regarding priorities over the next five years. Recommendations 1. As part of its deliberations regarding disposition of the current YRC funding agreements, the Yukon Government may wish to consider establishing a broader policy context for research and innovation funding to complement the current work on a YG research agenda. 2. Re-assess the requirement for YRC to reduce reliance on these core funding agreements, and focus instead on the need to secure project or research-based funding from other sources. 3. Include in any future agreements a requirement as well as funding to prepare an evaluation plan during the first years of the agreement 12

17 An Evaluation of the Yukon Research Centre - FINAL REPORT EQ2. Is program design appropriate? Conclusion: While the design of most YRC programs appears appropriate, the broad range of activity set out for CCI under the Economic Development Contribution Agreement is inconsistent with both YRC capacity and the logic of the innovation process. Overview The current YRC program line-up evolved over the initial years in which the two YG funding agreements were being implemented. As reflected in its organization chart (Table 3) and described in the Evaluation Framework, YRC operates six research units or programs and carries out a series of administrative and support functions, grouped within three thematic areas as follows: Table 4 YRC Program Overview Thematic Areas: Management & Research Services Programs/Functions: YRC management and oversight Research support services Facility management Sponsorship relations and external affairs Communications Northern Discoveries Northern Innovation Northern Climate ExChange (NCE) Resources and Sustainable Development in the Arctic (ReSDA) Cold Climate Innovation (CCI) Technology Innovation (TI) The program groupings reflect the dual mandate of YRC as reflected in the YG core funding agreements: climate change research (Northern Discoveries) and cold climate innovation (Northern Innovation). In addition, this program structure reflects a fundamental difference in financial orientation in that Northern Discoveries is primarily in the business of soliciting research funding from third party sources, whereas Northern Innovation is oriented to providing funding for suitable research and development projects. Science Adventures, NCE and Biodiversity Monitoring are legacy programs that have been operating since 2000 or previously; ReSDA was launched in The three mature programs have expanded their activities to varying degrees as the result of the Education (and other) funding. It is notable that, despite their longevity, none of the three has ever been formally evaluated. Nevertheless, the program line-up makes sense and no questions arose from any source during this review around their individual designs. 13

18 An Evaluation of the Yukon Research Centre - FINAL REPORT Within Northern Innovation, only Technology Innovation (TI) is a legacy program, launched in The Cold Climate Innovation (CCI) program is new, launched in 2010 with Economic Development funding. The two offerings have parallel purposes; both offer financial and technical assistance to approved projects; and they employ a common application/approval process. These factors suggest they should be treated as one program with two streams, rather than two distinct programs. Start-up operations and experience over the past three years have provided a good testing of CCI design and strategy. In the DED Contribution Agreement, the stated vision for CCI is: to become nationally and internationally recognized for the development, commercialization, and export of sustainable cold climate technologies and related solutions for sub-arctic regions around the world in order to support the development, enhancement and success of Yukon s technology industries. The scope of such a program when fully implemented would be extremely broad. Activities aimed at the...development, commercialization, and export of technologies are not specified in the CA, and would seem well beyond the capacity of a relatively small, college-based research centre. The vision statement also raises the question of what is an appropriate scope for YRC/CCI programming in fulfilling or contributing to the mandate of cold climate innovation. Scope of the CCI Program None of the following discussion is intended to question the YG s intention to support cold climate innovation. Rather, it calls for re-examining the innovation process and bringing greater focus and clarity to the specific role that YRC/CCI can and should play within that process. Such thinking is particularly important to the development of performance measures, as required in the CA: measures should not be based solely on activities (inputs); they should also reflect desired outputs and outcomes, and these are not defined in the current CA. This is a common state of affairs with new programs and is not a criticism of CCI programming, but it is an area that requires attention as the program evolves. Because the CA does not define such terms as research, innovation, commercialization (or others), the expected scope of CCI programming is somewhat open to interpretation. In the absence of alternate definitions, we have assumed common, standard definitions of the key terms. The R-I-C process cited by DED is commonly referred to simply as innovation. It is a wideranging endeavor pursued by industry which plays the central role and supported by most levels of government worldwide. A recent study prepared by ASTRA The Alliance for Science and Technology Research in America - for the US Department of Commerce 2 suggested this definition: Innovation is a process by which value is created for customers through public and private organizations that transform new knowledge and technologies into profitable products and services for national and global markets. A high rate of innovation in turn contributes to more intellectual capital, market creation, economic growth, job creation, wealth, and higher standard of living. 2. Innovation Vital Signs Project Final Report; July Prepared for: Technology Administration, US Department of Commerce. Prepared by: ASTRA - The Alliance for Science and Technology Research in America. 14

19 An Evaluation of the Yukon Research Centre - FINAL REPORT Innovation is a dynamic, complex and challenging quest. The above-noted study (referred to as the IVS Project) set out the following framework to help define the scope of and factors influencing innovation, and examined methods for measuring its inputs, processes, outputs and impacts. In principle, the process will be similar whether practiced nationally in the US or locally in the Yukon. The framework (which DED generally subscribes to) does not suggest specific roles for public agencies, but is useful for considering where CCI could and should fit in the process, and what types of activities the program should undertake. Table 5 Innovation Framework R&D TALENT CAPITAL NETWORKS Public Policies Innovation Framework Major Subsystems and Linkages INNOVATION PROCESSES AND MODELS Macro Economic Conditions VALUE MARKET DEMAND Infrastructure National Mindset The key activity and core skill set of CCI is research and development (R&D) one of four main inputs to innovation. CCI may also contribute to networks, another input; however, it has little or no influence on or connection to the remaining two inputs, talent and capital. Note that Public Policies and Infrastructure are two other major influencers of the process and neither of these is within the purview of CCI. The IVS Project goes on to describe the Implementation (Innovation Process and Innovating Enterprises) dimension of innovation: The general implementation process consists of market definition, design, engineering, production, marketing, distribution, and support phases. These activities can be viewed as linear steps, but the reality in most cases is much more complex. For each phase of the process there are numerous sub-processes, both internal and external to the enterprise, involving feedback loops and the coupling of each activity to downstream and upstream phases. Technical and economic problems that are uncovered in the development process often generate demand for additional research in engineering and even fundamental science Ibid. Page 9. 15

20 An Evaluation of the Yukon Research Centre - FINAL REPORT One can envision CCI providing support for some of these functions, e.g. research and engineering. Indeed, that has been the focus of their activities to date. The Centre, however, is not positioned or equipped to support such other functions as market definition, design, production, marketing, or distribution or such related aspects as business start-ups and financing. Yet all of these functions and more would be required to fulfill the CA s call for...development, commercialization, and export of technologies Within the Yukon, there is a distinct lack of many supports critical to innovation, among them design, engineering, production facilities, and financing. It is unrealistic to expect CCI to provide or even contribute significantly to such wide-ranging support functions, particularly with the limited Outside of a bit of R&D, there is a vacuum of support systems for innovation in the Yukon. resources provided under the current CA. Rather, the funding agreement as well as CCI strategy should focus on the two inputs to innovation that are consistent with the focus of YRC, namely R&D (where YRC can plan a lead role); and networking (a contributing role). It is, however, impracticable and even naive to expect YRC/CCI to play a lead role in the implementation of innovative products or technologies, i.e. the commercialization, and export of technologies under its current make-up. A key consideration is what outputs and outcomes CCI should be held accountable for producing; these are not defined in the CA. It is reasonable and appropriate that these should include outputs related to R&D and networking (Appendix 7 offers a few suggestions). It is not reasonable that they should include creating new businesses or exporting new technologies; there are simply too many external factors affecting these outputs to hold CCI accountable for them. At best, CCI may contribute to such achievements through its R&D and networking activities, as part of a multistage process involving many participants. DED and other public agencies provide some supports for industry as it seeks to commercialize R&D and/or export technology outside the Territory. When considering the CCI role in these endeavours, the focus should be on providing for a seamless transition from YRC/CCI to such initiatives. In practice, this hand-off has apparently occurred in a few projects. This experience should help the parties better define the process in both the CA and program strategies. As stated previously, a fundamental principle of innovation is that it must be led by industry. YRC can provide valuable input to and support for cold climate innovation in terms of R&D and networking; it cannot and should not be expected to lead other elements of the innovation process. As the IVS Project found: Entrepreneurs and innovating enterprises are the prime agents for transforming knowledge and commercializing products, services and processes. Our new understanding of innovation, however, rejects the idea that innovation simply flows from some earlier process of scientific discovery. Innovation is not just a linear process that unidirectionally proceeds from science to the enterprise and then the marketplace. The framework here goes beyond knowledge creation (invention) and emphasizes the many additional factors that drive the transformation of knowledge into useful products and services and value for society Innovation Vital Signs Framework Report An Update. June Page 9. 16

21 An Evaluation of the Yukon Research Centre - FINAL REPORT In re-examining the scope of and strategies for CCI programming, the parties should: Agree on the logic of the program and the innovation process: inputs, activities, outputs and outcomes (the YRC logic model in the Evaluation Framework provides a good starting point) Define such key terms as innovation and commercialization, particularly if the concept being used is different from the common understanding of these terms Determine a clear demarcation point for CCI support activities that enables YRC to focus on R&D and networking as inputs to the innovation process Adjust the YRC Strategic Plan II (specifically Long Term Goal #2: Develop and commercialize technology innovations and export them within and beyond Yukon ) as well as the CCI Strategic Plan to reflect the new focus Develop indicators of performance (as required in the CA and included in Strategic Plan II) that are appropriate to this revised program scope these should NOT include, for example, commercialization or export activity, or number of business start-ups; rather, they should focus on such aspects as: -- informing and assisting target clientele -- achievement of R&D project objectives -- delivering clear and relevant research results -- providing an efficient transition for the client to a subsequent phase of activity. Additional suggestions for measuring innovation are put forward in Appendix 7. Recommendations 4. Consider revising both the DED Contribution Agreement and the YRC/CCI Strategic Plans to reflect a more discreet and specific role for the CCI program, namely that of: (1) leading the cold climate research and development function; and (2) contributing to appropriate networks. Provide for a seamless transition from YRC support to other agency(s) or initiatives that have responsibility for the implementation/commercialization dimension of innovation. 5. Consider carrying out formal evaluations of those YRC programs that have operated for more than a decade 17

22 An Evaluation of the Yukon Research Centre - FINAL REPORT EQ3. Is the governing structure effective? Conclusion: The governing system, primarily in the form of the YRC Oversight Committee established in 2010, is confusing and requires re-thinking. Although not directly involved in governance, the CCI Advisory Council operates substantially as intended. The Governance System The governance system for YRC was initially established in 2010 by the (previous) YC President. Its purpose was not entirely clear, other than the fact it reflected the two new YG funding agreements. Each of the two program streams was to feature a designated group with responsibility for oversight and advice on programming. These were originally designated Governing Councils although neither group has fulfilled that role as modelled elsewhere in YC. 5 Over the past three years, the two groups have evolved to become quite different in their composition and purpose, as shown in Table 6 below. Table 6 Two YRC Councils Northern Discoveries Northern Innovation Title Oversight Committee CCI Advisory Council Composition Mandate Yukon College President ADM, Advanced Education (Advisor: Senior Science Advisor, ECO) Grand Chief, CYFN Chief Resources: Vice President, Research and Director, Research Services General/financial oversight of YRC Senior direction, policies & priorities Liaise with Board of Governors ADM, DED (Co-Chair) Vice President, Research (Co-Chair) Yukon College President CYFN appointee 4 private sector appointees Director, CCI Advisory body only; recently confirmed it has no governance role Advise on research project selection and management; strategy; priorities; finances; and networking Terms of Reference Meetings Minutes No written terms of reference exist; the Committee operates informally No set schedule; as required Last meeting Spring 2012 No formal minutes recorded; decisions are acted upon by individual members Prepared and distributed in June 2012 Planned quarterly; in practice 3 annually In person; some members teleconference Formal minutes taken & distributed Info package distributed in advance of meetings 5. YC features Governing Councils in the institutes of Social Justice, Mining Innovation, First Nation Initiatives, and School of Visual Art (SOVA). 18

23 An Evaluation of the Yukon Research Centre - FINAL REPORT The two groups perform distinct and different functions: The Oversight Committee is primarily a form of governance in that the group has implicit authority to make decisions, approve policies and procedures, and make recommendations to and receive direction from the YC Board of Governors. The CCI Advisory Council is one of many advisory bodies within YC that provide advice to senior managers on designated programs. None of these advisory bodies has a governance role (i.e. they have no authority to make decisions, nor are they held accountable for program performance); governance is the purview of the Board of Governors, senior YC leaders, and governing councils where appointed. Both groups perform functions important to YRC. The CCI Advisory Council not only offers wide-ranging advice that benefits the program, it also serves as a key and to this point, only formal networking mechanism with industry. While some private sector members of the Council argue that it should have some decision-making authority, such a move would be inconsistent with YC governance policy and several decades of precedents. The situation may be muddied by the CA s use of the term oversight in describing the Advisory Committee s purpose; this function is not normally associated with an advisory group. Furthermore, it is curious that the YC President is named as a member of this Advisory Council, although the designation may simply be held over from the original governing council concept that was never implemented. The usual protocol is that the President is an ex officio member of all YC advisory groups. The Oversight Committee is something of a hybrid group and its rationale is not entirely clear. Its composition suggests that it exists to carry out the various functions of: Governance: YC s President and Vice President, Research have decision-making authority, and also act as liaison with the Board of Governors Advice: Advanced Education may advise on research matters relevant to its funding agreement Networking: with First Nations, through the Grand Chief. The rationale for the governance function would appear to be that YRC is a new, expanded unit that is now the largest division within YC. Research is a distinctly different endeavour than traditional community college operations, and both senior YC managers and the Board of Governors need to learn more about research and YRC operations. YRC may require a degree of oversight beyond what the Board of Governors is able to provide. YRC goals include engaging YC faculty and students in research initiatives, and facilitating the exchange of research and educational knowledge. All of these factors can be addressed to some extent by a senior management group with widespread access and authority. A question arising from this Committee s membership is whether agencies outside YC in this case, Advanced Education and CYFN should be included in governance activities. These agencies may not wish to have authority, and be held accountable, for YRC operational decisions. At the same time, these and other members of the Yukon research community could contribute valuable advice in pursuit of YRC s goals re collaboration and integration of research. Apart from its governance function, this Committee s experience to date raises several questions: What is oversight and how is it different from governance? Is the ADM, Advanced Education the right position to advise on climate change research matters? Are networking and engagement with First Nations best achieved through the Grand Chief s participation on a committee, whatever its seniority and authority? Can any of these functions be adequately addressed through a committee that has not met in more than a year? 19

24 An Evaluation of the Yukon Research Centre - FINAL REPORT The fact that the Committee has not met in more than a year suggests that its agenda is neither extensive nor urgent. Whatever its raison d être, its purpose(s) and operation need re-thinking. If its governance function can be effectively carried out solely by the three YC senior managers (and that appears to be the case), the Committee should be re-constituted. If a sound rationale exists to involve Advanced Education, CYFN and perhaps other bodies (e.g. DED) in a governance or oversight role, the Committee should be reorganized and guided by a formal terms of reference. Governance of and within YRC, like many aspects of the Centre, is still evolving. The CCI program has settled on an advisory committee model that appears to be working well during this start-up period. A similar advisory model rather than a governing council would seem an appropriate choice to help guide other (non-cci) programming at this time. The group s mandate should include provision to advise on YRC research strategies and priorities, as well as coordination of scientific research throughout the Territory. YRC Structure As with its governance provisions, YRC s structure has steadily evolved in recent years. Since its re-launch as the Yukon Research Centre of Excellence in 2010, both structure and certain terminology have seen refinements; these include: Dropped the term of Excellence Branding the whole organization as the Yukon Research Centre, a division of YC Identifying streams or program groupings to reflect different areas of focus and funding Dropping the term centre from several program titles (i.e. Yukon Cold Climate Innovation Centre; Yukon Technology Innovation Centre; Northern Climate ExChange Centre) in order to reduce confusion around program terminology. Despite these measures, the organization is still seen by many staff and stakeholders as consisting of two units: the YRC side and the CCI side. As discussed previously, this situation is essentially created by the differing mandates and orientations of the two main YG funding agreements. The CCI side is led by an active Advisory Committee but does not fall under the purview of the YRC Oversight Committee or any other governance or oversight group. While it is accountable within YRC, CCI actually operates quite independently. The YRC side is directed to a degree by the Oversight Committee but none of its programs feature local advisory bodies. A former NCE advisory group has been inactive since pre-2010; and ReSDA features a pan-northern advisory group (see Appendix 3). These incongruities lead to the belief held by many with some justification that YRC actually consists of two organizations instead of one centre. While the situation stems directly from the differing mandates of the two funding agreements, uneven governance could over the longer term undermine effective management within the organization. It could be addressed by measures designed to balance and bring under one banner the two sides of YRC; examples might include: Creating an oversight or governance body responsible for all of YRC, including CCI (at the very least, any such body that includes senior staff should include the Director of CCI) Introducing advisory committee(s) for other YRC programs such as NCE As discussed in EQ2, focusing the scope of the CCI program on R&D, and adjusting the YRC Strategic Plan accordingly Ensuring that policy development initiatives (such as that underway regarding research ethics) reflects and engages all units of YRC Continuing to put forward and emphasize the YRC brand as the primary identification for all YRC projects, with individual programs given a secondary identification. 20

25 An Evaluation of the Yukon Research Centre - FINAL REPORT While this situation is not an imminent threat to YRC, it would best be addressed promptly so that it does not build to something problematic. It would be a problem, for example, if a division between the two sides inhibited collaboration on research or innovation projects. A structural issue that arose during the conduct of this evaluation was the YC re-organization implemented on 1 July This matter is discussed under EQ4. Organizational Cohesion A key question about any newly-formed organization is whether staff members display cohesion in working towards defined goals. This is particularly relevant to the YRC situation because it was melded together from existing, new and transferred programs; it added new senior leadership and more than doubled its staff; it operates under a new relationship with YC; and it has rapidly grown its research workload. One might well wonder, is it all coming together and working as planned? Most indications are that the answer is yes. Observation as well as feedback from various sources consistently suggests that staff is a cohesive, high-performing group that works well together. The various programs/units regularly share research knowledge and pursue opportunities to work jointly on projects. Morale is positive, the atmosphere is upbeat, and many staff members freely Staff members with whom I worked were all very professional, nice, openminded and efficient. This is what I most remember from working with YRC. Great people! stated that they enjoy their jobs. Perhaps most significantly, they have not allowed the uncertainty and changes around structure and other organizational issues to interfere with the work of research projects. All of these are signs of a healthy organization. Recommendations 6. Re-assess the rationale for and operation of the YRC Oversight Committee with a view to re-constituting it to fulfill a defined purpose. An appropriate model would appear to be a President s advisory council charged with advising on research priorities, strategies, and coordination. 7. Identify and implement measures designed to overcome the organizational risks associated with the YRC dual mandate; and balance governance mechanisms between the two sides. 21

26 An Evaluation of the Yukon Research Centre - FINAL REPORT EQ4. How effectively has YRC implemented its strategic plan? Conclusion: YRC has met or exceeded most short-term expectations in successfully implementing its strategic plan. Much remains to be done with regard to moving towards achievement of some longer-term goals. As described previously, YRC has undergone a more-or-less perpetual process of strategic planning since late 2010, and its current Strategic Plan II awaits Board of Governors approval. The Plan sets out six strategic directions as its focus (these are unchanged from Strategic Plan I): 1. Building a community of researchers 2. Supporting research with Yukon First Nations 3. Conducting quality research and commercialization 4. Communicating research projects and findings 5. Building and operating a research and service hub 6. Operating an effective and sustainable organization Each strategic direction contains one or more long-term goals as well as objectives and success indicators. Feedback on both strategic objective and long-term goal achievement was obtained through the online survey as well as interviews. Full survey results are presented in Appendix 1. Responses by those stakeholders who providing ratings on achievement of selected strategic objectives are shown in Table 7. Table 7 Survey Ratings: Achievement of Strategic Objectives (6-point scale) Strategic Objective: Collaboration with community organizations Support innovation and commercialization Number of responses Average Rating Rating of 5 or 6 Rating of 1 or % 3% % 5% Collaboration with industry % 2% Dissemination of research % 3% Engaging sponsors % 8% Knowledge exchange between research & education % 13% Leverage of research funding % 7% Yukon College faculty/student involvement in research Source: Online survey (Appendix 1); Section % 8% 22

27 An Evaluation of the Yukon Research Centre - FINAL REPORT The survey also provides feedback on a number of other strategic issues; highlights include: Project stakeholders were generally positive about the research support they had received for their projects. Support included partnerships or coordination as well as information and contacts. Project stakeholders most often reported that the YRC was achieving its short-term outcomes of producing credible research/innovation outputs and contributing to the grown/ enhancement of Yukon knowledge. Stakeholders most often agreed that YRC research was accessible (less so that it was used or well-disseminated). These data paint a relatively positive picture of the extent to which YRC has successfully implemented its strategic plan. There were no significant differences between the responses of project stakeholders (i.e. those whose involvement was at a research project level) and community stakeholders with broader interests. Informants who participated in interviews generally confirmed these findings and added detail and context to the issue. The majority expressed the view that YRC has met or exceeded most expectations for the short term, including start-up organization; launching an ambitious research agenda; and securing outside funding. While this start-up period has not been totally free of difficulty, transition to the new YRC has been remarkably effective. Some of the aspects of the strategic objectives were not applicable to our project, but this does not indicate a negative feedback. The project was designed to fit our needs, and this is in fact very positive. EQ5. Are CCI processes used to evaluate proposals effective and transparent? Conclusion: The CCI proposal application and review process is well-documented and transparent. Some clients, however, reported problems with delays and conditions imposed by external funding agencies within the application process. The CCI program has a clear process for evaluating proposals. The CCI Project Lifecycle was documented in 2012 and is shown in Appendix 2. The online survey (Appendix 1) asked project-related respondents their views of the application and review process. As described in Table 7, their ratings were mostly positive with approval at about the 80% level. There was no significant difference in responses from CCI clientele (of whom 9 are included in Table 9) and those who accessed other YRC programs. In addition to providing feedback to YRC staff and stakeholders, these ratings serve as baseline data against which future ratings can be compared. 23

28 An Evaluation of the Yukon Research Centre - FINAL REPORT Table 9 Survey Ratings: Project Proposal/Application Process (6-point scale) Project Attribute: Number of responses Average Rating Rating of 5 or 6 Rating of Process was described in advance % 4% Approval criteria were clear to me % 4% Process was completed in a timely manner % 6% Rationale for the decision was clear % 4% Source: Online survey (Appendix 1); Section 2.2 These ratings are largely positive although there is room for improvement in all of them. Furthermore, several informants professed ignorance about the process and suggested that YRC should better communicate both the process and the approval criteria Recommendations 1 or 2 Communication on how the YRC selects and evaluates projects would be a useful tool to understanding operations of the YRC. 8. For application-driven programs: (1) prepare an information package for prospective applicants describing the process and success criteria; and (2) make this information available in website and paper form. EQ6. Are projects monitored effectively? Conclusion: Although most project proponents maintain that their projects are well-monitored, the majority of project files contain scant evidence of monitoring activity. Two sources of feedback inform this question: the online survey and the file review (see Appendix 3). Survey respondents involved in research projects rated several factors related to YRC project monitoring; results are summarized in Table 10 below. 24

29 An Evaluation of the Yukon Research Centre - FINAL REPORT Table 10 Survey Ratings: Project Monitoring (6-point scale) Project Monitoring Attribute: Number of responses Average Rating Rating of 5 or 6 Rating of A final report was required % 0% Clear project objectives were required % 2% Status reports were required % 4% Activities to be documented % 2% Advance planning of activities % 4% Source: Online survey (Appendix 1); Section or 2 The file review, however, suggests that much of the monitoring documentation reported in the survey never made its way to project files: 62% of the 21 files reviewed contained little or no such documentation. Reports required by funding agreements were present; status reports and evidence of communications were not, in the majority of cases. In other words, while researchers may report that their projects are well-monitored by YRC, the monitoring is poorly documented. Recommendations 9. Implement procedures to ensure that appropriate documentation of project monitoring activities, such as status reports and relevant communications, appears in project files. EQ7. Were Project Objectives Achieved? Conclusion: Multiple sources of feedback confirm that YRC projects have been markedly successful in achieving their objectives. The online survey featured 48 project-related respondents, of whom 43 reported on achievement of their project objectives. As shown in Table 11, achievement was overwhelmingly positive with a rating of 5.4/6 and no ratings of 1 or 2. Potential for commercialization/innovation was considered only slightly less positive with an average rating of 5.2/6 from 34 respondents. 25

30 An Evaluation of the Yukon Research Centre - FINAL REPORT Table 11 Survey Ratings: Project Objective Achievement (6-point scale) Project Attribute: Number of responses Average Rating Rating of 5 or 6 Rating of Project met its objectives % 0% Project shows potential for commercialization/ innovation Source: Online survey (Appendix 1); Section or % 2% The file review (Appendix 3) recorded two sets of data relevant to this question: Whether project objectives were stated in the file: 76% were clear; 24% were not Objective achievement for 15 project files that were sufficiently complete (71%; i.e. 29% not applicable) to include this information; results were as follows: -- Fully achieved or exceeded: 48% -- Mostly achieved / satisfactory: 19% -- Partial / unsatisfactory achievement: 5% -- Minimal / not achieved at all: 0% Interview findings with project informants were consistent with these data. Research objectives are typically expressed in terms of generating new information or insights, and according to multiple sources, YRC projects in recent years have largely achieved this purpose. A technical paper reporting on the project was produced, and presented at an international forum. The innovation was well received, and sparked considerable interest. EQ8. Is YRC achieving its longer-term goals? Conclusion: Although some progress can be seen in achievement of all goals, most require considerably more activity and attention to attain success over the longer term. The online survey asked respondents to rate the extent of achievement of longer-term goals. As shown in Table 12, the majority of the applicable responses provided were 5 or 6 on a 6-point scale where 6 meant that YRC has been achieving this goal very much. 26

31 An Evaluation of the Yukon Research Centre - FINAL REPORT Table 12 Survey Ratings: Achievement of Longer-term Goals (6-point scale) Longer-term Goal: Expanded research innovation culture in Yukon College Greater decision-making capacity of Yukoners Development and commercialization of innovations Support to First Nations identifying and implementing their research priorities Source: Online survey (Appendix 1); Section 3.5 Number of responses Average Rating Rating of 5 or 6 Rating of 1 or % % 10% % 8% % 9% These average ratings are among the lowest recorded for any attribute mentioned in the survey. Notably, a small but significant 8-10% of respondents saw little or no progress on several long-term goals. Interviews of informants most of whom were well-informed about YRC reinforced these survey findings. Most felt that, while some progress has been made on all longer-term goals, there remains much to do in the more challenging areas of its goals, including involvement of YC faculty and staff; support to First Nations; and community engagement. Activities in recent years have been positive and show promise but their scale has been relatively small. This view is generally shared by senior YRC leaders. The YRC has been a strong connection between our First Nation and the scientific community. The emphasis on partnership development and project coordination has built Effective July 2013, Yukon College amended its structure so that the VP, Research added the Extension Division (which includes all community campuses) to his responsibilities. A key part of the rationale for this move was to enable more ready access for YRC staff to communities, campuses and individual First Nations outside Whitehorse, as well as greater community engagement in research. The change recognizes and addresses the evaluation findings noted above (even though these findings were not known at the time of the decision), namely that progress has been limited in these aspects of the strategic plan. While the intention of the change makes sense, a counter argument is that the VP s added duties may detract from his attention to YRC and research issues. Nevertheless, the move is a proactive response to an emerging strategic issue, which in itself must be viewed as a positive initiative. Recommendations 10. Identify ways to monitor implementation of the recent YC structural change that aligned the Extension and YRC Divisions, with the intent of assessing incremental achievement of the longer-term goals regarding community and First Nations engagement in research. 27

32 An Evaluation of the Yukon Research Centre - FINAL REPORT EQ9. What do stakeholders think about the YRC? Conclusion: The vast majority of feedback from stakeholders was positive, as documented throughout this report, although a number of areas were suggested for improvement. Stakeholders views of the YRC are recorded primarily through the online survey, to which 63 responded. Feedback and suggestions for improvement related to specific evaluation questions are presented in appropriate prior sections of this report. Detailed results for survey questions are presented in Appendix 1. Stakeholders Comments The open-ended comments of which there are many make particularly interesting reading and provide considerable insight into stakeholders thinking. A number of comments were glowing: The quality of support is amazingly high, very personable. Running projects through the NRI/ YRC was a wonderful experience, and I wish the process worked as well at other institutions! They are small, but that is not a bad thing. I appreciate the existence of a research centre in the territory. There has been a significant gap in this area and the YRC offers the territory an opportunity to focus its research efforts and act as a storing house for research information. I like the atmosphere of can do and innovation [YRC] has credibility and it is a valuable resource for the College and community. very open to participation with Industry in the search for commercially viable products able to supply research funding, allowing projects to proceed great facility to host research projects minimal paperwork while others pointed to suggested improvements: The YRC seems to lack focus or clarity of mandate. The work undertaken by YRC seems far too focused on staff personal interest then it does on objective evaluation of northern research... At times YRC s work seems to extend past the bounds of academic research and into community development and/or policy making. Posture of YRC needs to include corporate-level awareness, partnership, and engagement to a higher degree. This requires a focused and leveraged partnership strategy as well as the resources to maintain and grow partnerships. I know that YRC is reliant not only on YG funding I would encourage YG or other funders to maintain or enhance core funding so that the researchers can spend more time researching and producing valuable project results rather than searching for funding. My hope is that entrepreneurial and applied business research in a northern context will be developed further Commercialization needs more entrepreneurial scrutiny at project startup, more coaching and mentoring of the inventor-entrepreneur during the project, and follow through to a point where implementation/commercialization is... likely to succeed. 28

33 An Evaluation of the Yukon Research Centre - FINAL REPORT Project Application Process - Revisited Questions about the project application and review process drew comments from several respondents who pointed to difficulties they experienced with their applications. These difficulties were not necessarily with YRC but with related funding agencies; for example: One respondent described a particular funding problem: The first projects went well, but there have been approval and funding delays for subsequent proposed projects. The problem has been finding a second partner to provide funding. Finding an interested second partner with money has been hard, and completing the application process has been hampered by funding restrictions imposed by one or more of the funding agencies that YRC is using. We are willing to work with YRC on this, but it would be great if more funds were available with fewer requirements for partners and funding by others. Finalizing agreements with partners can, and has, taken anywhere from several months to two years. Another noted that the main funding came from CanNor and EcDev the application process was very cumbersome and took 20 months. Yet another stated, 3 to 4 months to approve a project is far too long. I might as well go elsewhere for funding. Such problems originating with the external funding agencies are awkward for YRC to address, particularly in cases where a senior government agency is involved. CanNor (Canadian Northern Economic Development Agency) project funding decisions, for example, are made by the Minister 6, and several sources confirmed that decisions frequently take more than a year. Lengthy approval timelines are a perennial challenge in application-driven funding programs. While YRC s record appears satisfactory in this regard, it seems that not all YRC clientele are well-served by some of its partner funding agencies. Recommendations 11. Include within a comprehensive YRC evaluation/monitoring plan mechanisms to regularly capture and report on stakeholder feedback. 6. The CanNor website states: SINED [Strategic Initiatives for Northern Development] focuses on strengthening the driver sectors of the territorial economies, economic diversification and encouraging Northerners participation in the economy. Projects are prioritized based on five-year territorial investment plans developed with input from territorial stakeholders, and are approved by the Minister of CanNor. 29

34 An Evaluation of the Yukon Research Centre - FINAL REPORT EQ10. How could the program be improved? Conclusion: Recommendations for improvement are cited relevant to each evaluation question, and are repeated in the Executive Summary. A few additional areas for improvement emerged during this evaluation that are not directly related to the evaluation questions, but are related to internal systems and emerging issues associated with the YRC goal of Operating an effective and sustainable organization. An Effective Organization A recurring theme in the recent expansion of YRC refers to internal YC systems that are designed to fulfill college as opposed to research centre administrative requirements. These systems include financial accounting and reporting; contracting and procurement; human resources; policy development; and evaluation. The issues are known and are being addressed to varying degrees through such measures as: The recent appointment of a full-time accounting position within YRC, to fulfill accounting and reporting functions previously carried out by central YC personnel, or by individual program managers maintaining spreadsheets of project expenditures. This move recognizes that research accounting is different from mainstream college accounting, and has in a very short period of time demonstrated its effectiveness. Contracting and procurement can be challenging to a research centre, due in part to the need to act promptly to secure partners and cement funding agreements. Current procedures call for all contracts to be issued through the YC Procurement office. In the early days of YRC, at least one funding opportunity was lost because YC decision-makers could not react quickly enough to secure a project funding agreement. While the situation has reportedly improved, it bears monitoring to ensure that these support systems are able to respond to YRC needs. Human resource support was an essential ingredient in staffing YRC to its present level. The longer-term challenge will be to keep and replace as required key personnel by converting short-term contract positions to permanent or similar appointments. As YRC matures and grows, it will require and benefit from development of formal policies and procedures. Some of these needs, such as development of an ethics policy, are already underway. Others are recommended in this report, e.g. preparation of an information package for application-based programs. There are numerous additional policy issues on the horizon that will require time and resources to adequately address. The YRC Strategic Plan II acknowledges the need to develop a performance monitoring system and regularly evaluate program performance. As discussed previously (see Recommendation #3), a sound move would be to develop such a system in the very near future (see Appendix 7). All of this suggests that YRC s senior leaders are aware of the need to build appropriate supports and procedures into a growing organization. Current strategy and initiatives indicate that the need for improvement in these areas is being adequately addressed. 30

35 An Evaluation of the Yukon Research Centre - FINAL REPORT Organization Sustainability Previous discussions about organization and governance (EQ 1, 2, 3 and 8) demonstrate that these aspects of YRC have seen adjustments since 2010 and continue to evolve. The recommendations contained in this evaluation point to several areas where improvement is possible and YRC leaders have been receptive to these proposals. It is important to recognize that organizational factors such as governance, structure, policies, program design should be a function of strategy. In other words, these factors should be designed to enable and implement the institution s goals and plans. YRC has continually observed this principle in its start-up period. The question around sustainability is not whether YRC s current organization is capable of sustaining achievement and growth; rather, the central issue is whether YRC is adapting its organization to address current and emerging needs related to pursuit of its strategies. All indications are that YRC leaders are doing just that adjusting and tweaking organizational factors to meet emerging conditions. The July 2013 YC reorganization is a prime example, namely implementing a structural change to enable more effective pursuit of selected YRC goals. The Yukon research environment is dynamic and growing. It would be a mistake in this situation if YRC s organization was to remain static. YRC s actions to date indicate that it is capable of adapting to and sustaining itself within this environment in the foreseeable future. 31

36 Conclusion The central finding of this evaluation is that Yukon Government core funding agreements have enabled YRC to change and grow in significant ways over the past three years. Most knowledgeable informants agree that the pace of growth and extent of accomplishment has exceeded their expectations. During this start-up period for the rekindled research centre, management has focussed its attention on short-term priorities, namely strategic planning, assembly of staff, facility development, securing research funding and partners, and of course greatly expanding the number and scope of research and innovation projects. Organizational and governance issues, on the other hand, are being addressed on an as required basis. These aspects of YRC continue to evolve in an iterative fashion as strategies point to areas within the organization that require attention and adjustment. YRC has demonstrated a number of strengths over this period, including adherence to both the terms and the intent of its core funding agreements; a productive and cohesive staff; a clear focus on the work of research and innovation; and a readiness to address emerging issues and challenges. YC and YRC leaders, for example, were keenly interested in the findings of this evaluation and receptive to its recommendations. There are a number of areas within YRC that require improvement, adjustment or clarification. There are no areas in crisis or that require major overhaul or drastic changes in direction. The new YRC is progressing nicely, ahead of most expectations, and is well-positioned to fulfill the mandates of its core funding agreements over the longer term. 32

37 YRC Final Report Appendices 1 - Stakeholder Survey Summary Report 2 - Third Party Funding Detail 3 - ReSDA Organization 4 - CCI Project Lifecycle Chart 5 - File Review Summary Report 6 - Case Studies: Landscape Hazards Hydro Security PlastoVac BioChar 7 - Towards an Evaluation System for YRC

38 Appendix 1 Stakeholder Survey Summary Report Highlights 1.0 Introduction Most of the 63 YRC stakeholders responding to the survey were partners, producers or users of YRC research. These YRC stakeholders most often considered themselves to be somewhat knowledgeable about the YRC and its operations. Many of the responding stakeholders had been involved with one or more projects at the YRC over the past three years, most often as a project sponsor or researcher/project staff member. Project stakeholders had been involved with all types of projects including those completed as well as those that are still underway or ongoing. 2.0 Management and Delivery Project stakeholders reported their most recently completed or most significant current projects were often in the Northern Climate ExChange, Cold Climate Innovation and Technology Innovation areas. Project stakeholders reported that one-half of these projects had received financial support from the YRC and about one-third had received research support. Project stakeholders were generally positive about the project proposal/application process, on applicable projects. This included the process being described in advance and the rationale for the decision being clear. Project stakeholders were generally positive about the research support provided, on applicable projects. This included arrangements for partnerships or coordination as well as information and contacts. Project stakeholders were generally positive about the YRC s monitoring of these projects. This included requirements for final reporting and clear project objective requirements. A34

39 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES 3.0 Results Most project stakeholders (self-)reported that their projects had met their objectives and that more than one-half had shown potential for innovation/commercialization. YRC stakeholders most often reported that the YRC was achieving its short-term outcomes of producing credible research/innovation outputs and contributing to the growth/enhancement of Yukon knowledge. YRC stakeholders most often agreed that YRC research was accessible (less so that it was used or well-disseminated). YRC stakeholders most often reported that the YRC was achieving its strategic objectives of dissemination of research, collaboration with industry, and leveraging of research funding. YRC stakeholders most often reported that the YRC was achieving its longer-term goal of expanded research innovation culture in Yukon College. Achievement of other long-term goals was rated somewhat less. 4.0 Considerations for the Future When asked what they like most about the YRC, stakeholders most often reported that they like YRC s staff, that the research centre exists and that it is Yukon-based/focused. When asked what they would most like to change about the YRC, stakeholders most often reported that they would like increased and/or more focused research capacity, better communications, and increased or more stable funding. A35

40 1.0 Introduction This summary report sets out findings from the Yukon Research Centre (YRC) evaluation s online survey. It describes the responses made to each evaluation question. The methodology used, questionnaire, and cover s are appended. 1.1 Stakeholder Involvement with the YRC Most of the 63 responding stakeholders were partners, producers or users of YRC research. Of the 63 stakeholders who responded to the survey, 37% were partners in YRC research, about one in five (22%) were researchers or project staff members, and the same proportion (22%) were users of YRC research. About one in ten (11%) were funding agency representatives and 6% or fewer held Board, oversight or management positions. One-quarter (22%) reported other ways that they were involved with the YRC including 6% who reported they were YRC funding recipients. Respondents frequently reported they were involved with YRC in more than one manner on average individuals reported being involved in 1.5 ways. Exhibit 1.1 Stakeholder Involvement with the YRC A36

41 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES 1.2 Level of Knowledge About the YRC These YRC stakeholders most often considered themselves to be somewhat knowledgeable about the YRC and its operations. Of the 63 stakeholders who responded to the survey, about one in three (35%) of these respondents rated their knowledge as a 5 or 6 on a 6-point scale where 6 is very knowledgeable having a sound understanding of the YRC s purpose and activities. A further 54% provided a rating of 3 or 4 on the same scale. Only 11% provided a rating indicating not much knowledge (a 1 or 2 on the 6-point scale where 1 was not at all knowledgeable having no understanding of the YRC s purpose and activities). Exhibit 1.2 Level of Knowledge About the YRC Open-ended comments on this question related most often to stakeholders being knowledgeable about their area, who or how they specifically interact with the YRC. A37

42 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Selected Stakeholder Open-Ended Comments: While I am knowledgeable of the processes in place guiding the interaction between our office and YRC, it is possible that other aspects of YRC are totally unknown to us. This should not be viewed as a negative, since we may not need to know about any other aspect of the internal workings. I ve worked with the research arm (NCE, CCIC, etc.) at the college for several years; however, I can t say that I fully understand the mandate or operating strategy of YRC and how it fits into the greater context of the College and/or Yukon government. The Yukon Research Centre has been involved in a number of vital and relevant research programs in our small Yukon community. Representatives of the YRC have helped us to understand implications of climate change; develop renewable energy projects; engage in biodiversity monitoring; and support sustainable development initiatives. Communication on how the YRC selects and evaluates projects would be a useful tool to understanding operations of the YRC. 1.3 Project Involvement in the Past Three Years Many of the responding stakeholders had been involved with one or more projects at the YRC over the past three years, most often as a project sponsor or researcher/project staff member. Of the 63 responding stakeholders, 48 or 76% had been involved with specific YRC projects. Exhibit 3 Involved with Projects at the YRC (Past Three Years) Most often the 48 project stakeholders were involved with projects over the past three years as a project sponsor (38%) or as a researcher/project staff member (31%). One stakeholder may have been involved with YRC projects in more than one way. A38

43 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Exhibit 1.4 Type of Involvement with Projects at the YRC (Past Three Years) Many of these project stakeholders had been involved with multiple projects at the YRC over the past three years. About two-thirds (69%) of these project stakeholders had been involved with two or more projects at the YRC over the past three years. Only 31% had been involved with only one project. Exhibit 1.5 Number of YRC Projects Involved With (Past Three Years) A39

44 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Project stakeholders had been involved with all types of projects including those completed as well as those that are still underway or ongoing. Thirty-eight of the 48 project stakeholders reported on the current status of the projects they have been involved with in the past three years. At least one-half (56%) of these 48 project stakeholders reported the current status of one or more of these projects was that they were complete (results have been reported/submitted). At least 29% of these project stakeholders reported the current status of one or more of these projects was that they were incomplete and underway. At least 33% of these project stakeholders reported the current status of one or more of these projects was that they were ongoing (continuing or recurring activity). Exhibit 1.6 Current Status of Projects A40

45 Management and Delivery 2.1 Most Recently Completed or Significant Projects Project Areas Project stakeholders reported their most recently completed or most significant current projects were most often in the Northern Climate ExChange, Cold Climate Innovation and Technology Innovation areas. The 48 project respondents were asked to consider the most recently completed or the most significant current project they have been involved in over the past three years. They reported that these were primarily projects in the Northern Climate ExChange (40%), Cold Climate Innovation (29%) and Technology Innovation (29%) areas. Please note that more than one area may have been reported for one project. As well, more than one project stakeholder may have been responding about the same/one project. Exhibit 2.1 Specific Project Areas A41

46 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Project Connections to YRC Project stakeholders reported that one-half of these projects had received financial support from the YRC and about one-third had received research support. One-half (50%) of these project stakeholders reported these projects received financial support and 31% reported projects receiving research support from the YRC. These included 23% of project stakeholders who reported their projects received both. A further 42% of these project stakeholders reported that their project was a collaboration/partnership only with the YRC it received neither financial nor research support from the YRC. Exhibit 2.2 Specific Project Connections to YRC 2.2 Project Proposal/Application Process Project stakeholders were generally positive about the project proposal/application process, on applicable projects. This included the process being described in advance and the rationale for the decision being clear. More than one-half (52% to 56%) of the 48 project stakeholders did not provide ratings on the proposal/application process attributes with respect to their most recently completed or significant current project. A majority of the remaining responses provided were 5 or 6 on a 6-point scale where 6 was very much : Process was described in advance (42%) Rationale for the decision was clear (42%) Process was completed in a timely manner (40%) Approval criteria were clear to me (36%) Only a few (4% to 6%) of the responses provided were 1 or 2 on the 6-point scale where 1 was not at all. Similarly, among the 15 other (non-project) stakeholders, 11 did not provide an overall rating on how well the YRC s proposal/application process works, 1 provided a rating of 6 (very well) and 3 provided a rating of 4 on the 6-point scale. A42

47 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Exhibit 2.3 Project Proposal/Application Attributes Among only project stakeholders providing ratings on these attributes of the proposal/application process, average ratings 1 were as follows: Rationale for the decision was clear (5.0 average rating based on 26 responses) Process was completed in a timely manner (4.8 average rating based on 27 responses) Process was described in advance (4.7 average rating based on 27 responses) Approval criteria were clear to me (4.7 average rating based on 25 responses) Thirteen project stakeholders reported on these attributes for Cold Climate Innovation projects. For 4 of these project stakeholders, the proposal/application process was not applicable. A similar proportion of the project stakeholders provided responses that were 5 or 6 on a 6-point scale where 6 was very much : Process was described in advance (5 responses) Rationale for the decision was clear (6) Process was completed in a timely manner (4) Approval criteria were clear to me (5) The remainder gave lower ratings on these proposal/application process attributes. Open-ended comments on this question related most often to the complexity of the requirements and the length of time it took to have projects approved. 1. Average ratings are based on a maximum 6, and exclude don t know and not applicable responses. A43

48 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Selected Stakeholder Open-Ended Comments: The first projects went well, but there have been approval and funding delays for subsequent proposed projects. The problem has been finding a second partner to provide funding. Finding an interested second partner with money has been hard, and completing the application process has been hampered by funding restrictions imposed by one or more of the funding agencies that YRC is using. We are willing to work with YRC on this, but it would be great if more funds were available with fewer requirements for partners and funding by others. Finalizing agreements with partners can, and has, taken anywhere from several months to two years. What proposal/application process? As far as I m aware YRC accepts projects based on YRC staff preference. I m not aware of any formal proposal/application process. Main funding came from CanNor and EcDEv - the application process was very cumbersome and took 20 months. 3 to 4 months to approve a project is far too long. I might as well go elsewhere for funding. 2.3 Research Support Project stakeholders were generally positive about the research support provided, on projects they had used it for. This included partnerships or coordination as well as information and contacts. Nineteen percent of the 48 project stakeholders did not provide a rating on the overall research support provided to their most recently completed or significant current project. A majority 2 (63%) of the remaining 39 responses provided were 5 or 6 on a 6-point scale where 6 was that YRC has provided research support to the project very well. In comparison, 6% provided responses of 1 or 2 on the same 6-point scale were 1 was not at all well. Among only these 39 project stakeholders providing an overall rating on the research support provided, the average rating was Interestingly, this is substantially higher than the proportion of project stakeholders who considered their project had received research support from the YRC (31%) as reported in Section 2.2. This may be because the specific types of research asked about on the same page of the survey prompted higher recognition of these services. A44

49 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Exhibit 2.4 Overall Research Support Project stakeholders reported they had most often used YRC partnerships or coordination, information and/or contacts, followed by administrative support or technical advice/guidance. They had used work space, long/short-term storage, contracted local field assistants and accommodation least often on these projects. A majority of the applicable responses provided were 5 or 6 on a 6-point scale where 6 was that YRC has provided this support very well : Partnerships or coordination (69%) Information (59%) Contacts (54%) Administrative support (48%) Technical advice or guidance (46%) Only a few (4% to 6%) of the responses provided were 1 or 2 on the 6-point scale where 1 was not at all. A45

50 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Exhibit 2.5 Research Support Attributes Among only project stakeholders providing ratings on these attributes of the research support provided, average ratings were as follows: Partnerships or coordination (5.0 average rating based on 43 responses) Information (4.9 average rating based on 40 responses) Contacts (4.8 average rating based on 39 responses) Work space (4.8 average rating based on 13 responses) Long and short-term storage (4.8 average rating based on 13 responses) Administrative support 4.7 average rating based on 36 responses) Contracted local field assistants (4.7 average rating based on 10 responses) Technical advice or guidance (4.6 average rating based on 33 responses) Accommodation (4.3 average rating based on 6 responses) Among the 15 other (non-project) stakeholders, 9 did not provide an overall rating on how well the YRC provides research support to projects, 3 provided a rating of 5 and 3 provided a rating of 4 on a 6-point scale where 6 was very well. Open-ended comments on this question related most often to the availability of research support for projects and partner contributions A46

51 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Selected Stakeholder Open-Ended Comments: My organization has primarily used YRC as a source of either director(sic) or pass through funding for projects. I don t have the sense that YRC currently has the focus or capacity to offer much in the way of research support. YRC takes a 15% cut off of all my research contracts so they benefit for minimal administrative demand. Research support was not ever offered. The YRC has been a strong connection between our First Nation and the scientific community. The emphasis on partnership development and project coordination has built capacity within our remote Yukon community. 2.4 Project Monitoring Project stakeholders were generally positive about the YRC s monitoring of these projects. This included final reporting and clear project objective requirements. Twenty-three percent of the 48 project stakeholders did not provide ratings of YRC s overall monitoring of their most recently completed or significant current project. A majority (54%) of the remaining responses provided were 5 or 6 on a 6-point scale where 6 was that the YRC s monitoring of this project had been very effective. In comparison, 4% provided responses of 1 or 2 on the same 6-point scale were 1 was not at all effective. Among only these 37 project stakeholders providing an overall rating on monitoring, the average rating was 4.9. Exhibit 2.6 Overall Project Monitoring A47

52 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Project stakeholders most often reported that the YRC had required activities to be documented, followed by advance planning of activities, clear project objectives and a final report. A majority of the applicable responses provided were 5 or 6 on a 6-point scale where 6 was that YRC required this monitoring activity very much : A final report (63%) Clear project objectives (62%) Activities to be documented (60%) Status reports (58%) Advance planning of activities (56%) Only a few (2% to 4%) of the responses provided were 1 or 2 on the 6-point scale where 1 was not at all. Exhibit 2.7 Project Monitoring Attributes Among only project stakeholders providing ratings on these monitoring attributes, average ratings were as follows: A final report (5.4 average rating based on 36 responses) Clear project objectives (5.1 average rating based on 36 responses) Status reports (5.1 average rating based on 35 responses) Activities to be documented (5.0 average rating based on 37 responses) Advance planning of activities (4.9 average rating based on 36 responses) A48

53 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Among the 15 other (non-project) stakeholders, 11 did not provide an overall rating on how well the YRC monitors projects, 3 provided a rating of 5 and 1 provided a rating of 4 on a 6-point scale where 6 was very well. Open-ended comments on this question related most often to project monitoring roles and requirements. Selected Stakeholder Open-Ended Comments: These items have been provided/required, but also are important for a good project. Most of my experience is with [a project], which was led by the NCE. It was an incredibly collaborative and well-planned process, but one undertaken primarily by the NCE, and we were partners. So NCE did a lot of these activities, and didn t require partners to do them. Again, staff turnover led to a stop/start planning, reporting, and final reporting. We got our project completed, but progress was delayed/hindered by changes in staff. A49

54 3.0 Results 3.1 Project Objectives Most project stakeholders reported that their projects had met their objectives and more than one-half that they had shown potential for innovation/commercialization. Eleven percent of the 48 project stakeholders did not provide a rating on the extent to which their most recently completed or significant current project has met its objectives. A majority (77%) of the remaining responses provided were 5 or 6 on a 6-point scale where 6 was that their project had met its objectives very much. In comparison, no project stakeholders provided responses of 1 or 2 on the same 6-point scale were 1 was not at all. Twenty-nine percent of the 48 project stakeholders did not provide a rating on the extent to which their project has shown potential for innovation/commercialization. A majority (60%) of the remaining responses provided were 5 or 6 on a 6-point scale where 6 was that their project had shown such potential very much. In comparison, 2% of the project stakeholders provided responses of 1 or 2 on the same 6-point scale were 1 was not at all. Exhibit 3.1 Project Objectives A50

55 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Among only project stakeholders providing ratings on these project objectives, average ratings were as follows: Met its objectives (5.4 average rating based on 43 responses) Shown potential for innovation/commercialization (5.2 average rating based on 34 responses) Open-ended comments on this question related most often to future commercialization potential and the challenges of achieving it. Selected Stakeholder Open-Ended Comments: Our objectives are consistent with those of YRC and we are strongly motivated to achieve them. We know that our projects have significant potential for innovation/ commercialization. [The project] met its objectives, but now it s up to partners to implement it, and that hasn t happened. The other project has been transferred over to another partner, and that has commercialization potential. Some issues with the equipment tested in our program caused us to complete the project without field testing, however the potential exists for commercialization once equipment issues are resolved. 3.2 Short-Term Outcomes Other (non-project) stakeholders most often reported that the YRC was achieving its short-term outcomes of producing credible research/innovation outputs and contributing to the growth/ enhancement of Yukon knowledge. The survey asked only the 15 stakeholders who had not been involved with one or more projects at the YRC over the past three years about the YRC s short-term outcomes. These stakeholders were most positive about the YRC: Producing credible research/innovation outputs Contributing to the growth/enhancement of Yukon knowledge On these two short-term outcomes, a majority of the applicable responses provided were 5 or 6 on a 6-point scale where 6 was that YRC very much. As well, they had fewer don t know/not applicable responses, indicating that they were more often relevant. Exhibit 3.2 Short-Term Outcomes Short-Term Outcome Don t Know/ Not Applicable (Not at all) 1 or 2 3 or 4 5 or 6 (Very Much) Produced credible research/innovation outputs Contributed to the growth/enhancement of Yukon knowledge Made a significant contribution to the Yukon College vision Built supportive partnerships Coordinated most scientific research in the Yukon A51

56 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES 3.3 Dissemination of Research Other (non-project) stakeholders most often agreed that YRC research was accessible (less so that it was used or well-disseminated). The survey asked only the 15 stakeholders who had not been involved with one or more projects at the YRC over the past three years about the YRC s dissemination of research. These stakeholders were most positive that they: Are confident they can access YRC research at any time Have accessed YRC research information in the past 6 months On these two aspects of dissemination, a majority of the applicable responses provided were 5 or 6 on a 6-point scale where 6 was that YRC strongly agree. As well, they had fewer don t know/not applicable responses, indicating that they were more often relevant. Exhibit 3.3 Dissemination of Research Short-Term Outcome Don t Know/ Not Applicable (Not at all) 1 or 2 3 or 4 5 or 6 (Very Much) I have accessed YRC research information in the past 6 months I am confident I can access YRC research at any time I have used YRC research information in my work or vocation YRC does a good job of disseminating its research information Strategic Objectives YRC stakeholders most often reported that the YRC was achieving its strategic objectives of dissemination of research, collaboration with industry and leveraging of research funding. The 63 responding project and other stakeholders most often reported that the YRC had achieved the following strategic objectives: dissemination of research, collaboration with industry, leverage of research funding and Yukon College faculty/student involvement in research. A majority of the applicable responses provided were 5 or 6 on a 6-point scale where 6 was that YRC has been very effective at achieving this objective or that the project has contributed to the objective very much : Dissemination of research (53%) Collaboration with industry (42%) Leverage of research funding (49%) Yukon College faculty/student involvement in research (38%) Engaging sponsors (43%) Fewer (2% to 13%) of the responses provided on each objective were 1 or 2 on the 6-point scale where 1 was not at all. A52

57 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Exhibit Strategic Objectives Among only stakeholders providing ratings on these strategic objectives, average ratings were as follows: Collaboration with community organizations (4.9 average rating based on 48 responses) Support innovation and commercialization (4.9 average rating based on 43 responses) Collaboration with industry (4.9 average rating based on 41 responses) Dissemination of research (4.7 average rating based on 38 responses) Engaging sponsors (4.6 average rating based on 34 responses) Knowledge exchange between research/education (4.5 average rating based on 41 responses) Leverage of research funding (4.5 average rating based on 40 responses) Yukon College faculty/student involvement in research (3.9 average rating based on 31 responses) Open-ended comments on this question described the successes and challenges projects have had related to these strategic objectives. A53

58 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Selected Stakeholder Open-Ended Comments: We have had trouble finding other funding partner(s) in order to have the amount of leveraging that YRC wants. It would be great if this constraint could be modified or removed. Some of the aspects of the strategic objectives were not applicable to our project, but this does not indicate a negative feedback. The project was designed to fit our needs, and this is in fact very positive. It is important to recognize, that our products were closely guarded intellectual property during their development phase and YRC / YTIC is aware of this. They then would not disseminate any info or seek assistance unless the project was in danger. Innovation and commercialization is a good thing, but it can t be the main focus. This country needs fundamental/basic research even if the government doesn t understand its value. The collaborative efforts exhibited by the YRC are progressive and commendable. Throughout our work relationships with the YRC constant evaluations on project relevance and community engagement were applied. We were able to establish communication and connection with the community, industry, government, and sponsors through innovative approaches to engagement and information sharing. A technical paper reporting on the project was produced, and presented at an international forum. The innovation was well received, and sparked considerable interest. 3.5 Longer-Term Goals YRC stakeholders most often reported that the YRC was achieving its longer-term goal of expanded research innovation culture in Yukon College. The 63 responding project and other stakeholders most often reported that the YRC is achieving its longer-term goal of expanded research innovation culture in Yukon College, followed by greater decision-making capacity of Yukoners, development and commercialization of innovations, and support to First Nations identifying and implementing their research priorities. A majority of the applicable responses provided were 5 or 6 on a 6-point scale where 6 was that YRC has been achieving this goal very much : Expanded research innovation culture in Yukon College (60%) Greater decision-making capacity of Yukoners (40%) Development and commercialization of innovations (43%) Support to First Nations identifying and implementing their research priorities (30%) Fewer (8% to 10%) of the responses provided were 1 or 2 on the 6-point scale where 1 was not at all. A54

59 An Evaluation of the Yukon Research Centre - FINAL REPORT APPENDICES Exhibit 3.3 Longer-Term Goals Among only stakeholders providing ratings on these longer-term goals, average ratings were as follows: Expanded research innovation culture in Yukon College (4.8 average rating based on 53 responses) Greater decision-making capacity of Yukoners (4.4 average rating based on 47 responses) Development and commercialization of innovations (4.4 average rating based on 43 responses) Support to First Nations identifying and implementing their research priorities (4.1 average rating based on 35 responses) Open-ended comments on this question described the achievements and obstacles to these longer-term goals. A55