Analyzing outcomes and impacts
We have a responsibility to demonstrate the impacts of our science and technology investments, to assess the efficiency of the public spending we oversee, and to assess our contribution to achieving social and economic objectives for Canada. But measuring and evaluating the outcomes of CFI investments is complex. Research and innovation is inherently risky and outcomes and impacts linked to research infrastructure are difficult to measure. For example, there’s the time lag between when the investment is made and when the outcomes are realized, and it is often challenging to attribute the results of research to any particular investment. That’s why we use a range of data and assessment approaches to evaluate progress at the organizational level through to the societal level.
A new approach to mapping investments to impacts
Our presentation at the French National Institute for Agriculture Research conference in Paris
A socioeconomic impact assessment is a systematic analysis of the economic, social and cultural impacts, outputs and outcomes related to a particular set of investments. Understanding and measuring the impacts of public research and development is necessary to evaluate the efficiency of public spending and assess its contribution to achieving social and economic goals.
Impact Analysis of funding for research and development in medical imaging
In partnership with the Canadian Institutes of Health Research, we undertook a pilot project to quantify the socioeconomic impacts of our collective investments in medical imaging.
The study highlights the importance of an open and competitive research system to move innovation along the continuum from new ideas, to working prototypes, to products which benefit Canadians.
The project benefitted from an advisory committee and expert peer review who contributed both their time and their expertise to this pilot study.
To request a complete copy of the technical report, O’Connor, Alan C. and Albert N. Link. 2013. Pilot socioeconomic impact analysis of CFI and CIHR Funding: Medical Imaging. Prepared for CFI and CIHR by RTI International, please contact evaluations [at] innovation.ca, Director, Evaluation and Outcome Assessment.
Mapping investments to impacts is an approach we use to explore our contribution to the national research landscape in a given thematic area with specific emphasis on the pathways to impact.
Agricultural research in Canada
We first undertook a scoping exercise to look at possible themes that would be suitable for a national study aimed at capturing benefits to Canadians from research infrastructure funding. Of all themes considered, agriculture was the one that offered the best geographic distribution of CFI investments, including significant investment in Eastern Canada. Since 77 percent by value of CFI investments in agricultural research were made prior to 2010, the likelihood was high of being able to identify benefits from the research enabled by the infrastructure investment.
Platform outcome measurement studies allow us to examine large-scale, specialized, purpose-built research infrastructure that serves specific needs of the Canadian research community, while considering the uniqueness of each platform.
The Canadian Coast Guard Ship (CCGS) Amundsen contains a comprehensive pool of specialized scientific equipment and facilities for research in the Canadian Arctic. The research platform was made possible after a consortium of 15 Canadian universities and research centres, in partnership with the federal government, received funding from the CFI for the retrofit of the decommissioned Canadian Coast Guard icebreaker Sir John Franklin as an Arctic Ocean research vessel. This contribution of $27.5 million from the CFI International Joint Venture Fund included nearly $19.4 million for structural transformations to the ship for science, and for scientific equipment as well as $5.5 million in support for the operations of the platform.
We conducted a platform outcome measurement study that assessed the activities and achievements of the Canadian Research Icebreaker, Canadian Coast Guard Ship (CCGS) Amundsen.
Key findings of the Expert Panel:
- The Amundsen platform is enabling science of the highest international quality and is facilitating the translation and application of new knowledge to address societal issues of major consequence for the Arctic regions of Canada and for other Arctic settings.
- The Amundsen research program has had a major impact on the productivity, reach and influence of Canadian Arctic science as shown by the strong publication record and by the seminal papers produced on such topics as sea ice and ecological research in the Beaufort Sea.
- The Amundsen program has engaged a diverse set of end-users encompassing federal and provincial science-based government departments and agencies, industry, and communities.
Canadian Research Knowledge Network
The Canadian Research Knowledge Network (CRKN) is a partnership of Canadian universities dedicated to expanding access to digital content for the academic research enterprise in Canada. Between 2000 and 2012, CRKN has received two CFI contributions for a total of $97.7 million, including partner funding from provincial governments and Canadian universities. We assembled a panel of experts to assess the outcomes and impacts of CRKN in accordance with the POMS framework.
Key findings of the Expert Panel:
- CRKN is among the leading information-enabling organizations worldwide and is recognized as a “game changer” for the Canadian research community.
- The investment in CRKN by the CFI and its provincial government partners was essential, timely and catalytic, and has already been returned many times over.
- CRKN has well-developed and efficient operations to address its current mandate; however, recent changes to the governance structure were viewed as potentially detrimental to its future.
- CRKN may not have the resilience with its current level of resources to deal with the ongoing transformation in scholarly communication and the ways in which digital content is used by the research community.
Prior to 2011, we conducted a series of Outcome measurement studies designed to assess the degree to which our investment in research infrastructure is a critical contributing factor in the realization of five outcomes: strategic research planning, research capacity, highly qualified personnel, research productivity, and innovation. These early efforts to measure outcomes and impacts have helped to shape our current approaches to outcome and impact assessment.
Methodology for Outcome measurement studies
In December 2010, we published a paper about the outcome measurement studies methodology in Research Evaluation, an international, peer-reviewed journal.
The Canada Foundation for Innovation’s outcome measurement study: a pioneering approach to research evaluation (PDF)
Outcome measurement study summary report
In 2008, independent consultants examined the findings of the Outcome measurement studies conducted to date. Here is their report: