The Impacts of Innovation — Measuring the Effects of Challenges

Noah Conrad, Junior Analyst

What are Challenges? What are their desired outcomes?

Impact Canada Challenges are outcomes-based competitions run by the Impact and Innovation Unit (IIU), in partnership with other government departments to find solutions to some of Canada’s most pressing issues. Using an open innovation approach, challenges leverage financial and non-financial incentives to entice innovators to direct their talents towards helping to solve major societal problems. Challenge applicants typically apply to a staged competition where the ultimate winners are awarded large financial prizes.

My name is Noah Conrad, and for the last four months, I have been supporting the IIU in the measurement of challenges. What follows is a brief overview of the challenge impact measurement approach developed by the IIU, and some options being considered for future use.

Why is the IIU Measuring Impact?

As challenges are completed, formally measuring their direct and indirect effects will help the IIU to assess whether challenges are having their desired impacts and whether they are being designed and delivered in the most effective manner. Indeed, to continually enhance its programs, the IIU needs to understand the aspects of its methods that are functioning as intended and the ones that require further refinement. Moreover, given the monetary prizes associated with each challenge, formally measuring the effects of challenges will help maximize their effectiveness and efficiency within a public policy context.

Barriers to overcome

To date, there have been relatively few recent studies that have rigorously studied the impacts of challenges. By their very nature, challenges work on multifaceted problems that may not have one single cause or solution. Consequently, it can be difficult to attribute positive changes within an issue area to challenges themselves when many other variables and factors could have influenced the observed outcomes. Moreover, since each challenge works on a distinct problem and can employ one of three basic structures (challenge prizes, grand challenges, and competitive accelerators), all challenges are likely to have somewhat divergent pathways to impact, making it hard to develop a standardized approach to impact measurement. For example, one challenge may impact an issue by making high quality information more readily available (see Housing Supply Round One) while others may change an area by incentivizing a technological breakthrough (see Sky’s the Limit Challenge). Clearly then, determining the causal impacts of challenges will necessitate the use of multiple methods and lines of inquiry. The following sections can be read as a limited subset of the approaches to impact measurement that the IIU has employed to date and is considering in the future.

Approaches to Impact Measurement

In 2020, the IIU published a document called Logic Model and Narrative — Impact Assessment of Challenges under Impact Canada, which presents a basic theory of what we expect to achieve with our suite of challenges. Each of the methods of inquiry below is discussed in relation to the outcomes from the logic model that it is designed to measure.

Social Network Analysis: Measuring increased awareness, talent mobilization, and the formation of collaborative networks and partnerships

A desired process outcome from the logic model is the formation of partnerships both internal and external to the challenges themselves. Indeed, studies have shown that the formation of knowledge communities and networks through which actors exchange information are an important driver of innovation. As part of their design, challenges have developed various opportunities for participants to form connections with end users, potential financiers, local communities, and other competitors. Ideally, connections with financiers should help attract initial investments into start-ups while networking with fellow participants may assist in the exchange of ideas. Additionally, challenges should raise attention and generate increased awareness about important issues. Open communication from Impact Canada with strong engagement from innovators, end users, and the media contribute to raising the public profile of policy priorities and problems that a challenge has been designed to address or solve.

Another primary motivation behind the creation of challenges is to mobilize actors who do not typically apply to government programs as well as those who lie outside the normal constituency that may develop solutions to the issue in question. Challenges feature a streamlined application process designed to incentivize non-traditional actors to apply to them. It is expected that new actors who apply to challenges will bring new ideas, capacities, and experiences. These unconventional ways of thinking may help uncover new solutions to the difficult problems that challenges are designed to solve.

A line of inquiry which could help the IIU to measure all three of these outcomes is social network analysis. For those unfamiliar, social network analysis is a research method which outlines the exchanges and connections between actors[i]. Undertaking social network analysis would help paint a picture of the kinds of actors that are engaging in Impact Canada challenges as well as the interconnections between them. Beyond illustrating the connections between actors, social network analysis could reveal the relative centrality and importance of different actors within a network.

Surveys: Measuring the Mobilization of Talent

In addition to social network analysis, the IIU is making use of surveys to determine whether challenges are succeeding in attracting a wide variety of innovators. Surveys can help gather important information about applicants to understand who applies to challenges and what some of their motivations are. Some of the variables that can be analyzed from a survey include the age and gender distribution, location, background, and prior activity of applicants. Analysis of these variables will allow the IIU to measure the effectiveness of challenges in attracting new talent.

Quasi-Experimental Approaches: Measuring Economic Value

By de-risking investments into nascent technologies, challenges should help participants to raise new capital and financing. Evidence from a Nesta case study has shown that challenges are capable of helping participants generate investment into their products and solutions[ii]. Moreover, the staged approach that Impact Canada has taken to challenges in which non-winners are provided with various forms of support should help improve economic outcomes even amongst non-grand-prize-winners. Determining whether challenges have succeeded in generating greater public value will require the use of quasi-experimental approaches in which counter-factual outcomes from a control group are compared to the outcomes seen from applicants. By determining whether participants have experienced increases in economic indicators such as R&D spending, productivity, payroll spending, exports, patents, intangible assets, revenue, capital raised, and job creation, the IIU will be able to determine whether challenges have supported sustainable, innovative businesses that have experienced positive economic outcomes. Some initial work in this direction has begun in partnership with Statistics Canada (StatsCan). Moving forwards, the IIU will continue to collaborate with StatsCan to use data available in its Linkable File Environment to determine the most effective ways to measure the economic impacts of challenges while accounting for relevant covariates that could influence outcomes.[iii]

Short-Term Case Studies: Examining the Innovation Process

Many innovation oriented organizations have used case studies to evaluate the impacts of their programs. One reason for launching challenges is to change the incentives and conditions that currently exist around innovation in Canada. For example, the Women in Clean Tech challenge seeks to strengthen the innovation ecosystem in clean tech by creating a space for talented female entrepreneurs to develop and cultivate their ideas. In light of these goals, the IIU will consider using case studies that examine the impact of challenges on the innovation process. A pioneering approach here has been the short-term case study undertaken by Sitra of their challenge Ratkaisu 100[iv] which documents the impact of a challenge on participants as they are going through it. To date, the IIU has produced two case studies and will be publishing more in the future.

Long-Term Impact Studies: Measuring Public Value

As solutions developed throughout challenges come to be deployed in the real world, it will become imperative to examine the concrete impacts that these solutions are having on their respective issue areas. Through a longer term impact study, the IIU could examine the ways in which technologies, products, and services developed during challenges are being utilized. For example, if data eventually becomes available on the number of daily users the winning solution from the Drug Checking Technology Challenge has, the IIU could conduct statistical analysis about how the technology has been deployed to prevent overdoses within specific geographic areas. Each long-term impact study is likely to be very different given divergent pathways to impact that every challenge is likely to have. Nonetheless, long-term impact studies will ideally show how challenges have had an influence on some of Canada’s must pressing issues, responding to the longer-term outcomes of the challenges logic model, focused on societal, environmental and economic outcomes

Conclusion

As the IIU accumulates evidence on the effects of challenges, this knowledge will be used to understand the process and outcomes of challenges and to improve future challenges and initiatives. However, the importance of impact measurement extends beyond the need for institutional learning. Over the coming years, the insights obtained during impact measurement will help tell the story about how challenges have played a role in improving the lives of Canadians.

Notes

[i] Mohammed Saqr and Ahmad Alamro. ”The role of social network analysis as a learning analytics tool in online problem based learning,” BMC Medical Education 19, no.1 (2019): 1–11. https://doi.org/10.1186/s12909-019-1599-6

[ii] Nesta, Attracting Investment with challenge prizes, by Vidal Kumar. London, England: 2021, https://challenges.org/reports/attracting-investment-with-challenge-prizes/

[iii] Susan Athey and Guido Imbems. “The state of applied econometrics: Causality and policy evaluation,” The Journal of Economic Perspectives. 31, no.2 (2017): 3–32. https://doi.org/10.1257/jep.31.2.3

[iv] Sitra, Sparking social innovation through challenge prizes: Evidence on Teams, Ideas, and Incubation from Finland, by Tuuka Toivonen, Emma Nordbäck, and Ville Takala. ISBN: 978–952–347–052–1. Helsinki, Finland: 2018, sparking-social-innovation-through-challenge-prizes.pdf (sitra.fi)