INNOVATION ACCOUNTING METHODS TO ASSURE VALIDATED LEARNING: THE CASE OF FINNISH STARTUPS.

Bachelor's thesis International Business Degree programme Innovation and Technology Management 2015 Esteban Soto INNOVATION ACCOUNTING METHODS TO ASSURE VALIDATED LEARNING: THE CASE OF FINNISH STARTUPS.

BACHELOR'S THESIS ABSTRACT TURKU UNIVERSITY OF APPLIED SCIENCES International Business Degree programme Innovation and Technology Management 2015 61 Instructor: Alberto Gonzalez José Esteban Soto Hernández INNOVATION ACCOUNTING METHODS TO ASSURE VALIDATED LEARNING: THE CASE OF FINNISH STARTUPS This research takes a look at the different ways to measure progress while in a startup. Based on the different theories explained during the literature review, it is clear that a startup needs to measure its progress in a special manner. The theoretical background starts from briefly defining a startup, innovation and the sources of innovation. Then moves to explain the Lean Startup method s ideas, validated learning, measuring innovation and innovation accounting (IA) for startups. Multi-case qualitative study was selected to be the methodology for this thesis. Therefore, four Finnish startups were interviewed and its data analysed one by one. The purpose for the interview was to be able to know how Finnish new ventures are using validated learning to produce value to users, while taking a look at the metrics and targets these companies use to create a sustainable business. After the individual case analysis a multi-case analysis is performed during chapter 4.2. The multi-case analysis gives the opportunity to present the research findings in the conclusion, it is in the conclusion were the author matched the findings to the research objectives. The case study indicates that IA is a corner stone for the Finnish startups interviewed. There was a common approach to validate learning. In terms of accounting, the metrics and targets selected varied from company to company. Their approach to IA was significantly similar to the theory presented during the theoretical background part of the research. KEYWORDS: International Business Startups Lean Method Case Study Innovation Accounting Validated Learning Research

OPINNÄYTETYÖ (AMK) TIIVISTELMÄ TURUN AMMATTIKORKEAKOULU International Business Degree programme Innovation and Technology Management 2015 61 Ohjaaja: Alberto Gonzalez José Esteban Soto Hernández OPINNÄYTETYÖN NIMI ASIASANAT: Kirjoita asiasanat tähän. Etsi sopivia asiasanoja ONKI -ontologiapalvelun YSA (Yleinen suomalainen asiasanasto) ja MUSA (Musiikin asiasanasto) asiasanastoista.)

CONTENT LIST OF ABBREVIATIONS (OR) SYMBOLS 6 1 INTRODUCTION 7 1.1 Background of the thesis 7 1.2 Thesis Objective 8 1.3 Research questions 9 1.4 Thesis Structure 9 2 THEORETICAL BACKGROUND 10 2.1 What is a startup? 10 2.2 What is innovation? 11 2.3 Sources of innovation 12 2.4 Lean Startup 12 2.5 The Lean Startup method 17 2.5.1 Entrepreneurs are everywhere. 18 2.5.2 Entrepreneurship is management. 18 2.5.3 Validated Learning. 19 2.5.4 Build-Measure-Learn. 19 2.5.5 Innovation accounting. 21 2.6 Measuring innovation. 21 2.6.1 Output and input measurements in innovation 22 2.6.2 Innovation measurement based on econometrics 23 2.7 Innovation accounting for startups. 24 2.7.1 Establish the baseline 25 2.7.2 Tune the engine 26 2.7.3 Pivot or persevere 31 3 METHODOLOGY 32 3.1 Research Methodology 32 3.2 Research trustworthiness 34 3.3 Data Collection 34 4 EMPIRICAL ANALYSIS OF THE RESEARCH 36 4.1 Case by case analysis 36

4.1.1 Zadaa 36 4.1.2 Eliademy 39 4.1.3 LeadDesk 42 4.1.4 Smarp 44 4.2 Multi-case Analysis 48 5 CONCLUSIONS 53 5.1 Research findings 53 5.2 Further research suggestions 56 REFERENCES 57 APPENDICES Appendix 1. Interview questions to Finnish startups entrepreneurs 61 FIGURES Figure 1. Steve Blank s Customer Development (Brant Cooper, 2010) 14 Figure 2. Roger s adopter categorization on the basis of innovativeness 16 Figure 3. Three principles of a startup (Ries, 2011) 18 Figure 4. The Lean Startup Cycle: Build-Measure-Learn (Ries, 2011) 20 Figure 5. Dave McClure s Pirate Metrics (Maurya, 2010) 27 Figure 6. Weekly Cohort by join date. Running Lean (Maurya, 2010) 31 Figure 7. Overview of companies interviewed (Soto, 2015) 36 Figure 8. Eliademy s Impact Calculation Method Example (Gerasimenko, 2015) 41 Figure 9. Connected User Retention Cohort Table (Smarp, 2015) 46 Figure 10. Overview of each startup value creation, metrics and targeting methods (Soto, 2015) 48

LIST OF ABBREVIATIONS (OR) SYMBOLS Abbreviation KPI ROI EBITDA ARPU OECD AARRR VC MVP R&D OMTM SaaS CEO CPO UX UI ARR EA IA MUC Explanation of abbreviation Key performance indicator Return on investment Earnings before interests, taxes, depreciation and amortization Average revenue per user Organization for economic cooperation and development Acquisition, activation, retention, referral, and revenue Venture capital Minimum viable product Research and development One metric that matters Software as a service Chief executive officer Chief product officer User experience User interface Annual recurring revenue Employee advocacy Innovation accounting Monthly unique clicks

7 1 INTRODUCTION 1.1 Background of the thesis The problem is not an inability to take action but an inability to take appropriate action (Sull, 1999) Innovation has been present throughout our history since the time we invented the wheel. From this invention, to the industrial revolution, and building cars and rockets. That is the way we behave as humans, we are always seeking for better ways of doing things, of improving. It is common for individuals and organizations that are not under any particular pressure to adopt new ways of doing things because they have reason to believe that these would be improvements (Nelson, 2012). During the past years there has been a declining of new companies in Finland. In 2013 there were 3,274 less new companies than in 2010 and the amount of failing companies has increased from 21,449 in year 2010 to 25,441 in year 2012 (Tilastokeskus-Statistics Finland, 2014). Furthermore, a study by Nordic Credit Alliance established that in Finland, due to current economic situations, consumers are being more cautious and their purchasing power has weakened (Talous Sanomat, 2014). Although current economy is one of the main reasons of failure, it has been proven that companies also fail due to poor management of innovation processes. Some of these entrepreneurs do not seek for customer feedback, they keep building and fritter away time on something that they do not know is going to be of value to their users. (Sull, 1999). On top of this, the majority of entrepreneurs does not accept the data coming from their market research and experiments, therefore they will not know how to deal with a failed business hypothesis (Feinleib, 2011).

8 Several authors (Maurya, 2010; Ries, 2011; Blank & Dorf, 2012; Croll & Yoskovitz, 2013) have paid close attention to the difficulties that appear when managing startups. Their research made a great contribution to approach the innovation management issues that new ventures face when trying to measure their learning. This learning comes by performing tests to product assumptions. Eric Ries (2010) expresses that for a startup is crucial to learn as fast as possible to avoid overspending hours, money and other resources when providing their service. The commonly used KPIs of ROI, EBITDA, ARPU, etc. (See abbreviations on page 6) are not the ones in which a startup needs to focus, at least in the beginning when uncertainty is at its highest point. Based on the Lean Startup methodology, the key to create a successful product is in the innovation accounting (IA) measurements that are used to validate hypothesis-driven experiments and assumptions. But how is this possible? How to measure progress when no proper framework for startups can be found due to extreme uncertainty? The Lean Startup gives a set of steps to not only learn from assumptions but to validate them. This allows a startup to decide if the product needs to be modified (pivot) or not (persevere). 1.2 Thesis Objective The purpose for this thesis is to define what kind of IA methods startups use and how those methods are being used to validate assumptions to grow their business. Is there a common approach when measuring their innovation? This subject is highly appealing and personal to the author since he has been taking part in the entrepreneurship world from the beginning of his degree and now works for a SaaS startup. The author is eager to learn more on how to drive success in startups based on real scenarios. The motivation behind this research is to teach future entrepreneurs and startups about current IA practices from successful ventures.

9 1.3 Research questions This thesis goal is to find an answer to the following questions: 1. How are Finnish startups using validated learning to create value to users and a sustainable business? 2. What kind of metrics they use to know if they are making progress? 3. How have they decided the target for the metrics used? 1.4 Thesis Structure The theoretical background of this thesis will be included in chapter number two. Theories and methods of innovation management, measurement, and accounting will be introduced and explained. During the chapter, Innovation Management theories will be briefly discussed followed by explanations of the lean startup, measuring innovation and IA methods for startups. In chapter three the methodology for this research will be reviewed. There will be further explanation on the way the study was designed and how the data were collected. Research trustworthiness is an important topic that will be discussed as well in the third chapter. After having explained the methods used for the research in chapter three, the data collected from the interviews to startups will be presented and analyzed in chapter four. The fifth and final chapter of this study will be a complete summary of the whole thesis from its applied theories to the findings based on the interviews conducted. It is in this part where the research questions will be answered and a personal opinion on what can be added to continue with further research concerning IA.

10 2 THEORETICAL BACKGROUND 2.1 What is a startup? The phrase start up has been used in the English language since the 1550s, while the startup word came to life around the year 1845 (Douglas Harper Etymology Dictionary, 2010). As a word, startup means the act or process of starting or making something start (Cambridge Dictionary 2015). When you think and talk about new companies, it makes complete sense to call them startups. Now, what is a startup really? What do you consider as a new company? Can you call it a startup because of the months or years they have been running? Is it the amount of investment received? Or the quantity of profit made? Many authors have given their insight to help us divide a well-stablished company from a startup. Steve Blank (2010) expresses that a startup is an organization formed to search for a scalable and repeatable business model. To be considered a startup you are in a constant search of growth. A common aspect that startups have is that they are designed to increase size rapidly. To Paul Graham, Y Combinator co-founder and respected entrepreneur, a startup does not need to be a newly founded company, it could also be a new project inside a big corporation. A startup does not necessarily need to get investment, be technology oriented or be bought out. The only essential thing is growth. Everything else we associate with startups follows from growth. (Graham, 2012). Uncertainty is another characteristic for a venture to be treated as startup. Risks are always present and there is no way to ensure whether the product or service created will be successful. A startup is a human institution designed to deliver a new product or service under conditions of extreme uncertainty. (Ries, 2011). There is no need for a venture to be newly founded in order to be considered a startup. A company with a vision to grow, where its outcome is not predictable and has the capacity to enter new markets is a startup. Even a big corporation

11 can have startups within their business. The amount of resources can be significantly higher for these kind of new ventures, still what makes them startups is that they have the 3 key factors of growth, unpredictability and market disruption. 2.2 What is innovation? To define innovation we need to define invention too. Both terms can be very different from each other but in some cases there is a thin line between them. Invention is the first occurrence of an idea for a new product or process, while innovation is the first attempt to carry it out into practice (Fagerberg, 2004) In the Oxford Handbook of Innovation (2004) Fagerberg mentions that universities are among the organizations where inventions may be produced, while innovations happen in companies but also in places such as public hospitals. The combination of different types of knowledge, skills, infrastructure, ongoing market research and financial resources, for example, is what makes innovation possible in firms. Innovation, at the level of an individual firm, might be defined as the application of ideas that are new to the firm, whether the new ideas are embodied in products, processes, services, or in work organisation, management or marketing systems (DIST, 1996, p.2, and credited to Gibbons et al, 1994). Steve Blank (2015) sees innovation from a different perspective. For him, innovation is satisfying users current or future wants/needs by turning an idea into a product or service with speed and urgency, using minimal resources and costs. Finally, the business council of Australia highlights that to be considered innovative, the final product needs to be disruptive and commercially successful or valuable for clients. They define it as the following: In business, innovation is something that is new or significantly improved, done by an enterprise to create added value either directly for the enterprise or indirectly for its customers (BCA 1993, p.3)

12 2.3 Sources of innovation Innovation can be systematically managed if one knows where and how to look (Drucker, 1998) According to the European Commission s Directorate-general for Enterprise (2004), innovation has evolved because of the addition of social ingredients in its definition. Before social ingredients were included, innovation was defined only by palpable forms of capital. For Peter Drucker (1985) most innovations, including the most successful ones, come from a calculated and well-researched approach to find innovation opportunities. Of course there can be innovations out of a genius thought, an eureka moment but that does not happen too often. He then established 7 sources of innovation. Four areas where opportunity for innovation appears to companies or industries are: unexpected occurrences, incongruities, process needs, and industry and market changes. Other areas of opportunity can come from outside a company, things that they have absolutely no control of, such as demographic changes, changes in perception, and new knowledge (Drucker, 1985). By systematically monitoring these sources, innovation opportunities will occur. 2.4 Lean Startup The Lean Startup is a new way of looking at the development of innovative new products that emphasizes fast iteration and customer insight, a huge vision, and great ambition, all at the same time. (Ries, 2011, 30) In 2011, Eric Ries developed a framework based on lean manufacturing and all his years of experience in building software, entrepreneurship and failures. He called it the Lean Startup. Throughout my career, I kept having the experience of working incredibly hard on products that ultimately failed in the marketplace (Ries, 2011).

13 In the beginning, Ries thought that failures were caused by problems in the software architecture, engineering processes, focus, product vision and overall lack of discipline. Soon he realized that it does not matter how much he tried to fix the problems more issues appeared. This is why Ries looked for advice in books and from the best minds in Silicon Valley such as Steve Blank. Ries became cofounder of IMVU in 2004, since then the desire of knowing more of how to build a new company kept growing. IMVU is an online virtual-reality platform where users can interact through an avatar with other avatars in a pre-set digital environment (Ries, 2011). During the time when IMVU was founded, serial entrepreneur Steve Blank, came up with the customer development process. The approach inspired Eric Ries to work on a method which he called The Lean Startup. This led to a movement after he published the book explaining the method in 2011. The proposition of Ries method is that a startup s set of ideas are just hypotheses in need of testing (Blank, 2014). Here is where Blank s customer development approach makes an entrance to the Lean Startup world. As one can see from figure 2, the customer development process is often referred to a systematic get out of the building approach to test and validate a new product or service hypotheses. In other words, going out to ask a potential customer its opinion of the future product or service. This is done in order to build a stable business model which can grow over time (Blank, 2015).

14 Figure 1. Steve Blank s Customer Development. (Brant Cooper, 2010) About three-quarters of venture-backed startups fail. This is according to a study, which involved more than 2000 companies with an investment of more than $1 million during the period of 2004-2010 (Ghosh, 2012, HBR). This is what many specialists in the innovation field such as Ries, Blank, and von Hippel wanted to reduce with their innovation theories for startups. The whole idea of Lean startup comes from the lean manufacturing movement that Taiichi Ohno and Shigeo Shingo developed when working at Toyota in Japan. While Ohno and Shingo s method was applied to physical manufacturing, Ries shaped it to make it work with the entrepreneurial challenges startups face (Ries, 2011). The purpose of the new framework was to apply lean thinking into innovation processes. With this approach Ries wanted to eliminate the incredible amount of waste startups were producing. Some waste examples are hours of useless coding, products that nobody wanted, new products that were taken down from retailers and overall failure. Therefore the need of understanding the potential user point of view and its feedback is important. This to lower the risk of failure by creating a product people wants (Ries, 2011).

15 We can see similarities between von Hippel s (1986) ideas and the Lean Startup method. For von Hippel and Ries, it is crucial to identify the lead user to trial a product and adopt a position towards previously made assumptions that help answer questions such as: Is this something of value to the user? Will it have a good reception when the product is given to a larger target audience? Should we pivot, which means making a radical change to the business strategy, or persevere? Back in 1986 Eric von Hippel, economist and professor at MIT Sloan School of Management, presented his paper Lead Users: An important source of novel product concepts in which he focused on his theory of lead users. Von Hippel expresses the importance of understanding the need of a user to create a commercially successful product. The requirement of a better way to create market research, not just the conventional market analyses that companies were used to do, was what inspired him to present the term of lead users of a product or service. Lead users are defined as (1) users whose present strong needs will become general in a marketplace months or years in the future (von Hippel, 1986) and (2) They anticipate relatively high benefits from obtaining a solution to their needs, and so may innovate (von Hippel, 2005). Meaning that this need will grow into a general need and produce the opportunity to become a successful venture. As seen in figure 3, the lead users have characteristics in common with the innovators of the innovation diffusion theory. From the innovation diffusion point of view, 2.5% of the market is considered to be innovators, they are excited to try out new ideas or products. We know then that some users are more likely to adopt innovation before others (Rogers, 1981).

16 Figure 2. Rogers adopter categorization on the basis of innovativeness. Image source: www.mbaskool.com Now, looking more into IA, another important aspect of Ries method is also the way of measuring progress. The right way to measure progress in lean manufacturing is to produce high-quality physical goods. When using Lean Startup, progress needs to be measured with a different unit called validated learning. With scientific learning as our yardstick, we can discover and eliminate the sources of waste that are plaguing entrepreneurship (Ries, 2011, 28). The reason why it is hard for a startup to focus on learning gathered from future users is that this learning is intangible, while coding, manufactured prototypes, all of this is tangible and can be presented as progress. Based on the Lean Startup, entrepreneurs need to work as fast as possible and at the same time they should be measuring their productivity by learning if people would like to use or purchase their product. An entrepreneur must remember at all times, as obvious as it is, that a company will be successful only if it sells what it has created (Ries, 2011). In spite of declarations from innovation specialists like von Hippel, Blank, Croll and Yoskovitz, on how revolutionary Lead User Theory and Lean Startup is, there are other experts that have their own views about it. One of them is Marc Andreessen, co-founder of Netscape and VC firm Andreessen Horowitz. For him, not every startup should be a Lean Startup and VC firms need to be cautious in rejecting startups that are too ambitious (Andreessen & Kern, 2012).

17 Some new ventures need to start big, they cannot just go from trial and error, pivoting, quick iterations and inexpensive minimum viable products. Some examples such as Tesla Motors, SpaceX and many more companies, had to create a product in its entirety from the beginning so that people could see the benefit of it. I do not think the Lean Startup idea, as brilliant as it is, and as widely applicable as it is, should halt us from investing in these big ideas right out of the gate (Andreessen, 2012) 2.5 The Lean Startup method Failure comes most of the time after thinking that you have a good plan, strategy and have made a proper market research. In the current startup world this approach does not work, there is too much uncertainty that needs to be cleared before you start to see progress. The more we see into the future the more uncertainty you will have (Ries, 2011). Entrepreneurs think they can predict the future when they should be creating a future together with their clients. Entrepreneurs tend to be single-minded with their strategies wanting the venture to be all about the technology or all about the sales, without taking time to form a balance plan (Ghosh, 2011) This is why the method is targeted to entrepreneurs and people to whom they need to show results. It has been divided into the following 5 principles: 1. Entrepreneurs are everywhere. 2. Entrepreneurship is management. 3. Validated Learning. 4. Build-Measure-Learn. 5. Innovation accounting.

18 2.5.1 Entrepreneurs are everywhere. Someone who identifies itself with Ries (2011) following definition of entrepreneurship A human institution designed to create new products and services under conditions of extreme uncertainty is considered an entrepreneur. There is no need to be working in a garage or basement to be one. This is the reason entrepreneurs are everywhere, they can be independent, work in a small company or part of a big enterprise within any industry 2.5.2 Entrepreneurship is management. A startup is an institution (Ries, 2011). This is commonly misunderstood, a startup is not just a product it is an organization that requires management to focus on a vision that will take it out of its extreme uncertainty condition. For a startup to succeed in its innovation opportunity, a dynamic and disciplined management model needs to be created exclusively for them. General management is commonly used in companies because of its huge success in the past but this approach does not work in conditions of big uncertainty. Product Strategy Vision Figure 3. Three principles of a startup. (Ries, 2011) As one can see from figure 4, a startup has 3 main parts: vision, strategy and product. The only ones that change in time are strategy and product but vision will remain the same. If you know where you are heading, for example going by bike from point A to point B, you can change the route (strategy) or the vehicle

19 (product) and you will get to point B. However, the fastest you reach to point B the better. (Ries, 2011) The Lean Startup establishes that product can always be optimized, and they are in constant change and improvements, it is called tuning the engine. Regarding strategy, it does not change as often as the product but when it does it will pivot. Both of them, strategy and product, should be tested every time there is a new assumption. If the test fails to meet expectations a change, what he calls pivot, is needed as fast as possible. If the test succeeds then you will persevere. (Ries, 2011) 2.5.3 Validated Learning. This is the unit Eric Ries created to measure progress in a new venture. He got inspired by Taiichi Ohno and Shigeo Shingo s strategy of Lean Manufacturing. In Lean Manufacturing, the main goal is to reduce waste in a process and its progress is measured by the production of high-quality goods. Lean Startup aims to reduce waste too, like time used on developing features that gives no value to customers, but it needs to be measured differently (Ries, 2011). The reason why the unit of validated learning was created relates to the high uncertainty a new venture has; validated learning is essential to make sure that every element of a startup s vision is working as it should. Startups gather validated learning by testing their product and strategy, with this approach they can start moving towards being a sustainable business. A definition of sustainable business is one whose principal productive asset is not just the founder s skills, contacts, and efforts. (Bhide, 1996). Therefore, this is the goal for every entrepreneur. 2.5.4 Build-Measure-Learn. Ries (2011) mentions a cycle that every startup should have, Build-Measure- Learn (see figure 5). First it is needed to build the idea of the product or service,

20 he calls it a minimum viable product (MVP). The term, coined by Frank Robinson in 1987, means a product that has just the necessary features needed to do was it is intended to do. It is the simplest product possible that can create value to a potential customer. Build ideas Learn Minimize TOTAL time through the loop. (Clockwise) product Measure data Figure 4. The Lean Startup Cycle: Build-Measure-Learn. (Ries, 2011) After the MVP, measurement has to take place to get to know the way clients are responding to the product. Then gather enough data to learn if it is necessary to change the business strategy completely or continue moving forward with the original idea. In other words, pivot or persevere. Finally, optimize based on the validated learning, this is known as the Build part of the loop once again. By introducing a MVP and following optimizations an entrepreneur can continue learning from its two most important assumptions: 1. Value hypothesis, is this viable to the market? Is it delivering value to customers? 2. Growth hypothesis, is this able to scale up without having to make too much effort for customers to use it again and again? All successful startup processes should be geared to accelerate that feedback loop (Ries, 2011).

21 2.5.5 Innovation accounting. Within the build, measure, learn cycle there is a stage where a startup will need to measure its progress. The faster this measurement becomes, the more efficient the venture. This measurement is a way to keep the feet on the ground and the results provided from these analytics objectively tells how well a startup is doing. For Croll and Yoskovitz (2013), entrepreneurs need to stop lying to themselves, they need to be careful of vanity metrics. These are metrics that look really good on paper for investor hunting, board meetings, press releases but have accomplished absolutely anything to create more value for the startup. It is understandable to like positive numbers, it is a common entrepreneurial behavior to be optimistic. However there are eight important vanity metrics that should be avoided: number of hits, page views, visits, unique visitors, followers/likes, downloads, emails collected and time spent on the site. This point of view seems to be in line with Ries, who refers to vanity metrics as something to avoid because of the harm they can create to the health of a startup (Ries, 2011). The methods of measuring innovation and how they allow a business to get to a healthy and sustainable growth will be amplified in point 2.6. 2.6 Measuring innovation. If you measure better, you re more likely to succeed (Croll and Yoskovitz, 2013). It is interesting that publications such as the third edition of the Oslo Manual (2005), which focuses on guidelines for collecting and interpreting innovation data, there is a vague study on how to measure innovation. Instead they focus more on measuring expenditures for innovation. By measuring these they try to give a guideline on how to track innovation in intangible investments such as training or marketing campaigns and tangible investments such as machinery. The issue here is that this kind of innovation measurement does not give any

22 value to a startup. A startup does not have too much capital to invest, this is why they have to test that the strategy followed is progressing. One person who studied how innovation could be measured was Mark Rogers (1998). He also focused on firms, already established companies with a track of records. He even accepts that the measurement of innovation is likely to be difficult due to the broad nature of the scope of innovative activities. For this reason, Rogers divided his approach into two ways to assess innovation: outputs of innovative activity and inputs to innovative activity. 2.6.1 Output and input measurements in innovation For him, the most important output of innovative activity is the success of the company. To measure output, indicators such as market capitalization, revenue growth, profit or productivity need to be taken into account. He also agrees that these indicators can have their own problems, since they can be caused by factors other than the level of innovativeness (Rogers, 1998). On the other hand, the most used criteria to measure input to innovative activity is research and development. Based on the Australian Bureau of Statistics (ABS) definition of R&D, Rogers insists that this kind of input isn t a good way to measure innovation since sometimes it will not create a useful innovation, it will rather create just knowledge and to be considered innovation, the product needs to have a practical application. systematic investigation or experimentation involving innovation or technical risk, the outcome of which is new knowledge, with or without a specific practical application of new or improved products, processes, materials, devices or services. (ABS, 1996 p26) Other input criteria are intellectual property statistics, acquisition of technology from others, intangible assets, marketing expenses, training, cost of tools to do the job and managerial change. The way Rogers insists on dealing with consistency through the process of measuring innovativeness is to use specific econometric techniques. When using these techniques it is then possible to quantify

23 a firm innovativeness by comparing it directly to other firms who used the same process. 2.6.2 Innovation measurement based on econometrics One of the issues when using output and input criteria is that they are considered separate or partial measurements. The reason is that they can provide only partial data of the overall innovation of a company. One option to solve this is to add different type of criteria together. However, sometimes the values of these criteria such as managerial change (input) or introduction of improved products (output) can be measured only by a yes or no response, so the units of measurement do not match. Therefore, an alternative method which uses econometric techniques can match these measurements to the overall performance of the company. This allows inferences about the value of the different innovation activities, as well as an assessment of the overall value of innovation activities (Rogers, 1998). Econometrics has been described as the mix of economics, mathematics, and statistics (Tintner, 1953). Marschak (1948) takes an interesting approach to explain econometrics, by defining mathematics and statistics. Mathematics teaches how to derive propositions from other propositions; statistics teaches how to derive propositions from observed facts. Mathematics would then coincide with deductive logic, and statistics with inductive logic. Econometrics would then be simply the application of rules of logic to economics. Econometrics is commonly used with R&D data. Some variables that can be used together to measure success of a venture are: market value and productivity, patent data instead of R&D or R&D and patent data together (Rogers, 1998). This also is in line with the definition of a good metric by Croll and Yoskovitz (2013). A good metric is comparative, understandable, it is a ratio or rate and it changes the way you behave. Hagedoom and Cloodt (2002) express that the econometrics approach has the advantage of giving the possibility to analyze innovative performance through a more complete measurement system. Then the measurement is not left to just a

24 single indicator which can be selected conveniently to show the most positive result but takes multiple ones, which will add credibility to the results. 2.7 Innovation accounting for startups. Going back to the startup reality, there is absolutely no way that a startup can use indicators such as Rogers input and output criteria to measure their progress (Ries, 2011). The reason is that they do not have any previous data, they are a new venture. How can they start their IA process without data? In the Lean Startup method, one of the most important aspects is to commit to iteration. This should be agreed previous any test is performed, no matter what happens after the test you will not lose hope but work to solve the issue (Ries, 2011). The first step into IA is to have a MVP, with it a startup will be able to modify its strategy and measure what is working or not. Accounting has to be implemented to have control of every aspect of the business, therefore it contributes greatly to its success (Ries, 2011). The key is to translate a startup assumptions into a quantitative model. The real analytical work starts the minute you develop and launch an MVP, because every interaction between a customer and MVP results in data you can analyze (Croll & Yoskovitz, 2013) When launching a product there are many things that may not work. Sometimes entrepreneurs do not think of these things as part of the product but they are. Some examples of these are pricing, design or positioning. Thinking of solving all of these issues can be overwhelming. This is why it is important to keep focus and identify the key metrics that will help the startup to understand how to make the product better (Maurya, 2010). Ries (2011) explains that startups usually answer to his question of How do you know you are making your product better? in a way in which they cannot explain why they are getting specific results. Startups think they might be going on the right direction if overall numbers get higher and customers seem to like the new

25 features but this is not an indicator of progress. This is the reason IA is needed. It allows a better understanding of a new venture s current situation and gives the ability to prove that they are on the right track in growing a sustainable business. 2.7.1 Establish the baseline IA has 3 basics steps that can work for any industry (Ries, 2011): establishing the baseline, tuning the engine and pivot or persevere. As mentioned before, a MVP is needed in order to have information on the current position of the startup, the first information gathered is also known as baseline data. Baseline data includes for example customer lifetime, value, sign ups, trial rates and conversion rates. This is data that serves as the foundation to learn about how customers react and think of the product, for better or for worse. With the collection of customer reactions and feedback, it is then possible to modify the MVP into something more valuable to them. The best part is that this decision of optimization is backed by real data. The road to the MVP is full of qualitative learning. However, an entrepreneur needs actionable metrics to measure the activity of its users or user lifecycle. The objective after finding the product and market fit is to optimize for conversion guided by actionable metrics. An actionable metric is one that ties specific and repeatable actions to observed results, it is the opposite of vanity metrics (Maurya, 2010). Ash Maurya (2010) also describes user lifecycle as the path a user takes from first landing on your website to eventually becoming a passionate user. To measure a user lifecycle quantitatively it is possible to track customer behavior. Methods such as Dave McClure s (2008) Pirate Metrics: AARRR (Acquisition, Activation, Retention, Referral, and Revenue), Lean Startup or Lean Analytics have been developed to track actionable metrics.

26 2.7.2 Tune the engine Everything you build into the initial MVP should relate to and impact the one metric that matters, OMTM. (Croll & Yoskovitz, 2013) A startup can tune its engine towards what the founders have envisioned as the main goal. This means that by handing over the product to clients, they will get feedback that contributes to the product optimization and company growth. The learning gathered in this second step will dictate the kind of changes needed to improve the metric selected, if the metric does not improve then it means that the product or service optimization was a failure. (Ries, 2011) New ventures can have tens of assumptions, because of this, the uncertainty is high. This is why they need to be focused on trying to test the most risky assumption. The reasoning behind this idea is that if there isn t a way to reduce the risk in direction to the ideal sustainable business vision, then it makes no sense in testing the rest of assumptions. (Ries, 2011) Croll and Yoskovitz (2013) also highlight the fact that a startup needs to focus on what is important, they call it the One Metric That Matters (OMTM). The OMTM is the one number you re completely focused on above everything for your current stage. It is fine to track many KPIs at the same time, however, a startup has to focus and select the most important metric for their current stage. By keeping KPIs at a minimum and focused on the most important metric, the organization can have more control in their experimentation. Subsequently, this will allow to move to the ideal business vision in the most effective way possible. One way to start measuring is to have a macro conversion funnel. McClure s (2008), see figure 7, categorization of AARRR is an example of macro metrics and it is the basis for defining a macro conversion funnel. This helps startups to keep track of their progress quantitatively. A macro metric is a roll-up of several steps or sub-funnels (Maurya, 2010). McClure s framework does not necessarily need to follow a strict order, referral may come before revenue for example. The important aspect is to know what each of the 5 elements mean to be able to

27 manage and measure a company s growth. Entrepreneurs need to remember by this point that to start measuring, they need to establish what means to be successful on each of the key metrics in use. If they get successful results they should keep making tests to improve those (Croll & Yoskovitz, 2013). Figure 5. Dave McClure s Pirate Metrics (Maurya, 2010). Description of McClure s 5 key elements: Acquisition, is the path a user follows from finding the product to become an interested user. A way to measure this is to identify what means acquisition for the startup. For some startups a successful acquisition can be that a user sees their sign up page or read a certain article. Even if the user stayed at the website for a few minutes, if the behavior wanted did not happen, then it cannot be considered as success (Maurya, 2010). Other metrics relevant to this element are traffic, mentions, cost per click or cost of acquisition, search results and open rate (Croll & Yoskovitz, 2013). Activation happens once a user is interested. It involves users who got interested enough to sign-up to the service and then they go through the process of having the first happy experience. Key metrics to track are pages per visit, time on site, conversions (McClure, 2008). Retention, when enrolled consumers keep using the service over and over again. It is one of the key metrics to measure product/market fit (Maurya, 2010). Some examples of retention goals can be 1 3 or more visits per month or 20% email open rate with 2% Click-through rate. The main idea

28 is to know how users come back. How Often? and how to keep them coming? (McClure, 2008). Metrics commonly used to measure are engagement, time since last visit, daily/monthly active use, and churn (Croll & Yoskovitz, 2013). Revenue, measures the events that get a startup money. Most of the products have a trial period where users can have a first look at what the service has to offer. Revenue events typically occur after Activation and some repeated-use - Retention (Maurya, 2010). The events depend on the business model, they can be purchases, ad clicks, content creation or even subscriptions. Referral, it happens when happy customers invite other potential users to use the service or product, leaving space for conversion. It is also considered as an advanced type of acquisition channel (Maurya, 2010). McClure (2008) suggests to focus on driving referrals after getting at least 8 customers out of 10 being happy with the experience. Word-of-mouth, viral campaigns, it can start from a simple share with a friend feature on social media. Important metrics to track would be invites sent, viral growth factor, viral cycle time (Croll & Yoskovitz, 2013). When talking about tracking data, McClure (2008) also suggests 4 measurement types: qualitative, quantitative, comparative and competitive. Qualitative measures a small amount of users through usability testing. By monitoring the session, a startup can begin to optimize the service based on the problems seen during the test. Quantitative is how users are engaging with the service in terms of traffic, conversions, percentage of conversion, etc. Comparative means doing a test to compare what users are doing in one controlled situation and others in the other defined controlled situation. Comparative testing is usually done by A/B tests, segmentation or cohort analysis. The importance of doing this will give enough data to select what scenario is the most effective for the result wanted. (Croll & Yoskovitz, 2013). Finally, competitive

29 measurement is basically track what competitors are doing and compare that activity against the startup. Comparisons can be from channels, keyword traffic, demographics, customer satisfaction, etc. (McClure, 2008) Having selected what metric or metrics to measure is not the ultimate step. A startup needs to define a target for its metrics. If there is no target then it will be difficult to know what will be the next step after an experiment. Was it a failure, a success or something in between? Croll and Yoskovitz (2013) mention that there are two ways to know what a metric target should be. First it may come from the business model or the second is by comparing with other companies in the same industry. To select your target based on the business model, a good example can be of a company that needs 5% of their users to get the paid version of the service in order to get to the business goal then that 5% is your target. Do what you do, if you cannot get it up to 5% then the experiment is a failure and you will need to think about other ways to increase that percentage. If a business is at the early stages, it is hard to know what the business model should be. Therefore, the other way to know what target to aim, is to know the baseline of the industry. By comparing the startup to it, is then possible to get an idea of how the business model should look like and this makes the target selection easier. (Croll & Yoskovitz, 2013) To choose a target, a startup needs to pick up a number and be completely sure that if that target is reached it means success. It is not just a simple question of how many new clients per week I would like. But how many clients do I need? And to be more specific, How many new customers per week (per acquisition channel) do you think defines a level of success that enables you to double down on user acquisition and move to the next step in the process? (Croll & Yoskovitz, 2013) Studies suggest that optimizing should happen one metric at a time. By focusing on the OMTM and reaching the target, the startup will notice the next metric to focus on. OMTM changes all the time, the change will bring the next metric to pay

30 attention to. This is how a business can work its way to strengthen all of its metrics into a sustainable model. (Croll & Yoskovitz, 2013). Paul Graham (2012) has a similar approach in regards to OMTM when dealing with startups within his Y Combinator seed accelerator. Y Combinator startups measure growth rate per week, one reason for this is to get feedback from users as fast as possible and be able to modify their approach. For him, a good growth rate during Y Combinator is somewhere between 5-7% per week. If you can hit 10% a week you are doing exceptionally well. If you can only manage 1%, it is a sign you have not yet figured out what you are doing. This comes back to Blank s (2010) definition of a startup s main goal: the search for a scalable and repeatable business model. In the case of growth rate, the best way to measure it is revenue. If the startup is not charging initially as part of their strategy then the next best thing to measure growth is active users. However, when choosing active users as proxy to growth rate, the rate has to be higher because these kind of new ventures need a very large amount of users to be successful. The way Y Combinator advise startups to think about a target, in this case growth rate, they think they can reach. After that it is a matter of reaching it every week, if they do not then the startup should be alarmed because they failed at the OMTM (Graham, 2012). For Maurya (2010), doing a cohort comparison is the best way to see the changes in the metrics values over a specific amount of time. By matching the cohort results with specific activities or tests done during an established period of time, a startup will get important data of whether it is progressing or not. Ries (2011) calls this validated learning. As we can see from the example on figure 8, it is possible to understand that acquisition and activation has remained almost the same while revenue grew continually during the last 3 weeks of June. If there is no change among the selected metrics to measure, then it means that the startup is not making progress. Consequently, some more testing needs to be done to improve that specific metric.

31 Figure 6. Weekly Cohort by join date. Source: Running Lean (Maurya, 2010). 2.7.3 Pivot or persevere Finally, after all the optimizations and tests are done, an entrepreneur has to make a crucial decision. Is the startup succeeding? Is the progress big enough that can prove the original vision and assumption is true? Or is it needed to make a major change to the business model? A major change in the business model is called pivot (Ries, 2011). A pivot can be considered as a complete change to one or more of the nine criteria of the business model canvas. Pivots are driven by the learnings and insight from a continuous stream of pass/fail tests you run throughout discovery and validation (Blank & Dorf, 2012). Although the IA, as a scientific method, might give the idea of providing a clear blueprint to make a decision whether to pivot or persevere, this thinking is false. We are humans after all, the power of intuition and vision makes this decision one of the most difficult ones to do. My goal in advocating a scientific approach to the creation of startups is to channel human creativity into its most productive form (Ries, 2011).

32 3 METHODOLOGY 3.1 Research Methodology When conducting a research, the author can base the results on qualitative or quantitative data. In this research, qualitative data was used because it is meant to study people experiences in the most natural setting possible and subsequently tries to analyze these experiences through the meaning people bring to them (Denzin & Lincoln, 2011). The selection of a qualitative research gave the opportunity to represent the opinion of the participants in the best accurate way possible. In the case of this qualitative research the category selected was a case study method with structured interviews. As described by Saunders (2009) interviews can be commonly categorized as structured, semi-structured and unstructured or in-depth interviews. A structured interview uses a questionnaire, for the purpose of this study the interview consists on a formal list of 16 questions that was used with every participant (Saunders, 2009). When gathering data, four sources of information were used instead of one, this to increase the accuracy and reliability of the study (Yin, 2011). In order to better explain the interviewee s approach, some pictures of their measurement processes were presented at the time of analyzing the data collected. Pictures or videos are also considered qualitative sources (Saunders, 2009). A qualitative research gave more opportunity in the topic selection of interest because other research methods might have being limited due to inability of experimentation, insufficient data or coverage, difficulty to get a high sample and response rate, or being limited to study the past and not present events (Yin, 2011). Case study was the method selected for this research. The reason for this was to have an in-depth study about the topic of IA in new ventures. Case study is defined as the study of the particularity and complexity of a single case, coming to understand its activity within important circumstances (Stake, 1995). The

33 structured-type interview and case study method allowed the author to ensure comparability of each startup s approach in measuring progress. This research involves many questions in regards to how startups measure innovation, what kind of metrics they use and how they set up their targets to those metrics. The best way possible for the researcher to answer these questions was to meet in person with the people in charge of the startup. The meeting would give the author the ability to completely understand their point of view in regards to IA within their companies. This is the main reason for selecting case study as the most suitable approach. Qualitative data has its own implications at the time of its analysis. It might be possible to make use of statistics and diagrams but the most common way to analyze qualitative data is by creating a conceptual framework, which can be formulated at any point of the research. Due to the non-standard and complicated nature of the information gathered, the data was summarized, divided into categories based on the questions asked (Saunders, 2009). Then it was analyzed company by company followed by an overall analysis which compares every Finnish startups IA approach. The questions of the interview were structured based on what the literature review established as important criteria to measure progress in a startup. Some of the question where strategically made to have a more in-depth understanding of the company history that would help to analyse the data gathered. The questionnaire (appendix 1) was divided as the following: Questions 1-2: Company background Questions 3-4: Business model Questions 5-7: Validated learning Questions 8-12: Metrics and measurement Questions 13-14: Metric targets Questions 15-16: Company s current position and future Since the research involves four case studies, this type of research can be placed into the multi-case study category (Yin, 2009). The reason to investigate several

34 Finnish startups was to have an in-depth understanding of the reality of each individual company in terms of IA. At the same time a multiple case study helps to get the overall picture of what the SaaS and mobile services industry is doing and identify common areas among each startup. The researcher believes this is the right approach to make the topic understandable for the people directly involved in the study and the readers. 3.2 Research trustworthiness For a qualitative research it is important to evaluate its worthiness by taking a look at the credibility, dependability and confirmability of it (Lincoln & Guba, 1985). In terms of credibility, the author recorded and transcribed the interviews. The information gathered will not be presented as part of this thesis due to the limit of document length. However, the author will archive the data collected for future use. If needed, this qualitative data can be used as reference to test the validity of the findings. This also tackles the dependability of the research. If at any point an external audit should be done by a researcher not involved in the process, this person will have the data collected available. After using the information gathered by the author the audit will get the same results as this research. The purpose of this research is not to produce a theory that can be generalized in what other startups are doing to measure progress. The main purpose is to understand, explain and analyze the current way that selected Finnish startups are handling IA. This way other startups within the field of SaaS or mobile services can compare their IA approach to those described during this study. 3.3 Data Collection For this research a total of four face-to-face interviews were done. All interviewees were selected by the author. Although more than 10 interview requests were sent to other companies, these four startups were the only ones who accepted to

35 be interviewed for this research. In order to get the opportunity to have a thorough comparison between similar business models, the startups selected are in the high tech area of software as a service (SaaS) or mobile app service. The 16 questions asked were designed to provide insights on 3 different areas that helped understand their past, current and future position, while focusing on IA, metrics and setting their metrics targets (see appendix 1). Due to the topic of this study, the profile of the interviewee needed to be that of a founder, an employee who started early in the startup or has an overview of the whole business strategy. This criteria presented challenges and opportunities. Quarter 4 is a time of the year were startups are very busy focusing on closing their year as well as possible. Therefore getting founders, most of them in C-Suite level, meaning chief executive (CEO) or product (CPO) officers, interested in taking the time to be interviewed was a challenge. The opportunity in this case was that they had all the information needed and experienced their company growth from the beginning. This made them the perfect candidates to get to know more of their company s IA approach. The process to get these key people to participate lasted several weeks and the interview request was made through email. After a number of companies were contacted and refusals from a few of them, the following four startups agreed for a recorded interview. 1. Iiro Kormi: Co-founder and CEO of Zadaa (Mobile app service) 2. Sergey Gerasimenko: Co-founder and CPO of Eliademy (SaaS). 3. Olli Nokso-Koivisto: CEO of LeadDesk (SaaS) 4. Roope Heinilä: Co-founder and CEO of Smarp (SaaS)

36 4 EMPIRICAL ANALYSIS OF THE RESEARCH 4.1 Case by case analysis The companies seen in figure 7 will be analysed one by one, this to help the reader to get a full understanding of each case. The author will use the information gathered from the interviews and each case analysis to create a multicase analysis. Figure 7. Overview of companies interviewed (Soto, 2015) 4.1.1 Zadaa Zadaa was founded on May of 2015 in Helsinki, Finland and currently has 3 employees. Their business is an e-commerce mobile application for second hand clothing. To get revenue, they have a commission-based business, Zadaa earns a percentage of the sales their users make through the app. Their application matches people with similar size and style so that users can see, sell or buy clothes within the app easily. Through Iiro s interview the author got to know that the way Zadaa s founders noticed the need for their application, was to simply look at what was happening on Facebook groups, where people was selling second hand clothing. They had the first assumption that Facebook wasn t a good place for second hand e-commerce and decided to create a MVP which they tested with 100 users during summer 2015. These 100 first users can clearly be placed in von Hippel s (1986) group of lead users or also known as innovators (Rogers, 1981). The first users anticipated the benefit of a service like Zadaa s. Then based on the good

37 uptake from the trial users, the founders saw the opportunity in the market and released the ios app. In this case, Zadaa s source of innovation was to exploit the need for a better process to buy and sell second hand clothing (Drucker, 1985). The process existed already through Facebook for example but it was slow and inefficient. Iiro Kormi expressed that the app wasn t completely ready when released but they wanted to go fast to the market to see if the concept work and if there was a possibility for revenue. Since the app is not completely ready, Zadaa s approach to implement new features is to do interviews with users, see the activity on the app and how they behave. They learned based on user feedback that the assumption that people will use their app to find other people with same body size wasn t completely true. What they have learned during past weeks is that people use Zadaa because the process of buy/sell is very easy compared to other options. There are no exact book instructions that they are following. There is the vision of what Zadaa is looking to become and they work towards that. By implementing user feedback they can make adjustments and continue to improve the product to something the user wants and needs. Since the company is in a fast-paced environment and there is no main office yet, the 3 founders meet in person once or twice a week to discuss about important areas. They are in constant contact everyday by other means of communication. The way Zadaa measures progress is by having a set of metrics that tracks every kind of activity around the application. User registration, customer retention in a daily, weekly and monthly basis. Other metrics used are active users, meaning how many of the registered users puts clothes on sale. Also sales, user acquisition (organic vs paid), what screens/buttons users use more, user logins and average session time. Besides these user-based metrics, Zadaa also has implemented team metrics. Customer support response time, new app feature or update release time and number of meetings with investors per week are the most important team-based

38 metrics at the moment. This was also a new finding since all the theories used during the theoretical framework didn t mention team metrics as a way to measure progress. For Iiro something like startup metrics should be a basic thing to know. Of course everyone wants to know how many users you have and is it growing? How many sales you make, what is the cost of customer acquisition? He learned more about metrics not by reading books but by meeting people and asking. Since the company is a few months old, there has been no change on the type of metrics used. In Zadaa there isn t just one most important metric but 2 or 3. User growth which has goals for daily and weekly is one of them and content creation by users. Sales is a metric they are following closely but it is too early to place it as important. The reason Zadaa follow sales is not to know how much money the company is doing, it is to measure how many people listed clothes and successfully sold them. For Iiro this means that the platform is valuable to the users. When having to set the targets to metrics, the method they follow is: common sense. Another way to know that their targets are within good levels of acceptance is by discussing with people who has experience with startups. Usually people say that an app business has to grow at the beginning like 15% every week. So that is the minimum thing, then it is our minimum goal too. Up until now it is going really well in reaching that target. Iiro explained. The software they use to measure user activity is Flurry analytics by Yahoo. They put all the key points and the software makes all the statistics. Zadaa s targets are goals are set on weekly basis but keeps track of metrics on a daily basis. As for experimentation, they haven t made any yet, it will come in time but it is still early to start creating tests. They will release an update of the app soon with improvements and new features, then after this point they will be able to tell if the enhanced product is driving more customer satisfaction for example.

39 In overall, the company has been growing at a pace of 15% minimum, the exact growth percentage wasn t disclosed during the interview. In six months Zadaa s goal is to have a solid and loyal user base. The long term vision of the startup is has been clear for the beginning. This is to continue collecting data and with it, enhance the app so that it selects the right size and style of clothes for the user instead of the user having to search for it. 4.1.2 Eliademy This SaaS startup was founded in 2012 and it has developed an online academy where teachers can create courses online for people around the world. The courses have a range of categories like languages, business, law, technology, art, etc. The platform is free but organizations can purchase a premium version. In that case their business model is subscription-based, charging 1 euro per month per user. The whole idea started as a way to replace Moodle, the e-learning platform that many universities use in Finland. They saw the opportunity on the market because of the user dissatisfaction with Moodle, just by searching in Facebook groups you can see an I hate Moodle one, Eliademy CPO Sergey Gerasimenko told. They worked with a Finnish university to experiment their first product and after some time the founders realized that the business wasn t scalable. As described by Drucker (1985), the source of innovation Eliademy used was through an incompatibility between the efforts of an industry and expectations of customers. Finnish universities take a long time to decide whether to buy and implement a platform and this is where the startup changed the business model. Since they were getting good users reviews they decided to pivot from their previous business model and released the platform to the public. This way everyone could have access to it for free. Monetization wasn t a concern for Eliademy at the beginning.

40 The interesting part is that none of the founders are teachers or has a background as a professional educator. To make progress and know what to do, they started to work with teachers and users. Sergey tracks user support tickets, questions, feedback and complains to be able to know what kind of enhancement is needed to keep users engaged. In addition to user feedback they also have implemented a crowd-based development model. Their user voice service is where users can express what they feel about the platform or features they would like to have and other users can vote if they agree. This way Eliademy can really know what is important to a majority of users and what isn t. The process of deciding which feature to implement is not left to the users, some feature s importance is sometimes more obvious than others. Decisions of features can be also based on what competitors have implemented. Eliademy has a special way to prioritize what feature is worth implementing first or if it is worth implementing at all. This is done by calculating the impact the feature will have on the service. To calculate if a feature will create value they needed to first define what the most important metrics are for the company to measure. Eliademy s focuses on three key metrics: amount of users (students), amount of courses and revenue. As an internet startup it was obvious for them what kind of metrics they needed for their business. This 3 metrics have been continuous through time, they haven t changed. Using the key metrics and a value-based weighted average method they can know the impact of a new feature. It is not common to use this kind of method for product development, Sergey said, he has seen it mostly for offer estimations in consultancy companies. Eliademy adapted the method to work with their key metrics.

41 Figure 8. Eliademy s Impact Calculation Method Example (Gerasimenko, 2015) As seen in the example of figure 9, to measure the impact of a potential feature they first rank the key metrics in order of importance from 1 to 3, where 3 is the most important. In this case revenue is the most important metric they want to focus on. There are 2 features that were asked to be implemented: Facebook sign up and upload of private videos. To know which one would be more valuable for Eliademy s goals, every feature is given a value from 1 to 10 depending on its ability to increase users, courses and revenue. This is how having a Facebook sign up might be very important to get users and received a 5, but it will not drive much revenue therefore it got a 1 in that category. While for private video upload on the other hand is the opposite. The way they select the target for each feature per key metric is by doing a simple guess and the same goes for the 3 key metric targets. By gut feeling and by taking a look on what the competition is doing or what is out there, Eliademy is able to take an informed guess to base their targets. Eliademy has been sustainably growing with almost no money placed to marketing or business development but its growth is based on word-of-mouth from teachers and students. They have been doubling the amount of users every years and based on their previous data and projections this kind of growth is going to continue. The vision is also clear, which is to keep running the platform as a free service, while slowly putting some monetization features to allow the platform to be self-sufficient and sustainable.