This report was prepared by:

Peer review of Small Business Innovation Research programme (SBIR) Netherlands Pilot Peer Review Report 9 JUNE 2010

This report was prepared by: Christiaan Holland, Dialogic Contact details: Dialogic Innovation and Interaction Hooghiemstraplein 33-36, 3514 AX Utrecht - Netherlands Phone: +31 30 2150582 e-mail: holland@dialogic.nl Disclaimer: The views expressed in this report are those of the author and the Peer Review Team. They do not necessarily reflect the opinion or position of the European Commission and in no way commit the involved organisations. 2

TABLE OF CONTENTS TABLE OF CONTENTS...3 1. Introduction...4 2. SBIR Overview of the measure...5 3. SBIR Process...8 4. Results and impact of the measure...10 5. Results of the review exercise...12 6. Pilot Peer Review Process suggestions for improvement...14 Annexes...15 I) Participants to the Peer review visit...15 II) Agenda...16 III) Internal Protocol...18 3

1. INTRODUCTION The Pilot Peer Review Visit described in the present report is part of the INNO-Partnering Forum (IPF). The INNO-Partnering Forum (IPF) is an INNO-Net. It was established as a project under the auspices of the European Commission (DG Enterprise and Industry) for the period 2009 2012. The aim of IPF is to identify, develop and exploit synergies between public innovation agencies in Europe and propose new approaches to innovation support for SMEs. The project will in particular explore and test new ways of service delivery, aiming to accelerate the take-up of the most advanced innovation mechanisms with proven efficiency and impact. IPF is carried out by a consortium consisting of the following six partners Vinnova (SE, co-ordinator), Tekes (FI), Technology Strategy Board (UK), Enterprise Ireland (EI), NL Agency (NL) and FFG (AT). WP2 of IPF consists of three pilot peer reviews. A pilot peer review is a full fledged peer review, at its learning stage. This implies that some of its procedures might be adjusted on the basis of experience. Also the relationship of the peer review to other activities under IPF will be under development during these pilot PRs. For the pilot peer reviews the cases (programmes) are provided by the partner organisations and the members of the review teams come exclusively from the partners. The first pilot peer review was on the Small Business Innovation Research programme (SBIR) in the Netherlands. This programme has its origin in the US, but it has been tailored to the Dutch situation and requirements. The visit took place the 18 th and 19 th of March 2010 in The Hague, Netherlands, at the premises of NL Agency. The visiting organizations were three European innovation agencies: Technology Strategy Board, UK (Mr. David Golding, chairman); Enterprise Ireland (Mr. Terence O Donnell); FFG, Austrian Research Promotion Agency (Mr. Wolfgang Knapp). The Review Team also included: NL Agency (Mr. Eelco Denekamp); Dialogic, Netherlands (Mr. Christiaan Holland) as external consultant. Our discussion and conversation partners were: Koen de Pater - NL Agency, manager International Innnovation Carla Dekker - NL Agency, SBIR coordinator Andre Roos - Ministry of Economic Affairs Ruben Prins - NL Agency, staff member Coen Faber - Ministry of Transport, public works and water management Suzanne Verboon - NL Agency, SBIR manager Frans Gerritse NL Agency, staff member Freek van Leijen Hansje Brinker Dijkmonitoring Willem Zwalve NL Agency, director of NL Innovation 4

2. SBIR OVERVIEW OF THE MEASURE The SBIR is the Netherlands programme on precommercial procurement. It is procurement of R&D, of feasible solutions. Not the procurement of products and services. The actual procurement part runs through other programmes, such as Tendernet and networks, such as PIANOO. SBIR is an innovation-oriented procurement program to develop solutions to societal issues. It is specifically designed to give government an active role in the innovation process. The approach of the Dutch SBIR program combines the experiences of SBIR program in the United States with the European vision on the use of pre-commmercial procurement of innovations. The Netherlands government invites companies in an open tender to develop products and services to help solving social or societal issues. By issuing an open tender the government wants to challenge not only the usual suspects, but also companies that want to innovate but don t have the financial means to do it all by themselves. The SBIR programme finances the most riskfull stages of a development. The Dutch SBIR programme comprises a three-stage process: First technical experiments and a study into the technical, economic and organisational feasibility of a project idea. The maximum duration of this phase is six months. R&D up until a first, non-commercial prototype. The R&D has to conform to the European definition and the maximum duration of a phase 2 project is two years. Developing a market-ready product. (not financed by SBIR programme) The Netherlands SBIR Programme has three objectives: Create solutions for societal issues Encourage innovation in SMEs (90% of winning proposals come from SME s) Valorisation of knowledge (by companies) Targeted beneficiaries: Society Companies; preferably smaller companies, but that s not a criteria in the judging process because of European rules against discrimination require that applications from companies of any size and from any European country should be accepted. 5

SBIR in the context of other innovation stimulation measures (Dutch innovation policy context) 6

Remarks by Review Team: Scheme is set up on the basis of US scheme, rather than addressing a particular gap in support or a perceived need. The SBIR scheme will clearly support the SME in taking forward the development of new and innovative products or services. At the end of the funding, it often seems the case that more needs to happen before the product or service is in a position where it can be commercialised. Ideally there should be a way of linking stages 2 and 3. The individual competitions on specific themes can either be narrow or broad in scope. Need to find a good balance, but will differ depending on the subject area. 7

3. SBIR PROCESS For more details see Annex III Internal Protocol Theme (problem) selection process: Societal themes are selected by the department responsible for the societal issue. The department that is responsible for responding to the societal problem allocates budget for an SBIR tender. Selection process participants: Companies respond to an open tender, that is issued on the website and by press release and e-mail. Jury of 4-5 people make the ranking / selection based on criteria Companies with the best proposals are awarded a contract. There are 4 selection criteria: 8

contribute to the solution of public demand and entrepreneurship; (technological) quality and degree of innovation; economic perspective; added value for society. Activities supported by NL Agency: Funding is possible for the costs of a feasibility study with first experiments (phase 1) and R&D (phase 2). Around the projects there is a lot of communication and networking activitiy to raise the profile of the projects and to help introduce the companies to potential clients. (e.g. brochures, supported projects being at exhibitions). There are also wider activities to bring procurement officials together. Engagement of procurement officials is seen as important. There are also some activities in terms of creating a dialogue with the market beforehand. Remark by Review Team: The selection of the themes by ministries and within ministries seems to be bottom up. Ideas come forward on the basis that someone in the ministry has a problem to solve and some money to make it happen. So it currently requires someone in the ministry to initiate and champion the idea rather than a top down strategic selection by the ministry. Is the ministry therefore addressing its most important challenges? Response Coen Faber (Ministry of Transport, public works and water management): Within the Ministry of Transport, public works and water management there is a strategic innovation agenda on the subjects of mobility and water. This agenda defines the need for a SBIR approach. Also, SBIR can be part of a wider ranging interdepartmental MIA (societal innovation agenda). So the subjects for a possible SBIR are defined within these strategic innovation agendas. 9

4. RESULTS AND IMPACT OF THE MEASURE The programme is relatively young to measure the impact. When we move away from impact and look at the number of projects and the increase in budget we see the following: Number of projects increase over the last years: 100 90 80 70 60 50 40 30 20 10 0 2005 2006 2007 2008 2009 Phase 1 Phase 2 Phase 3 Budget has grown significant over the last years: 25 20 15 10 5 0 2004 2005 2006 2007 2008 2009 2010 2004 2005 2006 2007 2008 2009 2010 The budget is in million euros based on commitment. 10

Other Results of SBIR (Self assessment NL Agency): We see companies in SBIR, that we don t see in our innovation subsidy programmes. We also see new partnerships between companies that usually don t co-operate. E.g. a small family firm that co-operates with a high tech, quoted on the stock exchange, company. We perceive that the programme generates a lot of positive energy. Because of the close involvement of the department (even Ministers) and the close contacts with the entrepreneurs and R&D people within the companies. Willem Zwalve: it s a win-win situation; SBIR solves societal problems and stimulates innovation in companies. Assessment of the added value of SBIR by owners of SBIR (representatives of Ministries): Andre Roos (Ministry of Economic Affairs): SBIR is an ideal instrument to close the so-called knowledge gap. To make better use of existing knowledge and existing technology and experience (possibly from other sectors) to solve specific problems. Coen Faber (Ministry of Transport, public works and water management): SBIR provides greater opportunities to create connections between government and the market, much more than you would get from other forms of research funding. Remarks by Review team: Impact is too early to tell. There are just 5-7 projects in phase 3. There seems to be evidence that the scheme can stimulate and has stimulated the establishment of new companies. Impact on society is difficult to assess and also too early to tell. Impact on companies should be easier to assess by measuring how many successfully make it through phase 3. 11

5. RESULTS OF THE REVIEW EXERCISE The review at a glance Strengths SBIR scheme is more than just funding also brokerage, coaching, etc. PR and communication has helped to position this instrument SBIR is a contract to deliver, not a subsidy Short time to contract (6-8 weeks after deadline submitting proposal) Simple procedures Enthusiasm within other ministries (personal contacts and networks) Unique place in the whole set of instruments (policy context). SBIR doesn t seem to overlap or duplicate other mechanisms in the Dutch innovation system as there is nothing else quite like it. There is a buzz around this programme (companies are eager to participate) Innovation is placed in a broader context of relevant ideas and relevant problems (you can consider SBIR to be an example of problem oriented or demand driven innovation policy) New solutions to existing problems out of the box thinking and also reaching new companies Weaknesses Gap between R&D and procurement (from phase 2 to 3). Hard to make a connection because of Innovation Policy context (stimulation) versus Procurement Policy context (regulation); There is a need to look at exploitation much earlier, including what might happen at phase 3. Mechanism to formulate (and prioritize) problems. Anything goes? The measure does not create any new markets as there is very little, if any government procurement of the eventual products/services. Relevant issues How to fill the gap between phase 2 and 3 1? Additional instrument(s) for phase 3? 1 NL Agency is still learning on this point. Interesting presentation (video in dutch) on how to move on in phase 3 and the (perceived) obstacles can be seen on: HTTP://STOWA.MEDIAMISSION.NL/MEDIASITE/SILVERLIGHTPLAYER/DEFAULT.ASPX?PEID=A51C5DACEDB044FC8591C0BC0F379 14A 12

Broad versus narrow themes. How to balance the number of proposals? Innovation Policy context versus Procurement Policy context How to balance both objectives: societal benefit versus innovation/commercial benefit Other suggestions Make use of the opportunities to involve the judging panel as mentors and ambassadors for the projects they have assessed. SBIR projects have the potential to change legislation as they make regulators more aware of what is possible in a specific application area. Government officials are looking for new ways to support innovation. Move away from supporting the usual suspects (large companies) to SMEs who are faster on their feet and have better ideas. It would be good to have a bigger group of support people within ministries (and also maybe external experts) to explore ideas for new SBIR calls with. 13

6. PILOT PEER REVIEW PROCESS SUGGESTIONS FOR IMPROVEMENT The fact that we are dealing with a pilot peer review implies that we have to develop further the peer review as a method. During and after the visit, the following suggestions for improvement were made: More emphasis on transferability. This means more information on and analysis of the (national) context of innovation policy. Collection of questions of peer review team members before meetings. Send (some) questions beforehand to scheduled interlocutors. Ask them for introductory presentation; not just Q&A. More room for discussion/analysis within team. Possibly a survey on strong and weak points of the measure according to (more than 1 member of) target group. Eight people in the review team would be too many. A number of 6 would be just fine. On a more practical note: review team members in the same hotel please (to stimulate informal contacts). Analysis beforehand of answers in SA-qu. Make use of it. Invite a member of the panel selecting the proposals. More than one business representative as client of the measure. Report can be in a concise, bullet-point format rather than an elaborate, written-out format (as we have done in this report). Build in an extra round for remarks and questions after the visit of the peer review team (which we have done in this first pilot peer review). 14

ANNEXES I) PARTICIPANTS TO THE PEER REVIEW VISIT The visiting organizations were three European innovation agencies: Technology Strategy Board, UK (Mr. David Golding, chairman); Enterprise Ireland (Mr. Terence O Donnell); FFG, Austrian Research Promotion Agency (Mr. Wolfgang Knapp). The Review Team also included: NL Agency (Mr. Eelco Denekamp); Dialogic, Netherlands (Mr. Christiaan Holland) as an external consultant. Our discussion and conversation partners were: Koen de Pater - NL Agency, manager International Innnovation Carla Dekker - NL Agency, SBIR coordinator Andre Roos - Ministry of Economic Affairs Ruben Prins - NL Agency, staff member Coen Faber - Ministry of Transport, public works and water management Suzanne Verboon - NL Agency, SBIR manager Frans Gerritse NL Agency, staff member Freek van Leijen Hansje Brinker Dijkmonitoring Willem Zwalve NL Agency, director of NL Innovation 15

II) AGENDA AGENDA PILOT PEER REVIEW of SBIR 18-19 MARCH, 2010 NL AGENCY, The Hague Thursday, 18th March 12:30 Arrival of Review Team. Internal preparatory discussion. Sandwich luncheon. 13:45 OPENING. Welcome. Introductory statements. 14:00 SBIR. Overview presentation by Carla Dekker, SBIR co-ordinator. 14:45 Q&A 15:15 Statement by André Roos (MEA), owner of SBIR. Plus discussion. [short break] 16:00 Ruben Prins, staff member responsible for a current call (presentation, questions and visit) 16:30 Statement by a commissioning department (Ministry of Transport, Public Works and Water Management), Coen Faber. 17:00 Informal discussion between team, speakers and others (with drinks) 18:00 Close 19:00 Dinner with team (plus Suzanne, Carla, André) Friday, 19 th March 09:00 Internal session of team 09:30 SBIR in broader context, short presentation by Suzanne Verboon, SBIR manager 10:00 SBIR, as compared to other programmes, plus discussion (Frans Gerritse) 10:45 Experiences of a participating entrepreneur 16

[short break, allowing internal reflection by the team] 11:45 Closing session with Suzanne Verboon and Carla Dekker (and Willem Zwalve, director Agency NL Innovation) 13:00 Closing luncheon (Team) in nearby restaurant 17

III) INTERNAL PROTOCOL Netherlands SBIR Programme: Internal protocol from start to contracts phase 1 projects in ten steps Background SBIR programme Introduction SBIR Program Office SBIR is a precommercial procurement: correct wording Protocol: An SBIR procurement in 10 steps STEP 1 Contact and commission from Department to NL Agency 1.1 Contact with Department 1.2 Request for Quotation by Department to NL Agency 1.3 Make an offer for implementation of an SBIR tender to the Department 1.4 Commission from the Department 1.5 First announcement SBIR tender on website 1.6 Send notice to companies by e-mail STEP 2 Administrative and financial preparations 2.1 Time-keeping 2.2 Preparing project database 2.3 Access to database en disks 2.4 Planning and dead-lines 2.5 Announcement of Information meeting for companies on website STEP 3 Judging Committee members and colleagues for project analysis 3.1 Recruitement and appointment of judging committee members 3.2 Excel file for members of the committe 3.3 Engage collegues for preparation of the committee STEP 4 Publish and information 4.1 Communication strategy 4.2 Announcement of the tender 4.3 Information on website 4.4 Press Release 4.5 Inform branches and companies 4.6 Formation of a helpdesk team for companies 4.7 Information meeting(s) for companies and notes of this meeting(s) STEP 5 more administrative, financial and logistical preparations 20 5.1 Reservations of meeting room for the committee 5.2 Cupboard space 5.3 Instructions to the mailroom about incoming proposals STEP 6 Preparing the work of the judging committee 6.1 The letter to the committee with instructions 6.2 Document on assessment method 18

6.3 Prepare excel file for the committee 6.4 Instructions for collegues on assessment method and their task STEP 7 Receiving the proposals 7.1 Day of the deadline: answering questions from companies 7.2 Day of the deadline: mailroom and reception receive proposals 7.3 Update Website: tender is closed 7.4 Check for completeness while submitting proposals to database 7.5 Digital copies of the proposals on disk for collegues 7.6 Analysing proposals by fellow workers 7.9 Acknowledgment of receipt of proposal to companies 7.10 Invite companies for meeting judging committee (if number of proposals does not exceed 25) STEP 8 The meeting of the judging committee 8.1 Preparotory meeting with the chairman of the committee 8.2 Discuss last details on the meeting with the chairman after receiving individual ranking results 8.3 Meeting of the judging committee: ranking the proposals 8.4 Notes of the meeting 8.5 Letter with advice on the ranking of the committee to the Department STEP 9 Decision of the Minister and contracts with the companies 9.1 Letter with decision of the Minister 9.2 Informal feedback to the companies by telephone 9.3 Formal feedback to companies: decline or contract 9.4 Cash advance (80%) 9.5 Data collection for monitoring purposes STEP 10 Informing the public 10.1 Press Release on winners of the contracts 10.2 Project Summary on the website 10.3 Organisation of an event to congratulate and motivate the winners Internal protocol from phase 1 to closing contracts phase 2 (in preparation) 1 Control of phase 1 projects 1.1 Start of the SBIR project: explaining project leaders the rules 1.2 Visit to the companies 1.3 Extra: organisation of a workshop for project leaders 1.4 Explain steps: how to get from phase 1 phase 2 1.5 Answering questions from the companies about the phase 1 report 1.6 Receiving and evaluating phase 1 reports 1.8 Approval of feasibility reports and pay out last 20% 1.9 Informing contact at ministry about the results 2 Preparing, receiving and processing of phase 2 proposals 2.1 Preparing project database 2.2 Decision on sending request for a quotation to phase 1 companies 2.3 Are there changes in main and sub-contractor? 2.4 Write and send a request for a quotation to phase 1 companies 19

2.5 Answering questions from the companies about the phase 1 report 2.6 Planning and processing of phase 2 proposals 2.7 Instructions to the mail room 2.8 Submitting proposals into the database 2.9 Acknowledgment of receipt of proposal to companies by e-mail 3 Preparing assessment phase 2 3.1 Reservation of meeting room for the committee 3.2 Instructions on assessment for the committee 3.3 Preparing excel file 3.4 Analysing proposals (incl. results on phase 1 project and impressions company visit) 4 Judging committee phase 2 4.1 Preparotory meeting with the chairman of the committee 4.2 Inviting companies to a 15 minute discussion per proposal with committee 4.2 Discuss last details of the meeting with the chairman after receiving individual ranking results of the members of the committee 4.3 Meeting of the judging committee: discussion with companies, ranking of the proposals, formulate an advice to the minister 4.4 Letter with report on proposals and the ranking of the committee to the Department 4.5 Receiving letter with decision of the Minister 4.6 Negotiation on the price of the proposals: costs must be related to R&D 4.4 Informal feedback to the companies by telephone 4.5 Formal feedback to companies: decline or contract 4.6 Cash advance (20%) 4.7 Data collection for monitoring purposes Informing the public 10.1 Press Release on winners of the contracts 10.2 Project Summary on the website 10.3 Organisation of an event to congratulate and motivate the winners 6 Control of phase 2 projects 1.1 Start of the SBIR project: explaining the rules to the project leaders 1.2 Visits to the companies (at least once a year) 1.3 Receiving and evaluation phase 2 mid term reports 1.4 Approval of mid term reports and pay out more 20% cash advances 1.3 Receiving and evaluating phase 2 reports 1.4 Final meeting with company and (potential) clients: demonstration of prototype and 1.5 Approval of phase 2 reports and pay out last 20% 1.6 Informing contact at ministry about the results 20