metadata: authors: 0: author: Cusolito,Ana Paula
1: author: Dautovic,Ernest
2: author: Mckenzie,David J.
count: Albania,Bulgaria,Bosnia and Herzegovina,Kosovo,North Macedonia,Montenegro,Croatia,Greece,Italy,Romania,Serbia,Slovenia,Turkiye
docna: 0: docna: Can government intervention make firms more
            investment-ready ? a randomized experiment in the Western Balkans
docty: Policy Research Working Paper
subtopic: ICT Applications,Private Sector Economics,Private Sector Development Law,Marketing,Educational Sciences,Securities Markets Policy & Regulation,Capital Flows,Capital Markets and Capital Flows
teratopic: Law and Development,Private Sector Development,Information and Communication Technologies,Finance and Financial Sector Development,International Economics and Trade,Education
abstracts: cdata!: Many innovative start-ups and small and
            medium-size enterprises have good ideas, but do not have
            these ideas fine-tuned to the stage where they can attract
            outside funding. Investment readiness programs attempt to
            help firms to become ready to attract and accept outside
            equity funding through a combination of training, mentoring,
            master classes, and networking. This study conducted a
            five-country randomized experiment in the Western Balkans
            that worked with 346 firms and delivered an investment
            readiness program to half of the firms, with the control
            group receiving an inexpensive online program instead. A
            pitch event was then held for these firms to pitch their
            ideas to independent judges. The investment readiness
            program resulted in a 0.3 standard deviation increase in the
            investment readiness score, with this increase occurring
            throughout the distribution. Two follow-up surveys show that
            the judges' scores predicted investment readiness and
            investment outcomes over the subsequent two years. Treated
            firms attained significantly more media attention and were 5
            percentage points more likely to have made a deal with an
            outside investor, although this increase is not
            statistically significant (95 confidence interval of -4.7 to
            +14.7 percentage points).
geo_reg_key: 517208,119229,517198,517191
repnme: repnme: Can government intervention make firms more
            investment-ready ? a randomized experiment in the Western Balkans
display_title: Can government intervention make
            firms more investment-ready ? a randomized experiment in the
            Western Balkans
listing_relative_url: /research/2018/08/30329472/can-government-intervention-make-firms-more-investment-ready-randomized-experiment-western-balkans
topicv3: Private Sector Development m1326060 1921,Education, Skills Development and Labor
            Market m1326135 1484,Education m1327048 470,Social Media m1327099 662,Equity Finance m1327686 334
publishtoextweb_dt: 2018-08-06T00:00:00Z
historic_topic: Law and Development,Private Sector Development,Information and Communication Technologies,Finance and Financial Sector Development,International Economics and Trade,Education
no_of_pages: 73
author_keys: 0: author_key: 1003705
1: author_key: 1366474
2: author_key: 437864
model_extractions: WPS8541 Policy Research Working Paper 8541 Can Government Intervention Make Firms More Investment-Ready?

A Randomized Experiment in the Western Balkans _Ana Paula Cusolito Ernest Dautovic David McKenzie_ Development Economics Development Research Group August 2018
Policy Research Working Paper 8541

# **Abstract**

Many innovative start-ups and small and medium-size enterprises have good ideas, but do not have these ideas fine-tuned to the stage where they can attract outside funding. Investment readiness programs attempt to help firms to become ready to attract and accept outside equity funding through a combination of training, mentoring, master classes, and networking. This study conducted a five-country randomized experiment in the Western Balkans that worked with 346 firms and delivered an investment readiness program to half of the firms, with the control group receiving an inexpensive online program instead. A pitch event was then held for these firms to pitch their ideas to independent judges. The investment readiness program resulted in a 0.3 standard deviation increase in the investment readiness score, with this increase occurring throughout the distribution. Two follow-up surveys show that the judges’ scores predicted investment readiness and investment outcomes over the subsequent two years. Treated firms attained significantly more media attention and were 5 percentage points more likely to have made a deal with an outside investor, although this increase is not statistically significant (95 confidence interval of −4.7 to +14.7 percentage points) .

This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at dmckenzie@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Can Government Intervention Make Firms More Investment-Ready?**

# **A Randomized Experiment in the Western Balkans**<sup>**#**</sup>

Ana Paula Cusolito, _World Bank_ Ernest Dautovic, _European Central Bank_ and _University of Lausanne_ David McKenzie, _World Bank_ _Keywords:_ Investment readiness; start-ups; innovation; equity investment; entrepreneurship; randomized controlled trial.

_JEL codes_ : L26, M2, M13, O1

> # Funding for this project was received from the European Commission. Funding to support the impact evaluation from the World Bank i2i Trust Fund is gratefully acknowledged. We gratefully acknowledge comments from seminars at Duke, the IGL/Nesta conference, the IPA SME conference, Lausanne, Toronto and from Marius Starke, Janko Milunovic and Peter Trapp. This experiment and a pre-analysis plan were registered in the AEA RCT registry on October 2, 2015: https://www.socialscienceregistry.org/trials/895.
# **1. Introduction**

Innovative start-ups and SMEs in developing and transition countries often have good ideas, but may not have these ideas fine-tuned to the stage where they can attract outside funding. This is the case in the Western Balkans, where there is a perceived lack of investment readiness of innovative start-ups to be in a position where they can compete for, and take on, outside equity ( <mark>Karajkov, 2009).</mark> The most common reasons for a lack of investment readiness include a reluctance of entrepreneurs to surrender partial ownership and control of their business, lack of knowledge about the availability of external sources of finance, low investability of business development propositions, a lack of understanding about the key factors investors look for in making investment decisions, and presentational failings such as deficiencies in business pitches (Mason and Kwok, 2010).

Investment Readiness Programs which provide individualized training, mentoring and coaching are designed to overcome these constraints, but the programs can be expensive to provide, and to date there is no rigorous evidence as to their effectiveness. We conduct a five-country randomized experiment in the Croatia, Kosovo, Macedonia, Montenegro and Serbia to test the effectiveness of such a program. A sample of 346 innovative SMEs were randomly divided into two groups: a treatment group that received a high-cost and intensive program that involved help developing their financial plans, product pitch, market strategy, and willingness to take equity financing, along with master classes, mentoring, and other assistance; and a control group which received access to an inexpensive online-only basic investment readiness course. After this program, both groups of firms competed in a pitch event, where they were scored by independent judges (blinded to treatment status) on their investment readiness, with the top 50 firms then going onto a finals stage where they pitched to investors.

The independent judges scored the pitches on six aspects of investment readiness: team, technology, traction, market, progress, and presentation, with each firm scored by five judges. We find that firms that went through the investment readiness program receive an average of 0.3 standard deviations higher investment readiness scores at this event, and are more likely to get selected to proceed to pitch in front of investors. We then track firm outcomes over the next two years via a six-month and two-year follow-up survey, and through measuring their subsequent media mentions and social media attention. We find that treated firms do get significantly more 2
media mentions and social media buzz over the next two years (our measure excludes any mention related to the competition itself). The judges’ scores are found to be statistically significant predictors of investment readiness and investment outcomes over the next two years in the control group, suggesting that improvements in these scores should result in improved firm investment outcomes. Our point estimates show positive, but statistically insignificant, impacts on firm survival, three categories of investment readiness, and on steps towards receiving external financing, with treated firms being 5 percentage points more likely to receive external financing (95% confidence interval of -4.7p.p., +14.7p.p.).

We reconcile the significant impact on judges scores and significant predictive effect of judge scores on firm outcomes with these results through discussion of a funnel of attribution, through which large changes in investment readiness are predicted to result in smaller changes in investment outcomes. The results highlight the difficulty in designing experiments to measure the impacts of such programs.

The remainder of this paper is structured as follows: Section 2 discusses what investment readiness programs are, their use around the world, and contrasts them to other types of programs studied in the literature; Section 3 outlines the experimental design and provides details of the intervention; Section 4 provides the impacts on investment readiness; Section 5 examines how investment readiness translates into firm performance; and Section 6 concludes.

# **2. What Are Investment Readiness Programs and What Is the Evidence on Their Effectiveness?**

While much policy attention around the world has been given to efforts to expand the supply of equity finance for innovative start-ups and SMEs (through seed and venture capital co-investment funds and other activities to attract capital), the effectiveness of these programs can be hampered by a lack of readiness of these firms to receive equity investment. Mason and Kwok (2010) highlight three main aspects of this lack of readiness: first, many entrepreneurs are believed to be equity-averse, unwilling to surrender any ownership stake in or even partial control of their firms; second, many businesses that seek external finance are not considered “investible” by external 3
investors due to deficiencies in their team structure, marketing strategy, financial accounts, intellectual property protection, and other business areas; thirdly, even if entrepreneurs are willing to consider equity and have investible projects, presentational failings mean that many firms are unable to pitch their ideas successfully to investors.

# **2.1 What Are Investment Readiness Programs?**

Investment readiness programs are intended to increase the effective demand for equity financing by helping firms overcome the factors that result in a lack of investment readiness, thereby enlarging the size and quality of the pipeline of potential funding opportunities for investors and increasing the likelihood of new equity investments being made.

These programs are a relatively new form of intervention, but there are now a number of examples in the U.S. and Western Europe. Appendix 1 provides details on a number of these programs, and we summarize some of these examples here. In the United States, the Larta Institute uses a combination of personalized mentoring, webinars and learning modules, and market connections to help National Science Foundation grantees in the Small Business Innovation Research program to develop Commercialization Plans. Several universities offer online investment readiness platforms, including the program we offer to our control group. In addition, there are a number of accelerators and incubators that offer investment readiness training as part of their _broader_ array of specialized services. Examples in Europe include investment readiness support services provided by Enterprise Ireland, the Invest Academy Programme of the European Business Angel Network, the European program InvestHorizon and several demonstration programs provided by the UK Government’s Small Business Service.

Such programs are rarer in less developed countries, but pilot programs have been introduced in a number of recently developed or higher middle-income countries. For example, the Romanian Innovation Commercialization Assistance Program (RICAP) worked with 30 firms to help technology innovators address commercialization needs,<sup>1</sup> and the Malaysia Bioeconomy Accelerator Programme provides mentoring services to assist the commercialization of innovations developed in Malaysia.<sup>2</sup> The Getting Ready for Capital (GReaC) project funded by

> 1 <u>http://portal.larta.org/ricap#what</u>

> 2 <u>https://portal.larta.org/malaysia</u>

4
the EU aimed to help entrepreneurs in Bulgaria, Poland (and Belgium) understand the private equity market and effectively present their business propositions to investors.<sup>3</sup> The World Bank is preparing an investment readiness component to a program in Morocco.

While there is substantial heterogeneity in the content of these programs, the most comprehensive programs usually cover four dimensions, based on the core reasons that many investment deals do not materialize (Mason and Harrison, 2001; Mason and Kwok, 2010). The first dimension aims at reducing equity aversion, by explaining to entrepreneurs the potential advantages that equity can bring to the firm, both as a source of funding, and also because of the knowledge outside investors can bring to the firm. The second dimension addresses the investability of the business by helping to train the entrepreneur to demonstrate that they have a viable revenue model, can measure market traction, have dealt appropriately with property right issues, have a competitive strategy, etc. The third dimension works on the presentational skills, teaching the entrepreneur how to effectively pitch their business ideas and provide the key information investors are looking for. Finally, some programs also offer a networking dimension, aiming to facilitate the matching process between entrepreneurs and investors through events such as venture forums.

These programs are offered in two modalities: “hard” and “soft” programs. Hard programs usually involve a package of support that combines online tools and training, customized and face-to-face mentoring, group training through masterclasses, and investor demonstration days or pitch events. Soft programs are self-learning online tools structured in modules that entrepreneurs can work through at their own pace.

Both types of programs tend to be subsidized by governments, even in developed economies like the U.S. and U.K. There are several possible reasons to justify subsidies. The first is that the targeted firms are frequently liquidity constrained, and therefore unable to pay. Some incubator programs like Y-Combinator overcome this constraint by investing seed capital in the firms in exchange for an equity stake in the business. But since equity-aversion is one of the key constraints investment readiness programs are trying to overcome, investment readiness programs have typically not required equity stakes in exchange for participation. Secondly, since many of these programs are new in nature, potential entrepreneurs may find it hard to assess in advance the

> 3 <u>http://greac.eu/</u>

5
overall quality of the program, and their payoffs from participation are highly uncertain, making them unwilling to pay the costs of participating. Finally, governments may justify the subsidies in terms of the public benefits (more innovation, higher tax revenues, greater employment) that can come from successful ventures.

# **2.2 Existing Evidence about Their Effectiveness**

Currently there is no causal evidence as to the effectiveness of these investment readiness programs. The existing literature consists of several case studies and descriptive evidence. Several studies attempt to argue that a lack of investment readiness hampers equity investment, focusing in particular on presentation skills. Mason and Harrison (2003) use a case study to argue that poor presentational issues dominate the reactions of potential investors to a business proposal and constrain the likelihood of a deal taking place. Clark (2008) uses questionnaires submitted to business investors after an investor forum, and shows that presentation skills are significantly correlated with investment decisions.

Mason and Kwok (2010) provide a descriptive evaluation of the U.K. Government’s Small Business Service’s Investment Readiness program. Consultants judged the program to have had success in awareness raising, business development, and funding, but acknowledge that they do not have a counterfactual, and that it was difficult for businesses to reflect on what their behavior would have been without the program. They also report that tracking participants in the Finance and Business program of the North East Regional Development authority in England found businesses reported increases in funding, sales, and jobs two years after the program, but do not have a control group against which to compare the before-after comparisons.

More rigorous non-experimental evidence comes from work on related programs.<sup>4</sup> One set of related programs are business accelerators and incubators. These differ from investment readiness programs in typically being more intensive and expensive, often offer some seed capital and workspace in addition to training and mentoring, and work with a much smaller number of firms at any one time. For example, accelerators like Y-combinator take an entry cohort of 10 to 20 firms, who then receive seed capital, move to Silicon Valley for 3 months, and culminate with a

> 4 Somewhat, but less closely, related is regression-discontinuity work that shows the impact of financing to innovative firms on subsequent firm outcomes (Kerr et al, 2014; Howell, 2017); and experimental work on business training and consulting services to firms (reviewed in McKenzie and Woodruff, 2014).

6
demo day in which they present their ideas to selected investors. Several studies have used matching approaches to compare firms going through accelerators to similar firms which did not. Hallen et al. (2014) compare accelerator-backed new ventures to a matched set of non-accelerator ventures and find the former are faster at raising venture capital and gaining customer traction. Smith and Hannigan (2015) match firms going through Y-Combinator and Tech-Stars to start-ups that instead received financial assistance from angel groups, and find participation in a top accelerator program increases the speed of receiving follow-on funding and the speed of exit. In contrast, Yu (2016) matches accelerator and non-accelerator companies and find the former raise less funding, and close down earlier. <mark>Gonzalez-Uribe and Leatherbee (2018) use a regressiondiscontinuity approach to compare start-ups enrolled in the Start-up Chile accelerator program to those just below the qualification threshold, finding a positive relationship between the mentorship and the scale of the start-up and access to seed and venture financing.</mark> <mark>Finally, two recent experiments examine effects of short, cheap interventions to potential business ventures.</mark><sup>5</sup> <mark>Wagner (2016) conducts an experiment with 88 Start-up Chile grantees, and finds giving written feedback on their business plans makes firms more likely to survive, as measured by web presence, but is not able to measure other outcomes.</mark> Clingingsmith and Shane (2017) provide 30-minute pitch training to undergraduate students in Ohio, who then deliver 90-second pitches to judges. They find training actually lowers the judges scores on average, by helping judges better distinguish bad from good ideas, and having a more negative impact on low quality ideas than the positive impact on better quality ideas.<sup>6</sup> Our work differs from this work on accelerators in at least four key aspects. The first is in the type of program being analyzed: investment readiness programs of the type we study here are designed to be able to be scaled up and operate with sizeable numbers of firms at a time, compared to the more intensive focus on a small number of firms at a time in incubator and accelerator programs. Second, ours uses a randomized experiment, overcoming concerns about how well nonexperimental methods are able to overcome biases induced by the explicit selection mechanism that aims to choose better firms for the program than the non-participants. Thirdly, the existing

> 5 Another example of an experiment with early-stage ventures is Bernstein et al. (2017), who randomize the information potential investors on AngelList receive about start-ups, and find that information about the founding team matters for their decision to find out more about the company.

> 6 Using a difference-in-differences approach, Howell (2018) also finds that negative feedback leads to more abandonment of business ventures.

7
literature has largely relied on a small number of outcome measures that can be collected without the use of firm surveys – survival, whether or not they received venture funding, and web traffic measures. We have much more detailed data, including the use of judge scores and intermediate outcome indicators that allow us to focus on investment readiness, and not just investment outcomes. Finally, the majority of studies focus on the U.S. which has a well-developed venture capital market, whereas we focus on an area of the world where firms are only just starting to engage with outside investors, making investment readiness programs potentially more important.

# **2.3 Why an investment readiness program in the Balkans?**

Increasing innovation is a key regional priority in the Balkans region as a means to boost firm productivity and sustain economic growth. While it is generally accepted that debt finance is not the optimal source of funding for early-stage SMEs and start-ups, equity finance is only marginally used in the region. A regional report noted that there is a debate as to how much this lack of use of risk capital reflects a lack of supply of equity finance, versus a lack of readiness of entrepreneurs to attract and accept this financing (Karajkov, 2009). Based on the viewpoint that action was needed on both the supply and demand sides, the Enterprise Development and Innovation Facility (EDIF) initiative financed by the European Commission includes efforts to increase the supply of private equity to the region, improve the legislative frameworks to better encourage venture capital activity, and undertake efforts to increase investment readiness. This paper provides an evaluation of the investment readiness component of this initiative.

# **3. Experimental Design**

To implement this intervention, we ran a competitive procurement process where companies specializing in investment readiness programs provided bids. We shortlisted five companies, and together with evaluations of these proposals from Josh Lerner from Harvard University and his team at Bella Research Group, and with advice from experts in the national innovation agencies of the participating countries, chose as the winning firm the company _Pioneers JFDI GmbH_ (Pioneers henceforth). Founded in 2009 and based out of Vienna, they are one of Europe’s leading platforms for entrepreneurship, organizing an annual “Pioneers Festival” (with 3,000 attendees), as well as providing mentoring, pitch training, and opportunities for presentation and networking with European and international founders and investors. They launched a specific investment readiness program called _Pioneers of the Balkans_ for this project.

8
# **3.1 Generating the Sample**

Eligibility criteria for the program were developed by the World Bank and Pioneers team, conditional on the rules of the European Commission. To participate in the program, a firm had to be legally registered in at least one of the five countries: Croatia, Kosovo, Macedonia, Montenegro or Serbia. The firm had to be a micro, small, or medium-enterprise, defined as having fewer than 250 employees, and an annual turnover below 50 million euros. It had to be innovative, meaning that “it will in the foreseeable future develop products, services, or processes which are new or substantially improved compared to the state of the art in its industry, and which carry a risk of technological or industrial failure”, and could not be on a sanctions list or operating in a set of negative activities (e.g. gambling or alcohol production).

To launch the program, the brand _Pioneers of the Balkans_ was created, and a dedicated website set up.<sup>7</sup> The program was marketed as a competitive program designed especially for innovative entrepreneurs seeking or considering venture financing. The main communications therefore promoted a major pan-regional start-up competition due to take place in two stages, with a Semifinals in Belgrade and subsequent Finals event in Zagreb. It included a preliminary list of investors who had already confirmed their attendance at the Finals and noted that selected firms would receive a training and preparation package.

We had set a target of 300 to 350 participating firms. In designing the program, both providers of investment readiness services and experts in the innovation agencies agreed that there was a limit on how many firms potential investors would be willing to listen to pitches from. They also noted a concern that randomly choosing a firm to pitch in front of investors that was not of high quality could have reputational risks to the region, with potential investors observing such firms as a signal more generally that firms in the region are not of high enough quality to merit investments. A twostage process was designed to overcome these issues: the Semi-finals would be the main phase of our study, with all firms in the study having a chance to present their ideas in the semi-finals and get scored by independent judges on their investment readiness. Then only the top-50 would progress to the finals, with these firms selected on merit.

> 7 <u>http://www.pioneersofthebalkans.io [accessed May 5, 2018]</u>

9
Pioneers aimed to create broad awareness of the program among entrepreneurial firms in the region, launching the program at the start of August 2015 (see timeline in Appendix 2) and marketing the program rapidly. It used five major instruments to achieve this goal: public sources of information for applicants, direct electronic and physical mailings, social media marketing, a roadshow spanning all five target countries, incentives for early applications (a raffle for a dinner with two leading entrepreneurs from the region), and media relations. A list of more than 1,200 potential contacts was directly emailed using firm names provided by the local innovation funds and government counterparts, and other contacts in the region. LinkedIn and Facebook advertising was used, and “multipliers” were asked to spread the word to their contacts in the region.

Applicants had to apply online, with the data from this application form providing the baseline data for this study. More than 1,200 applications were started online, and a total of 584 full applications were received. These were screened for eligibility, resulting in 346 firms being selected as eligible for the program.

This process succeeded in generating a sample of young firms involved in a wide range of innovative activities. At the time of application, firms had an average of 6 employees, with a 1090 percentile range of (1, 12). They had been in business for 2.5 years on average, and are involved in high-tech innovative industries such as cloud computing and big data, app development for a wide range of business and personal services, pharmaceutical products, etc. Half of the founders have post-graduate education, and 60 percent have a global rather than regional focus as their key market. To make clear the types of firms involved, it is worth giving some more specific examples of the types of innovation these firms are doing. Some examples are as follows:

- A firm that is developing virtual reality software that can be used in outdoor interactive missions, with the aim of deploying this in military training exercises and theme park adventures (e.g. a team-based maze/obstacle course where dragons and other objects are flying around)

- A firm developing an app that geo-locates users on ski fields in Europe, and provides a way for them to see where all their family members are at any point in time, and to direct them to common meeting places.

- A bio-tech firm that has developed a new coating for common medicines that allows the body to better regulate the dose-intensity, to reduce under- and over-dosages of medicines

10
- An architecture firm that has developed an innovative luxury “boatel” that runs on an electric motor and can be used on lakes

- A firm that has developed solar-powered benches for public spaces that can charge phones and also monitor air and noise quality.

A number of the firms were developing apps for the Balkan and global markets, covering a wide range of activities such as making it easier to use public transport, a local version of Uber, an app to connect consumers with producers of organic products, online sports coaching, and an app to manage freight logistics. But there are also firms involved in physical manufacturing of products, such high-end electrical bicycles, smart vending machines, indoor pet houses, and a USB charger that charges while bicycling.

# **3.2 Random Assignment**

Applications closed on September 6, 2015 and were then screened to ensure they met the eligibility requirements. All applicants which met the formal eligibility criteria were accepted into the study. Eligible applications were then scored on four criteria to measure their initial level of investment readiness: market attractiveness, product technology, traction, and team. Appendix 3 describes the scoring methodology. The top 10 proposals overall in terms of score were then randomly assigned to 5 in treatment and 5 in control, in order to ensure that some of the very top proposals were in both groups. Then the remainder of firms were divided into strata based on country (Serbia, Croatia, or the rest), and on whether or not they already have a private investor. Within these six stratum firms were ranked into groups of four on the basis of their investment readiness score. Within these quartets two firms were randomly allocated by computer to treatment and two to control. This was done for an initial batch of 333 firms, allocating 167 to treatment and 166 to control. An additional batch took longer to verify their eligibility requirements and were received after this assignment, these were then also randomly allocated and form a separate strata. This resulted in 346 firms, with 174 treatment and 172 control. A pre-analysis plan was registered with the AEA trial registry on October 2, 2015 to pre-specify the initial outcomes of interest.<sup>8</sup> This process resulted in treatment and control groups that are evenly balanced and comparable in terms of their initial characteristics. This is seen in Table 1. Figure 1a shows that the two groups

> 8 <u>https://www.socialscienceregistry.org/trials/895</u>

11
are also similar across the entire distribution in terms of initial investment readiness. As a result, any difference in investment readiness at the conclusion of the program can be reliability assessed as the impact of the program and not due to any pre-existing differences across groups.

# **3.3 Details of the Treatment and Control Offerings**

The treatment and control groups were blinded to treatment status, and both were offered a form of investment readiness training – the difference being in the intensity, cost, and medium of the offerings. We summarize both treatment and control programs here. A key issue with understanding the impact of different training programs is that much of the literature does not provide sufficient detail on what was offered, leaving the program as a black box for others seeking to learn or compare. Therefore, in Appendices 4 and 5 we provide much more detailed information on each program.

The treatment group received an investment readiness program provided by Pioneers, but branded under the name _Startup Live Mini-Accelerator_ . This was an intensive two-month program that aims to prepare companies to be in a position where they are ready to talk with potential investors. The first phase (“qualification”) was structured around an online training platform called _WhatAVenture._ Using this tool, individuals are asked to outline and self-critically assess their businesses by describing the problem or need addressed by their product or service, the commercialization concept and expected revenue streams, conduct a market sizing exercise, and describe their competitive positioning. Each business was assigned a lead mentor who supports them through this process and provides feedback and help.

After completing this first phase, firms were then brought into an “acceleration phase”. In this phase they had individualized mentoring from both their lead mentor, and from a pool of more than 100 specialized mentors who could help out on specific concrete and sector-specific needs. Mentoring took place both on-site and via video calls. During this phase, there were four masterclass weekends, which took place every week in October from Friday evening through Sunday afternoon. These masterclasses rotated around the different countries, and were recorded so that those who could not attend in person could access the contents online. Each workshop followed a similar format, but with the topics varying. On Friday evenings the attending entrepreneurs would have a chance to introduce themselves and their businesses in just 90 seconds with no presentation materials, and also see examples of the same from the mentors, followed by 12
informal discussions. Saturdays would involve five to eight lectures and/or workshops, with themes such as sales and marketing, team building and human resources, and investment and finance. On Sundays, all participants and mentors focused on presentational skills as well as pitch deck structure and design. The final phase was a “pitch preparation phase” and took place in the last two weeks, in the run-up to the semi-finals. This included working on their pitch decks with their mentors, delivering practice pitches, and then on-site training in Belgrade the day before the semi-finals performance as a final practice run.

The total cost of the treatment is estimated to be $614,000, or approximately $4,000 per active participant.<sup>9</sup> The main component of the cost is the individual mentoring, which averaged $3,072 per beneficiary, with the masterclasses costing $793 per beneficiary and pitch training $230.

The control group companies were offered an e-learning course developed and distributed by the Global Commercialization Group (GCG) of the University of Texas at Austin. This course is distributed under the label Innovation Readiness Series<sup>TM</sup> and was launched in 2011. It is targeted to a broad audience of entrepreneurs, scientists, engineers, and students, with the goal in helping transform their innovative and technology-based concepts into a viable commercialization plan and a convincing pitch. The content is delivered online through 10 modules of 45-60 minutes each, with a multiple-choice quiz at the end of each module. Appendix 5 provides descriptions of the content of each module. They cover key issues such as how to articulate the benefits of an innovation to customers and investors, intellectual property protection, market validation, comparing to competition, and how to pitch and present. The cost of the course was a one-time $5,000 set-up charge to customize to our program, and then $153 per firm.

There were several reasons for offering the control group an online investment readiness program rather than not providing any service at all. The first was that, from a public policy point of view, a key question was whether an expensive and intensive program was needed, or whether identical results could be obtained by cheap and accessible online alternatives. This was considered the more interesting policy counterfactual than offering nothing at all. Second, from an evaluation

> 9 The exact cost per firm differs in terms of services contracted vs services actually delivered, since not all firms used all the mentoring hours they were allocated. Pioneers retrospectively estimates that the actual services delivered to the firms were approximately $3,000 based on actual hours mentoring used. Note further that this calculation does not include the costs of advertising the program through roadshows, or of putting on the semi-final and final events, which were important in attracting firms to the program. These overhead costs are estimated at approximately $1,500 per firm (in both the treatment and control groups).

13
standpoint, offering both groups an investment readiness program lowers the risk of Hawthorne and John Henry effects, since both groups were told they were being provided with an investment readiness program. Finally, we also believed that offering the control group something would minimize the risk of differential attrition compared to the treatment group.

# **3.4 Take-Up**

Of the 174 firms randomized into treatment, 157 (90.1%) completed the _WhatAVenture_ online training platform, and 79.3% received individual mentoring. Conditional on receiving individual mentoring, entrepreneurs received a median of 8 and mean of 11 hours of individual mentoring from the lead mentor and pool of specialist mentors.<sup>10</sup> 76 out of the 174 (43.7%) attended at least one masterclass in person (videos of the masterclasses were also available online, with typically 10-20 firms watching each). There were approximately 1,150 mentoring hours provided during the masterclasses, of which around 390 hours were individual mentoring, and 760 hours were in the form of lectures and presentations. This represents an average of 15 hours per attendee. In addition, before the semi-finals, 76 firms (43.7%) attended a 3-hour final pitch presentation training.

Table 2 examines the correlates of take-up of the _WhatAVenture_ tool and of masterclass participation amongst those in the treatment group. We run a probit of take-up on all the baseline characteristics in Table 1, and then run a stepwise procedure to progressively drop the largest coefficient with a p-value above 0.2 to end up with the sparser specifications in columns 2 and 4. For the initial stage of using the _WhatAVenture_ tool, the only variable that is consistent in the sparse model is the initial investment readiness score: firms with higher initial readiness are more likely to complete this first phase. However, we then see in columns 3 and 4 that higher initial investment readiness is associated with a lower probability of attending a masterclass. This might reflect that firms who already are more ready feel they have less need to learn from such workshops. We see attendance is lower for firms from Croatia. This perhaps reflects the masterclass weekend in that country being held in Split, rather than the larger city of Zagreb: a one-day workshop was held additionally in Zagreb, and including this reduces the gap slightly. Attendance is higher for firms whose owners have post-graduate education, and for those who

> 10 Note firms were eligible to receive up to 30 hours of individual mentoring time, so the majority of teams used considerably fewer hours than allocated to them.

14
have participated in a mentoring or acceleration program before, potentially reflecting a taste or revealed preference for training, or complementarities with existing skills. Finally, companies which use cloud technology were more likely to attend. When asked in our follow-up survey why they did not attend, the most important reasons given were that they did not want to take the time away from their businesses (and in some cases second jobs as employees), and that the locations were too far away.

Out of the 172 participants assigned to the control group, 120 (70%) accessed at least once the online Innovation Readiness Series<sup>TM</sup> platform. However, even conditional on accessing the platform, overall usage was relatively low. Conditional on accessing the online platform, 118 participants viewed at least once the modules’ section and 55 viewed it at least 10 times; the mean number of views of the modules section was 21 and the median 9. Each module last approximately half an hour, so we can approximate that the mean time spent on the modules was 10 hours while the median 4.5 hours. Only 63 (37% of the control group) participated in one of the seven quizzes at the end of a module. A total of 51 control group entrepreneurs passed at least 4 quizzes with 45 attaining the threshold of 70% correct answers in all quizzes, necessary to receive a certificate of completion from the IC<sup>2</sup> Institute at the University of Texas at Austin. The main two correlates of taking and passing the quizzes among the control group are having postgraduate education (positively correlated), and having previously participated in a mentoring or accelerator (negatively correlated). The online courses are thus done by those who have more schooling and have not previously had exposure to such content.

# **4. Impacts on Investment Readiness as Scored by Judges**

# **4.1 The Semi-finals and Judging Procedure**

The semi-finals were held in parallel to, and in cooperation with, the Belgrade Venture Forum, an annual venture capital conference that took place from November 12 to 14, 2015. Participants were invited to present in a pitch event that follows the standard format of such events, with firms giving a 5-minute pitch of their business case, followed by 5 minutes of questions from a jury of judges.

Participation required the founder of the firm or a representative to be physically present in Belgrade. To encourage participation, firms received multiple reminders and calls, were sent an invitation letter with a ticket voucher that allowed them one day of free access to the adjoining 15
Belgrade Venture Forum, and were provided with a transport subsidy that was sufficient to cover the cost of bus travel to the event. The travel time was approximately 4 hours from Croatia, 5 hours from Macedonia, and 6 to 7 hours from Kosovo and Montenegro. In total 211 of the 346 invited firms (61%) attended the semi-finals: 110 firms from the treatment group (63.8%) and 101 firms from the control group (58.1). The attendance rate was similar for Serbia (64%) and Croatia (67%), and lower for the other three countries (51%). Attendance rates were higher amongst those who had participated more in the intervention. Amongst the treatment group, 81.6% of those who had attended at least one masterclass attended the semi-finals, versus 49.0% of those who had not. Among the control group, 88.9% of those who had taken any of the quizzes attended, versus 41.3% of those who had not. We discuss robustness to this attrition in the next section.

A group of 66 independent judges was used to do the scoring. Panels of five judges were assigned to judge a session of six firms at a time, with judges then being rotated so that they are on panels with different judges for their next sessions. Each batch of six firms consisted of three treatment and three control firms, selected to have a similar range of initial investment readiness scores, and grouped according to industry and country of operation. Judges were assigned to batches based on their availability (some were giving talks at the venture forum), industry, and technology used. Appendix 6 provides details of characteristics of these judges. They were a mix of investors, successful business owners, and experts in mentoring and coaching start-ups. Thirty-seven percent lived in one of the five countries taking part in the competition, while two-thirds were based in other countries. Eighty percent of them regularly mentor start-ups, 64 percent were part of companies that make venture investments, and three-quarters had founded their own companies. They were therefore experienced in what outside investors are looking for in terms of investment readiness.

Judges were blinded to treatment status, and were not provided with any information about the company in advance of scoring. They were briefed and asked to score each firm on six factors:

- 1) Team: the skills and capabilities of the entrepreneur and his or her team

- 2) Technology: the degree of innovativeness and technological advancement

- 3) Traction: indications of measureable market success

- 4) Market: the commercial market attractiveness and size of the potential market

16
- 5) Recent business progress: the amount of progress firms had made during the last three months (the time since initial application)

- 6) Presentation performance

An aggregate investment readiness score was then formed using the following weights: (team) 28%, (technology) 21%, (traction) 14%, (market) 7%, and (progress) 30%. These weights were not revealed to the judges, but were based on what seed- and early-stage investors would commonly focus on (Kaplan and Strömberg, 2004). They tend to emphasize the quality of the team and their technology (Gompers et al, 2016), and the extent to which the business is continually improving. The presentation score was added to allow judges to independently assess how well the firm presented its ideas, and as “hygiene” factor that could be used if necessary to avoid placing someone unable to present in front of investors at the final. The correlation between this weighted score and an equally-weighted score is 0.995, and we show in Appendix 6 that our results are robust to this choice of weighting.

There were two ways for firms to be selected for the finals. The main path was through an overall ranking based on the aggregate investment readiness score. Secondly, judges scored each firm after watching its pitch, and then at the end of the batch of six presentations, discussed the set of six. They then were asked to collectively rank the three best they had seen out of the six, and could choose to directly nominate the top-ranked firm to directly be sent to the finals. They were asked to use this direct nomination selectively, reserving it only for firms they believed should certainly be granted the opportunity to present in the finals. The idea behind direct nomination was to allow for the possibility that through collective discussion, the strength of a firm may be more apparent.

Sixteen firms were directly nominated to the finals, of which only four were not in the top-50 overall based on the individual ranks.<sup>11</sup> Then firms ranked in the top 46 based on the overall score were also chosen to give a total of 50 finalists. We then examined how sensitive these rankings were to allowing for differences in scoring amongst judges, and re-ranked firms on their residual scores after subtracting judge fixed effects. Four additional firms were chosen as finalists based on having judge-fixed-effect-adjusted scores in the top-50 even though their raw scores were not in the top 50. This gave a set of 54 firms that were invited to the finals.

> 11 They ranked between 58 and 74.

17
# **4.2 Estimating the Impact on Investment Readiness as Scored by Judges**

To estimate the impact of the program on investment readiness as scored by the judges, we use the following (pre-specified) base specification for firm _i_ in stratum _s_ : Where 1����� are strata dummy variables. Note that stratification implicitly controls for baseline investment readiness, country, and whether or not the firm has an outside private investor at baseline. Robust (Eicker-White) standard errors are used. As a robustness check, we also reestimate equation (1) after controlling for judge fixed effects.

The parameter _β_ is then the intention-to-treat effect (ITT). This measures the impact of being assigned to the treatment group, and being offered the expensive and intensive investment readiness program rather than the online course offered to the control group. We could also attempt to measure the local average treatment effect (LATE) of actually receiving treatment. Recall that 90.1% of the treatment group completed the _WhatAVenture_ tool. However, all but one of the treatment group firms that attended the semi-finals (99.1%) had completed this tool, so the noncompliers to treatment status are firms for which we do not have investment readiness scores. As such, the ITT and LATE are almost identical for the firms attending the semi-finals. We therefore just report the ITT results.

The first column of Table 3 presents the impact of treatment in our overall measure of investment readiness, as scored by the judges. This is our main outcome in this table, and so our main approach to multiple hypothesis testing for this set of outcomes is to rely on this aggregate. The control group has a mean investment readiness score of 2.9 (s.d. 0.9). We find that treatment increases this score by 0.284, which is significant at the 5 percent level. The magnitude is thus equivalent to 0.31 standard deviations. The second row of estimates show that this impact continues to hold after controlling for judge fixed effects, with a larger magnitude of 0.41. Figure 2 compares the distributions of investment readiness scores for the treatment and control groups, and shows there is a rightward shift in the distribution, so that these gains appear to be occurring everywhere except at the very top.

The next five rows of Table 3 examine which components of the overall score have improved with treatment. We find positive impacts on all five components (team, technology, traction, market, 18
and progress), with the impacts statistically significant for three out of five measures, and significant for all five measures after controlling for judge fixed effects. The seventh row then examines the impact on the team’s presentation score. Recall this is not included as part of the overall score, but was scored separately. We find that treatment resulted in a 0.37 unit (0.32 s.d.) increase in the team’s presentational score, which is statistically significant at the 5 percent level. Treated firms are therefore more investment ready in terms of both being able to present their idea, and in terms of the quality of the idea presented.

We had hypothesized that the treatment might also reduce the variability among judges in their assessment of how investment-ready firms are. To examine this, in column 8 we consider as an outcome the standard deviation of the individual judge scores for a firm, with a higher standard deviation indicating more divergence amongst judges in their assessment of the firm. However, we see a small and not statistically significant impact of treatment on this measure.

Finally, the last column examines whether treated firms were more likely to be selected as one of the top 54 firms that were invited to pitch to investors in the finals. Only 12 percent of the control group firms were selected for the finals, and treatment has an 11.5 percentage point increase in this likelihood. This is a large effect, doubling the likelihood of making the finals, but it is only significant at the 10 percent level.

The investment readiness scores are only available for firms which participated in the semi-finals. This raises the concern of bias arising from differential participation patterns among treatment and control firms. The last columns of Table 1 examines balance on baseline characteristics by treatment status for the firms which participated in the semi-finals. We see that, overall, the sample still looks balanced on most observable characteristics, although the overall joint orthogonality test has a p-value of 0.086. Most importantly, the mean of the baseline overall investment readiness differs only by 0.02 between the two groups, and Appendix Figure 6.1 compares the full distribution of the baseline investment readiness score by treatment group and participation status, and shows the distributions also look similar. Our pre-analysis plan specified two approaches to examining the robustness of our results to this attrition: imputing scores for those who did not attend, and using Lee (2009) bounds. Appendix 6 shows the results are robust to both approaches, and are also robust to using alternative weighting schemes to aggregate the different components 19
of the overall score. The program therefore succeeded in making firms more investment-ready, as judged by independent experts.

# **4.3 Heterogeneity in Impact by Initial Investment Readiness**

In our pre-analysis plan, we hypothesized that the impact of the program is likely to be greater for firms that were less investment-ready to begin with, since firms that already had very high scores on all components would have had little room to improve. Conversely, this impact could be negative, if training causes less investment-ready firms to present their ideas more clearly to judges, allowing judges to more easily recognize them as low quality as in Clingingsmith and Shane (2017). To test this hypothesis, we interact treatment with an indicator of whether or not the firm had a baseline investment readiness score below the median of 3 (45.1 percent of firms), and include this interaction, along with the level effect of having a below median readiness score in an expanded version of equation (1).

Table 4 reports the results of examining this heterogeneity for the same outcomes as were tested in Table 3. The point estimate in column 1 is consistent with this hypothesis, with the estimated effect of treatment on investment readiness being twice as large for below median firms as above median firms. However, our power to detect this heterogeneity is low, and we cannot reject the null hypothesis that there is no difference in treatment effects by initial readiness. The next six columns show positive point estimates on four out of five of the interaction effect for the different subcomponents of the overall score, along with a positive point estimate on the interaction with the presentation score. However, the only significant effect is when looking at the technology subcomponent as an outcome. Moreover, after correcting for multiple hypothesis testing using Holm’s (1979) method this impact is no longer significant. Finally, in the last column we see that the interaction is negative for being selected for the finals. This is consistent with the idea that firms that were far from investment-ready to begin with would not be able to improve enough to get into the top group, although this interaction is not statistically significant and so we cannot reject that the treatment effect on progression to the finals is the same regardless of initial score.

# **5.  Longer-Term Impacts on Investment Readiness and Firm Performance**

The immediate impacts on investment readiness are seen in the performance in the semi-finals. We then track the firms over time in a variety of ways to see whether this short-term improvement 20
in investment readiness translates into longer-term investment readiness, into the chance of receiving investments, and into firm performance.

# **5.1 Performance in the Finals Event**

The Finals event was held in cooperation with the Balkan Venture Forum on December 3 and 4, 2015 in Zagreb. This was the largest venture capital conference in the five target countries to date, with more than 400 attendees. The pitching slots were spread over two days and grouped into batches based on industry segments (business and productivity, lifestyle and entertainment, life science and energy, environment, and mobility and transportation). Jury members consisting of partners at venture capital firms and managers of accelerators/incubators choose a category winner for each batch. Out of eight category winners, 6 came from the treatment group and 2 from the control. These category winners were publicly awarded with a large-format printed award and a bottle of sparkling wine following the slogan “honor, fame, and champagne”. The three lead investors of the conference had each publicly committed to choose at least one firm each to give an “invitation to negotiate” for investment by the end of the conference. They extended these invitations to four finalists in total, of which 3 were from the treatment group and 1 from the control. The treatment group therefore did better, but because the absolute number of firms winning is so low, these impacts are still small in absolute magnitude (1 to 2 percentage points), and are not statistically significant (the smallest p-value is 0.157 for being a category winner).

Following the finals, a short survey was sent to investors who had attended the finals. Responses were received from 32 investors. Out of these investors, 66 percent said they had talked about a potential investment with at least one firm, 28 percent planned to negotiate with a firm, and 50 percent said they might invest and had added new firms to their watchlists. Only 40 percent had previously invested in the region, and when asked what the main barrier to investing in the region was, the modal answer was in generating deal flow and identifying good investment prospects. Investors were asked whether they planned to increase their investment in the region as a result of attending. Twenty-five percent said they would, 31 percent said they would reallocate their investment from other investments they might make towards firms in the event, and the rest would not change their investment strategies. This provides suggestive evidence that the project may have increased the overall amount of investment towards these types of firms in the region.

# **5.2 Impact on Media Buzz**

21
We examine whether the firm is gaining attention and traction through several measures of media attention and social media buzz. One advantage of these measures is that they are available for the full sample, with no attrition. The media intelligence specialist firm _Meltwater_ was contracted to collect online media mentions of the firms in our sample over the six-month period March 1 to August 31, 2015 (pre-intervention), and then one year and two years later (March 1 to August 31, 2016; and March 1 to August 31, 2017). Note that these time periods exclude the period of the intervention, semi-finals, and finals, so are independent of any media coverage of the program or pitch events, and correspond to an average of 6 and 18 months post-intervention. Meltwater tracks more than 250,000 global news sources in 190 countries in 25 languages (including Serbo-Croatian and Albanian). Thirteen percent of the firms in our sample had at least one media article about the firm during the six months prior to application, with a median of three articles conditional on having any media.

Column 1 of Table 5 show that 9.9 percent of the control sample was mentioned at least once in the media during the six-month intervals in 2016 and 2017, and treatment results in a 4.7 percentage point (2016) and 3.9 percentage point (2017) increase. These increases are large relative to the sample mean, but not statistically significant. Column 2 of Table 5 shows that we do see a statistically significant increase in the total number of media mentions in 2017, which are three times as high for the treatment group as the control.

Two-thirds of the firms had some form of social media presence at baseline, with Facebook (which 58% use) and twitter (42% use) being the most common. Column 3 shows a small and statistically insignificant impact of treatment on the number of Facebook followers a firm has, and Column 4 shows that treated firms have 20% more twitter followers after two years, but this is also not statistically significant. We pre-specified an overall index of _media buzz_ by taking standardized z- scores of these first four columns.<sup>12</sup> The last column of Table 5 shows that treated firms have more media buzz, with this significant at the 5 percent level in 2017.

# **5.3 Tracking Firm Performance through Follow-up Surveys**

> 12 Our pre-analysis plan also noted we would look at the impact on web-traffic, and being included on _AngelList_ , a web platform for fundraising, but that these would not be included in our overall index of media buzz. We find no significant impact on these other outcomes (Appendix 7).

22
We conducted two rounds of follow-up surveys of these firms. The first, intended to measure shortterm effects, was taken between April and August 2016, corresponding to a period of approximately six months after the end of the investment readiness program and judging, and enables us to measure short-term effects. The overall survey response rate was 79.2 percent, and does not differ significantly between treatment (79.9%) and control (78.5%). In addition, we collected information on operating status, number of employees, and whether negotiations for an outside investment had occurred for a further 12 percent of firms, resulting in basic data being available for 92.2 percent of firms. The second follow-up survey took place between August 2017 and March 2018, corresponding to an average of two years since the intervention. Catalini et al. (2017) show that 75 percent of firms that receive venture capital financing in the U.S. receive their first financing within the first two years after incorporation, so this timing covers a window where we should expect many firms to receive external financing if they will ever do so. The overall survey response rate for this second follow-up was 85.0 percent, and again does not differ significantly between treatment (86.2%) and control (83.7%), with data on firm operating status and receipt of equity available for 94.5% of firms. Appendix 8 shows no significant difference in response rates by treatment status, and that treatment and control firms remain balanced on baseline observable data for those responding to the survey.

The follow-up surveys focused on measuring changes in the firm in three domains. The first is whether or not the firm is still operating (regardless of whether or not it has been sold to another owner). The second is investment readiness, where we focus on three aspects identified by Mason and Kwok (2009): (1) willingness and interest in taking on equity investment; (2) general investability, as measured whether there is a viable business of interest to investors in terms of employment, sales, and profits; and (3) whether the firm has put in place specific measures investors want to see before making investments, such as separation of outcomes, revenue projections, knowledge of customer acquisition costs, tracking key metrics of traction, and covering intellectual property. The third and final domain looks at steps towards receiving external funding and then external financing received. Steps towards financing include contacting outside investors, making pitches, working with mentors or experts to help obtain financing, and entering into negotiations. Receipt of external financing considers new debt and equity investments, as well as receipt of incubator and accelerator grants.

23
We ask several questions under each domain and sub-domain. Our pre-analysis plan then specifies aggregating these measures to form standardized indices. This reduces concerns about multiple hypothesis testing by focusing on one aggregate outcome in each family of questions. Appendix 3 provides the exact questions used in forming each question, and Appendix 9 provides treatment impacts on each specific question used in these aggregate measures.

# **5.4 Do Higher Investment Readiness Scores Predict Better Firm Investment Readiness and Investment Outcomes?**

The investment readiness program resulted in higher investment readiness scores from judges. To investigate whether these judges’ scores are informative about future outcomes for the firm, we use the control group sample to run the regression: �������� �������������� ���������� ���′�� ��� (2) We carry out this estimation first with no additional controls, and then with controls _X_ for country (dummies for Serbia and for Croatia), whether or not the firm had received funding from an outside investor at baseline, and the business sector (dummies for business and productivity, and lifestyle and entertainment sectors). We estimate this separately by survey, to examine results at different time horizons.

Table 6 presents the results. Column 1 shows that 10% of control firms had died by the first followup, and 25% by the second follow-up, two years post-intervention. These high death rates are higher than the average rates in developing countries, and likely reflect the firms being young and in relatively developed countries (McKenzie and Paffhausen, 2018). We then see no significant association between higher investment readiness scores and subsequent survival. Columns 2 through 6 then examine the associations between higher investment readiness scores and our different measures of subsequent investment readiness actions and investment steps and outcomes. We see that the judges’ scores of investment readiness are statistically significant predictors of the subsequent willingness and interest of the owner in taking on equity investment, whether the firm is meeting specific needs of investors before investment can take place, whether the firm has taken steps towards external financing, and whether they have received external financing. This is true both in the short-run (six-month) survey, and in the two-year survey. The magnitudes range from 0.14 to 0.33, suggesting that a one-unit change in the judge scores (which had a mean of 2.9 and 24
standard deviation of 0.9) would predict a 0.14 to 0.33 increase in these indices (corresponding to 0.2 to 0.5 standard deviations). The only measure where we do not find a significant association is in general investability, which is an index of measures of firm employment, profits, and sales.

Finally, in the last column we examine whether the firm had made at least one deal with an outside investor since the start of the program (August 2015). 24.4 percent of the control group have made such a deal after two years. A one-mark higher investment readiness score from the judges significantly predicts a 17 to 18 percentage point increase in the likelihood of making such a deal, which is large relative to this baseline rate.

We therefore have that treatment has a causal impact (�<sup>�</sup> �0.28) on the investment readiness score received from judges, and that this investment readiness score in turn is a significant predictor (with coefficient �<sup>�</sup> ) of firm outcomes in the control group sample. Combining these two estimates allows us to obtain an estimate of the _predicted treatment effect_ �<sup>�</sup> �<sup>�</sup> . This predicted effect is shown for each outcome in Table 6. It assumes that the only impact of the investment readiness program on firm outcomes is captured through the investment readiness score, that the association between score and outcomes observed in the control group is causal, and that the sequential ignorability assumption of Imai et al. (2011) holds.<sup>13</sup> Although these assumptions can be questioned, we believe such an exercise is useful in providing a sense of the magnitudes we might expect to see for treatment effects, given how much our program affected investment readiness scores, and how much a change in scores in turn predicts future outcomes. We see that the predicted treatment effects are small in absolute terms: each of our index measures is predicted to increase by only 0.04 to 0.09 over two years, and the predicted increase in the likelihood of receiving outside funding is 4.6 percentage points. We compare our estimated treatment effects to these benchmarks in the next section.

# **5.5 Treatment Effects on Investment Readiness and Investment Outcomes**

Table 7 presents the treatment effects of the investment readiness program on these survey outcomes after estimating equation (1). Panel A shows the short-run impacts six months after the intervention, and panel B the impacts two years post-intervention. Treatment results in a 7.2

> 13 The sequential ignorability assumption requires that if there are heterogeneous treatment effects, it is not the case that the firms for which treatment increases investment readiness scores are different from the firms for which an increase in investment readiness scores would increase future outcomes.

25
percentage point increase in firm survival over two years, but this is not statistically significant, with a 95 percent confidence interval of (-1.7p.p., +16.1p.p.). We see a reduction in external investment in the very short-run, which comes through less debt financing, but no significant impact on any of our investment readiness or investment outcomes over two years. After two years, the treatment group is 5 percentage points more likely to have made a deal with an outside investor, with a 95 percent confidence interval of (-4.7p.p., +14.7p.p.). The estimated point estimates on all our index measures at two years are all positive, but small, ranging from 0.003 for our external investment index, to 0.089 for general investability. These magnitudes are similar to those of our predicted treatment effects, and in all cases the predicted treatment effect �<sup>�</sup> �<sup>�</sup> lies within the 95 percent confidence interval for our estimated treatment effects.

Appendix 9 shows impacts on the individual measures that make up these aggregate indices. The intervention has a large and significant (p=0.013) impact on employment after two years of 4.5 workers, which almost doubles the employment level in the control mean. Employment is often a key policy outcome by itself, and so this program would compare favorably to a number of other programs when judged on employment alone. However, if we correct for testing 25 different outcomes that make up the aggregate indices, this impact is no longer statistically significant (p=0.425).

# **5.6 How Should We Interpret the Lack of Treatment Effect?**

Our results show that the investment readiness program increased investment readiness scores from the judges, that these scores are predictive of future investment readiness and investment outcomes, but that we do not find any significant impacts of the program on these future outcomes. To reconcile these findings, we note that our estimated treatment effects are in line with the predicted treatment effects �<sup>�</sup> �<sup>�</sup> – while we increased investment readiness scores, we did not increase them by enough to register large enough changes in investment outcomes to be detectable. Our confidence intervals enable us to rule out the program having large absolute impacts on these outcomes, but are wide enough to allow for the program to have moderate sized impacts that are not possible to detect with the sample size we have.

One difficulty in detecting end outcomes comes from the issue of statistical power weakening the more steps one takes along a “funnel of attribution” (McKenzie, 2018). Our experiment starts with 26
a group of firms who apply to a program. To get from this stage to receiving equity investment requires satisfying a number of steps – they must be interested in receiving investment and become investment ready, take steps towards receiving investment, and then actually receive investment. The effective number of firms drops as we pass from one step to the next, making it harder to detect impacts for end outcomes than initial outcomes. This program is the first randomized experiment of its kind, but like a number of other experiments involving larger firms, the sample size is set by external constraints in terms of the number of firms that the program attracts and caters to, rather than being a choice parameter.

Given the sample size, our funding proposal calculated that we would have 80% power to detect a 0.23 increase in the investment readiness score, based on the mean and standard deviation of the baseline score measure and not accounting for the power gains from stratification. Our estimated treatment effect of 0.28 exceeds this level. In contrast, our funding proposal assumed that it would be very rare for control group firms to receive outside funding, assuming a mean of 3 percent, and then estimated a minimum detectable effect size of 8 percentage points at 80% power, not accounting for the power gains from stratified randomization (since we did not know how strongly our strata would be correlated with the end outcome). Ex post, our randomization strata have an R<sup>2</sup> of 0.29 in a regression of making a deal with an outside investor on strata dummies. Given this, our anticipated power would have been 91 percent to detect an 8 percentage point increase.

In practice, our estimated impact on receiving outside funding is 5 percentage points (similar in magnitude to the predicted impact �<sup>�</sup> �<sup>�</sup> �0.046�, which is less than this minimal detectable effect. But the larger reduction in power comes from the control mean being much higher than anticipated. While we expected very few control firms to receive external financing, in practice 24.4 percent of control firms had made a deal within two years. It is much harder to detect an 8 percentage point increase from a control mean of 24.4% than from a control mean of 3%: under our baseline assumptions, power would drop to 33.3% at this mean level. So a key reason for not being able to detect a treatment effect on external investment is that control firms found it easier to get investment than we had anticipated.

27
We explore the types of investment received in Table 8.<sup>14</sup> Firms in the two-year long follow-up survey were asked about whether they had made deals with different types of outside investors, and if so, what type of deal. We see that the most common deals occurred with other business owners (17%), angel investors (10%), and venture capital funds (10%).<sup>15</sup> Most of these deals were for a share of equity in the firm, with royalty deals, convertible notes, and licensing deals not very common. Both the type of investor and type of deal are similar across the treatment and control groups. We have only partial data on the amount of these deals, but know that 16.8% of firms (69% of those receiving an outside investment) received an amount of at least 25,000 euros, with this again not differing significantly by treatment status (appendix Table 9.5).<sup>16</sup> Finally, firms in the long survey were asked what is the main challenge their business faces in their ability to grow over the next two years. Getting financing is seen as the main challenge by 41% of the control firms and 24% of the treatment firms, with this difference statistically significant (p=0.002). In addition to the positive point estimate on getting access to external financing, it is possible that the investment readiness program led some treatment firms to realize that they need to improve other areas in their business first.

# **6. Conclusions**

Investment readiness programs have been offered in a range of developing and emerging markets, based on the idea of a gap between the quality of ideas entrepreneurs have, and their readiness to attract and receive outside investment in those ideas. Despite their growing use, there has not been any rigorous study of their effectiveness. Our five-country randomized trial enables measurement of the effect of such a program. We do find that investment readiness increases, as measured by scores in a pitch competition, and that these scores are in turn predictive of future investment readiness and outcomes among firms. Nevertheless, despite finding positive point estimates, our estimates of the treatment effects of the investment readiness program on these firm investment outcomes over the next two years are not statistically significant. Our analysis suggests that this in

> 14 Note that this table was not pre-specified, and is intended to explore the higher than anticipated rate of outside funding received by firms.

> 15 Our measure of receiving external financing excludes financing received from family and friends. 10.8% of firms in the long survey received financing from family and friends, but in less than half of the cases this was for an equity stake- terms tend to be less formal in such cases. 16 Using the long follow-up survey only, 43% (42% treatment, 44% control) of those receiving outside funding received at least 100,000 euros in investment. This information is not available for firms doing the short survey.

28
part reflects that the change in investment readiness score is not large enough to generate sizeable impacts on subsequent firm outcomes, and also that more of these firms are able to obtain financing without the program than was originally anticipated.

We believe these results offer lessons for governments deciding whether and how to use such policies. Starting with firms that express interest in outside funding, but that require many steps to take place before being in a position to receive funding may end up including many firms for which investment readiness is not the main constraint to receiving outside funding and to firm growth. As a result, investment readiness programs that start from the demand side of outside financing may have stronger impacts on getting firms to take steps towards investment readiness, than on investment outcomes, where other constraints also play a role. A possible alternative to test would be to start from the supply side, starting with investors and asking them for a list of firms that are at the margin of being investible, but for which assistance on specific aspects of investment readiness is needed.

# **References**

Bernstein, Shai, Arthur Korteweg and Kevin Laws (2017) “Attracting Early-Stage Investors: Evidence from a Randomized Field Experiment”, _Journal of Finance_ 72(2): 509-538.

Catalini, Cristian, Jorge Guzman and Scott Stern (2017) “Hidden in plain sight: venture growth with or without venture capital”, Mimeo. MIT.

- Clark, Colin (2008) “The impact of entrepreneurs' oral ‘pitch’ presentation skills on business angels' initial screening investment decisions”, _Venture Capital_ , 10:3, 257-279

Clingingsmith, David and Scott Shane (2017) “Training Aspiring Entrepreneurs to Pitch Experienced Investors: Evidence from a Field Experiment”, _Management Science_ , advance online access. Holm, Sture (1979) “A simple sequentially rejective multiple test procedure”, _Scandinavian Journal of Statistics_ 6: 65-70.

Gompers, Paul, Will Gornall, Steven Kaplan and Ilya Strebulaev (2016) “How do Venture Capitalists Make Decisions?”, Mimeo. Harvard Business School.

29
<mark>Gonzalez-Uribe, Juanita, and Michael Leatherbee (2018) " The Effects of Business Accelerators on Venture Performance: Evidence from Start-Up Chile"</mark> _<mark>, Review of Financial Studies</mark>_ <mark>31(4): 1566-1603.</mark> <mark>Hallen, Benjamin, Christopher Bingham, and Susan Cohen (2014) “Do Accelerators Accelerate? A Study of Venture Accelerators as a Path to Success?”,</mark> _<mark>Academy of Management Proceedings</mark>_ <mark>12955.</mark> <mark>Howell, Sabrina (2017) “Financing Innovation: Evidence from R&D Grants”,</mark> _<mark>American Economic Review</mark>_ <mark>107(4): 1136-64.</mark> <mark>Howell, Sabrina (2018) “Learning from Feedback: Evidence from New Ventures”, Mimeo. NYU. Imai, Kosuke, Luke Keele, Dustin Tingley, and Teppei Yamamoto (2011) “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies.”</mark> _<mark>American Political Science Review</mark>_ <mark>105 (4): 765–89.</mark> <mark>Kaplan, Steven and Per Strömberg (2004) “Characteristics, Contracts, and Actions: Evidence from Venture Capitalist Analyses”,</mark> _<mark>Journal of Finance</mark>_ <mark>59(5): 2177-2210.</mark> <mark>Karajkov, Risto (2009)</mark> _<mark>Bridging the Gap: Access to Risk Finance for Early-Stage SMEs in Southeast Europe.</mark>_ _<u><mark>http://www.incubat.ro/uploads/Publicatii_Investment_SMS_report.pdf</mark></u>_ <mark>Kerr, William R., Josh Lerner, and Antoinette Schoar (2014) "The consequences of entrepreneurial finance: Evidence from angel financings."</mark> _<mark>Review of Financial Studies</mark>_ <mark>27.1 (2014): 20-55.</mark> Lee, David (2009) “Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects", _Review of Economic Studies_ 76(3): 1071-1102.

Mason, Colin and Richard Harrison (2001) “Investment Readiness': A Critique of Government Proposals to Increase the Demand for Venture Capital”, _Regional Studies_ , 35(7): 663-668 Mason, Colin and Richard Harrison (2003) “Auditioning for money: what do technology investors look for at the initial screening stage?” _Journal of Private Equity_ 6(2): 29-42.

Mason, Colin and Jennifer Kwok (2010) “Investment Readiness Programmes and Access to Finance: A Critical Review of Design Issues”, _Local Economy_ 25(4): 269-92.

30
McKenzie, David (2018) “Statistical Power and the Funnel of Attribution”, Development Impact blog, January 8 https://blogs.worldbank.org/impactevaluations/statistical-power-and-funnel- <u>attribution.</u>

- McKenzie, David and Anna Luisa Paffhausen (2018) “Small Firm Death in Developing Countries”, _World Bank Policy Research Working Paper no._ 8236.

McKenzie, David and Christopher Woodruff (2014) “What are we learning from business training evaluations around the developing world?”, _World Bank Research Observer_ , 29(1): 48-82 Smith, Sheryl Winston, and TJ John Hannigan (2015) “Swinging for the fences: How do top accelerators impact the trajectories of new ventures?”, <u>http://druid8.sit.aau.dk/druid/acc_papers/5ntuo6s1r5dvrpf032x24x5on5lq.pdf</u> Wagner, Rodrigo (2016) “Does Feedback to Business-Plans Impact New Ventures? Evidence from a Field Experiment”, Available at SSRN: http://ssrn.com/abstract=2766566 Yu, Sandy (2016) “How do Accelerators Impact High-Technology Ventures?”, Mimeo. UC Berkeley 31
# **Figure 1a: Baseline Distributions of Investment Readiness for Treatment and Control Groups**

<!-- Start of picture text --> Baseline Score Distributions by Treatment Status<br>1 2 3 4 5<br>Baseline investment readiness score<br>Treatment Control<br>.5<br>.4<br>.3<br>.2<br>Proportion of Firms<br>.1<br>0<br><!-- End of picture text --> Note: Kolmogorov‐Smirnov test of equality of distributions has p‐value of 0.988

**Figure 1b: Baseline Distributions of Investment Readiness for Those Attending Semi‐Finals**

<!-- Start of picture text --> Baseline Score Distributions for Those Attending Semi-finals<br>1 2 3 4 5<br>Baseline investment readiness score<br>Treatment Control<br>.6<br>.4<br>.2<br>Proportion of Firms<br>0<br><!-- End of picture text --> Note: Kolmogorov‐Smirnov test of equality of distributions has p‐value of 0.959 32
# **Figure 2: Distribution of Investment Readiness Scores after Program as Scored by Judges**

<!-- Start of picture text --> Post-treatment Investment Readiness Distributions by Treatment Status<br>1 2 3 4 5<br>Semifinals investment readiness score<br>Treatment Control<br>.4<br>.3<br>.2<br>.1<br>0<br><!-- End of picture text --> Note: Kolmogorov‐Smirnov test of equality of distributions has p‐value of 0.017 33
|**Table 1: Balance Test on Application Data**|F|ull Sampl|e|Semi‐|Final Partic|ipants|
|---|---|---|---|---|---|---|
||Treatment|Control|P‐value|Treatment|Control|P‐value|
|_Variables stratified on_|||||||
|Incorporated/Registered in Croatia|0.25|0.24|0.612|0.25|0.30|0.920|
|Incorporated/Registered in Serbia|0.46|0.46|0.626|0.48|0.48|0.513|
|Baseline Readiness Score|2.95|2.92|0.150|2.99|2.97|0.476|
|Has an outside private investor|0.10|0.09|0.178|0.14|0.06|0.170|
|_Other variables_|||||||
|Market attractiveness score|3.08|3.05|0.851|3.13|3.18|0.579|
|Product technology score|2.47|2.43|0.835|2.56|2.71|0.085|
|Traction score|3.34|3.27|0.507|3.28|3.06|0.382|
|Team score|3.04|3.05|0.878|3.08|3.02|0.207|
|Sector is business and productivity|0.48|0.39|0.107|0.45|0.36|0.436|
|Sector is lifestyle and entertainment|0.18|0.23|0.295|0.20|0.27|0.215|
|Uses Cloud Technology|0.20|0.26|0.231|0.20|0.21|0.984|
|Uses Big Data|0.18|0.21|0.642|0.17|0.20|0.915|
|Place in value chain is developer|0.61|0.55|0.171|0.60|0.57|0.677|
|Place in value chain is service provider|0.59|0.54|0.372|0.60|0.54|0.108|
|Age of firm (years)|2.61|2.66|0.887|2.24|2.29|0.346|
|Early stage firm|0.30|0.33|0.475|0.35|0.37|0.554|
|Revenues in 2014|178073|184760|0.959|37642|144012|0.303|
|Number of employees|6.47|5.88|0.539|4.65|5.32|0.800|
|Age of main founder|38.22|36.81|0.204|38.02|36.67|0.362|
|Main founder has post‐graduate education|0.49|0.48|0.816|0.54|0.55|0.740|
|At least one founder is female|0.16|0.22|0.128|0.16|0.30|0.071|
|Company has a global focus|0.60|0.58|0.576|0.59|0.63|0.569|
|Have accepted outside financing|0.34|0.37|0.656|0.42|0.40|0.836|
|Previouslyin mentoring/accelerator program|0.15|0.16|0.704|0.18|0.22|0.202|
|**Sample Size**|**174**|**172**||**110**|**101**||
|Joint test of orthogonalityof treatmentp‐valu|e||0.621|||0.086|

Notes: Full sample denotes the full experimental sample. Semi‐final participants are the sample that were scored by judges during the semi‐final pitch event. Variables stratified on were the variables used in randomized assignment.

34
|**Table 2: Correlates of the Take‐up Decision among the Treatm**|**ent Group**<br>Take‐up W|hatAventure|Attend M|asterclasses|
|---|---|---|---|---|
||Full Model S<br>|tepwise Model <br>|Full Model <br>|Stepwise Model<br>|
|Baseline Investment Readiness Score|0.494**|0.573***|‐0.217|‐0.238*|
||(0.194)|(0.169)|(0.144)|(0.132)|
|Incorporated/Registered in Croatia|0.424||‐0.757**|‐0.771***|
||(0.397)||(0.296)|(0.286)|
|Incorporated/Registered in Serbia|0.409||‐0.331|‐0.393|
||(0.354)||(0.261)|(0.245)|
|Sector is business and productivity|0.128||0.040||
||(0.254)||(0.262)||
|Sector is lifestyle and entertainment|0.191||0.010||
||(0.366)||(0.327)||
|Uses Cloud Technology|0.137||0.577**|0.610**|
||(0.372)||(0.275)|(0.259)|
|Uses Big Data|0.740||0.278||
||(0.521)||(0.266)||
|Place in value chain is developer|‐0.181||‐0.360*|‐0.309|
||(0.292)||(0.215)|(0.207)|
|Place in value chain is service provider|‐0.211||‐0.108||
||(0.294)||(0.235)||
|Age of firm (years)|‐0.005||‐0.026||
||(0.056)||(0.044)||
|Early stage firm|‐0.147||0.024||
||(0.335)||(0.238)||
|Number of employees|0.008||‐0.005||
||(0.013)||(0.010)||
|Age of main founder|‐0.020|‐0.018|‐0.008||
||(0.014)|(0.013)|(0.012)||
|Main founder has post‐graduate education|‐0.141||0.487**|0.485**|
||(0.278)||(0.221)|(0.212)|
|At least one founder is female|‐0.674**|‐0.455|‐0.001||
||(0.302)|(0.337)|(0.270)||
|Company has a global focus|0.280||0.073||
||(0.305)||(0.225)||
|Have accepted outside financing|0.211||0.210||
||(0.324)||(0.252)||
|Has an outside private investor|‐0.123||‐0.121||
||(0.581)||(0.373)||
|Have participated in mentoring/accelerator program before|0.558||0.399|0.488*|
||(0.579)||(0.308)|(0.276)|
|Sample Size<br>Notes: Robust standard errors in parentheses. Coefficients ar<br>*, **, *** indicate significance at the 10, 5, and 1 percent leve<br>90.1% of firms took up WhatAVenture, and 43.7% attended at|174<br>e marginal effe<br>ls respectively<br>least one mast|174<br>cts from probit<br>erclass.|174<br>estimation.|174|

35
|**Table 3: Impact of Progra**|**m on Investm**<br>Overall|**ent Readi**|**ness as Score**|**d by Judge**|**s**|||Std Dev|Selected|
|---|---|---|---|---|---|---|---|---|---|
||Readiness<br>Score|Team<br>Score|Technology <br>Score|Traction<br>Score|Market<br>Score|Progress <br>Score|Presentation <br>Score|of Judge<br>Scores|to go to<br>Finals|
|_Base Specification_||||||||||
|Assigned to Treatment|0.284**|0.167|0.372**|0.206|0.268*|0.373***|0.372**|0.006|0.115*|
||(0.126)|(0.150)|(0.152)|(0.130)|(0.137)|(0.137)|(0.164)|(0.049)|(0.068)|
|_Including Judge Fixed Eff_|_ects_|||||||||
|Assigned to Treatment|0.409***|0.369**|0.476***|0.295**|0.463***|0.440***|0.514***|‐0.017|0.090|
||(0.135)|(0.158)|(0.174)|(0.142)|(0.139)|(0.143)|(0.191)|(0.051)|(0.076)|
|Sample Size|211|211|211|211|211|211|211|211|211|
|Control Mean|2.908|3.042|2.970|2.541|3.406|2.794|3.042|0.723|0.122|
|Control Std. Dev.<br>Notes:<br>Robust standard errors in<br>5, and 1 percent levels re<br>firm.|0.903<br>parentheses<br>spectively. Ju|1.068<br>. Regressi<br>dge fixed|1.031<br>ons control fo<br>effects contr|0.947<br>r randomi<br>ols for wh|0.940<br>zation stra<br>ich five of|0.937<br>ta. *, **, *<br>the sixty‐fi|1.145<br>** indicate sign<br>ve judges judg|0.317<br>ificance a<br>ed a parti|0.328<br>t the 10,<br>cular|

36
|**Table 4: Heterogeneity in Impacts on Investment Readiness**<br>Overall|||||||Std Dev|Selected|
|---|---|---|---|---|---|---|---|---|
|Readiness|Team|Technology|Traction|Market|Progress|Presentatio|n of Judge|to go to|
|Score|Score|Score|Score|Score|Score|Score|Scores|Finals|
|Assigned to Treatment<br>0.203|0.014|0.405**|0.138|0.009|0.314|0.249|0.020|0.177*|
|(0.178)|(0.208)|(0.193)|(0.192)|(0.180)|(0.197)|(0.230)|(0.062)|(0.101)|
|Assigned to Treatment*Baseline Readiness below Median<br>0.210|0.378|‐0.083|0.183|0.646**|0.169|0.310|‐0.019|‐0.179|
|(0.254)|(0.305)|(0.317)|(0.251)|(0.275)|(0.270)|(0.335)|(0.105)|(0.127)|
|Sample Size<br>211|211|211|211|211|211|211|211|211|
|Control Mean<br>2.908|3.042|2.970|2.541|3.406|2.794|3.042|0.723|0.122|
|Control Std. Dev<br>0.903|1.068|1.031|0.947|0.940|0.937|1.145|0.317|0.328|
|Notes:<br>Robust standard errors in parentheses. Regressions control for random<br>5, and 1 percent levels respectively. Regressions also control for level|ization<br>effect of|strata. *, **, *<br>having a bas|** indicate<br>eline inves|significan<br>tment rea|ce at the 1<br>diness scor|0,<br>e below the|median of|3.|

37
# **Table 5: Impacts on Media Mentions and Social Media Buzz**

||Any media<br>mention|Number of<br>Media mentions|# Facebook<br>likes|# Twitter<br>Followers|Media Buzz<br>Index|
|---|---|---|---|---|---|
|**Panel A: Impact at Six Months**||||||
|Assigned to Treatment|0.047|0.786|‐38.0|15.110|0.085|
||(0.031)|(0.483)|(145)|(18.495)|(0.053)|
|Sample Size|346|346|346|346|346|
|Control Mean|0.099|0.663|1119|112.471|‐0.060|
|stddev|0.299|3.410|2388|260.201|0.546|
|**Panel B: Impact at Eighteen months**||||||
|Assigned to Treatment|0.039|0.736**|0.889|22.106|0.112**|
||(0.030)|(0.291)|(218)|(18.974)|(0.047)|
|Sample Size|346|346|346|346|346|
|Control Mean|0.099|0.320|1430|106.866|‐0.073|
|Control S.D.|0.299|1.566|3106|249.504|0.528|

Notes: robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels. All regressions control for randomization strata fixed effects and for baseline values of outcome of interest.

**Any media mention** denotes firm was mentioned in news media in 6 month window, **number of media mentions** is the number of times the firm was mentioned, winsorized at the 99th percentile. **# Facebook likes** and **# Twitter Followers** are the number of Facebook likes for the firm's Facebook page, and number of Twitter followers for the firm, both winsorized at the 95th percentile. **Media Buzz Index** is an index of standardized z‐scores of these first four columns.

38
**Table 6: Judges Scores Predict Firm Outcomes 6 months and 2 years after program**

||Firm<br>survival|Interested<br>in equity|General<br>Investability|Specific needs <br>of investors|Investment<br>Steps|External<br>investment|Made a deal<br>with investor|
|---|---|---|---|---|---|---|---|
|**Panel A: Association at Six Mo**<br>|**nths**|<br>||||||
|_without controls_|0.024|0.201**|0.076|0.336***|0.222***|0.213**|0.093**|
|Score assessed by Judges|(0.037)|(0.076)|(0.072)|(0.065)|(0.082)|(0.098)|(0.038)|
|_with controls for country, prio_|_r funding, a_|_nd sector_||||||
|Score assessed by Judges|0.017|0.168*|0.052|0.300***|0.179**|0.187*|0.085**|
||(0.037)|(0.087)|(0.074)|(0.069)|(0.087)|(0.110)|(0.039)|
|Sample Size|92|83|83|81|73|82|82|
|Control Mean|0.898|‐0.015|‐0.039|‐0.059|0.008|0.084|0.083|
|Control S.D.|0.303|0.764|0.634|0.682|0.720|0.741|0.276|
|Predicted Treatment Effect|0.007|0.056|0.021|0.094|0.062|0.060|0.026|
|**Panel B: Association at Two Ye**|**ars**|||||||
|_without controls_|0.061|0.153*|0.040|0.136*|0.322***|0.322***|0.166***|
|Score assessed by Judges|(0.041)|(0.088)|(0.073)|(0.082)|(0.100)|(0.072)|(0.048)|
|_with controls for country, prio_|_r funding, a_|_nd sector_||||||
|Score assessed by Judges|0.053|0.128|0.027|0.140*|0.324***|0.339***|0.175***|
||(0.044)|(0.094)|(0.078)|(0.077)|(0.099)|(0.077)|(0.049)|
|Sample Size|100|92|86|88|80|99|99|
|Control Mean|0.753|‐0.005|‐0.058|‐0.059|‐0.032|0.018|0.244|
|Control S.D.|0.433|0.783|0.650|0.692|0.760|0.698|0.431|
|Predicted Treatment Effect|0.017|0.044|0.011|0.038|0.090|0.090|0.046|

Notes: robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels respectively.

**Firm survival** is a binary variable that takes value one if the firm is operating, and zero otherwise. **Interested in equity** is a standardized index of whether the firm is interested in equity financing, the maximum equity share they are willing to have owned by outside investors, whether they have specific deal terms for investors, and whether they would consider a royalty‐ based investment. **General investability** is a standardized index of number of employees, whether the founders work full‐ time in the business, whether the firm had positive sales in the first quarter of the year, whether total sales exceed 10,000 euros in that quarter, whether the firm made a positive profit in the past year, and whether the firm made sales to Western Europe or the United States. **Specific needs of investors** is a standardized index of whether business and personal accounts are separated, whether the firm has made a revenue projection for the next year, whether it knows customer acquisition costs, the number of key metrics tracked, whether it has found out if the product or service can be covered by intellectual property protection, and whether it has at least one form of intellectual property protection received or pending. **Investment steps** is a standardized index of having contacted at outside investor, made a pitch to an outside investor, have a mentor or external expert supporting them to obtain financing, and entered into negotiations with an outside investor. **External investment** is a standardized index of having taken on new debt, having made a deal with an outside investor, have received at least 25,000 euros in outside financing, and have received an incubator or accelerator grant (all since August 2015). **Made a deal with an investor** indicates having made a deal with an outside investor since August 2015 (program start).

39
**Table 7: Impacts on Survey Outcomes 6 months and 2 years after program**

||Firm<br>survival|Interested<br>in equity|General<br>Investability|Specific needs <br>of investors|Investment<br>Steps|External<br>investment|Made a deal<br>with investor|
|---|---|---|---|---|---|---|---|
|**Panel A: Impact at Six Months**||||||||
|Assigned to Treatment|0.049|0.051|0.026|0.082|‐0.017|‐0.152*|‐0.024|
||(0.030)|(0.094)|(0.085)|(0.080)|(0.098)|(0.087)|(0.033)|
|Sample Size|319|278|277|269|240|279|279|
|Control Mean|0.898|‐0.015|‐0.039|‐0.059|0.008|0.084|0.083|
|Control S.D.|0.303|0.764|0.634|0.682|0.720|0.741|0.276|
|_Predicted Treatment effect_|0.007|0.056|0.021|0.094|0.062|0.060|0.026|
|**Panel B: Impact at Two Years**||||||||
|Assigned to Treatment|0.072|0.032|0.089|0.084|0.044|0.003|0.050|
||(0.045)|(0.084)|(0.082)|(0.079)|(0.092)|(0.080)|(0.049)|
|Sample Size|340|309|291|298|282|330|330|
|Control Mean|0.753|‐0.005|‐0.058|‐0.059|‐0.032|0.018|0.244|
|Control S.D.|0.433|0.783|0.650|0.692|0.760|0.698|0.431|
|_Predicted Treatment effect_|0.017|0.044|0.011|0.038|0.090|0.090|0.046|

Notes: robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels respectively. All regressions control for randomization strata fixed effects.

**Firm survival** is a binary variable that takes value one if the firm is operating, and zero otherwise. **Interested in equity** is a standardized index of whether the firm is interested in equity financing, the maximum equity share they are willing to have owned by outside investors, whether they have specific deal terms for investors, and whether they would consider a royalty‐ based investment. **General investability** is a standardized index of number of employees, whether the founders work full‐ time in the business, whether the firm had positive sales in the first quarter of the year, whether total sales exceed 10,000 euros in that quarter, whether the firm made a positive profit in the past year, and whether the firm made sales to Western Europe or the United States. **Specific needs of investors** is a standardized index of whether business and personal accounts are separated, whether the firm has made a revenue projection for the next year, whether it knows customer acquisition costs, the number of key metrics tracked, whether it has found out if the product or service can be covered by intellectual property protection, and whether it has at least one form of intellectual property protection received or pending. **Investment steps** is a standardized index of having contacted at outside investor, made a pitch to an outside investor, have a mentor or external expert supporting them to obtain financing, and entered into negotiations with an outside investor. **External investment** is a standardized index of having taken on new debt, having made a deal with an outside investor, have received at least 25,000 euros in outside financing, and have received an incubator or accelerator grant (all since August 2015). **Made a deal with an investor** indicates having made a deal with an outside investor since August 2015 (program start).

_Predicted Treatment effect_ is the treatment effect predicted from association in the control group between the judges score and the outcome, multiplied by the treatment effect of the program on the judges score.

40
|**Table 8: Details on Types of Externa**|**l Financing **<br>Sample<br>Size|**Deals Made**<br>Treatment<br>Group|Control<br>Group|P‐value|
|---|---|---|---|---|
|_Who was deal made with?_|||||
|Sold Firm|221|0.017|0.038|0.547|
|Deal with Other Business Owner|221|0.165|0.179|0.385|
|Deal with Angel Investor|221|0.096|0.104|0.804|
|Deal with Crowdfunding|221|0.043|0.019|0.164|
|Deal with Accelerator|221|0.087|0.038|0.096|
|Deal with VC Fund|221|0.096|0.113|0.967|
|Deal with Government Fund|221|0.070|0.075|0.706|
|_What type of deal was made?_|||||
|Equity‐Share|221|0.209|0.236|0.765|
|Licensing Deal|221|0.043|0.009|0.113|
|Royalty Deal|221|0.035|0.057|0.737|
|Convertible Note Deal|221|0.026|0.019|0.868|
|Other Deal|221|0.043|0.057|0.826|
|_Says financing is main challenge_<br>Notes: Data are for firms that answ<br>approximately two years post‐inter<br>deals, and so numbers given are pr<br>deal type. Says financing is main ch<br>viewed as the main challenge the b|204<br>ered the full<br>vention. So<br>oportion of f<br>allenge is an<br>usiness face|0.239<br>survey in the<br>me firms made<br>irms which m<br>indicator of w<br>s in its ability t|0.411<br>second foll<br>multiple<br>ade at least<br>hether fina<br>o grow.|0.002<br>ow‐up,<br>one of this<br>ncing is|

41
# **ONLINE APPENDICES**

**Appendix 1** : Examples of Investment Readiness Programs Around the World

**Appendix 2:** Timeline

**Appendix 3:** Scoring Methodology and Variable Definitions

- **Appendix 4:** Additional Details on Treatment Intervention

- **Appendix 5:** Additional Details on Control Intervention

- **Appendix 6:** Additional Details on the Semi-Finals and Finals

**Appendix 7:** Impact on web traffic and being included on AngelList

**Appendix 8:** Follow-up survey completion rates and balance

**Appendix 9:** Treatment effects on individual survey outcomes

42
# **Appendix 1: Examples of Investment Readiness Programs around the World**

The text provides some examples of investment readiness programs offered in other countries. We provide more discussion of these examples here, with Mason and Harrison (2001) and Mason and Kwok (2009) also providing reviews of some programs.

# **Australia:**

The “Impact Investment Readiness Fund” offers grants of up to $100,000 for enterprises to purchase specialized capacity building support from providers such as advisory, financial, intermediary or legal services. The program aims to bridge the gap in the Australian market that exists between mission‐driven organizations in need of funding and investors actively seeking impact investment opportunities.<sup>17</sup>

# **United Kingdom**

In the UK, there are different types of investment readiness programs. Some of them discriminate in favor of entrepreneurs that have a social mission, while others focus on all types of firms. Within the first group, the “Investment and Contract Readiness Fund”<sup>18</sup> , supported by the Office for Civil Society, assists social ventures to build their capacity to be able to raise capital. The “Impact HUB Westminster” also offers an investment readiness program, the “Impact Investment Readiness”, which aims to accelerate investment into social and environmental businesses based in London. It helps entrepreneurs to learn which type of investment is right for them, discover how to write investable business plans, and articulate their business mission as an attractive impact investment. They usually offer two days of free in‐depth content on relevant topics, led by experts and supported by peer‐to‐peer learning.<sup>19</sup> Within the second group, the “Growth Accelerator”<sup>20</sup> provides investment readiness services that help the entrepreneur understand which type of finance is right for her/him, build strong business plan and investment pitch, ensure the financial information provided to potential investors is credible and robust, pitch to the right type of investor for the entrepreneur’s business, connect with a wide range of finance institutions and investors across the country and secure finance. Another example in the UK is the investment readiness program provided by the “Angel Capital Group”, which focuses specifically around three key dimensions: (i) positioning in the market, (ii) developing attractiveness to the investor, and (iii) pitching the message and opportunity correctly. From its headquarters in central London, the Angel Capital Group works both nationally and internationally, providing access to leading‐edge services designed to improve investment readiness, facilitate access to early stage investment, and create opportunities for the development of new early stage co‐investment funds, with a key focus on the angel investment market.<sup>21</sup> The “Greater London Enterprise”<sup>22</sup> is also another provider of investment readiness services, usually through a combination of an e‐learning model and legal and financial advisors, who are also investors.

Mason and Kwok (2010) also provide details on several other programs in the U.K. These include the different variants of investment readiness programs tried by the U.K. Small Business Service’s Investment Readiness Demonstration Project, the University of Warwick’s Science Park’s Investment Readiness

> 17 http://impactinvestingaustralia.com/iirf/.

> 18 http://www.sibgroup.org.uk/investment‐readiness/.

> 19 https://westminster.impacthub.net/impact‐investment‐ready/

> 20 http://www.ga.businessgrowthservice.greatbusiness.gov.uk/what‐we‐offer/access‐to‐finance/

> 21 http://www.angelcapital.co.uk/.

> 22 https://www.gle.co.uk/gle‐business‐support.html

43
program; and the Finance and Business program delivered in the North East of England by the North East Regional Development Agency.

# **Europe**

In Ireland, “Enterprise Ireland” offers investment readiness support to entrepreneurs by giving them access to the network “Enterprise Ireland Advisers” and allowing them to get specialist support in a range of key strategic business development functions, including equity raising, technology development, market research, and export sales. The “Invest Academy Programme”, is an investment readiness program sponsored by the European Business Angel Network (EBAN), Sun&Sup, and Eurada geared to train entrepreneurs to understand sources of financing for their company by building their knowledge of financial sources, and helping them to refine their business propositions and business plans to make them attractive to potential investors and/or lenders. “InvestHorizon” is a program designed to increase investments made in Innovative European SMEs through Investment Readiness development and investor sensitization. The program assists companies getting started, raising awareness amongst SMEs about investment sources, options and requirements, providing coaching services to get funded, and matching entrepreneurs with specialized and active investors through investment forum events.

A European Union financed project led to the Ready for Equity<sup>23</sup> program which now offers training programs for fund‐seeking entrepreneurs throughout Europe, with an 8‐module course that includes an introduction to equity, discussion of the investment process, team building, how to do the perfect presentation, and how to value the enterprise and manage exit.

# **United States**

In the U.S, there are also several initiatives to foster investment readiness. For example, the “Lean Startup” methodology developed by Steve Blank<sup>24</sup> offers entrepreneurs a framework to focus on what’s important to be ready. Teams use the Lean Startup toolkit: the Business Model Canvas + Customer Development process + Agile Engineering to prepare themselves and be ready to present their business propositions to potential investors. These three tools allow start‐ups to focus on the parts of an early‐ stage venture that matter the most for investors: the product, market fit, customer acquisition/base, revenue and cost models, channels and partners. The “Larta Institute” also offers investment readiness services.<sup>25</sup> By working side‐by‐side with entrepreneurs to identify and address their unique challenges and opportunities, they help them to be ready to raise equity finance. The Larta Institute gives entrepreneurs access to top‐notch specialists in financial planning and mentors that support the entrepreneur in building a credible and attractive business plan. They have also worked with NSF grantees to help them commercialize their ideas.

> 23 http://www.readyforequity.eu/article/2010/start_page/

> 24 http://steveblank.com/about/.

> 25 http://www.larta.org/services/entrepreneurs

44
# **Appendix 2: Timeline**

Aug 14, 2015: Applications launched August 2015: Roadshows, advertising Sept 6, 2015: Applications closed Sept 10, 2015: Random assignment done by computer Oct 2, 2015: Registration in AEA RCT registry Sept 10‐Nov 13, 2015: Investment Readiness program implemented, master classes, mentoring, etc.

November 12‐14, 2015: Semi‐finals and pitch event in Belgrade December 2‐4, 2015: Finals with the top 54 firms from the semi‐finals pitching in front of the investors VC fund managers and Business Angels.

April‐August 2016: First follow‐up survey (approximately 6 months post‐program) August 2017‐March 2018: Second follow‐up survey (approximately 2 years post‐program).

# **Appendix 3: Scoring and Data Appendix**

The key variables are measured and defined as follows:

# **_Baseline Investment Readiness_**

The applications were scored by a team from Pioneers Ventures, a seed‐stage venture capital investment unit. Two professional investment managers reviewed each eligible application independently and assigned a score, based on for sub‐scores using an agreed scoring metric as detailed below in Appendix Table 3.1. Where the independent scores differed by more than one unit, they discussed the cases to arrive at a consensus score, otherwise the scores were averaged. Each business was scored on four sub‐ components as follows:

# **Appendix Table 3.1: Description of the Investment Readiness Scoring Scale**

|_Category_|_Weight_|_Points_|_Threshold description_|
|---|---|---|---|
|**Market attractiveness**|**10%**|1<br>2|Market does not exist/ no market need<br>Small market well served by competitors or equally<br>good substitutes|
|||3|Large market well served by competitors or equally<br>good substitutes|
|||4|Attractive niche in small market with unique solution/<br>positioning|
|||5|Attractive niche in large market with unique solution/<br>positioning|
|||6|Very large and mostly untapped/ underserved market<br>with right offering|

45
|**Co‐founder(s) and team**|**20%**|1<br>Single founder, no team|
|---|---|---|
|||2<br>Team of 2+ people|
|||3<br>Complimentary team with little experience|
|||4<br>Complimentary team with significant experience|
|||5<br>Serial entrepreneur(s)|
|||6<br>Serial entrepreneur(s) with exit|

|**Product/ technology**|**30%**|1<br>No/ low innovation ‐ Imitation of existing products or<br>services|
|---|---|---|
|||2<br>Low innovation ‐ Localization of proven business<br>models from abroad|
|||3<br>Some innovation ‐ Incremental improvements of<br>existing products or services|
|||4<br>Innovative new solutions or business models that<br>address customer needs|
|||5<br>Competitive technological innovation/ advantage|
|||6<br>Patented/ patent‐pending technological innovation<br>or otherwise protected IP|
|**Traction**|**40%**|1<br>No traction|
|||2<br>Soft traction (press coverage, facebook likes etc.)|
|||3<br>Test users/ prototype testing|
|||4<br>Non‐financial KPIs (e.g. downloads, pre‐orders)|
|||5<br>Generating revenues|
|||6<br>Sustainable business (generated revenues in 2014 ><br>GPD/capita for each founder)|

The _baseline investment readiness score_ was then calculated as a weighted average of these four sub‐ components, using the weights detailed above.

# **_Semi‐Finals Scores Provided by Judges_**

Judges scored each of the following on a six‐point scale, with the score being the simple average of the scores of each of the five judges scoring the pitch:

1. _Team:_ a score for the skills and capabilities of the entrepreneur and team

2. _Technology:_ a score for the degree of innovativeness and technological advancement 3. _Traction:_ a score for indications of measureable market success 4. _Market:_ a score for commercial market attractiveness 5. _Progress:_ a score for recent business development progress (in the last 3 months) 6. _Presentation:_ a score for the presentation performance.

The following two variables were then calculated: _Overall readiness score:_ this is calculated as a weighted average of the team (28% weight), technology (21% weight), traction (14% weight), market (7% weight), and progress (30% weight) scores.

46
_Std dev of judge scores:_ the overall readiness score is calculated for each judge. We then calculate the standard deviation of the five judge scores for a firm to get this measure of how much disagreement amongst judges there was in the scoring.

Finally, we also construct a dummy variable _Selected to go to Finals_ to denote whether or not the firm was selected by virtue of having a top overall score or by direct nomination to go through to the Finals event.

# **_Media Mentions and Social Media Buzz_**

_Any media mention_ is a dummy variable that takes value one if the firm is mentioned in any of the over 250,000 global news sources covered by Meltwater during the six‐month period March 1 to August 31. This is measured for 2016 in panel A of Table 5, and for 2017 in panel B.

_Number of media mentions:_ the number of times the firm is mentioned in any of the global news sources covered by Meltwater during the six‐month period March 1 to August 31. This is winsorized at the 99<sup>th</sup> percentile to reduce the influence of outliers.

_# Facebook likes:_ the number of likes for the firm’s Facebook page, measured approximately 6 months and 18 months post intervention. This is recorded as zero for firms without Facebook pages (including firms that have closed down), and is winsorized at the 99<sup>th</sup> percentile.

_# Twitter followers:_ the number of followers the firm’s twitter account has, measured approximately 6 months and 18 months post intervention. This is recorded as zero for firms without twitter profiles (including firms that have closed down), and is winsorized at the 99<sup>th</sup> percentile.

_Media buzz index:_ Standardized z‐scores of each of the above four variables are obtained by subtracting their mean and dividing by their standard deviation (separately by time period). The media buzz index is then the mean of the standardized z‐scores for any media mention, number of media mentions, # facebook likes, and # twitter followers.

# **_Survey oOtcomes_**

_Firm survival_ : this is a dummy variable coded as one if the firm is still operating (regardless of whether or not it has the original owners), and 0 otherwise.

_Interested in equity:_ this is an average of standardized z‐scores from the following variables:

- _Interested in equity financing for the business:_ a dummy variable which takes value one if the owners say they are interested in receiving new equity financing for the business.

- _Maximum equity share willing to have held by outside investors:_ this variable ranges from 0 to 100, and is the percent of equity the firm owner reports being willing to have held by an outside investor. It is coded as 100 for individuals who have sold their whole firm, and as the share of equity currently held by investors for those who are not interested in receiving new equity.

- _Have specific deal terms of offer outside investors:_ this is a dummy variable, coded as one if the firm owner reports having specific deal terms (e.g. a draft term sheet) to offer outside investors, and zero otherwise. It is coded as zero for firms that have closed.

- _Would consider a royalty‐based investment:_ a dummy variable, coded as one if the firm owner reports willingness to consider a royalty‐based investment, and zero otherwise. It is coded as zero for firms that have closed.

47
_General investability:_ this is an average of standardized z‐scores of the following variables:

- _Number of employees in the company:_ the number of employees in the company, coded as zero for firms that are closed, and winsorized at the 99<sup>th</sup> percentile.

- _Founder/co‐founders work full‐time in the company:_ a dummy variable that takes value one if at least one of the founders works full‐time in the company, and zero otherwise.

- _Positive total sales for first quarter:_ this is a dummy variable which takes value one if the firm made positive sales in the first quarter of 2016 (first follow‐up survey), or in the first quarter of 2017 (second follow‐up survey), and zero otherwise. It is coded as zero for firms that have closed.

- _Total sales for first quarter of at least 10,000 euros:_ a dummy variable which takes value one if the firm made sales of at least 10,000 euros in the first quarter of 2016 (first follow‐up survey), or in the first quarter of 2017 (second follow‐up survey), and zero otherwise. It is coded as zero for firms that have closed.

- _Business made positive profit in last year:_ a dummy variable which takes value one if the firm made a positive profit in 2015 (first follow‐up survey) or in 2016 (second follow‐up survey), and zero otherwise. It is coded as zero for firms that are closed.

- _Sales made in Western Europe or U.S.:_ a dummy variable which takes value one if the firm makes sales in European Union countries (excluding Croatia and Slovenia) or in the United States, and zero otherwise. It is coded as zero for firms that are closed.

_Meeting the specific needs of investors:_ this is an average of standardized z‐scores of the following variables:

- _Accounts of the business are separated from those of the owners:_ a dummy variable that takes value one if the business accounts are kept separately from those of the owner, and zero otherwise. It is coded as zero for closed firms.

- _Revenue projection made for the next 12 months:_ a dummy variable that takes value one if the firm has in place a revenue projection for the next 12 months, and zero otherwise. It is coded as zero for closed firms.

- _Business knows customer acquisition costs:_ a dummy variable that takes value one if the firm knows the cost of acquiring a customer, and zero otherwise. It is coded as zero for closed firms.

- _Number of key metrics (out of 11) being tracked:_ the number of key metrics being tracked such as newsletter sign‐ups, pre‐orders, free user downloads, requests for samples or free trials, free pilot projects with customers, current active users, new sales leads per month, sales meetings per month, paid pilot projects with customers, paid customer sign‐ups or paid downloads, and customer life‐time value. This is coded as zero for closed firms.

- _Found out whether product or service can be covered by intellectual property protection:_ a dummy variable that takes value one if the firm has found out whether their product or service can be covered by some form of intellectual property protection, and zero otherwise. This is coded as zero for closed firms.

- _Has at least one form of intellectual property protection or application pending:_ A dummy variable that takes value one if the firm has, or has pending, a copyright, trademark, industrial design right, patent, or other form of IP protection, and zero otherwise. This is coded as zero for closed firms.

_Investment Steps:_ this is an average of standardized z‐scores of the following variables: 48
- _Has contacted an outside investor to see if they are interested in making an investment:_ A dummy variable taking the value one if, in the last year, the firm has contacted an outside investor to see if they are interested in making an investment, and zero otherwise. Firms that say they are not interested in investment and that do not answer this question are assumed to have not contacted an investor. Coded as zero for closed firms.

- _Has made a pitch to outside investors outside of our program:_ A dummy variable taking the value one if, in the past year, the firm made a pitch to outside investors at an event. Firms were explicitly asked to exclude pitches made during the semi‐finals and finals of the Pioneers program. Firms that say they are not interested in outside investment are assumed not to have made a pitch.  It is coded as zero otherwise, including if the firm is closed.

- _Have a mentor or external expert supporting them to obtain external financing:_ a dummy variable that takes the value one if the firm has a mentor or external expert helping them to raise funding, and is zero otherwise, including if the firm is closed.

- _Entered into negotiations with outside investor since August 2015:_ a dummy variable which takes the value one if the firm has entered into negotiations with any outside investor since August 2015, and zero otherwise. It is coded as zero if the firm is closed. Firms which have been sold, or which have received outside equity investments, and which did not answer this question, are assumed to have entered into negotiations.

_External investment:_ this is an average of standardized z‐scores of the following variables<sup>26</sup> :

- _Taken on new debt since August 2015:_ a dummy variable which takes value one if the firm has taken on new debt since August 2015, and zero otherwise. It is assumed to be zero for firms closed.

- _Have made a deal with an outside investor since August 2015:_ a dummy variable which takes value one if the firm has made a deal with an outside investor (who is not family or friends) since August 2015, and zero otherwise. This takes value one if the firm has been sold, and zero if the firm has closed before being sold.

- _Received at least 25,000 euros in new outside investment since August 2015:_ a dummy variable that takes value one if the firm has received at least 25,000 euros in outside investment since August 2015, and zero otherwise. It is set at zero for firms that have closed and not been sold for more than 25,000 euros.

- _Received incubator/accelerator grant since August 2015:_ a dummy variable that takes the value one if the firm has received a grant from an incubator or accelerator since August 2015, and zero otherwise.

_Have made a deal with an outside investor since August 2015:_ a dummy variable which takes value one if the firm has made a deal with an outside investor (who is not family or friends) since August 2015, and zero otherwise. This takes value one if the firm has been sold, and zero if the firm has closed before being sold. Note that this is also considered as part of the external investment index, but given its role as a

> 26 Our pre‐analysis plan also originally added a fifth variable to this index: total amount of outside investment received. However, after our first follow‐up survey found firms were very reluctant to specify the exact amount of funding received, this question was dropped from the second follow‐up survey, and so is not included in the overall index.

49
summary statistic of whether investment readiness leads to new investment, is also considered as an outcome by itself.

# **Appendix 4: Additional Details of the Treatment Program**

_Selection of Content_ The treatment group intervention was designed to reflect best international standards for investment readiness programs and guarantee quality of training and mentoring. One of the main concerns for us was to find an implementer having the capacity to train more than one hundred firms across five countries in the Western Balkans in a limited amount of time. This required the availability of a considerable quantity of mentors, both local and international, willing to travel to the region and with a wide‐ranging background of skills in business development. We also needed to find a partner with demonstrated capacity on organizing internationally renowned pitch events, where small and nascent enterprises have the opportunity to pitch in front of international investors and opportunity to network their product and ideas, witness successful stories from established young entrepreneurs and the investors’ community.

The selection procedure consisted in three phases: a call for an Expression of Interest (EOI), followed by submissions of Technical Proposals (TP) and a final phase where we made a comprehensive assessment of the technical proposals and their compatibility with the Terms of Reference (TOR). The first phase saw eight companies submitting their EOI. We selected five out of the nine companies that expressed their interest for the second phase: all of them shared a few characteristics like an international focus, and a team with experiences in the region and familiarity with the SMEs and VC eco‐systems of the Western Balkans.

The World Bank team reviewed these technical proposals, and also sought an outside evaluation from Professor Josh Lerner and his team at the Bella Research Group. They have worldwide experience in assessing venture capital eco‐systems and business accelerator programs. In addition, we referred to the expert opinion of country officials in the Western Balkan region, experts in the local national innovation agencies, familiar with the regional eco‐systems and hence able to detect incongruences of the technical proposals with local conditions. The final overall assessments merged the feedbacks of these three main sources: it listed the positives and negatives of each proposal and identified specific questions to be submitted to the applicants in case there were aspects to investigate further. The final ranking that emerged from the series of consultations and assessments identified the Austrian company Pioneers JFDI GmbH as the best suitable candidate for the planned intervention.

Pioneers JFDI GmbH was the best candidate because of the experience of their team in the region and in providing small businesses personalized training and advice, the competences and logistical as well as human capital capacity to deliver a widespread training program across five countries. Prior to 2011, the Pioneers team was involved with STARTeurope, which offered the Startup Live events, a series of training workshops and pitch events. Pioneers’ mentors have deep experience as venture‐funded startups entrepreneurs and represent the countries of interest in the Western Balkan region and in addition Austria, Germany, Greece, Israel, Lithuania, Poland, Slovak Republic, Turkey, United Kingdom and the 50
United States. Many of their mentors come through the Pioneers JFDI GmbH program already, so they already know the curriculum and thus do not need to be trained.

# _Treatment Website_

The treatment was operated under a separate brand to ensure separation and clearly communicate the difference between the “Pioneers of the Balkans” competition and the investment readiness program for the treated group. The “Startup Live Mini‐Accelerator” provided a dedicated website that also provides a central point of access to all the treatment resources. It was password‐protected to ensure that only invitees (i.e., Treatment Group participants, mentors, the program management team and World Bank Group team members) could access it.

At the beginning of the program each beneficiary of the treated group was provided with a starter kit including a detailed booklet with instructions and description of all the four parts of the investment readiness program: qualification phase, mentoring phase, masterclasses, and pitch training; and details of the Pioneers team and their contact details.

# _WhatAVenture_

WhatAVenture asks a simple set of question about the business in order to i) match the entrepreneur with the appropriate mentor ii) understand the phase of development and the preparation of the entrepreneur in order to tailor to each firm the subsequent individual mentoring phase, iii) bring the treatment group firms to a similar level of qualification before proceeding with individual mentoring in the second stage of the training period.

The application WhatAVenture and the methodology therein was developed and tested in the context of post‐graduate studies at the University of Economics and Business in Vienna, in close collaboration with leading academics and practitioners from the innovation and entrepreneurship field. It is an online interactive course for start‐ups to put in words the details of their business idea, from the development of the business plan to marketing strategy and their financing needs.<sup>27</sup> The application is designed for self‐ paced progress along its steps. Once registered, startups assigned to the treatment group were granted access to the tool until 31 December 2015 independent of their progress or advancement. After completing each step, they had the opportunity to discuss their progress, findings and potential questions or difficulties in short online mentoring sessions (typically 30‐45 minutes). The main questions addressed with the WhatAVenture application are:

1. Customer Exploration: the first step requires the team to answer questions on the targeted customers, to identify the customer segment and to customer needs related to their product

2. Solution: develop a solution to the problem and match it to the customers’ needs 27 Since the beginning of its external commercialization in 2014, the WhatAVenture has already been rolled out at several academic institutions as well as leading European corporates like Deutsche Telekom that use it for standardizing and professionalizing their intrapreneurship processes. Furthermore, several (corporate) accelerator programs like Bayer’s Grants4Apps and two Austrian governmental equity financing and R& funding institutions (Austrian Federal Promotional Bank; Vienna Business Agency) have chosen the tool as their central application for tracking startups progress and coordinating mentoring sessions throughout their programs.

51
3. Business model – frame a sound business model around the value proposition of the company 4. Competitor analysis – Elaborate on the competitive advantage of the firm, organize an idea of marketing, sizing and competitive positioning 5. Market size: define the target size of the customer segment 6. Financials: quantify the costs and revenue structures, expected profitability and financing needs until break‐even In the first meetings of the WhatAVenture the mentor takes some time to ask questions and understand in detail the product the company plans to market and the possible value generation. This is important for providing a better mentorship in the successive acceleration phase.

# _Assignment to Mentors_

In the qualification phase each company was assigned a lead mentor from the beginning who takes the role of a direct contact person for getting started in the mentoring program. The lead mentors support their mentees not only as their personal sparring partner during the qualification phase but also as primary contact person and advisor during the acceleration and pitch preparation phases. Match‐making is conducted based either on relevant professional experience (e.g., an entrepreneur in the dairy industry might be assigned a lead mentor with an academic background in dairy product management), personal interests (e.g., a participating business active in the area of design might be assigned a lead mentor with a passion for sailing), technical expertise (e.g., a team that lacks even a basic online presence might be assigned a web‐/graphics designer as a lead mentor) or proximity.

In addition, a “Mentors Catalogue” was distributed to each firm. It contains relevant biographical and professional information of the 100+ mentors forming the pool of regional and international experts from where the participants can draw in addition to the assigned lead mentors. The catalogue was sent to the treated group beneficiaries in the welcoming package just before the beginning of the program and they were provided with an internet interface where they have access to the network of dedicated mentors, and where they have the possibility to screen the qualifications and the field of expertise of the mentors through a short CV and contact them directly to book a mentoring session.

In total the treatment group could benefit from 141 mentors, who came from 26 different countries. Most of them live in Austria (43.3%) followed by Serbia (10.6%) and Germany (9.2%). They can be divided in four main groups: standard teachers and mentors (i.e. business consultants, university and business school professors), successful entrepreneurs (i.e. CEOs of their companies), successful young enterprise investors (e.g. business angel investors, venture capitalists etc.), leading public speakers and pitch trainers. All of them cover a wide range of expertise and have at least three years of mentoring experience, while more about half of mentors have, individually, more than 10 years of experience in business mentoring. The majority having experience in business development and management in the IC&T industry; there are more technical mentors with a science background as software or hardware experts, payment systems and financial industry experts. Other industries were also covered, as for instance health care and pharmaceuticals, automotive and transportation, shipping and apparel sectors. All mentors have a good knowledge of business development, but a dedicated group of mentors was highly 52
specialized in sales, marketing and e‐commerce as well as intellectual property, competitive strategy and marketing. A smaller subset has experience in human resources, relationships and team building.

# _Acceleration Phase_

Upon successfully completing the qualification phase all beneficiaries are inducted into the acceleration phase. The individual mentoring sessions were scheduled on the online dedicated website to the program and were carried out either remotely via phone, video call<sup>28</sup> or on‐site mentoring depending on the availability of mentors in the cities where the entrepreneur is located. It is important to note that among the pool of 100+ mentors many of them are internationals living and residing in the Western Balkan region, hence there was still the possibility to get international mentoring in English within the city of residence of the entrepreneurs. We ensured that every startup in the program gets some on‐site mentoring exposure, partly also as an instrument to ensure their continued personal commitment to the program and to allow for the development of personal relationships beyond voice and video calls.

Average mentoring sessions typically lasted approximately 90 minutes and required additional work between sessions from the entrepreneur to improve the business proposal before the next session. In total we had more than 1800 hours of individual mentoring. Once a mentor submits his feedback to the central database, the information entered into the first part of the form is be forwarded via e‐mail to the mentored entrepreneur, along with the request to likewise provide feedback to the mentor in question. This bidirectional feedback process not only serves the purpose of assessing mentees’ satisfaction with the mentor and the benefit gained from a particular session, but also to validate the mentor’s feedback and data entered by means of a counterparty review process.

Examples of the discussion in the acceleration phase were:

- Some companies were developing more products so needed advice on what would be best to focus on or whether to spin‐off part of their business.

- Explore value proposition for different customer segments and how to structure it (i.e. B2B or B2C), how to implement it and what channels of communication to use. When necessary narrow down customer segment.

- Some firms needed a market validation ‐ to take a prototype or mockup to target customers and test the outcomes.

- Formulating and analyzing the competitor’s matrix, set up a market research plan to investigate competition in target markets.

- Identifying local partners for collaboration and regional expansion.

- Defining a clear pricing strategy for different markets (e.g., Western Balkans, Europe, U.S. etc.)

- For companies in a more advanced stage discussions on possible financing options for current expansions plans, the amount to be asked and the form of partnership.

- Discuss legal ways to achieve monetization: early‐stage selling, licensing to interested parties worldwide for franchising etc.

> 28 Remote session were arranged between both parties to take place either via Skype or, especially for remote group mentoring hosted on our software solutions to provide video calls (e.g. WebEx).

53
- Making sure the startup product abides to and will operate according to existing regulations and the differences in regulations between the EU and the Western Balkans for limited liability partnerships and equity financing.

- Practice to present the company in 5 minutes and in an elevator pitch of 90 seconds; preparation for the questions time to understand what investors want, and working on telling a clear story.

# _Masterclass Weekends_

Additional training during the acceleration phase is delivered in from of classes and lectures, these take place during 2.5 days “masterclass weekends” organized in the participating countries. At the masterclass weekends general business education is taught, courses such as marketing, finances, team building, sales, competition as well as rhetoric, body language and design.

A dedicated website for masterclasses and the material was set up, the portal also offers information about and access to a dedicated community communication channel “ChatGrape”. This is an instant communication tool available as a browser‐based application as well as native application for most mobile devices and allows for private as well as group communication in a structured way by allowing all users to set up and join subject‐specific groups and to tag information and questions posted with key word expressions.

Masterclasses took place in the following four locations and dates:

- 9 to 11 October ‐ Split, Croatia

- 16 to 18 October ‐ Novi Sad, Serbia

- 23 to 25 October ‐ Pristina, Kosovo

- 30 October to 1 November ‐ Skopje, Macedonia

Each weekend had a main theme but was not exclusively dedicated to it with lectures, panels and presentations covering other topics as well. For example, the weekend in Split dealt with the business model, while the masterclass in Novi Sad with sales and marketing. The Pristina masterclass had the main lectures on team building and human resources while the Skopje weekend dealt with investment and finance. The final program of each masterclass weekend was set up at the end of September and published on the information portal so that beneficiaries can gather information and decide which masterclass weekends they want to attend. Before each masterclass the mentors and the beneficiaries are provided with a guide that helps them to understand the organizational structure of the weekend and the benefits of participating in the weekend. Some examples of the content of the masterclasses are described below.

The lecture on “Research and networking” introduced the importance of research and networking for the best business model. It explained the difference between a business model and a business plan, how to prepare an action plan and structure a business model canvas step by step. It urged entrepreneurs to think about the weakest points of their plans and possible solutions. It then touched upon the importance of customers, competition, sales, marketing, traction, business development and finances.

The class “Rapid Prototyping” described how to move from an idea to a market validated product. It explained the concept of rapid prototyping, the importance and the methods of prototyping and using examples from the cinema, cars and smartphone applications sectors. It then covered the concept of minimum viable product (MVP) and the need to frame business hypotheses on the market reaction to 54
their product, the customers and financial hypotheses too. All of them should be tested in the market to get feedback and fine tune product development.

The team building panel addressed questions on how to create a team and what are the most important features a new company must develop in order to have the investors’ attention. Two main things emerged: the first is that a successful enterprise has to form an eclectic and competent team encompassing all possible functions that a nascent company must have. The spectrum of functions proposed ranges from not only having a developer and an idea but also in having a good lawyer, a technician, a person familiar with the financing. But the most important of all seem to be having a very good member acting as a sales person. This figure should end up being most of the time the CTO of the company if not a co‐founder because dedicated persons are really difficult to find, in those cases is the founder itself that must acquire sales skills and complement them with partners acting as supporters in this role. The importance of having a team with a wide‐ranging expertise that complement each other turned out to be one of the best ways sending a positive message of confidence and investment readiness to the investors, a message saying that if you put the money in my company you are minimizing the risk of wasting your money.

The traction presentation emphasized the importance of the three Ts: team, technology and traction. Traction because it is strictly linked with the term growth, with the importance of scaling up and having sustainable growth and having a “product‐market fit” which is another way of saying that the product should be in line with the demand coming from the market. However, it was stated that one size does not fit all and there are no general rules, what works for one company is not always good for others, as well as a channel to gain traction today is not guaranteed that will work for the same company some time down the road. The focus shifted then to the need to update the targets, reset the objectives forward every time a target is reached. The channels to increase traction were also covered, 19 of those channels were mentioned and briefly explained (social and display ads, offline ads, email marketing, targeting blogs, direct sales, trade shows etc.).

The presentation “The quantified startup” delved into trying to use data driven decision frameworks into strategic decision making of a startup. The presentation is directed mostly, but not exclusively, to web service providers, that is companies that can track their users online. What kind of metric are important to identify which stage your startup belongs at the moment, what metrics are important to scale up or increase traction? The presentation provided references of papers and books the presenter recommends to identify the metrics needed for every stage in the startup development. Measures such as churn, acceptance rate, viral coefficient, cost per user and similar were overviewed.

The presentation “How to sell to corporations” covered the topic of how to get access to established corporates for nascent startups. How to ally with them and exploit the market potential and value they have. One of the main points is that corporations, despite investing money in in‐house accelerators and alliance partnership programs for startups, they do not really understand fully the value the startups that approach them have. So, it is up to the startups to get ready for this kind of partnership, it is they that have to explain and convince the corporations of the value of their idea. The presenter described a process 55
toward strategically thinking about approaching a corporation. How to convince corporations? Set the targets, find the best match, do your research, be well prepared, set our sales steps. An important aspect touched upon was that, once arranged a meeting you need to frame the meeting in order to get the idea convened, speak about concrete and clear things.

The lecture on B2B marketing saw a short introduction on the history of marketing. Some general information was given and the difference between the B2B and B2C marketing was explained. Introductions to new paradigms like the C2B and C2C was also described. The speaker explained processes of customer decision making and affiliation with a brand, with few examples from the most established companies and their marketing strategies. The importance of tradeshows for marketing was emphasized despite being an expensive option. But it is one of the best ways to get in touch with professional buyers informally.

The lecture on “EU funding” delved into the landscape of funding opportunities for startups and SMEs at the institutional EU level through EU structural funds for development. Information on different type of funding, the application process and the best way to approach these funding minimizing the load of work for the application. The need of a consultant for the application was also pushed forward as a good idea to develop these proposals and how much consultancy is needed.

The presentation “How to craft a pitch” as delivered in all four masterclasses and described how to structure the pitch and what to emphasize in it. The second part of the talk dealt with the 90 second elevator pitch. The emphasis for the 5 minute pitch was on seven main points to take into account: i) product/service what it is and explained it in detail to make the audience understand it, ii) market opportunity, what are the prospects, the vision and the demand for the product, iii) team, who are the main components of the team, what’s their expertise and role, iv) competition, v) finances and cost structure, vi) development stage: where you are, at what stage, what you need, and vii) future, where you will be, or expect to be, in 6 months to a year.

# _Pitch Preparation_

The mentoring program transitions into the Pitch Preparation Phase after the last masterclass weekend. This phase is intended to ensure that all beneficiaries focus their attention entirely on their pitch performance in the remaining two weeks before their appearance on stage in front of jury members in the semi‐finals pitch event.

In the course of this phase, a standard pitch training approach was implemented, this was developed and tested in the context of the annual Pioneers Festivals and consists of the following steps:

1. The entrepreneurs were asked to upload the pitch decks (tailored to a 5‐minute on‐stage presentation followed by another 5 minutes of questions and answers with the jury). This pitch deck is then made available to the lead mentor for initial review.

2. The entrepreneurs schedule a video call with their lead mentor to begin practicing the pitch together.

3. During the sessions, the entrepreneur delivers his/her pitch and receive feedback on both the oral pitch performance as well as the pitch deck.

56
4. Lead mentor and entrepreneur may schedule additional sessions bilaterally to review progress as the entrepreneur implements recommendations.

5. In parallel, the program management team assigns each entrepreneur two additional mentors, one it has worked with already and one new mentor.

6. Also these mentors are asked to schedule pitch training sessions with the entrepreneur and request the latest version of the pitch deck.

7. The program management team collects and reviews feedback protocols to assess the entrepreneurs’ preparedness for their Semi‐finals appearance.

The entrepreneurs were encouraged to use the time between sessions to work on their pitch decks and practice their oral delivery of the pitch further. Additionally, to this standard pitch training cycle and the live "dress rehearsal" on the day prior to their pitch in the Semi‐finals, entrepreneurs can request further support from specialists on rhetoric, body language or slide deck design by approaching relevant mentors from the mentors’ catalogue if needed, or upon recommendation by one of their pitch preparation mentors.

# _Detailed Dost Greakdown_

The cost of offering the program is provided in appendix Table 4.1

# **Appendix Table 4.1: Detailed Program Cost Breakdown**

57
# **Investment readiness programs - Calculatory program cost**

|**A. Individual mentoring**|**Unit**|**Quantity**|**Rate [USD]**|
|---|---|---|---|
|1. Direct cost of individual mentoring|hours per beneficiary|30|1'917|
|2. Overhead cost per mentor|per beneficiary|1|326|
|3. Overhead cost of mentoring program|per beneficiary|1|492|
|4. Online mentoring tool|per beneficiary|1|338|
|**_Subtotal per beneficiary_**|||**_3'072_**|
|**B. Masterclasses**||||
|1. Organization|per beneficiary|1|321|
|2. Venue & catering|per beneficiary|1|107|
|3. Lectures|per beneficiary|4|175|
|4. Travel and accomodation cost|per beneficiary|1|191|
|**_Subtotal per beneficiary_**|||**_793_**|
|**C. Pitch training**||||
|1. Organization|per finalist|1|170|
|2. Venue & catering|per finalist|1|73|
|3. Pitch training|per finalist|1|279|
|4. Travel and accomodation cost|per finalist|1|168|
|**_Subtotal per finalist_**|||**_690_**|
|**Grand total per beneficiary**||||
|A. Individual mentoring|per beneficiary|1|3'072|
|B. Masterclasses|per beneficiary|1|793|
|C. Pitch training|per beneficiary|0.33|230|
||||**_4'095_**|
|**Grand total per investment readiness program**||||
|A. Individual mentoring|Number of beneficiaries|150|460'865|
|B. Masterclasses|Number of masterclasses|4|118'932|
|C. Pitch training|Number of finalists|50|34'513|
||||**_614'310_**|

58
# **Appendix 5: Additional Details of the Control Program**

# _Selection of Content_

We organized the control group intervention design around a few simple guidelines: i) an online course, ii) relatively cheap or free to use, iii) offering general knowledge of simple investment readiness concepts and iv) providing e‐guidance toward a start‐up pitching competition. The World Bank team conducted market research together with Innovative Ventures Incorporated, a specialized investment advisor to international financial institutions and governments in private equity and venture capital funds. After this initial screening of available alternatives, the decision was made to use a paid online course since the alternatives without fee did not offer the necessary quality standards. For the paid alternatives we carefully evaluated the contents and undertook the full demo versions to understand the specific differences among the candidate courses.

The program chosen is an e‐learning course developed and distributed by the Global Commercialization Group (GCG) of the IC<sup>2</sup> Institute at the University of Texas at Austin. The group is an internationally active facilitator for growth of innovative and technology‐based businesses and it offers a wider range of technology commercialization training programs for managers around the world. The Innovation Readiness Series<sup>TM</sup> was created to bring the work of the Global Commercialization Group to a global customer base at a cheaper price vis‐à‐vis delivering training and international business development programs in‐country. Since its launch in 2011, the Innovation Readiness Series<sup>TM</sup> has trained more than two thousand entrepreneurs and students from 20 countries worldwide. The content can be offered in three different languages: English, Spanish or Russian. For the Pioneers of the Balkans cohort we opted for the English based course.

# _Course Details and Content_

The program introduces students to common terminology used in the start‐up eco‐system, and the requirements to commercialize innovations, including protecting intellectual property, describing an innovation and the benefits it provides (vs. features), navigating development, understanding competition (substitutes and direct competitive products), market validation, creating a ‘pitch’ and presenting to investors, customers and others.

This content is delivered online through 10 modules of 45‐60 minutes each. The modules have a set of slides that are read and explained via a recorded voice. Each module has detailed steps to work through for creating a business proposition and includes assignments in two formats: quizzes with multiple‐choice answers beneficiaries can take to test their understanding of the material, and in the case of some of the ten modules (i.e. technology brief and description, benefits, competition and presentation skills) written exercises to be voluntarily handed in. Finally, in the last module there is the possibility to record and upload a video sample of the planned pitch. Nevertheless, for the Pioneers of the Balkans cohort the program was customized to allow feedback only after the multiple‐choice quizzes in form of number of correct answers. Written exercise and the video of the pitch were voluntarily uploaded on the platform but were not commented or discussed with the participant.

While this program is not a substitute to one‐on‐one mentoring, it gives a basic introduction to business planning and pitching, is well‐structured and cheap alternative to a mentorship‐based investment readiness program, it is comprehensive and allows beneficiaries to create a sketch of business model 59
which can be presented to investors, customers and other interested parties. Moreover, it is a self‐ learning tool, beneficiaries can work at their own pace, the ten‐module series introduces the key concepts of innovation, and explores each of the primary issues that impact bringing a technology to the market allowing for a self‐paced learning environment.

In terms of curricular incentives, at the completion of all the modules, beneficiaries who answer correctly at least 70% of quiz questions and take active part in all of them, receive a certification of Investment Readiness from IC2 Institute at the Texas University through the World Bank Group program “Pioneers of the Balkans”.

The list of the ten modules and short description of the content is provided.

Module 1 – Introduction: the introduction module explains how the Innovation Readiness Series works, and the objectives for the course. It explains what commercialization is and helps distinguish between innovation and invention.

Module 2 ‐ Technical Description: the technology description module helps participants describe their innovation using technical jargon and key words.

Module 3 – Benefits: the benefits module teaches how to articulate the benefits of an innovation in a way that conveys value to customers and users.

Module 4 ‐ Development Status: the development status module delivers an overview of the product development cycle with an eye to the market.

Module 5 ‐ Intellectual Property, Part 1: explains what IP is, the different types of ownership, and what can be protected. It also explains Trademarks and Copyrights.

Module 6 ‐ Intellectual Property, Part 2: the focus is on Patents and Trade Secrets, and provides a foundation to designing an individual IP strategy.

Module 7 – Competition: the competition module will help the participant discover and compare key benefits to those of the competition.

Module 8 ‐ Market Validation: the market validation module explains the validation process and how to discover exactly what the market expects from an innovation.

Module 9 ‐ Pitching Your Innovation: the planning and pitching module helps prepare a technology brief of the innovation and can be used in the next steps to commercialization.

Module 10 ‐ Presentation Skills: the presentation module is taught by an internationally established and experienced public speaker, demonstrates how to deliver presentations in an effective and captivating way.

Depending on the previous experience of the participant and their commitment to hand in a written set of answers, a minimum of four weeks is recommended to deliver a basic course and the total envisioned time to complete the course lectures, answer the quizzes and compile the written exercises is 15‐30 hours. However, recall that among the set of assignments only the quizzes after each session were graded and participants receive feedback on the number of correct answers. In case of written exercises and uploaded pitch video no feedback was offered so that the only incentive in that case was self‐motivation. Moreover, 60
only quizzes counted toward the receipt of the final completion certificate, given this incentive structure we expect a lower usage of the written exercises and video pitch uploads than multiple‐choice quizzes.

# _Communication and Reminders_

During the deployment of the intervention our team sent weekly motivational announcements to the students on the platform and on their email address, the aim of the announcements was to promote learning and active participation. They were structured as progress reports where we showed the top ten performing firms in the last week in terms of correct answers in submitted quizzes, and explained the reasons why it is important to take part in the course. Firms were told that going through the modules would both help provide matching of their businesses with judges who had sectoral expertise in their business, and that going through the contents of the modules would likely increase their chances of getting a higher score in the semi‐finals and getting selected for the finals.

# _Usage_

Appendix Figure 5.1 summarizes the proportion of students that submitted assignments (either quizzes or written exercises <mark>), each bar corresponds to an assignment. O</mark> ut of the 120 participants that connected at least once to the online platform, 63 (36.6% of the total) actively participated in one of the quizzes, with 45 of them completing the threshold of 70% correct answers. For the non‐graded written exercises, the technology description was completed by 40 participants, the technology brief by 20, benefits exercise by 29 and the competition exercise by 22 participants. Lastly, only 8 students uploaded a video of their pitch.

# **<mark>Appendix Figure 5.1: Participation of the Control Group in Online Course Content</mark>**

<!-- Start of picture text --> Proportion of students submitted assignemnts<br>Technology Brief Exercise<br>Competition Exercise<br>Benefits Exercise<br>Technology Description Exercise<br>Prepare a Pitch<br>Competition Quiz<br>Tech Description Quiz<br>Market Validation Quiz<br>Intellectual Property 2 Quiz<br>Intellectual Property 1 Quiz<br>Development Status Quiz<br>Benefits Quiz<br>% 20% 40% 60% 80% 100%<br><!-- End of picture text -->

# _Satisfaction_

A short survey was administered after the semi‐finals to assess their satisfaction with different elements of the program. Respondents are therefore only the entrepreneurs that participated in the semi‐finals. The survey was answered by 102 treated group firms (92.7% of the treated semifinalists) and 87 control 61
group firms (86.1% of control semifinalists) **.** Appendix Table 5.1 compares the overall satisfaction of the treated and control group semifinalists over few dimensions on a scale from 1 to 6. The treated group values more the communication, the structure and design and the training materials provided, the difference is statistically significant. However, the mean grade given by the control group to those dimensions is well above 4. Recall that firms were blind to treatment assignment. Where there is no significant satisfaction difference between the treated and control group is in the feedback received from the jury at the semifinals and the organization of the semifinals. These features were common to both groups. As such, the satisfaction survey indicates the value added of the treatment also in the subjective assessment of the program by participants.

**Appendix Table 5.1:  Treated vs. Control Satisfaction survey – How satisfied are you with each of the following?**

|||Treatm<br>|ent<br>||Contr<br>|ol<br>||
|---|---|---|---|---|---|---|---|
|Dimension|Obs.|Mean|Std. dev.|Obs.|Mean|Std. dev.|p‐value|
|Communication overall|**102**|5.17|.95|**87**|4.55|1.44|0.014|
|Structure and Design of PotB|**101**|5.00|1.14|**86**|4.43|1.26|0.005|
|Training Resources|**102**|5.31|1.02|**84**|4.36|1.25|0.000|
|Jury Feedback|**101**|4.45|1.43|**86**|4.11|1.68|0.486|
|Semi‐Finals (Belgrade Venture|**101**|4.45|1.42|**83**|4.34|1.36|0.861|
|Forum)||||||||

Note: PotB denotes Pioneers of the Balkans program

# **Appendix 6: Additional Details on the Semi‐Finals and Finals**

Appendix Table 6.1 summarizes the characteristics of judges used for the scoring.

|**Appendix Table 6.1: Semi‐Final Judge Chara**|**cteristics**<br>**Mean**|**Std. Dev.**|
|---|---|---|
|Lives in the Western Balkans|0.37|0.49|
|Lives in European Union (except Croatia)|0.48|0.50|
|Male|0.88|0.33|
|Age|39.1|10.4|
|Has Founded a Company|0.75|0.43|
|Years of Experience in their industry|11.5|8.5|
|Company makes venture investments|0.64|0.48|
|Is an Angel Investor|0.37|0.49|
|Regularly Mentors Start‐ups|0.80|0.40|
|**Sample Size**|**65**||
|Note: data unavailable for one judge.|||

Appendix Figure 6.1 shows that the baseline distribution of investment readiness scores is similar for those that participated in the semi‐finals (and therefore received judges’ scores) and those that did not.

62
<!-- Start of picture text --> 4<br>oo \<br>7<br>7<br>7<br>7<br>7 7 /? \<br>179 \<br>ZL \<br>YO<br>4 Z \\\O<br>UZ NON<br>Ye SSN<br>SRS<br><!-- End of picture text --> OverallScore; = a + bBaselineTeamScore; + cBaselineMarketScore; + dBaselineProductScore; + eBaselineTractionScore, + f Croatia; + gSerbia; + hBaselineOutsidePrivatelnvestor; + &;
The second approach is to compare the participation rates of treatment and control and use Lee (2009) bounds to adjust for differential attrition. The participation rate in the semi‐finals was 63.2 percent for the treatment group, and 58.7 percent for the control group. The difference of 4.5 percent is not statistically significant (p=0.39, or 0.37 after controlling for strata fixed effects). Nevertheless, we test sensitivity to this difference in attrition rates by dropping the top or bottom eight (4.5% of 174) scores from the treatment group. The next two columns of Appendix Table 6.2 then show the Lee upper and lower bounds respectively are 0.41 and 0.18. Since Table 1 and appendix Figure 1 shows that the differential attrition is not coming from the tails of the baseline investment readiness score distribution, we think it highly unlikely that it would be coming from either tail of the follow‐up distribution.

As a final robustness check, we show in the last two columns of Appendix Table 6.2 that our results are not sensitive to how we aggregate the different sub‐scores. Column 5 aggregates the five sub‐scores using equal weights instead of the different weights in our main specification, while Column 6 also includes the presentation score. We see the estimated effects of 0.277 and 0.293 are very similar in sign, significance, and magnitude to those using the unequal weights.

Taken together, these results show that the impact of treatment on the investment readiness score is unlikely to be driven by differential participation patterns in the semi‐finals between the treatment and control groups, nor by the weighting, and so our finding that the investment readiness program has improved investment readiness is robust.

|**Appendix Table 6.2: Robustness of Impact on Investment Readiness to Attrition and to how scores are weighted**|
|---|

||Score|Imputed<br>Score|Lee<br>Upper|Lee<br>Lower|5 components<br>Equally|6 components<br>weighted|
|---|---|---|---|---|---|---|
|Assigned to Treatment|0.284**|0.193***|0.408***|0.176|0.277**|0.293**|
||(0.126)|(0.065)|(0.119)|(0.130)|(0.123)|(0.124)|
|Sample Size|211|343|203|203|211|211|
|Control Mean|2.908|2.865|2.908|2.908|2.950|2.966|
|Control Std. Dev|0.903|0.750|0.903|0.903|0.884|0.894|

Notes: Robust standard errors in parentheses. Regressions control for randomization strata.

*, **, *** indicate significance at the 10,5, and 1 percent levels respectively

Score is the investment readiness score in the semi‐finals. Imputed score imputes missing scores based on regressing the score for the control group on baseline team, traction, market readiness, product technology, country, and having an outside investor and using predicted score for missing observations. Lee upper and Lee lower bounds trim the bottom 8 and top 8 scores respectively from the treatment group to adjust for higher attrition in the control group. Equally weighted scores weight the five (team, technology, traction, market and progress) or six (also presentation) sub‐scores equally.

# **Appendix 7: Impact on Web Traffic and Being Included on AngelList**

Our pre‐analysis plan also noted that we would consider several measures of web traffic and web presence that have been used by other researchers (e.g. Kerr et al. (2014), Gonzales‐Uribe and Leatherbee (2015)), but which may be less appropriate for firms in a less developed market: whether or not the firm is included in AngelList, a popular web platform for fundraising, startup jobs and investing allowing startups to raise capital from angel investors; and the global web‐traffic rankings of the company’s webpage as collected by Alexa and SimilarWeb. We see no significant impacts on any of these measures.

64
# **Appendix Table 7.1: Impacts on Web Traffic and AngelList**

||Appears<br>on<br>AngelList|Has<br>Alexa<br>Rank|Alexa<br>Global<br>Ranking|Has<br>Similar<br>Rank|Similar<br>Web<br>Ranking|
|---|---|---|---|---|---|
|**Panel A: Impact at Six Mo**|**nths**|||||
|Assigned to Treatment|‐0.022|‐0.042|‐304.7|0.013|447.5|
||(0.035)|(0.043)|(1412)|(0.047)|(2068)|
|Sample Size|346|346|188|346|160|
|Control Mean|0.308|0.535|11161|0.442|12364|
|Control S.D.|0.463|0.500|8383|0.498|9434|
|**Panel B: Impact at Eighte**|**en months**|||||
|Assigned to Treatment|‐0.034|0.032|120.6|‐0.008|1952.4|
||(0.041)|(0.048)|(1438)|(0.048)|(1945)|
|Sample Size|346|346|132|346|156|
|Control Mean|0.372|0.343|7407|0.442|12614|
|Control S.D.|0.485|0.476|5431|0.498|9949|

Notes: Robust standard errors in parentheses. *, **, and *** denote significance at the 10, 5, and 1 percent levels respectively.

All regressions include controls for baseline level of outcome, and for strata used in randomization.

Alexa Global ranking and Similar Web ranking are expressed in 1000s, and are conditional on having a ranking at all.

# **Appendix 8: Follow‐up Survey Response Rates and Balance on Responders**

Appendix Table 8.1 reports the completion rates by treatment status for three definitions of completion. Initially we began with a longer follow‐up survey, which in addition to asking about our key outcomes, also asked a series of process questions about the Pioneers of the Balkans program and their reasons for participating or not participating. In order to encourage responses from more reluctant firms, we removed these questions to shorten the questionnaire for a second interviewing phase, with the short survey containing all the key outcomes in our pre‐analysis plan. Finally, for firms that we could not interview after multiple attempts, we attempted to collect basic information in a few minutes from them, asking for their current operating status, their number of employees, whether they had entered into negotiations with an outside investor to make an investment in their firm since August 2015, and how much new investment they had received since August 2015. In the second follow‐up, this basic information was restricted to whether the firm was still operating, and whether it has received external financing, and also used web searches and secondary contacts.

We see that the treatment group was more likely to respond to the full survey than the control group in the first follow‐up survey (p=0.066), but there is no significant difference in response rates for having at least the short survey, or at least basic information, and no significant treatment differences for the second follow‐up.

**<mark>Appendix Table 8.1: Follow‐up Survey Completion Rates</mark>**

65
||Overall|Treatment|Control|p‐value|
|---|---|---|---|---|
|_First Follow‐up Survey_|||||
|Completed Full Survey|0.65|0.70|0.60|0.066|
|Completed at least Short Survey|0.79|0.80|0.78|0.781|
|At least basic information|0.92|0.93|0.91|0.520|
|_Second Follow‐up Survey_|||||
|Completed Full Survey|0.64|0.66|0.62|0.282|
|Completed at least Short Survey|0.85|0.86|0.84|0.504|
|At least basic information|0.95|0.95|0.94|0.873|
|Sample Size|346|174|172||
|Note: p‐value is for test of equality<br>rates after controlling for randomiz<br>At least basic information denotes<br>is operating and whether it has rec|of treatme<br>ation strat<br>that inform<br>eived exter|nt and contr<br>a.<br>ation on whe<br>nal financing|ol complet<br>ther the f<br>is availab|ion<br>irm<br>le.|

Appendix Table 8.2 compares baseline observables for the treatment and control groups, conditional on completing at least the short survey. We cannot reject that these observables are orthogonal to treatment status for either definition of survey completion. Given the lack of significant difference in response rates by treatment status, and that the sample responding to at least the short survey is balanced on observables, we treat attrition as missing at random in our analysis of the survey data.

66
|**Appendix 8.2: Balance Test on Sample Interview**|**ed at Follo**<br>Answere|**w‐up**<br>d First Follo|w‐up|Answere|d Second F|ollow‐up|
|---|---|---|---|---|---|---|
||Treatment|Control|P‐value|Treatment|Control|P‐value|
|_Variables stratified on_|||||||
|Incorporated/Registered in Croatia|0.230|0.237|0.869|0.27|0.24|0.623|
|Incorporated/Registered in Serbia|0.446|0.481|0.619|0.48|0.50|0.637|
|Baseline Readiness Score|2.997|2.899|0.183|2.93|2.94|0.163|
|Has an outside private investor|0.122|0.067|0.145|0.10|0.10|0.227|
|_Other variables_|||||||
|Market attractiveness score|3.112|3.062|0.885|3.06|3.09|0.657|
|Product technology score|2.485|2.419|0.649|2.44|2.48|0.872|
|Traction score|3.433|3.233|0.818|3.32|3.17|0.135|
|Team score|3.090|3.008|0.971|3.00|3.11|0.630|
|Sector is business and productivity|0.460|0.393|0.435|0.47|0.38|0.172|
|Sector is lifestyle and entertainment|0.187|0.230|0.516|0.19|0.23|0.428|
|Uses Cloud Technology|0.201|0.252|0.617|0.19|0.26|0.187|
|Uses Big Data|0.187|0.222|0.959|0.19|0.24|0.186|
|Place in value chain is developer|0.647|0.533|0.056|0.63|0.57|0.270|
|Place in value chain is service provider|0.568|0.533|0.479|0.59|0.56|0.482|
|Age of firm (years)|2.712|2.622|0.445|2.55|2.50|0.951|
|Early stage firm|0.331|0.304|0.475|0.32|0.37|0.464|
|Revenues in 2014|197649|157401|0.955|181796|127478|0.630|
|Number of employees|6.856|5.467|0.341|6.08|5.35|0.218|
|Age of main founder|38.216|36.563|0.222|38.02|37.19|0.433|
|Main founder has post‐graduate education|0.525|0.496|0.934|0.50|0.50|0.770|
|At least one founder is female|0.137|0.222|0.066|0.15|0.22|0.063|
|Company has a global focus|0.583|0.578|0.850|0.59|0.60|0.815|
|Have accepted outside financing|0.374|0.348|0.559|0.35|0.39|0.614|
|Previously in mentoring/accelerator program|0.173|0.178|0.535|0.16|0.17|0.905|
|**Sample Size**|139|135||150|144||
|Joint test of orthogonalityof treatmentp‐value|||0.417|||0.167|
|Note: interviewed at follow‐up denote that fir|m completed|at least th|e short su|rvey|||

# **Appendix 9: Treatment Effects on Individual Survey Outcomes**

Appendix Tables 9.1, 9.2, 9.3, 9.4 and 9.5 report the treatment impacts estimated on each of the individual outcomes that make up the aggregate indices presented in Table 7. Our main approach to multiple testing is to use the standardized indices of z‐scores, which are contained in Table 7, and are presented again at the end of each table. Alternatively, since there are 25 outcomes presented in these appendix tables for each time period, using Holm’s (1979) step‐down method gives an adjusted p‐value for the most significant of the individual outcomes (the employment effect in round 2) of 0.425. Thus, none of the coefficients shown in these tables are individually significant after adjustments for multiple testing.

67
|**Appendix Table 9.1: Treatment I**|**mpacts on Willingness**<br>|**and Interest in**<br>|**Taking on Eq**<br>|**uity Investment**<br>||
|---|---|---|---|---|---|
||Interested in<br>equity investment|Maximum<br>equity share|Has specific<br>deal terms|Would consider <br>Royalties|Aggregate<br>Index|
|**Panel A: Impact at Six Months**||||||
|Assigned to Treatment|‐0.019|3.920|0.001|0.025|0.051|
||(0.066)|(3.169)|(0.061)|(0.065)|(0.094)|
|Sample Size|278|264|271|268|278|
|Control Mean|0.603|23.155|0.331|0.508|‐0.015|
|Control S.D.|0.491|23.439|0.472|0.502|0.764|
|**Panel B: Impact at Two Years**||||||
|Assigned to Treatment|‐0.034|‐2.175|0.050|0.105*|0.032|
||(0.055)|(2.972)|(0.051)|(0.056)|(0.084)|
|Sample Size|309|285|309|303|309|
|Control Mean|0.575|25.066|0.242|0.487|‐0.005|
|Control S.D.|0.496|26.591|0.430|0.501|0.783|

|**Appendix Table 9.2: Impacts on General**<br>|**Investabil**<br>|**ity**<br>|||||
|---|---|---|---|---|---|---|
|Number<br>Employees|Founder<br> full‐time|Positive<br> Revenue|Revenue<br> >10,000 euros|Positive<br>Profit|Sales<br>US/Europe|Aggregate<br>Index|
|**Panel A: Impact at Six Months**|||||||
|Assigned to Treatment<br>1.100|0.061|0.008|0.035|‐0.061|‐0.019|0.026|
|(1.215)|(0.052)|(0.061)|(0.068)|(0.059)|(0.060)|(0.085)|
|Sample Size<br>318|269|277|277|272|265|277|
|Control Mean<br>6.111|0.750|0.699|0.353|0.289|0.386|‐0.039|
|Control S.D.<br>10.596|0.435|0.461|0.480|0.455|0.489|0.634|
|**Panel B: Impact at Two Years**|||||||
|Assigned to Treatment<br>4.554**|0.018|0.032|‐0.017|0.085|0.051|0.089|
|(1.814)|(0.061)|(0.071)|(0.068)|(0.061)|(0.055)|(0.082)|
|Sample Size<br>291|291|232|242|276|310|291|
|Control Mean<br>4.683|0.620|0.526|0.361|0.482|0.340|‐0.058|
|Control S.D.<br>6.381|0.487|0.502|0.482|0.502|0.475|0.650|

68
|**Appendix Table 9.3: Imp **|**acts on Meet**<br>Separates<br>Accounts|**ing Specific N**<br>Has revenue<br>projection|**eeds of Investors**<br>Knows customer<br>acquisition cost|Number key<br>metrics tracked|Found out if<br> can protect IP|Has IP or <br> pending|Aggregate<br>Index|
|---|---|---|---|---|---|---|---|
|**Panel A: Impact at Six Mo**|**nths**|||||||
|Assigned to Treatment|0.060|0.066|0.009|‐0.168|0.033|0.054|0.082|
||(0.053)|(0.066)|(0.064)|(0.299)|(0.065)|(0.061)|(0.080)|
|Sample Size|268|268|268|268|269|269|269|
|Control Mean|0.742|0.561|0.409|2.106|0.439|0.364|‐0.059|
|Control S.D.|0.439|0.498|0.494|2.598|0.498|0.483|0.682|
|**Panel B: Impact at Two Y**|**ears**|||||||
|Assigned to Treatment|0.086|0.018|0.061|‐0.092|0.059|0.049|0.084|
||(0.060)|(0.061)|(0.059)|(0.361)|(0.063)|(0.057)|(0.079)|
|Sample Size|291|291|291|269|271|275|298|
|Control Mean|0.577|0.486|0.352|1.667|0.444|0.244|‐0.059|
|Control S.D.|0.496|0.502|0.479|2.854|0.499|0.431|0.692|

69
|**Appendix Table 9.4: Impacts on Steps**|**Towards Investmen**<br>Contacted|**t**|Has mentor|||
|---|---|---|---|---|---|
||outside<br>investor|Made<br>a pitch|helping raise<br>finance|Entered into<br>negotiations|Aggregate<br>Index|
|**Panel A: Impact at Six Months**||||||
|Assigned to Treatment|‐0.082|0.016|0.078|‐0.008|‐0.017|
||(0.074)|(0.068)|(0.063)|(0.057)|(0.098)|
|Sample Size|239|240|232|279|240|
|Control Mean|0.509|0.549|0.236|0.323|0.008|
|Control S.D.|0.502|0.500|0.427|0.470|0.720|
|**Panel B: Impact at Two Years**||||||
|Assigned to Treatment|‐0.019|0.006|0.050|0.068|0.044|
||(0.057)|(0.047)|(0.040)|(0.059)|(0.092)|
|Sample Size|282|282|279|279|282|
|Control Mean|0.324|0.184|0.097|0.328|‐0.032|
|Control S.D.|0.470|0.389|0.297|0.471|0.760|

70
|**Appendix Table 9.5: Impact on Ex**|**ternal Invest**<br>|**ment**<br>|||||
|---|---|---|---|---|---|---|
||Taken on <br>new<br>debt|Made deal<br>with<br>investor|Received<br>at least<br>25,000|Amount of<br>new investment<br>received|Received<br>incubator <br>grant|Aggregate<br>index|
|**Panel A: Impact at Six Months**|||||||
|Assigned to Treatment|‐0.118**|‐0.024|‐0.032|‐11232*|‐0.036|‐0.152*|
||(0.057)|(0.033)|(0.028)|(6425.486)|(0.037)|(0.087)|
|Sample Size|276|279|277|277|269|279|
|Control Mean|0.419|0.083|0.068|13452.273|0.090|0.084|
|Control S.D.|0.495|0.276|0.253|62358.927|0.288|0.741|
|**Panel B: Impact at Two Years**|||||||
|Assigned to Treatment|‐0.059|0.050|‐0.024|n.m.|‐0.005|0.003|
||(0.048)|(0.049)|(0.041)||(0.036)|(0.080)|
|Sample Size|278|330|317||268|330|
|Control Mean|0.182|0.244|0.168||0.076|0.018|
|Control S.D.<br>Note: n.m. denotes not measure|0.388<br>d in this surv|0.431<br>ey round.|0.375||0.267|0.698|

71