metadata: authors: 0: author: Miller,Amisha
1: author: Lall,Saurabh A.
2: author: Goldstein,Markus P.
3: author: Montalvao Machado,Joao H. C.
count: World
docna: 0: docna: Asking Better Questions : The Effect of
            Changing Investment Organizations’ Evaluation Practices on
            Gender Disparities in Funding Innovation
docty: Policy Research Working Paper
abstracts: cdata!: Female innovators raise fewer resources
            from investors, even when their ventures are similar to
            those of all-male teams. Efforts to mitigate the disparities
            have typically focused on changing how founders seek
            investment. However, the causes of gender disparities are
            systemic: in uncertain contexts, evaluators value
            women&#x2019;s competence or leadership potential lower
            than men&#x2019;s, and investors inquire more about
            risks when facing female founders than males. What is the
            effect of investment organizations&#x2019; evaluation
            practices on gender disparities in funding innovation? This
            paper examines a two-stage global field experiment with
            investors making 1,871 investment decisions on early-stage
            startups, which resulted in $320,000 invested in 16
            startups. The experiment changed an
            organization&#x2019;s evaluation framework to
            systematize investor inquiry across all ventures by
            including prompts about (1) risk and reward and (2) progress
            during the evaluation period. This caused treated investors
            to (1) assess startups more consistently and (2) assess
            startup competence more dynamically than control investors.
            It eliminated, even reversed, the gender gap in investment
            outcomes. These results have implications for organizations
            making decisions in uncertain contexts, and those aiming to
            reduce gender disparities.
display_title: Asking Better Questions : The Effect
            of Changing Investment Organizations’ Evaluation Practices
            on Gender Disparities in Funding Innovation
listing_relative_url: /research/2023/12/34208252/asking-better-questions-effect-changing-investment-organizations’-evaluation-practices-gender-disparities-funding-innovation
topicv3: Private Sector Development m1326060 1921,Angel and Venture Capital m1326977 383,Gender m1327621 615,Entrepreneurship m1582903 4424
publishtoextweb_dt: 2023-12-04T00:00:00Z
dois: 10.1596/1813-9450-10625
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 63
author_keys: 0: author_key: NA
1: author_key: NA
2: author_key: 769522
3: author_key: 1237527
model_extractions: Policy Research Working Paper 10625

# Asking Better Questions

The Effect of Changing Investment Organizations’ Evaluation Practices on Gender Disparities in Funding Innovation _Amisha Miller Saurabh A. Lall Markus Goldstein Joao Montalvao_ Africa Region Gender Innovation Lab December 2023
Policy Research Working Paper 10625

# **Abstract**

Female innovators raise fewer resources from investors, even when their ventures are similar to those of all-male teams. Efforts to mitigate the disparities have typically focused on changing how founders seek investment. However, the causes of gender disparities are systemic: in uncertain contexts, evaluators value women’s competence or leadership potential lower than men’s, and investors inquire more about risks when facing female founders than males. What is the effect of investment organizations’ evaluation practices on gender disparities in funding innovation? This paper examines a two-stage global field experiment with investors making 1,871 investment decisions on early-stage startups, which resulted in $320,000 invested in 16 startups. The experiment changed an organization’s evaluation framework to systematize investor inquiry across all ventures by including prompts about (1) risk and reward and (2) progress during the evaluation period. This caused treated investors to (1) assess startups more consistently and (2) assess startup competence more dynamically than control investors. It eliminated, even reversed, the gender gap in investment outcomes. These results have implications for organizations making decisions in uncertain contexts, and those aiming to reduce gender disparities.

This paper is a product of the Gender Innovation Lab, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at jmontalvao@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Asking Better Questions: The Effect of Changing Investment Organizations’ Evaluation Practices on Gender Disparities in Funding Innovation**<sup>∗</sup>

**Amisha Miller      Saurabh A. Lall      Markus Goldstein      Joao Montalvao**<sup>**†**</sup>

JEL Classification: G24, D22 Keywords: Gender gap, Entrepreneurship, Innovation.

> ∗ We thank our implementing partner Village Capital. We thank the World Bank Group’s Umbrella Facility for Gender Equality (UFGE) and the We-Fi initiative, as well as ANDE and SGB for funding. Thank you also to David McKenzie, Christopher Woodruff, the NESTA IGL Group, and the University of Oregon Policy Group for helpful comments. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. This paper is an output of the World Bank Africa Gender Innovation Lab, within the Office of the Africa Region Chief Economist.

> † Miller: New York University's Stern School of Business, amisha.miller@nyu.edu; Lall: Adam Smith Business School, University of Glasgow, Saurabh.Lall@glasgow.ac.uk; Montalvao: World Bank, jmontalvao@worldbank.org.
# **<mark>1. Introduction</mark>**

<mark>Women are underrepresented in leadership positions in innovation – among the population of funded ventures, under 12 percent of startups have female founders (e.g., Gompers and Wang 2017, Lerner and Nanda 2020, Luo and Zhang 2022). Early-stage startups with female founders are valued less than those with male founders even when ventures are similar or identical (e.g., Brooks et al. 2015, Guzman and Kacperczyk 2019, Ewens and Townsend 2020, Bapna and Ganco 2021). This disparity exists across the US and developing economies (Roberts and Lall 2018). It inhibits the ability of female-founded ventures to grow (Delecourt and Ng 2021), directs innovation away from novel solutions or female users (e.g., Jeppesen and Lakhani 2010, Koning et al. 2020), and, more broadly, can result in a large misallocation of resources within economies (e.g., Hsieh et al. 2019).</mark> <mark>Scholars have theorized systemic reasons that produce and reinforce gender disparities in economic outcomes (e.g., Fernandez-Mateo and Kaplan 2018). Gatekeepers play an important role in resource allocation and are overwhelmingly male (Gompers and Wang 2017). They tend to socialize with, hire, or invest in people who share their gender (Ibarra 1993, Greenberg and Mollick 2017, Howell and Nanda 2019, Bapna and Ganco 2021). Beyond homophily factors, in contexts where quality is uncertain (such as innovation), all evaluators tend to rely on easily accessible indicators of expected quality, including status (Podolny 1993, Simcoe and Waguespack 2011, Kim and King 2014). Because men are typically perceived as higher status and/or more competent than women, this – often unconscious – reliance on gender can lead to lower evaluations for women than men (Ridgeway and Correll 2004, Correll and Benard 2006, Botelho and Abraham 2017, Snellman and Solal 2023). In addition, early evaluators often consider the preferences of others, which pushes evaluators to make more conventional choices, as encoded in status beliefs (Correll et al. 2017).</mark> 2
<mark>These gendered differences are also embedded in investors’ behaviors during evaluation processes. In the absence of information about organizational performance (Aldrich and Fiol 1994, Cohen et al. 2019a), evaluation typically involves interacting with founders to gather information on the potential of the innovator and their idea within a short time (Kirsch et al. 2009, Petty and Gruber 2011, Huang and Pearce 2015, Huang 2018). During these interactions, evaluators assess innovators, and pattern-match their behaviors to previous successful cases (e.g., Elsbach and Kramer 2003, Huang 2018), who are typically male. Investors also often ask founders questions early in interactions to better understand a venture (Miller et al. 2023), and tend to ask female founders more difficult questions compared to male founders (Kanze et al. 2018). Given these patterns of behaviors, investors’ processes of inquiry – how evaluators</mark> assess the potential of an innovator and their idea during interactions, in the absence of extant performance data <mark>– can disadvantage female innovators.</mark> <mark>The extensive research on the sources of women’s underrepresentation in innovation is not matched by how they might be mitigated (Jennings and Brush 2013). Scholars have examined female resource-seekers’ pitches on investors’ decisions (Kanze et al. 2018, Lee and Huang 2018, Balachandra et al. 2019, Huang et al. 2021). By focusing on how individuals might circumvent investors’ behavior, scholars have held investor behaviors constant. Yet many investors are embedded in accelerators, angel groups, or venture capital firms, which typically allocate organization-level funds as part of a designed, collective investment process (Drover et al. 2017). Investment organizations and their funders (often limited partners) have invested $4.8 billion in diversity strategies since 2018 (Cortes 2019, Biegel et al. 2020, DFC 2021), but the strategies they employ to do so, and the effects on investment outcomes are understudied. Without examining organizational evaluation practices – creating agreement on referents, negotiating about criteria, and establishing value by comparing entities (Lamont 2012) – s</mark> cholars and practitioners cannot fully understand how disparities in 3
investment outcomes are created or might be reduced. _What is the effect of investment organizations’ evaluation practices on gender disparities in funding innovation?_ <mark>Examining this question requires access to investment organizations’ evaluation templates, investors’ evaluation practices, and investment outcomes by gender. This research is d</mark> ifficult to conduct at <mark>scale because most investment organizations prefer not to share their evaluation processes and outcomes publicly (e.g., Da Rin et al. 2013). To examine this question, we employ unique data from Village Capital (</mark> Vilcap <mark>), a global investment organization that</mark> selects qualified early-stage startups for consideration by its own investors, and introducing startups to a broader and more diverse set of potential follow-on investors.

Typically, investment organizations evaluate startups using organization-level criteria over three months (Tyebjee and Bruno 1984, Fried and Hisrich 1994, Cohen et al. 2019b, Gompers et al. 2020). Investors receive information – i.e., a pitch deck or overview, meet founders, and then make a decision on whether to continue to diligence (or inquire about) the startup. This process repeats as investors progress through deeper stages of diligence before making a decision to invest in a startup. Working with Vilcap, we <mark>designed a two-part intervention to reduce gender disparities in their investment outcomes by changing their evaluation templates that structured inquiry, specifically how Vilcap prompted individual investors to inquire about risk in stage one, and startup progress in stage two. We examined the effects on investors’ evaluation, analyzing over 3</mark> 1,000 scores investors allotted to startups during the period of study.

In the first stage of the field experiment, we used a cross-sectional design to assess the first stage of investment outcomes – an investor’s decision to conduct further diligence on a startup, or to exclude it from further consideration. We randomized a diverse set of 278 investors into a treatment group, which Vilcap prompted to systematically inquire about risk and reward, and a control group 4
where investors evaluated startups as normal. After analyzing 87 startups – resulting in 1,341 judgestartup investment decisions on continuing diligence – treated investors sought consistent information across startups while control investors asked more risk-focused questions to startups with a female founder. This reduced gender disparities in evaluation, compared to the control group.

In the second stage, we leveraged a unique facet of Vilcap’s investment thesis – it trains local investors to invest $320,000 into 16 startups over three months. We tested the effect of an additional treatment to systematically inquire about startups’ progress, added to the first treatment. We assessed this treatment’s effect on investment outcomes of Vilcap’s actual investment capital. Our analysis of a panel dataset of 1,530 decisions (from 510 investor-startup dyads over three time periods) suggests that treated investors assessed startups in a dynamic fashion rather than static – this changed how they evaluated startup competence and its future potential. These small changes in evaluation templates produced changes in investors' processes of inquiry, which spurred differences in evaluation that eliminated, even reversed, the gender gap in investment outcomes.

Rather than focus on how to prepare startups for evaluation, our research explores a more systemic question: what is the effect of organizations’ evaluation practices on investments? We contribute to a growing literature in entrepreneurship that considers how to level the playing field for female entrepreneurs. This research highlights a novel mechanism to reduce gender disparities in investment outcomes: organizations can change evaluation templates to structure investor inquiry. This can affect which startups continue in due diligence, or receive investment. We extend prior research on designing organizational evaluation practices to reduce gender disparities in outcomes, to contexts where performance data is limited, and inquiry is a core part of evaluation. More broadly, we theorize how processes of inquiry can sustain or reduce status-based inequities in society.

5
# **2. The Role of Organizations in Evaluation**

<mark>Organizations play an important role in designing evaluation practices that shape collective evaluation (e.g., Lamont 2012, Zuckerman 2012). They can design templates and processes that shape how decisions are made (e.g., March and Simon 1958). For example, organizations can specify how data that informs decision-making is shared or analyzed (e.g., Fayard and Metiu 2014, Anthony 2021), and how data is presented, can affect the decisions that are made (Kaplan 2011).</mark> <mark>Organizations can also design evaluation practices to affect outcomes by gender. One common organizational process, similar to investment decisions, is hiring and promotion decisions – where evaluators make decisions about people under conditions of uncertainty, over a fairly short time frame, with ramifications for organizational funds and reputational outcomes over the long term. Organizations have made many efforts to reduce gender disparities in promotion and hiring, but many have been ineffective or had negative effects (e.g., Kalev et al. 2006, Dobbin et al. 2015, Stephens et al. 2020). Some interventions such as affirmative action policies, can lead to unintended consequences if they unintentionally increase the saliency of stereotypes that target groups lack competence, which can decrease target groups’ performance and increase disparities between groups (Leibbrant et al. 2018, Leslie 2019).</mark> <mark>Focusing on evaluation processes and the content of evaluation has proven more successful in these contexts (Stephens et al. 2020), perhaps because these efforts tackle the</mark> organizational processes that could unknowingly reproduce inequality (Amis et al. 2020). <mark>Successful interventions include limiting</mark> employee discretion when making decisions (Castilla 2008), or shortening the evaluation scales employees can use to increase equity in evaluation (Rivera and Tilcsik 2019). Organizations could also focus on <mark>developing evaluation criteria that are not exclusionary, such as moving away from “cultural fit” (Rivera 2012), and instead rewarding performance on tasks (S</mark> tephens 6
et al. 2020 <mark>). However, these recommendations do not perfectly apply to the context of evaluating innovation and venture potential.</mark> <mark>Implementing strict decision-making rules in changing environments could theoretically limit an organization’s ability to learn and adapt (March 1991, Canales 2014). This adaptation may be necessary, given that startup strategies are subject to change (e.g., Siggelkow 2002, Kirtley and O’Mahony 2020), and startups often operate in rapidly changing environments (Eisenhardt and Tabrizi 1995). Structuring clear rules to evaluate performance may also be difficult, as startups have little history of organizational performance (Stinchcombe 1965, Aldrich and Fiol 1994, Cohen et al. 2019a) and the potential value of an idea is difficult to ascertain before dedicating some resources to testing it (Gans et al. 2019). Using data on team performance could reward founders’ elite connections, which would reinforce inequities in the status quo (e.g., Higgins and Gulati 2003, Hallen 2008). Given these difficulties in assessing static information, in investment, interacting with founders is seen as a fundamental part of evaluation (Petty and Gruber 2011). In</mark> vestors pride themselves on seeking information beyond the business plan, and using their “gut feel” to source and evaluate investment opportunities rather than relying on data (Kirsch et al. 2009, Huang 2018).

# **3. The Role of Inquiry in Evaluation**

<mark>Investors e</mark> valuate through inquiry – assessing the potential of an innovator and their idea during interactions. <mark>During these interactions with founders, investors ask questions to gather information on the venture (Kanze et al. 2018, Miller et al. 2023), but also to assess founders’ potential to scale their venture (Huang 2018). Interactions are part of evaluation in many hiring contexts too. Managers typically use job interviews to hire new workers (Macan 2009), but these can introduce disparities in outcomes for both racial minorities and women (e.g., Rivera 2012). To reduce disparities, scholars have theorized that organizations could add structure to interaction by asking evaluators to</mark> 7
<mark>use structured interviews – asking the same open-ended questions to all applicants (Huffcutt 2011), or task-based interviews – asking applicants to complete a task or set of tasks that are similar to those required in the job (e.g., Ployhart et al. 2006). However, not all task-based interviews reduce disparities, and there is little causal evidence on the efficacy of structured interviewing (Stephens et al. 2020). This may be because the content of inquiry is important. For example, asking applicants to management consulting roles to evaluate a case on an industry heavily dominated by men was biased against women because they had less background knowledge about the industry (Rivera 2015). Given that inquiry is a crucial part of evaluation in innovation contexts, we examine how organizations might apply structure to inquiry processes, by focusing on two disparities in how investors inquire.</mark>

**<mark>Inquiry about Risk and Reward.</mark>** <mark>During interactions, investors tend to spend more time considering risk for startups with female founders than those led by male founders. Docsend, a platform that allows founders to share pitch decks with investors, has found that investors spend more time assessing traction and product slides for startups with female founders (to assess their current assets). In contrast, investors spend more time on fundraising request slides (to assess what founders might do in the future) for all male teams. In short, investors scrutinize startups with female founders differently than those with all male teams (Frost 2020). Similarly, in pitch competitions, investors typically ask prevention-focused questions – focused on maintaining non-losses and not changing to a worse state – to females to e.g., “How many monthly active users do you have?” They tend to ask promotion-focused questions to males to understand rewards or growth e.g., “How do you plan to acquire customers?” Investors’ patterns of inquiry produce conversations that differ by founder gender, and could cause investors to evaluate ventures with female founders as less valuable (Kanze et al. 2017, 2018). We hypothesize that prompting investors to systematically inquire about risk and reward could result in more consistent investor inquiry across startups, prompting investors to pay attention to both risk and reward for all startups, and reducing gender disparities in evaluation.</mark>

8
**Inquiry about Progress.** Investment organizations provide evaluation templates to investors to facilitate their assessment of competence by assessing the growth potential of early-stage startups in the absence of a history of startup performance (Cohen et al. 2019b, Gompers et al. 2020). However, evaluating potential can disadvantage female candidates. For example, in a retail organization, evaluating “potential” for leadership did not result in promotions for equally-performing female candidates. If the organization had promoted based on current job performance, it would have reduced disparities (Benson et al. 2022). Whereas performance ratings are backward-looking and based on demonstrable achievements, potential ratings are based on an evaluator’s forecast of a worker's future performance and contribution. This makes rating “potential” fundamentally more subjective and uncertain, which could increase reliance on ascriptive characteristics such as gender (Ridgeway and Correll 2004, Correll and Benard 2006, Botelho and Abraham 2017, Snellman and Solal 2023).

How investment organizations might assess performance is complicated, given the lack of organizational performance data on startups (e.g., <mark>Stinchcombe 1965, Aldrich and Fiol 1994, Cohen et al. 2019a). However, some organizations assess short-term</mark> signals of performance during hiring processes. For example, the classic Goldin and Rouse study (2000) demonstrates that when orchestra hiring managers evaluated candidates’ performance through blind auditions, this resulted in hiring more female performers. This suggests that investors may be able to assess competence differently – inquiring about and assessing short-term signs of performance during the selection process.

In fact, some investors already do so. For example, program managers in Vilcap shared that progress made during the three-month program is important to investment decisions: “[We] invest in people that make the most progress during the program”. A VC investor, Mark Suster, (2010) blogged:

> “The first time I meet you, you are a single data point... Because I have no observation points from the past, I have no sense for where you will be in the future. Thus, it is very hard to make a commitment to fund you.”

9
This suggests that some individual investors value signs of progress when making their investment decisions. However, most investment organizations design evaluation templates to assess static elements of a startup including team and venture characteristics (e.g., Tybjee and Bruno 1984, <mark>Gompers et al. 2020)</mark> , which they use to assess future potential. This could disadvantage startups with female founders. We hypothesize that if organizations prompt investors to inquire about startups’ progress as well as potential, this would focus investors’ attention on demonstrable achievements – dynamic progress during the selection process. This could reduce gender disparities in investment decisions. Overall, we hypothesize that organizations can systematize both the consistency and content of inquiry to reduce gender disparities in investment outcomes.

_H1: Investment organizations that systematize inquiry by prompting investors to inquire about risk, reward and progress will reduce gender disparities in investments._

**How Organizations Affect Inquiry.** Our main hypothesis suggests that organizations can create systems-level change in evaluation by systematizing the consistency and content of inquiry. By changing prompts in evaluation templates, organizations can reduce gender disparities in investors’ decisions, which feed into collective investment outcomes. However, this hypothesis assumes a mechanism – that changing organizations’ evaluation templates will cause individual investors to change their processes of inquiry, which will in turn, affect their evaluation of startups. This assumption may not hold, as investors pride themselves on using their intuition or gut feel to evaluate investment opportunities and do not follow templates (Kirsch et al. 2009, Huang 2018).

We test each part of the mechanism in the next two hypotheses. We theorize that if investors were to inquire about promotion and prevention for all startups, this could reduce gender disparities (Kanze et al. 2017), due to increasing the consistency of inquiry and investor attention to prevention concerns for all startups. To test the mechanism driving a possible change in gender disparities in 10
investments, we test whether organizations can affect the consistency of inquiry – how investors inquire about risk and pay attention to it – in hypothesis 2: _H2: Prompting investors to inquire about risk and reward will increase the consistency of inquiry across startups._ We test whether organizations can affect how investors inquire about progress as well as potential in hypothesis 3. There is evidence that if organizations change the content of inquiry, it can backfire. For example, when organizations positioned their hiring and promotion practices as meritocratic, hiring managers were even more likely to favor a male employee over an equally qualified female employee in pay increase decisions (Castilla and Benard 2010). To overcome this type of effect, organizations can create more transparency in evaluation processes and their effects (Castilla 2015). One way to do so is to set criteria in advance to reduce opportunities for retroactive criteria construction – for example by requiring evaluators to weight evaluation criteria before assessing applications (Uhlmann and Cohen 2005). We hypothesize that organizations can change the content of investor inquiry if they change evaluation templates to include new criteria and create transparency around evaluation practices in those templates. We test whether organizations can affect the content of how investors inquire – to prompt them to pay attention to progress – in hypothesis 3: _H3: Prompting investors to inquire about progress will increase investors’ attention to startup improvement when evaluating competence across startups._

# **4. Research Approach and Setting**

Understanding how investment organizations evaluate startups requires field research to examine organization and individual-level evaluation practices and link them to the outcomes produced. Following Yang and Aldrich (2014), we conceptualize organization-level evaluation frameworks as an input to decision-making. We designed and ran a two-stage field experiment to test whether systematizing inquiry would affect individual investors’ behavior, and investment outcomes. By using a field experimental setting, we demonstrate how effective interventions are under “real11
world” conditions, overcoming concerns of generalizing from experiments in laboratory settings with students or online survey participants (Hsu et al. 2017, Czibor et al. 2019). This is particularly important in this setting, because investors that are trained to assess startups often evaluate startups differently to an average individual (e.g., Kirsch et al. 2009, Clingingsmith and Shane 2018).

These experiments were only possible due to access to a unique field site – Vilcap. <mark>Vilcap is a</mark> global investment organization with investor training programs in Africa, India, the Middle East, and Latin America. <mark>Vilcap</mark> is the: “ **largest organization in the world supporting impact-driven, seed-stage startups** . Since 2009 our team has directly worked with more than 1,100 entrepreneurs in 28 countries, and our affiliated fund, Vilcap Investments, has **invested in 110 startups** that have gone on to raise more than $4 billion in follow-on capital.” _– Vilcap website_ Vilcap is an appropriate field site for this intervention as it provides access to two types of investor evaluation. Vilcap uses professional investor evaluations to facilitate introductions between startups and investors, and it trains local investors to invest Vilcap funds. This provided researchers with access to a setting using explicit evaluation templates to ensure effective communication between professional investors, Vilcap and trainees, and which facilitated discussions about evaluation with these stakeholders. Vilcap was also open to both field research and experimental methods with real investment funds to resolve the issue it faced: startups with all-male teams formed 70 percent of its portfolio, and it aimed to increase the number of startups with female founders in its portfolio.

In addition, Vilcap provided access to a similarly qualified set of startups. Vilcap uses a competitive process to identify startups with high growth potential to enter their program – with between 200 and 400 applicants for ten places. All startups have a product, are aiming to improve their product-market fit, and are currently seeking investment. Each investment program is focused on one industry or problem statement, so all startups within each program are working in a common 12
industry (but are not direct competitors). All startups were deemed by Vilcap to be of high quality and at similar stages. Baseline studies suggest that all startups accepted into the Vilcap program have similar observable characteristics, which does not differ by founder gender (Burns et al. 2019). Vilcap also provided access to a curated set of investors that had expressed an interest in startups at an early-stage and in a specific industry, which limits the sample to investors that have the potential to be interested in this stage of startup. Any differences in results should not be driven by differences in startup quality, nor investor interest.

Vilcap’s evaluation process is typical of the average investment organization in several ways. The average investment organization employs a collective evaluation process to decide whether to invest their organization’s funds into startups (Tyebjee and Bruno 1984, Fried and Hisrich 1994), which typically takes about 90 days (Gompers et al. 2020). Investment organizations assess startups using organization-level criteria which typically include assessments of the founding team, market size, product and business model (Cohen et al. 2019b, Gompers et al. 2020). As shown in a simplistic model in Figure 1, when evaluating, investors typically receive information such as a pitch deck or overview, meet founders, and then make a decision on whether to continue to diligence (or inquire about) the startup. This process repeats, as investors progress through deeper stages of diligence before they invest in a startup.

In Vilcap, this process unfolds with two types of investors. Professional investors,<sup>1</sup> embedded in a range of investment organizations, meet startups once. They receive a venture overview, written by Vilcap, meet approximately three founders for thirty minutes, and evaluate startups using a Vilcap survey where they decide if they wish to conduct due diligence, and ask for additional information

> 1 Professional investors were invited by Vilcap and included other accelerator managers, investors from angel groups, and early-stage venture capital funders.

13
from the startup. If investors say they wish to continue due diligence on a startup, Vilcap facilitates an introduction. In the first stage of the field experiment, we tested the effect of prompting this diverse set of investors to systematically inquire about risk and reward, and assessed the effect on the additional information requested from startups, and the likelihood of continued diligence. This models the beginning of the selection process, and this type of cross-sectional design is common to research attempting to unpack demographic disparities in investment evaluation in the field (e.g., Younkin and Kuppuswamy 2018, Ewens and Townsend 2020).

Vilcap also trains local investors to evaluate startups for Vilcap and to allocate their own investment capital.<sup>2</sup> Since Vilcap is training investors from the region and market to evaluate startups on its behalf, it requires investors to provide scores on specific elements of the venture, including team, problem and vision, product, market and business model – typical criteria used by other investors. Trainee investors evaluate over multiple periods and are required to explain their reasons for scoring and to provide transparent feedback to startups. This provides a unique setting not only to observe how evaluation is conducted, but also to experiment with heterogenous organization-level evaluation frameworks over time. In Vilcap, trainee investors typically receive a venture overview before meeting startups. To assess whether trainee and professional investors made similar decisions in the experimental program, all trainee investors were also asked to fill out the survey after initially

> 2 Vilcap trains founders that qualify for its program to be investors and to allocate Vilcap funds. Their website explains the rationale for this decision: “What if, instead of relying on investors to “pick winners”, we chose to rely on entrepreneurs themselves? That hypothesis led to the creation of a collaborative due-diligence model ... to shift decision-making power away from investors... and instead, give that power to entrepreneurs to forecast which ventures are most promising.” Vilcap has run collaborative due diligence more than 70 times. We model entrepreneurs as “trainee investors” as they are trained to evaluate startups, conduct due diligence, and invest money on behalf of the organization. Vilcap’s investment decisions since 2009 are highly correlated with follow-on investment outcomes, suggesting that entrepreneur-investors make similar decisions to “real” investors. We assess the validity of this finding in the first experiment, where we leverage a pooled sample of trainee and professional investors.

14
meeting startups. As part of Vilcap’s normal training program, trainee investors then continue to evaluate startups three more times, using standardized criteria.

In the second stage, we leveraged the panel dataset of trainee investors’ investment decisions and tested the effect of an additional treatment, added to the first treatment. We prompted treated investors to inquire about startups’ progress during the selection period. Using this panel dataset allowed assessment of whether gender disparities appeared at specific stages of the selection process (e.g., Botelho and Abraham 2017, Bohren et al. 2019).

To conduct the two-stage field experiment, we worked with Vilcap in eight of its investment training programs (two each in four regions – Africa, India, Middle East, and Latin America – allowing for one treatment and one control group in each region). Trainee and professional investors evaluated startups in these eight Vilcap programs. This resulted in a dataset of 31,714 evaluation scores. Our first stage leveraged the cross-section sample of 1,341 dyadic investor-startup decisions by investors made after the investor met a startup founder. This sample included both professional investors as well as trainee investors who were trained by Vilcap to allocate $320,000 to 16 out of 87 startups. The second stage leveraged the panel nature of the trainee investor dataset. We randomized investors into treatment and control groups, with a panel dataset of 1,530 decisions (from 510 investor-startup dyads over three time periods after the initial analysis we observed in the first stage). Both stages of the field experiment were pre-registered.

# **<mark>5. First Stage: S</mark> ystematizing Inquiry about Risk and Reward**

**5.1. Setting and Design.** We systematized how Vilcap prompted investors to inquire about

risk and reward in their evaluation template and assessed the impact on reducing gender disparities in continuing due diligence (H1), and the consistency of investor inquiry across startups (H2). Trainee investors met startups in a 90-minute welcome meeting where each startup founder was encouraged 15
to share a little about themselves and their startup. Professional investors met startups in 20-to-30minute sessions where startups shared an overview of their business and then sought advice from the investor: either on their target market, product growth map, or fundraising strategy, depending on investor expertise. Vilcap shared a venture overview document with all investors with one page on each startup that outlined team members, market, product and the funds the startup was aiming to raise. We randomized professional investors into a treatment or control condition after they met a startup and began to evaluate them. We randomized trainee investors into a treatment or control condition after they were selected for the Vilcap program, stratifying by region, gender and subsector.<sup>3</sup> In both cases, after meeting startups, treated investors received a slightly different evaluation form to the control group, which we outline below (see Appendix A).

**5.2. Dependent Variable.** After meeting startups, Vilcap’s evaluation form asked all investors to evaluate startups on a scale of 1 to 6: _“I would initiate due diligence on this venture.”_ This variable is part of a dependent variable made up of four questions used in previous research by Clingingsmith and Shane (2018) – the part closest to a real investment decision.<sup>4</sup>

**5.3. Intervention.** After they met startups, Vilcap’s evaluation form asked control group investors: _“what additional information would you want on this venture?”_ For the treatment group, Vilcap’s form prompted treatment investors to systematize inquiry about risk and reward: _“what additional information would you want on this venture’s potential for growth?”; AND “what additional information would you want on how this venture will mitigate risks?”_

# **5.4. Empirical Design.** We ran the following pre-registered regression:

> 3 Given that all startups were operating in the same industry, at Vilcap’s request, we stratified by subsector to ensure no competing startups appeared in the same cohort.

> 4 We saw the most variance in this part of the variable in exploratory studies / pre-tests, which we ran on different investor populations before the experiment.

16
The unit of analysis is the investor decision per startup. The dependent variable is Yis – the propensity to invest in a startup s by investor i, measured using a six-point scale for professional investors. F is a binary variable which equals one when a female founder represented the startup and zero when solely male founders represented the startup. T is a binary variable which equals one when investors were prompted to inquire about risk and reward systematically (treated) and zero otherwise. The coefficient of interest is _β3_ , on the interaction of inquiry about risk/reward and female founder. We included fixed effects for the region R.. Although we randomized startups into treatment and control, given the relatively small number of startups we assessed (87), it is possible that startup characteristics could affect the size of the estimates. Therefore, we also controlled for observable startup characteristics Xs – the number of employees and the log of funds raised at selection into Vilcap’s program.<sup>5</sup> We ran an ordinary least squares regression, which was pre-registered, but added an ordered logit because the dependent variable was ordinal. We report the ordered logit, as results were the same in both models. We clustered errors in all models by investor – the level at which the treatment was implemented.

**Mechanism.** To assess the mechanism, we assessed if prompting investors to systematize inquiry about risk and reward increased the consistency of questions investors asked across founders, Y is equal to one if a prevention question was asked to a startup, and zero otherwise. Two research assistants (one for Spanish, one for English) coded all investor questions posed to founders with prevention/risk or promotion/reward, following Kanze and coauthors (2018).<sup>6</sup> Any disagreements were discussed in a group with the first author so that codes were applied consistently. The final

> 5 Vilcap also collected funds raised by startups, but we did not include this as a control, as many startups had not yet raised funds, and there were many zeros. We also opted not to use Vilcap’s own evaluation score as a control, as it was highly correlated with employees.

> 6 We constructed the prevention variable following Kanze and coauthors (2018) but included some additional variables to better suit the setting – a series of dyadic entrepreneur-investor interactions, rather than a one-off pitch where investors asked questions to entrepreneurs in a group setting.

17
decision on whether a question was coded by promotion or prevention was made by the first author, who did not have access to the startup’s founder gender when making these decisions. We constructed a binary measure at the investor-dyad level to measure the incidence of a prevention focus, and used the same type of binary incidence measure for a promotion focus. Simply put, if an investor question to a startup did not have a promotion nor prevention focus, both variables would equal zero. If an investor question to a startup included both a promotion focus and a prevention focus, both variables would equal one.<sup>7</sup>

**5.5. Results.** We analyzed 1,341 decisions taken by 278 investors – combining 198 professional investors and 80 trainee investors – on 87 startups. As shown in Table 2, we assessed differences across all investor characteristics across treatment and control groups. As expected, given our randomized treatment assignment, here were no significant observable differences – using raw numbers, percentages or the p-value taken when regressing each characteristic on treatment.<sup>8</sup> As a result, any differences in evaluation practices between treatment and control groups are likely to be caused by our randomized treatments – not by the types of investors in treatment and control groups.

Investors in the control group scored startups with female founders significantly lower than those with all male teams. Startups with female founders received an average score of 3.7 out of 6, while startups with all male teams received 4.1. When including startup controls, investors in the control group gave startups with female founders significantly lower scores than startups with all male

> 7 Following Kanze et al. (2018), we also used a computerized method on English responses, utilizing a dictionary of 27 promotion and 25 prevention words developed and validated by Gamache et al. (2015), and uploaded these dictionaries into Linguistic Inquiry and Word Count (LIWC) software to determine their frequencies. Similarly to Kanze et al., this was not our preferred method, as the dictionary approach leaves LIWC vulnerable to a low detection rate; the software is not sensitive enough to capture intentions that do not directly overlap with the very specific 52 words in the regulatory focus dictionary (Gamache et al. 2015). We use this LIWC method as a robustness check for all responses in English, to verify the direction of results from the research assistants’ qualitative coding. We find the same directional results.

> 8 A small minority of professional investors met startups in multiple programs i.e., in the Middle East and Africa. As we randomized the investor according to the survey they received, 18 of the 276 investors encountered the treatment condition in one program, and the control condition in another program.

18
teams (0.6 on a 6 point scale – equivalent to 10 percentage points), as shown in Appendix B. This disparity held across trainee investors and professional investors, as well as male and female investors. This difference in scores was correlated with the fact that investors were more likely to focus on prevention when inquiring from startups with female founders than startups with all-male teams. As shown in Appendix B, investors in the control group were 15% more likely to ask a preventionfocused question to a startup with a female founder than a startup with an all-male team. This difference was directional, but not significant. Together, this suggests that when investors evaluate startups, they score startups with female founders lower than startups with all-male teams, and ask systematically different questions by founder gender. These differences in inquiry are directionally similar to those observed in US-based pitch contexts (e.g., Kanze et al. 2018).

We tested whether changing an evaluation framework to systematize inquiry around risk and reward would affect gender disparities in outcomes (H1) and whether it could change investor behavior (H2). Figure 2 shows the effect of systematizing inquiry by prompting investors to ask about risk and reward on an investor’s decision to continue diligence on the startup, by founder gender. While control investors scored female founders significantly lower than startups with all male teams, treated investors did not. This provides some evidence in support of H1, suggesting that if investment organizations systematize inquiry by prompting investors to ask about risk and reward, they can reduce gender disparities in evaluation outcomes.

Figure 2 also shows that investors in the control group were directionally more likely to ask prevention-focused questions to startups with female founders than those with all-male teams and that treated investors were more likely to inquire consistently across startups. This provides some suggestive evidence in support of H2; if investment organizations systematize inquiry by prompting investors to ask about risk and reward, investors will inquire more consistently across startups. This 19
difference between treated and control investors is driven by treated investors being significantly more likely to ask prevention-focused questions to _all_ startups – directionally, even more so for those with all-male teams. This provides some evidence in support of H1 and H2, suggesting that if investment organizations systematize inquiry by prompting investors to ask startups about risk and reward, investors will inquire more consistently across all startups, and reduce gender disparities in evaluation outcomes.

As shown in Table 3, regression analysis including startup controls showed similar results. Control investors were less likely to take a startup with female founders through due diligence compared to a startup with an all-male team, while treated investors were equally likely to take a startup through due diligence, regardless of founder gender. Control investors were only 65 percent as likely to increase their score by one unit (i.e., from agree to strongly agree to take the startup through due diligence) for startups with female founders compared to all male teams. However, treated investors were equally likely to do so (0.65 main effect multiplied by the 1.63 interaction).<sup>9</sup> This provides more evidence in support of H1. Regression analysis suggests that systematizing inquiry by prompting investors to ask about risk and reward increased the likelihood that investors would ask a preventionfocused question to all startups (by 265%), but that the likelihood increased less for startups with female founders.

These results provide evidence that simply prompting an investor to ask founders about “potential for growth” and “how this venture will mitigate risks” meaningfully affected the types of questions that investors posed to all startups, but particularly those with male founders. For example, the male founder of a platform startup that used mobile technology to connect handymen with work

> 9 Common to many experiments, we focused our design for this experiment on isolating the effects of gender on investment decisions (score), and the effect of our treatments. As an investment decision, we expect other unobservable preferences – including the weather – to add noise (e.g., Dushnitsky & Sarkar 2022).

20
opportunities received different questions from control and treated investors. A control investor asked: Can I see a “marketing plan clearly highlighting the marketing strategies?”. In contrast, a treated investor asked the same founder: How will the company “manage delayed payments (Risk) in case the company decides to partner with county or National Government”? Both questions were about scaling, but treated investors were more likely to use a prevention-focus frame – similar to the frames all investors used when assessing startups with a female founder. Treated investors were equally likely to agree or strongly agree to the statement “I will conduct due diligence” whether the startup had a female founder or all-male team. A simple change to an evaluation template affected investors’ use of prevention-framing, and whether they wanted to take the startup through due diligence. Together, this suggests that if organizations systematize inquiry by prompting investors to inquire about risk and reward in evaluation templates, this causes investors to inquire consistently across all founders, paying attention to risk and reward regardless of founder gender, which reduces gender disparities in investment evaluation practices.

**5.6. Alternative Explanations.** Differences in outcomes were robust to alternative measures

of the dependent variable (score – using OLS or a weighted score by investor), the independent variable (analysis of prevention-focused questions by LIWC) and the female binary variable (presence in venture overview – see Appendix C). The results held for heterogenous investor types (male and female, and trainee and professional). We do not have a large enough sample to conduct heterogeneity analysis by other investor characteristics, but we observed no directional difference on the relationship of treatments with the score.<sup>10</sup> This suggests that results could hold for a diverse range of investors.

**<mark>5.7. Investment Outcomes.</mark>** <mark>This experiment revealed that organizations can</mark> change evaluation frameworks to prompt investors to inquire about risk and reward systematically, and that

> 10 Heterogeneity analyses are available on request.

21
this affected the questions investors asked, and reduced gender disparities in evaluation. This effect was also meaningful. Investors that agreed or strongly agreed to the statement “I will conduct due diligence” were likely to actually do so in this context. In this sample, 31 control investors selected “strongly agree” and 73 “agree” for startups with female founders. In the treatment group, 37 investors selected “strongly agree” and 82 selected “agree”. This suggests that startups with female founders would have entered into 15 more due diligence processes in the treatment group than the control, which could have meaningful implications for future investment. If these results hold more broadly, this type of intervention could reduce disparities in investment outcomes by founder gender.

If investment organizations prompt investors to think about prevention and promotion, investors inquire more consistently from startups with female founders and those with all male teams. This results not only in more rigorous due diligence on startups with all-male teams, but also an assessment that startups with female founders and all male team pose similar risks. This assessment produces consistent evaluation across founder gender, and changes the number of startups with female founders that enter due diligence processes.

<mark>One important limitation to this experiment is that we did not observe final investment decisions, so findings can only generalize to early stages of the investment selection process – the decision to begin a due diligence process on a startup. For this reason, we conducted Stage 2, to evaluate the effects of systematizing inquiry on real investment decisions.</mark>

# **<mark>6. Second Stage: The Effect of</mark> Systematizing Inquiry on Risk, Reward and Progress**

**6.1. Setting and Design.** Vilcap trains local entrepreneurs to evaluate startups, conduct due diligence, and invest $320,000 of Vilcap’s money into 16 early-stage startups over three months. The first author observed the entire process in the same Vilcap programs as Stage 1, to further assess whether cross-section results applied over time (e.g., Bohren et al. 2019). We changed Vilcap’s

22
evaluation framework by systematizing how Vilcap prompted investors to inquire from startups during interactions. As a bundled treatment, we systematized both how Vilcap prompted investors to inquire about risk and reward, AND about progress. We assessed the impact on reducing gender disparities in investment decisions (H1), the consistency of investor inquiry across startups (H2), and whether investors assessed startup improvement (H3).

**6.2. Dependent Variable.** After baseline, three more times over the course of its 90-day program, Vilcap asked investors to complete due diligence again, and rank startups. Vilcap’s normal set of evaluation questions are focused on assessing potential: _“what is the company’s growth opportunity? And what is the company’s investment opportunity?”_ across eight categories (i.e., team, value proposition, market, scale).<sup>11</sup> Investors use a four-point scale per category, resulting in a 24-point scale overall – from 8 to 32. The final evaluation scores result in two investments into the most highly-scored startups.

**6.3. Intervention.** After Vilcap recruited entrepreneurs, we randomized startups/investors into treatment and control groups, stratifying by region and gender. Treatment investors received a bundled intervention using Vilcap’s changed evaluation template: 1) The same treatment that prompted investors to systematically inquire about risk/reward in Stage 1; and 2) Vilcap prompted investors to inquire about startup progress as well as potential during the program. For the treatment group, Vilcap added four questions (each on a 4-point scale, weighted to equal 1/3 of the overall evaluation score) to assess startups’ progress: _“Since the beginning of the program, how much has this company improved in...”: i) “understanding its path to growth?”; ii) “executing its path to growth?”; iii) “understanding its risks?” iv) “executing on risk mitigation?”_

> 11 Vilcap is unwilling to share its proprietary evaluation templates openly, but will share it with academic reviewers to demonstrate the similarities between its due diligence questions, and the additional questions created as part of the intervention.

23
**6.4. Empirical Model.** We ran the following ANCOVA regression, to increase statistical power, following McKenzie (2012). By including the baseline score from Stage 1 as a control variable, we assessed the change in scores after the additional bundled treatment was applied: i 1 i 0 2 3 4F i ii ii ii ii ii ii ii YY = ββ yy + ββ FF + ββ TT + ββ TT + XX + RR + tt + εε

Our unit of analysis is the investor decision per round. The dependent variable is Yist – the propensity to invest in a startup s by an investor i at a time period t. t is the stage of measurement – a scale variable collected over three time periods. Yis0 is the baseline measure of evaluation of a startups by an investor, also evaluated on a scale. All scale evaluations are normalized by Vilcap, using a zscore.<sup>12</sup> F is a binary variable which equals one when a female founder represented the startup and zero when solely male founders represented the startup, which varies over time. T is a binary variable which equals one when investors were in the changed evaluation framework treatment group and zero when investors were not. The coefficient of interest is _β4_ , on the interaction of treatment and female. We included fixed effects for the region R and time t and controlled for the same observable startup characteristics Xs as in Stage 1. We clustered errors in all models by investor – the level at which the treatment was implemented.

**Mechanism.** We added mechanism variables to assess if systematizing inquiry at the organization level changed how investors assessed startups. First, we examined whether investors inquired consistently across all founders using the same mechanism as Stage 1. Second, we examined whether investors assessed startups’ dynamic improvement by asking all investors to weight the criteria they used when evaluating: _“Please think about how you made your decisions and weight the criteria below with_

> 12 In Vilcap, each startup receives a zscore per round. The inputs are the average score and the standard deviation per judge per round. Then for each judge's score for each startup Vilcap creates a z_score = (score - avg_score)/sd_score. Vilcap then joins the ranks together by taking an average across all rankers. Vilcap’s zscore weights scores according to a judge’s baseline score. This type of weighted score can help to avoid heterogeneity in judges’ baseline scores driving results (Gonzalez-Uribe & Reyes 2021).

24
_percentages of how much weight you placed on each criterion. (Please make sure it adds up to 100%!) – [Growth opportunity, Investment opportunity, Improvement made during program]._ Given that investors do not accurately explain the criteria that are important to them (Petty and Gruber 2011), we followed recent field experiment research and conducted semi-structured interviews to provide more insight into mechanisms driving results (e.g., Dimitriadis and Koning 2022). The first author interviewed all 15 trainee investors in one region to further examine how they decided to give a startup a high or low score – what investors were paying attention to during evaluation.

To assess the effect of systematizing inquiry on individual investors’ evaluation, we compared how investors in the treatment and control group evaluated startups using a set of criteria, by founder gender, across three phases of evaluation. We used the same regression model, but changed the dependent variable to each criterion score i.e., score for “business model”. We then reran the regression to assess how investors re-evaluated startups from their first impressions at baseline.

**6.5. Results:** The balanced panel dataset after attrition was 1,530 investment decisions made by 65 investors on 69 startups over three time periods (510 decisions per period). Attrition was largely due to Covid-related absences. As shown in Table 4, there was no significant difference in observable characteristics between treatment and control groups – using raw numbers, percentages, or the p- value taken when regressing each characteristic on treatment.

To assess the baseline, we used trainee investors in the control group. Similarly to Stage 1, investors asked startups represented by females more prevention-focused questions and awarded lower scores than startups represented by all males. However, during the program, we observed this effect lessening, observing only directionally different effects by gender (see Appendix D). This suggests that the Vilcap program itself reduced gender disparities in the control group. The “Vilcap effect” could be driven either by the fact that all startups continue to be evaluated over time, or Vilcap 25
programming in which investors were provided with a standardized scoring system and questions when assessing each other. Overall, this suggests that VilCap is a conservative setting to test whether systematizing inquiry can reduce gender disparities in evaluation.

Investors in the control group evaluated startups represented by all male founders lower over the course of the program, from a baseline zscore of 0.12, which was reduced to 0.07. As demonstrated in Figure 3, investors very slightly increased their evaluation of startups represented by females on average from -0.09 to -0.07 on the zscore. This suggests that the Vilcap program itself acted to reduce gender disparities, largely by reducing zscores for startups with all male teams. Even so, startups with female founders received directionally lower zscores than startups with all male teams at the end of the program. By contrast, in the treatment group, the baseline score was already closer for startups represented by male founders (0.04) and female founders (0.01) because investors had been treated at baseline (prompted to ask about risk and reward). Even given this baseline difference, there were additional effects of the systematizing inquiry bundled treatment. Treated investors evaluated startups represented by all males lower over the course of the program (from 0.04 to -0.07), and evaluated startups represented by female founders higher over the course of the program (from 0.01 to 0.11). This effect was driven by changes in how startups with female founders were evaluated. Effects in the treatment group were larger than those observed in the control group, and were driven by increases in scores to startups with female founders.

As shown in Table 5, regression analysis showed similar results. Treated investors using Vilcap’s systematizing inquiry evaluation template scored startups represented by females lower than startups represented by males at the beginning of the program but not significantly so. Investors treated systematizing inquiry scored startups with all male teams lower than control group investors, 26
but the effect on females was positive, increasing their scores by 0.31 in the pre-registered model. This effect was significant at α=0.05. providing support for H1.

Organizational prompts to investors to inquire about “potential for growth”, “how this venture will mitigate risks”, and “startup progress” during the selection process, affected how investors evaluated and scored startups with female founders. Systematizing inquiry in these ways caused treated investors to evaluate startups with female founders as higher than those with all-male teams. The bundled treatment not only reduced gender disparities in scores, but reversed them. This suggests that organizations can change evaluation templates to reduce gender disparities in evaluation outcomes, even with real investments made over time. It also suggests that systematizing inquiry by prompting investors to inquire about progress may create a different effect than prompting investors to inquire about risk and reward.

**6.6. Investment Outcomes.** We next examined the effects of reversing gender disparities in evaluation scores, on gender disparities in investment outcomes. In the eight Vilcap programs in the sample, only 16 investments were made (to the two startups in each program with the highest scores). We cannot assess the effects of changing evaluation outcomes on this rare outcome. However, we conducted a simple calculation to assess if increasing a zscore by 0.31 – the difference in score for a startup with a female founder in the treatment group versus the control group – could affect the likelihood of investment for a startup. The average zscore for a startup that was ranked second and received investment was 0.76, compared to the average zscore of 0.43 for a startup that was ranked third and did not (Appendix E). The average difference was 0.33, close to the effect of the treatment for female startups. This suggests that the size of the increase in score could meaningfully change investment outcomes and reduce the gender disparities in Vilcap’s overall portfolio.

27
**6.7. Mechanism Analysis:** We conducted exploratory analysis to examine the mechanisms behind the effect of the bundled treatment that systematized inquiry on risk, reward and progress. We examined whether the differential effect in score by treatment and gender was driven by two mechanisms: the consistency of inquiry – by likelihood of asking a prevention-focused question to a startup (H2); or the investors’ assessment of dynamic improvement when evaluating startups (H3).

We reran the analysis from Stage 1 on the panel dataset and found that during the VilCap program (when VilCap provided due diligence questions for trainee investors to use), female founders were equally likely to receive prevention-focused questions as males, so the mechanism could not work as predicted (see Appendix F). Prompting investors to inquire about risk and reward had similar results to Stage 1, where males were asked more prevention-focused questions, and this effect was lower for females, but this did not affect the score. These results did not provide evidence in support for H2 in this setting, where investors and founders interacted repeatedly over a long period of time. Combining these results with results from Stage 1, we suggest two explanations. First, prompting systematic inquiry around risk and reward may be most valuable to retain startups presented by female founders at early stages in the evaluation pipeline, to keep them in consideration for the further selection process. Second, any prompt to systematically inquire (either from Vilcap during its program, or from the intervention we designed) may have a substitution effect. Both prompts increased scrutiny for startups led by all founders, but particularly for startups with all-male teams.

We next assessed whether investors’ assessment of dynamic improvement differed by treatment and control groups (H3). As shown in Appendix G, treated investors weighted the criterion of “improvement” as a higher part of their evaluation criteria (20.8%) than control investors (18.5%). This difference was close but not statistically significant at p = 0.108, on a sample of 65 investors, providing some directional evidence in support of H3. However, it is difficult to interpret the import 28
of this small percentage difference on how investors actually assessed startups. To provide more insight, the first author conducted 45-minute semi-structured interviews with investors in one region, to examine how investors conducted evaluation, and how they decided to give startups a high or low score. After interviewing all investors in the treatment and control program in one region, we found that many investors evaluated static startup elements of the venture to assess its potential. For example, a male investor explained: “I gave AA a four [top score] in most of the categories... I really like their solution... it has a lot of potential for scaling... I went through their website... I was quite impressed with the profiles of people that work in the team... their business model too.” A female investor explained that she rated a startup well when the problem was convincing: “I rated BB high... because I think the business idea is really necessary... I see its use and purpose.” These investors used similar criteria to that elaborated by Gompers and coauthors in their survey of earlystage investors (2020). In the control group, all nine investors evaluated static criteria (100%). As shown in Table 6, in the treatment group, three of the seven treated investors (43%) also described how they gathered data on static criteria in at least one of their responses.

Some investors took a more dynamic lens, focusing on progress or improvement when scoring ventures. A male investor described how he had ranked a startup well because he had observed improvement over time: “The question is, have you seen improvement in them during this business program? ... I’ve been in a breakout room with YY twice. And the suggestion I had noted to her in the first breakout room ... was repeated with a group of mentors... they asked a similar question.... I noted that when she’s answering the question, she’s answering it differently... it sounded much better than the first time... once you see those things from people you can see that they are improving and changing.” 29
In the treatment group, six of the seven investors (83%) explained that they valued startups that made progress over time. Only two of nine control group investors (22%) explained that they assessed progress. These results provide evidence that when Vilcap asked investors to evaluate progress as well as potential, investors assessed startups dynamically. This shifted the focus of evaluation from startup attributes and a forward-looking assessment of potential, to a backward-looking assessment of what startups had actually accomplished over a short period of time. Taken together with the finding on changes in the score, this suggests that when investors assessed startups dynamically, it positively affected their evaluation of female founders’ startups, which reduced gender disparities in investment outcomes. This provides evidence for H3.

To further examine how investors changed their scores of startups, we analyzed the effects of systematizing inquiry on assessments of startup characteristics. As shown in Table 7, systematizing inquiry affected how investors assessed startups that were represented by female founders. Over time, treated investors assessed startups with female founders significantly higher on “product” and “investor exit” than control investors. Treated investors also assessed startups with female founders directionally higher on “business model” and “scale”, compared to control investors. There was no significant difference in scores given to startups with male founders.

By systematizing inquiry (prompting investors to systematically inquire about risk, reward, and startup progress), investors changed how they evaluated startups with female founders. This was driven by changes in how they evaluated venture attributes like product, or potential for exit.

Organizational prompts to inquire about growth, risk and startup progress during the selection process affected how investors evaluated startups with female founders. In particular, asking investors to assess “since the beginning of the program, how much has this company improved?” in an evaluation template, significantly affected how investors scored startups on a set of criteria. Treated 30
investors were more likely to assess startups dynamically, paying attention to improvement during a selection process. In doing so, they evaluated startups with female founders more positively than those with all-male teams.

**6.8. Alternative Explanations.** We ran similar robustness checks to stage 1 on the panel data in stage 2. Differences in outcomes were robust to alternative measures of the dependent variable, the independent variable, and the female binary variable (as shown in Appendix H). It is possible that progress would be more salient to judges, or easier for startups, when startups entered the evaluation process at a less mature stage, but we found no statistically significant difference between startups with female founders and startups with male founders when they entered the evaluation process (as shown in Appendix I). This result is unsurprising because all startups passed through Vilcap’s common selection process prior to evaluation. It also suggests that this study’s results were not driven by differences in the stage at which startups entered the program.

# **7. Limitations**

This research is an early contribution on the effects of organization-level evaluation systems on investment decisions, and has several limitations. Similar to other research in this area, including research on crowdfunding or angel platforms (e.g., Younkin and Kuppuswamy 2018, Ewens and Townsend 2020, Bapna and Ganco 2021, Gornall and Strebulaev 2020), the results in Stage 1 are limited to the first stage of investors’ selection processes. To mitigate this limitation, we tested the effects of treatments in Stage 2 and leveraged a setting where real investments were made, over time. Vilcap provided a unique opportunity to design and test multiple evaluation templates, and observe their effects on evaluation practices in a transparent process. However, similarly to most field studies, we do not trace the long-term effects of treatments. Following investors over time will subject results 31
to noise stemming from a fading effect of the intervention (e.g., Ridgeway and Correll 2003), an investor’s follow-on experiences, or changes in the environment, so cannot fully mitigate this issue.

Also, since Vilcap observed persistent gender disparities over time, we bundled the treatment to structure inquiry, to create a strong intervention that would have a stronger chance of having a meaningful effect on a small sample in a noisy field setting. We find some evidence for both hypothesized interventions but are unable to distinguish their relative importance. We were also unable to determine a statistical effect of prompting investors to ask about risk and reward on consistency of investor inquiry. While not being able to separate out the exact mechanisms driving effects is a limitation, it is balanced by the benefit of identifying a cost-effective organization-level treatment, which affected investment outcomes and that was field-tested with real investors and entrepreneurs.

Like any field experiment, these experiments are limited by their context. In this paper, we cannot separate out the effects of the treatment by investor type (barring investor gender, where it did not change the main results). We tested the treatment in one organization’s templates. This organization had created an evaluation template, used it, mined its own data, identified a disparity in evaluation by gender, and was willing to change its evaluation templates to attempt to redress gender disparities. This suggests that these findings will only hold for organizations that identify gender disparities and which are willing to redress them by changing templates. Promisingly, the effects of the organization’s efforts held with investors from multiple organizations and geographic contexts, which suggests some generalizability. However, more research is needed in different contexts and with larger samples to assess the conditions under which this treatment, or other interventions to structure inquiry, can reduce gender disparities in investments.

Finally, because we tested this treatment with one dependent variable in mind – gender – we cannot assess how these or other evaluation practices might affect other disparities in investment outcomes. This treatment may have the potential to affect disparities driven by founder race (e.g., 32
Younkin and Kuppuswamy 2018), or other characteristics of a founder or venture. Future research is needed to assess the effect of systematizing inquiry in these contexts. Similarly, we designed this research to isolate the effects of gender and treatments on investment decisions. As such we do not theorize other inputs to investment decision-making such as organizational preferences (Tyebjee and Bruno 1984, Cohen et al. 2019), individual investor preferences (e.g., Huang 2018), or extraneous conditions such as weather (e.g., Dushnitsky and Sarkar 2022).

# **8. Discussion**

_What is the effect of investment organizations’ evaluation practices on gender disparities in funding innovation?_ We hypothesized that investment organizations could change their evaluation systems to invest into female-founded startups and designed and tested interventions in a two-stage field experiment. In the first stage, we tested whether prompting investors to systematically inquire about risk and reward would reduce gender disparities in investment decisions early in an investor-founder relationship. An analysis of 1,341 decisions taken by 278 investors suggested that prompting investors to inquire about both prevention (risk) and promotion (growth) in an evaluation template resulted in more consistent inquiry – investors asked more prevention-focused questions to all startups, but particularly those with all-male founding teams. This reduced gender disparities in evaluation in the treated group compared to a control.

In the second stage, we tested whether systematizing inquiry could affect collective investment decisions that allocated $320,000 to 16 of 69 startups over three months. Leveraging a panel dataset of 1,530 decisions, we tested the effects of a bundled treatment of prompting investors to systematically inquire about: 1) risk and reward; and 2) startup progress. Treated investors more positively assessed startups with female founders than the control. This resulted in meaningfully higher scores that affected the likelihood of investment for startups with female founders. Differential effects between treatment and control appeared to be driven by how the investors assessed startups’ venture 33
characteristics: control investors assessed static characteristics to assess competence, while treated investors were more likely to assess ventures dynamically. This focus on what startups had accomplished in a short period of time, rather than in assessing their potential, benefitted startups with female founders. Treated investors assessed that these startups had shown competence, and demonstrated future growth potential.

Together, these results suggest that organizations can change their evaluation systems to reduce gender disparities in investment outcomes. Using a global sample with real investment decisions, our paper makes three main contributions to our understanding of the effects of evaluation systems on investment outcomes: 1) changing the system – through organizational evaluation practices – can affect funding for female founders; 2) the promise of systematizing inquiry; 3) how status biases around competence can be counteracted.

# **8.1. Changing the System Can Affect Funding for Female Founders.** Research on

investment processes in entrepreneurship has explicated the importance of gatekeepers. Because investors source deals through their homophilous networks (e.g., Saxenian 1990, 1996; Sorenson and Stuart 2001, Hallen Davis and Murray 2020), the lack of female investors results in lower evaluation for startups with female founders (Gompers and Wang 2017, Greenberg and Mollick 2017, Howell and Nanda 2019, Ewens and Townsend 2020). However, research on the effects of having female investors is mixed (e.g., Bapna and Ganco 2021, Snellman and Solal 2023, Gornall and Strebulaev 2020). Other scholars have theorized the importance of founder narratives in accessing resources (e.g., Lounsbury and Glynn 2001, Martens Jennings and Jennings 2007, Hallen and Eisenhardt 2012), or combating investors’ biases through founder pitches (e.g., Kanze et al. 2018, Lee and Huang 2018, Balachandra et al. 2019, Huang et al. 2021). However, pitching tactics may not have the same effect with every investor (Panhke et al. 2015, Clough et al. 2019). Rather than focusing on networks or 34
interventions that put the onus on the founder, we instead theorized how investment organizations could engender changes in investment outcomes.

We considered evaluation not as a dyadic process between an investor and a founder, but as part of a collective process, designed by investment organizations, in which investors are nested. To structure evaluation processes, organizations design and use tools, templates, and evaluation frameworks (March and Simon 1958, Fayard and Metiu 2014, Anthony 2021). Scholars have found that the templates organizations use affect decisions that are made (Kaplan 2011), and the types of knowledge that organizations create (Anthony 2021). Similarly, in our research, we explain the importance of the structured evaluation templates that investors use. By designing and testing organization-level interventions to change evaluation templates, we reveal a novel organization-level mechanism that affects gender disparities in investments. Prompting investors to systematically inquire about risk and reward resulted in treated investors asking all startups more prevention-focused questions, particularly those will all male teams. Treated investors gave startups with male founders lower scores. This reduced gender disparities in both the types of questions asked by investors and the scores given – to startups with all male teams and those with female founders. Prompting investors to systematically inquire about progress, meaningfully affected how they evaluated startups with female founders. This was driven not by an improved assessment of their team, but by a dynamic consideration of their venture. Startups with female founders were considered to have demonstrated competence, and shown future potential.

By identifying how investment organizations’ evaluation practices can reduce gender disparities in investment decisions, we raise a broader question of how investment organizations’ evaluation tools, templates, evaluation frameworks and processes, often designed early in an organization’s development, might affect investors’ later assessments of startups. We suggest that 35
tracing a link between organizational systems of evaluation, and examining the effects on investor decisions, may be fruitful for future research that aims to explain how investment decisions are made.

**8.2. The Promise of Systematizing Inquiry.** When theorizing how to mitigate gender biases <mark>about people under conditions of uncertainty in a short time frame, s</mark> cholars have examined the effects of blinding applications (e.g., Goldin and Rouse 2000), limiting employee discretion (Castilla 2008), shortening evaluation scales (Rivera and Tilcsik 2019), structuring interviews so that evaluators ask the same questions to all candidates (Huffcutt 2011), or rewarding proven past performance on a task, rather than allowing evaluators to assess potential (Stephens et al. 2020, Benson et al. 2021). Each of these efforts focus on creating stricter regulations and processes around evaluation, and limiting individual evaluator discretion. We hypothesized how this might be applied in innovation contexts, where investors consider not only a person, but also the potential of their startup. Past data on the startup is insufficient to produce decision-making (e.g., Stinchcombe <mark>1965, Aldrich and Fiol 1994, Cohen et al. 2019a), and investors value the ability to use their discretion: using p</mark> rocesses of inquiry <mark>during interactions with applicants to assess potential (K</mark> irsch et al. 2009, <mark>Petty and Gruber 2011,</mark> Huang 2018, Miller et al. 2023). These processes of inquiry in investment can engender disparities in evaluation in innovation contexts (Kanze et al. 2018), and more broadly (Rivera 2012a, b, Rivera 2015, Stephens et al. 2020), yet their causal effects on economic outcomes are understudied.

In this context, we theorized that interventions that allowed investors to inquire freely – assessing the potential of an innovator and their idea during interactions – would be important in this context. We simultaneously hypothesized that systematizing how organizations prompt investors to inquire freely (on risk, reward and performance), can result in fewer gender disparities in individual evaluation and collective investment decisions. We theorized that organizations can create evaluation templates to reduce disparities in investment outcomes, by reducing gender disparities in investors’ 36
processes of inquiry. We intervened in two aspects of the inquiry process where we identified where investors typically evaluated startups differently by founder gender: (1) how risk and reward were assessed; and (2) assessing progress over time. We explain how structuring processes of inquiry in this manner caused treated investors to (1) inquire consistently across all founders and (2) assess startup competence dynamically. This eliminated, even reversed, the gender gap in investment decisions. These effects held across investors’ characteristics, including their organization and geography. In short, these interventions significantly changed how investors inquired, and their eventual decisions. Our research demonstrates the importance of processes of inquiry for investment outcomes by founder gender. Data collected through inquiry, particularly data collected over the selection process, using dynamic assessment, has the potential to increase the information used by investors in making collective decisions – which could outweigh the benefits of reducing individual evaluator discretion.

Extrapolating from this finding, we theorize a broader implication for many investment organizations – where evaluators assess multiple candidates, allocate funds and use processes of inquiry to gather data on potential, which is not readily available in static form. For example, such as when university hiring committees are attempting to assess the potential of an early-stage candidate’s research pipeline. Structuring processes of inquiry by prompting evaluators to inquire consistently about the same content to all candidates and to record it in templates, has the potential to reduce disparities in inquiry and outcomes across these contexts.

# **8.3. How Status Biases around Competence Can Be Counteracted.** Most scholars have

theorized that disparities are caused by disadvantage – mechanisms such as prejudice, stereotypes and structural barriers (Phillips et al. 2022). When evaluators use ascriptive characteristics like gender in evaluation, they undervalue females’ competence <mark>(England et al. 1994, Ridgeway and Correll 2004, Correll and Benard 2006, B</mark> otelho and Abraham <mark>2017, Snellman & Solal 2023</mark> ). Similarly to Botelho 37
and Abraham <mark>(2017), we focused</mark> on the processes of evaluation, and <mark>theorized disparities in the evaluation practices used by investors. This did not require an assumption that female founders’ competence was necessarily undervalued by investors – and we examined this question.</mark> <mark>In the first stage of the experiment, we found that disparities in investor inquiry around risk and reward were mitigated by treating male founders more like female founders, which resulted in less disparate investment outcomes by gender. This suggests that at least some of the discrepancy in investment outcomes is not d</mark> riven by undervaluing startups with female founders, but by underquestioning, and overvaluing all-male teams. It appears that startups with all-male teams may receive advantages that female founders do not – the benefit of the doubt – in early-stage evaluation processes. In the second stage of the experiment, we found that disparities in investment outcomes reversed when investors evaluated competence dynamically, driven both by increases in evaluation outcomes for startups with female teams, and decreases for all male teams. This finding is congruent with the previous, and suggests that startups led by male founders may receive a boost when competence is assessed statically, but are less valued when investors pay attention to demonstrated competence.

Together, these findings suggest that male founders may benefit from advantaging mechanisms such as permissiveness (Phillips et al. 2022), which can affect their ability to enter further investor due diligence. This can have implications for not only female founders, but also for investment organizations, who may spend more time evaluating startups with all-male teams and less time evaluating startups led by a more diverse set of founders than is warranted. Perhaps one way to engender more equitable evaluation would be to inquire more about risk and progress, to prompt male innovators to prove their competence. This has broader implications for how status-based biases around competence can be counteracted – by focusing on advantaging mechanisms occurring within organizational evaluation processes in innovation, and perhaps beyond.

38
**8.4. Policy Implications.** Policy makers and investors are increasingly recognizing the importance of improving gender diversity in investing. Development Finance Institutions (DFIs) like the International Finance Corporation (IFC), Development Finance Corporation (DFC) and British International Investment (BII) are committing billions of dollars to invest in female-founded ventures (DFC, 2021), as are private investment funds like Fidelity and Nia Impact Capital, and venture capital firms such as Andreesen Horowitz. In this context, our findings have a number of important policy implications for reducing gender disparities in investment organizations and systems by changing the way investors evaluate startups.

First, changing investment evaluation processes without explicitly focusing on gender reduces the chance of backlash, compared to affirmative action policies (e.g., Leslie 2019). Indeed, this “levelling the assessment field” might increase efficiency in markets as investors make decisions based on more complete information. Second, these interventions are designed to retain investment organizations’ focus on identifying the most promising ventures – investors are not expected to focus on multiple goals (i.e., diversity and investment potential together).

Third, while other interventions to improve gender outcomes in investment have focused on the actions of investment seekers, we show that systems-level change is possible by changing organizational processes. At first glance, it may seem difficult for investment organizations to implement an assessment of progress in their selection processes – part of the problem in selecting early-stage startups is that they do not have a history of performance to analyze. The Vilcap investment team initially worried that their own selection processes would not allow dynamic assessment. However, after further discussion, Vilcap identified that startups could make progress between filling out their application form and having an interview with a local investor. Vilcap added a question to their interview template to assess startup progress: “Do you have any updates for us since you filled out the application form? (Has there been any change in how you think about your business or how 39
you execute your strategy?)” If more investment organizations prompt investors to inquire about progress and assess it, rather than only assessing potential, they could detect startups that are able to make rapid improvement – a capability important to many investors. A more structured approach to assessing signs of demonstrated progress could change how investors evaluate a startup’s potential, which seems particularly important for startups with female founders, who tend to be less rewarded for their potential. Thus, a shift to focusing on short-term signs of performance (progress) might help!

Fourth, these interventions are relatively inexpensive changes to processes, compared to costly training programs, investment guarantees, or providing supplementary funding for female-founded ventures. For DFIs, these findings offer a promising avenue to improve gender diversity in their portfolios in a cost-effective and efficient way.

Finally, although the number and types of investment organizations have increased in the US and globally (Cohen et al. 2019b, Lerner and Nanda 2020, Guttentag et al. 2021), gender disparities persist across organizational type (e.g., Ewens and Townsend 2020, Bapna and Ganco 2021) and global regions (e.g., Lall et al. 2020). Yet much of extant research has focused on investment decisions made by VC firms in the US (Drover et al. 2017, Clough et al. 2019). Our field experiment takes place in four global regions – Africa, India, Latin America, and the Middle East and North Africa – bringing together investors and startups from over 30 countries. We tested the same treatments across regions with a range of investors and am confident that the key insights from this study can be applied in many contexts, within and outside these regions.

**8.5. Conclusion.** Rather than considering how to improve signals that a female resourceseeker can provide to an investor, either through changing their storytelling narratives or providing female-focused funds, we suggest a novel mechanism to reduce disparities in investment outcomes. Through a two-stage field experiment with real investment decisions, we found that organizations can reduce gender disparities in investments by changing their evaluation practices. Prompting investors

40
to systematically inquire about risk and reward reduced disparities in scores, largely by lowering male scores. This effect appeared to be driven by investors inquiring more consistently across all founders, and conducting more rigorous diligence on startups with all-male founding teams. Prompting investors to systematically inquire about startup progress increased scores allocated to startups with female founders. This appeared to be driven by prompting a dynamic assessment of the venture, which informed investors’ assessments of competence, and convinced treated investors that startups with female founders could make rapid progress and scale. Systematizing inquiry can meaningfully affect investment outcomes, which has implications for entrepreneurship theory, provides insight on the role of inquiry in evaluation, and how organizations can systematize inquiry. It has policy implications for any organization interested in investing in innovation.

41