metadata: authors: 0: author: Piraino,Patrizio
1: author: Burger,Rulof Petrus
2: author: Abel,Simon Martin
count: World
docna: 0: docna: The value of reference letters
docty: Policy Research Working Paper
subtopic: Labor Markets,Educational Populations,Gender and Development,Education for Development (superceded),Education For All,Rural Labor Markets
teratopic: Science and Technology Development,Industry,Rural Development
abstracts: cdata!: This paper shows that reference letters
            from former employers alleviate information asymmetries
            about workers' skills and improve both match quality
            and equity in the labor market. A resume audit study finds
            that using a reference letter in the application increases
            callbacks by more than 60 percent, with women driving the
            effect. Letters are effective because they provide valuable
            information about workers' skills that employers use to
            select applicants of higher ability. A second experiment,
            which encourages job seekers to obtain and use a reference
            letter, finds consistent results. In particular, employment
            rates for women encouraged to obtain a letter increase by 49
            percent, closing the gender gap in the sample.
repnme: repnme: The value of reference letters
display_title: The value of reference letters
listing_relative_url: /research/2017/12/29262962/value-reference-letters
topicv3: Education, Skills Development and Labor
            Market m1326135 1484,Labor Markets m1326488 855,Soft Skills m1327287 2137,Gender m1327621 615,Jobs m1503931 2670
publishtoextweb_dt: 2017-12-11T00:00:00Z
historic_topic: Science and Technology Development,Industry,Rural Development
no_of_pages: 48
author_keys: 0: author_key: 1487724
1: author_key: 1534238
2: author_key: 1373021
model_extractions: WPS8266 Policy Research Working Paper 8266 The Value of Reference Letters _Martin Abel Rulof Burger Patrizio Piraino_ Jobs Group & Africa Gender Innovation Lab December 2017
Policy Research Working Paper 8266

# **Abstract**

This paper shows that reference letters from former employers alleviate information asymmetries about workers’ skills and improve both match quality and equity in the labor market. A resume audit study finds that using a reference letter in the application increases callbacks by more than 60 percent, with women driving the effect. Letters are effective because they provide valuable information about workers’ skills that employers use to select applicants of higher ability. A second experiment, which encourages job seekers to obtain and use a reference letter, finds consistent results. In particular, employment rates for women encouraged to obtain a letter increase by 49 percent, closing the gender gap in the sample.

This paper is a product of the Jobs Group and Africa Gender Innovation Lab. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at mabel@middlebury.edu.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# The Value of Reference Letters

Martin Abel<sup>_∗_</sup> Rulof Burger<sup>_†_</sup> Patrizio Piraino<sup>§</sup><sup>_‡_</sup>

> _∗_ Middlebury College. Email: abel@middlebury.edu

> _†_ Stellenbosch University. Email: rulof@sun.ac.za

> _‡_ University of Cape Town. Email: patrizio.piraino@uct.ac.za

> §Acknowledgment: This paper greatly benefited from discussions with and comments by David Autor, Emmanuel Bakirdjian, Willa Brown, Eliana Carranza, Bruno Crepon, Robert Garlick, Rachel Glennerster, Rema Hanna, Lawrence Katz, Asim Khwaja, Michael Kremer, Amanda Pallais, Gareth Roberts, Volker Schoer. We also thank participants at the CSAE (2016), NEUDC (2016), World Bank/IZA (2017), Development Economics and Policy (2017), and EALE (2017) conferences, as well as seminar attendees at Harvard University, Middlebury College, Stockholm School of Economics, and University of Cape Town. Emma Lambert-Porter, Velenkosini Matsebula, Samantha Ndiwalana and Svetlana Pimkina provided superb research assistance. We thank the South African Department of Labour and J-PAL Africa, especially Laura Poswell, for assistance in the implementation of this project. This study was prepared in collaboration with the World Bank Jobs Group and the Africa Gender Innovation Lab. We thank EuropeAid for financial support. The experiment was registered under registry number AEARCTR-0000819. All errors and omissions are our own.
# **1 Introduction**

Information asymmetries about workers’ skills are prevalent in labor markets, especially in the market for low-skill and entry-level jobs.<sup>1</sup> Hiring firms can reduce these asymmetries through reference letters from previous employers. However, in various contexts (particularly in the developing world) this practice is largely absent and most firms resort instead to informal referrals, such as those from their existing workforce.<sup>2</sup> This has potential adverse effects on match quality as it limits the pool of candidates and as current employees may refer close friends or family members rather than their most qualified peers (Loury, 2006; Beaman and Magruder, 2012). In addition, informal referral systems may exacerbate inequity as they disadvantage less connected groups (Montgomery, 1991).

We conduct three experiments in cooperation with the South African Department of Labour (DoL) to investigate the value and usage of standardized reference letters among young job seekers. Specifically, we design a reference letter template and encourage young job seekers to have a former employer complete it. Belot et al. (2016) note that there are only a few studies in the literature that focus exclusively on the impacts of reducing information asymmetries in the labor market. In particular, the large literature on active labor market policies consists mostly of evaluations of programs that combine a variety of elements, such as advice, monitoring, and/or training. To our knowledge, this is the first study on the use and value of reference letters.<sup>3</sup> Our first experiment (Experiment 1) is an audit study where we submit applications with and without reference letters on behalf of job seekers to vacancies and compare firm responses. Importantly, we conduct the study with actual job seekers who visit the Labour Centres. This addresses the criticism that application materials designed by researchers may not be realistic or include all relevant information (Heckman, 1998) as well as ethical concerns about sending fictitious applications (Riach and Rich, 2004).

> 1In these markets, job seekers often have limited work experience and lack educational degrees to signal skills. Firms are less likely to invest in costly screening as employment relationships are often short-term (Autor and Scarborough, 2008).

> 2In developed economies about 50% of jobs are found through informal network (Topa, 2011). In South Africa, where our study is based, about 68% of jobs are found through informal referrals (Schoer et al., 2014).

> 3There is relatively little research on reference letters, defined as a “description or evaluation of an applicant that is completed by an observer and used as a source of information for personnel selection” (McCarthy and Goffin, 2001), despite its ubiquity in the selection process (Aamodt, 2015). Existing research focuses on the ability of reference letters to predict future performance. One exception is Kaas and Manger (2012), who find through an audit study that reference letters do not increase overall employer responses but may benefit applicants from minority groups.

1
The results from the audit study show that reference letters are valuable to both job seekers and hiring firms. For the _same_ applicant, attaching a letter increases the probability of receiving a response from the firm from 4.2% to 6.7% (a 60% increase) and the rate of interview requests from 2.4% to 3.9% (a 62% increase). The overall effect is driven by female applicants.

As reference letters become more common, applications attaching a letter may be less distinctive. We shed some light on this by randomly varying the share of applicants for whom we submit reference letters. We find no evidence that increasing the share of reference letters sent to a given vacancy affects the letter’s impact.

Firms use the information provided by past employers to update their beliefs of applicants and are more likely to respond to applications with positive letters. In addition, we find that reference letters in which the former employer gives the highest rating in every category are ineffective. We interpret this as a perceived lack of credibility, which harms employment prospects.

Our design includes an aptitude test that job seekers complete at the Labour Centre. This provides a measure of ability that is observable to us but not to the employer. Using the applicant’s score on this test, we show that reference letters lead firms to select candidates of higher ability. Performing one standard deviation higher on the aptitude test increases the likelihood of an employer response by 2.6 percentage points (a 62% increase) for applications that include a reference letter, as opposed to 0.6 percentage points (a 15% increase) for those not including a letter. This improvement in the firm’s capacity to screen better applicants suggests that letters are informative of workers’ unobserved ability. Indeed, we show that ratings from previous employers are highly correlated with aptitude scores, even after controlling for information that can be easily observed from the resume or school transcripts.

In sum, the results from the audit study (Experiment 1) show that reference letters can reduce information asymmetries, especially for women, and improve the employment prospects of higher ability candidates. Although our design cannot explicitly test for general equilibrium effects, theory predicts that the identification of higher ability workers should increase firm demand (Pissarides, 1985).

Since Experiment 1 uses employers’ callback as the main outcome, it does not estimate employment effects. Moreover, it abstracts from job seekers’ behavioral responses, as applications are submitted on their behalf. While these are limitations that are common to most audit studies (e.g. Bertrand and Mullainathan, 2004; Belot et al., 2016), we 2
conduct a second experiment (Experiment 2) in which we encourage half of job seekers to obtain a letter and subsequently follow their job search behavior and employment outcomes.

The results from Experiment 2 show that female participants who obtained letters are significantly more likely to receive job interviews and their employment rate doubles after three months. On the other hand, no significant impacts are found for men, thus fully closing the gender employment gap in our sample. This gender heterogeneity is very much consistent with the evidence from Experiment 1. We also show that the large employment effects for women are in part explained by a significantly higher usage of letters by female job seekers.

Finally, we run a third experiment (Experiment 3) to investigate why reference letters are not more prevalent in the applications of our target population. The results from this experiment show how providing information on the benefits of having a letter increases the share of participants that obtain one. By contrast, an arm of the same experiment, which offered cash incentives for obtaining letters, had no effect. Underestimating potential benefits may thus explain why job seekers are not asking former employers to provide (informative) reference letters.

The evidence presented in the paper contributes to the literature on job referrals. Previous studies have largely focused on whether social network links can be exploited to reduce information asymmetries, showing that although workers have information on the productivity of their peers (Pallais and Sands, 2016; Burks et al., 2015), they are less likely to pass on truthful information to firms unless sufficiently incentivized (Beaman and Magruder, 2012). Former employers may provide more credible information because their incentives are more aligned with the hiring firm. In addition, they can assess worker abilities more accurately as they observed them in a professional setting (Aamodt, 2015). However, few studies have looked at the role of former employers in reducing information asymmetries. Two notable exceptions are Pallais (2014), who finds that feedback on workers’ past performance in an online labor market increases their employment prospects, and Bassi and Nansamba (2017) who study the effect of certifying soft skills. We contribute to this literature by investigating a more traditional labour market setting in which workers can choose both the referee and whether to reveal the information to the market after they observe it.

In addition, we contribute to the literature on how search frictions affect employment (Mortensen and Pissarides, 1994). Information asymmetries between firms and workers 3
lead to socially sub-optimal hiring of people with limited work experience and an overall decrease in market efficiency (Pallais, 2014; Terviö, 2009). We show that a simple intervention can improve firms’ screening ability and reduce these asymmetries. This is a necessary precondition for reference letters to have general equilibrium employment effects, providing a rationale for the government to facilitate the information exchange.

This study also adds to an extensive literature evaluating the effectiveness of active labor market policies (ALMP) (see Card et al., 2015; McKenzie, 2017; for recent reviews). The evidence on ALMPs is mixed, in part because they typically include a package of interventions which makes it difficult to isolate the effectiveness of specific components. In this study, we are able to isolate one component of ALMPs, namely the reduction of information asymmetries. Our findings suggest that addressing information frictions through relatively simple interventions can translate into real employment results.<sup>4</sup> We acknowledge three main caveats in our study design. First, since we do not randomize the information content of the reference letter, we cannot investigate the relative importance of different types of information on employer’s demand. We partially address this concern in Experiment 1, where we can purge the effects of individual-specific characteristics and show that positive ratings on the skills listed in the letter increase employer’s responses. Secondly, some of our results are imprecisely estimated. While this warrants some caution in the interpretation of the results, it is reassuring that findings from different experiments (and samples) and on different outcomes consistently point in the same direction. In particular, the empirical results consistently show that women disproportionately benefit from reference letters. Third, one may object that the letters used in this study are based on an easy-to-digest template created by the authors rather than the more common narrative letters found in the market. However, we consider the development of a low-cost and highly-replicable new tool as an additional contribution of the paper (see Belot et al., 2016, for a similar contribution).<sup>5</sup> Screening job applicants is costly, as it requires information that is time-consuming to acquire. The template, which we designed based on feedback from firms, has precisely the intention to reduce this friction. In addition, the intervention is evaluated through the Labour Centres, which is the real environment where the letter templates would be introduced.

> 4In fact, J-Pal Africa and the authors are currently advising the South African Department of Labor on possible ways to implement the reference letter template in the Labour Centres across the country. 5Belot et al. (2016) develop and evaluate experimentally a tool that provides tailored job search advice in Job Centres in Scotland. These authors note that most interventions evaluated in the literature have been designed by policymakers or practitioners and that there is added value in developing new tools using insights from the academic literature.

4
Overall, we believe that our results show that letters are effective: they benefit job seekers and enhance firms’ screening ability. In particular, we find large employment gains for women, a group often excluded from informal referral networks. Reducing information asymmetries – through reference letters or other interventions – may thus improve equity by leveling the playing field for women in labor markets.

The remainder of the paper is structured as follows: Section 2 describes the study context and introduces a conceptual framework. Section 3 describes the data and experimental designs. We report and discuss the main empirical findings in Section 4, while Section 5 concludes.

# **2 Background and Conceptual Framework**

## **2.1 South Africa’s Labor Market**

The unemployment rate in South Africa is high (26.4%), especially for youths (36.9%) (StatsSA, 2015). The gender employment gap among black South Africans is substantial, despite the fact that black females are on average more educated than their male counterparts (Rospabe, 2001; Shepherd, 2008). One explanation is that firms appear to either underestimate or are more uncertain about the ability of female applicants (Malindi (2016)).

The labor market in South Africa offers a context conducive to investigating the role of information asymmetries. Most of the unemployed did not complete secondary education (55%) and have no or limited work experience (50.6%), which leaves firms with very little information to screen job applicants. In addition, the quality of education is low on average and highly variable, which limits the use of educational credentials as signals for productivity (van der Berg, 2007). Finally, unemployment spells in weak labor markets are less indicative of job seekers’ ability (Kroft et al., 2013).

Information asymmetries affect how firms and workers are matched. Some large firms in South Africa administer aptitude tests as part of the hiring process. While these tests can increase aggregate productivity and labor demand by improving match quality (Mortensen and Pissarides, 1994; Pissarides, 1985), they have not been widely adopted. This may be due to firms having fewer incentives to test candidates for jobs where investment in training is limited and employment spells are brief (Autor and Scarborough, 5
2008). Moreover, many small firms lack the expertise and resources to systematically test applicants.

Faced with these challenges, South African employers have increasingly turned to social networks and the existing workforce to fill vacancies. Schoer et al. (2014) report that up to 68% of workers found employment via networks. Yet, firms face a trade-off in their choice of hiring channels (Montgomery, 1991). Under the “good match” hypothesis (Rees, 1966), current employers can help overcome the asymmetric information problem and create better employment matches as they know both the firm and the people in their network. By contrast, the “limited choices” hypothesis stresses that finding employment through social networks limits the opportunities and match quality (Loury, 2006). In addition, current employees may have personal interests in referring family and friends that conflict with the interest of the firm (Beaman and Magruder, 2012; Fafchamps and Moradi, 2015).

A formal referral system with endorsements from former employers may thus be a more effective mechanism to reduce information asymmetries. Interviews with South African firms confirm the benefits of having former employers as references: if available, hiring managers report that they typically call them for the group of shortlisted candidates. However, focus group discussions with job seekers reveal that most do not have contactable references listed on their CV and less than 5% used a reference letter as part of the application process.

# **2.2 Conceptual Framework**

Markets differ in the extent to which references can mitigate information asymmetries. In many markets, sellers have no choice over the source of the reference and whether this information is publicly revealed (e.g. Amazon ratings). By contrast, job seekers typically choose referees and often observe their feedback before deciding whether to reveal it to the market. This is an important feature which may limit how effectively referral systems can reduce information asymmetries, and which has been largely omitted from the theoretical literature on references.

This section introduces a static illustrative framework for employer learning in this type of market. It will generate two important sets of results: (i) it identifies conditions under which letters have value and (ii) it derives predictions for how the letter affects the hiring decision and screening ability of firms. Here we limit ourselves to an intuitive explanation of the model, while the formal derivations are provided in Appendix A.

6
Consider a job-seeker who has (general) ability _a_ , which determines her productivity for any firm. At the time of applying for work, the job-seeker is endowed with an application signal _s_ 1 = _a_ + _e_ 1. This represents the content of a resume, including school transcripts and other easily observable applicant attributes. With probability _π_ she is also endowed with a reference letter signal _s_ 2 = _a_ + _e_ 2. Assume that _a ∼ nid_ (0 _,_ 1), _e_ 1 _∼ nid_ (0 _, σ_ 1<sup>2),and</sup> _e_ 2 _∼ nid_ (0 _, σ_ 2<sup>2).The job-seeker applies to a vacancy by sending application</sup><sup>_s_1to the firm</sup> and must choose whether to also attach a reference letter _s_ 2 ( _d_ = 1 if she does, otherwise _d_ = 0).

The firm chooses whether to hire the applicant based on available information. It will do so if the expected productivity exceeds some threshold, which we assume is firm-specific and exogenously determined. The job seeker’s utility depends only on whether or not she is offered a job, and there is no cost to applying or sending reference letters. The firm’s conditional expectation is rational and common knowledge, but the hiring threshold is private information.

We show in Appendix A that the firm’s equilibrium conditional expectation function, depending on whether they receive a reference letter ( _d_ ), can be expressed as where _κ_ 1 and _κ_ 2 capture the relative noise in resumes ( _s_ 1) and reference letters ( _s_ 2), respectively. _ω_ is the variance of the letter signal conditional on the information in the resume and _χ_ is a monotonic transformation of the probability that applicants who have a reference letter choose not to attach it.

When applicants include a reference letter ( _d_ = 1), firms form beliefs about ability using information from both the resume and reference letter, weighted according to the relative reliability of these two signals. If the application does not include a reference ( _d_ = 0), firms form beliefs using the information in the resume. They further penalize these applicants with a downward adjustment in expected ability, conditional on the quality of the resume. The magnitude of this penalty ( _χκ_ 1 _ω_ ) increases in the share of applicants who have access to letters but do not attach it, the relative reliability of the letter and the variance of the letter signal.

7
In equilibrium, applicants with access to letters will choose to send it if it improves the firm’s perception of their ability, i.e. if _E_ ( _a|s_ 1 _, s_ 2 _, d_ = 1) _> E_ ( _a|s_ 1 _, s_ 2 _, d_ = 0). This requires that the reference is sufficiently positive relative to the information in the resume. Implicit in the model setup are two testable assumptions about the information provided _∂E_ <u>(</u> _s_ 2 _<u>|a</u>_ <u>)</u> by reference letters: (i) letters are informative about the applicant’s ability, i.e. _δa >_ 0, and (ii) letters contain information that is not already contained in the applicant’s resume, _∂E_ <u>(</u> _s_ 2 _<u>|a,s</u>_ 1) i.e. _δa >_ 0. Under these assumptions (which we validate in our empirical analysis), the model makes the following predictions about how job seekers use letters and how firms respond to receiving letters (see Appendix A):

- Including a reference letter _will increase the hiring probability_ (unless the letter is sufficiently negative). Moreover, firms will be more likely to _hire candidates with stronger letters_ .

- Letters result in a _closer mapping from ability to job offers_ . That is, firms will be able to screen workers with a reference letter more accurately.

- Any attribute that casts doubt over the reliability of the reference letter (e.g. not providing contact information or being implausibly positive) reduces their effectiveness.

- If employers are more uncertain about the ability of a certain group of job seekers (e.g. females), then reference letters matter more for that group.

- As access to and usage of letters increase, the ability of firms to identify higher ability candidates improves.

This framework presents a rational benchmark model, which assumes that job seekers have correct beliefs about the value of reference letters. We revisit this assumption in Section 4.3.

# **3 Study Design**

This section first describes the sampling and the process of eliciting reference letters common to all three experiments. We then describe each of the experimental designs in detail.

8
# **3.1 Study Sample and letter template**

Our sampling frame is the Employment Services of South Africa (ESSA) database, consisting of more than 550,000 job seekers collected by the Department of Labour (DoL). We restricted this sampling frame to African unemployed youths between the ages of 18 and 34. We limit our study sample to job seekers who have some form of previous work experience (as our intervention tests reference letters from previous employers), have not completed university-level tertiary education and live within traveling distance from our implementing labor centers.

We further stratify the sample by gender to facilitate subgroup analysis. In the recruitment call, surveyors explain that the job seeker is invited to participate in an employment service study at the local labor center on a specified day. In return, they receive a stipend of 30 Rand (~ 2 USD) to cover travel costs. Across all experiments, 67% of the successfully contacted unemployed individuals agreed to participate.<sup>6</sup> A baseline survey is administered through an in-person interview at the labor center. In Experiment 1, this is followed by an aptitude test that evaluates basic math and literacy skills.<sup>7</sup> The study employs an encouragement design implemented in cooperation with the DoL. After the baseline survey, participants assigned to the treatment group have a brief individual meeting explaining how to use reference letters in the job search. This is followed by a discussion of the job seekers’ work history and identification of potential referees. We provide job seekers with several hard copies of the template. We conducted more than 30 interviews with employers who frequently mentioned the importance of contactable references in the screening process. When asked what information they collect from references, employers listed both non-cognitive skills like motivation, reliability and work ethic as well as cognitive skills like numeracy and literacy. They are also interested in the nature of the relationship between the referee and job seeker and why the employment relationship ended. Based on this feedback, we designed a reference letter template that employers can easily fill out. Figure A.1 in the Appendix shows the

> 6Using the limited demographic information provided in ESSA, we find that age and gender are not correlated with the decision to participate. By contrast, every year of additional education increases the probability of participation by 1.6 percentage points (p-value: 0.063). Of those that agree to participate, 63.5% showed up at the labor center on the specified day. None of the socioeconomic variables predict whether participants fail to show up at the agreed time and day.

> 7The test takes about 20 minutes and was designed by the researchers. It closely follows standard entry level tests used in the hiring process by large employers in South Africa. Test scores are approximately normally distributed with a mean (median) joint numeracy and literacy score of 61% (63%). For sample questions see Figure A.3.

9
template, while Figure A.2 gives examples of completed letters.

# **3.2 Experimental Designs**

Figure 1 describes how our intervention may affect employment and summarizes our experimental designs. In Experiment 1, we submit applications on behalf of job seekers to vacancies from online job sites and test if employers are more likely to respond when a reference letter is attached. This provides a “test case” whether reference letters can be valuable. Experiment 2 explores the effect of reference letters on job search behavior and estimates employment effects _after_ people adjusted their search strategy. Experiment 3 tests different forms of encouragement to investigate why only a small share of people in our target population obtain reference letters in equilibrium.

Figure 1: Experimental Design Overview

## **3.2.1 Experiment 1: Employers’ Response to Reference Letters**

To test the effect of the letter on employer demand, we employ a within-subject randomization design: we encourage 441 job seekers across three labor centers to obtain a reference letter and provide instructions on how to return the completed letter to us. After one week, participants receive a text message reminding them to obtain and return the reference letter; 31% of encouraged job seekers returned the completed letter.

Table A.1 provides summary statistics of the reference letter content, converting employer ratings into numeric values (0=below average, 1=average, 2=good, 3=very good). Overall, ratings tend to be positive: on a scale from 0 to 6, the average aggregate hard and soft skill rating is 4.9; 11% have a perfect score of 6. We find that hard skills are slightly less positively rated than soft skills (2.3 vs. 2.6 on a 3 point scale). While for most categories women receive slightly more positive ratings, only one gender difference is significant at 10
the 10% level (Team Ability) and one at the 5% level (How highly recommended). Note that we do not verify the authenticity of the reference letters. In Section 4.2, we will explore whether the letter provides truthful information.

Table A.2 investigates which characteristics are correlated with the probability of returning a letter. Age is the only statistically significant predictor. However, there are likely unobservable variables correlated with the propensity to obtain a letter.

For the participants who return the letter to us, we send out applications with and without the reference letter. While selection at the encouragement stage may affect the generalizability of results, the within-subject randomization ensures that results are internally valid. This also has the advantage that we can control for individual specific factors that determine employer responses and thus estimate the effect of reference letters more accurately.

Figure 2 summarizes the randomization design. We search the four most popular South African job websites to identify vacancies for entry positions from one of the following sectors: administration, call center, cleaner, driver, retail, security and unskilled. The vacancies are randomly assigned to vacancy slot 1 through 6. Next, we select four of the job seekers who returned the letter and have previous work experience in a related sector. We create email addresses for each participant and send out six applications following the pattern described in Figure 2. For example, for Participant A we send four applications with the CV (and any additional supporting documents the job seeker provides) and two applications for which we attach the reference letter as an additional document. Importantly, we are invisible to the employer in the entire application process. Vacancies 1 through 4 offer a straightforward test of the effect of reference letters as we can compare employer responses between applications with and without the attached letter (e.g. compare cell A1 to cell A2, A3 and A4). For vacancy 5 we only send CVs. This provides us with a test for displacement effects at the interview stage, i.e. whether being in an application pool with somebody with a reference letter reduces the chances of getting an employer response. To test for this, we can compare employer responses in cell A5 to A2, A3, and A4. Vacancy 6 receives three applications with reference letters. Comparing application A1 and A6 allows us to test whether employers respond to reference letters differently once they represent a higher proportion of the applicant pool.

We submitted a total of 2,050 applications between June 2015 and April 2016.<sup>8</sup> We

> 8We included vacancy 6 starting in January 2016.

11
||||Vacancy<br>||||
|---|---|---|---|---|---|---|
|ee|ee|ee|ee|ee|e|ee|
|a|CV+Ret<br>Letter|cv|cv|cv|cv|<br>CV+ Ret<br>Letter|
|=<br>o<br>a<br>|cv|CV+Ref<br>Letter|cy|cy|cy|CV+Ref<br>Letter|
|oa<br>Cc|cv|cv|CV+ Ref<br>Letter|cv|cv|CV+ Ref<br>Letter|
||cv|cv|cv|CV+Ret<br>Letter|cy|cy|
treatment.<sup>11</sup> One potential shortcoming of any survey data is that it is self-reported. We therefore complement the survey data with an observed measure of job search. Specifically, study participants in Experiment 2 receive a notification about a vacancy and are asked to submit their full application via email in case they are interested. This message was sent from a third party email address not associated with the research project in order to mitigate concerns about surveyor demand effects.<sup>12</sup> This allows us to test whether participants apply and whether they submit the reference letter as part of their application.

# **3.2.3 Experiment 3: Barriers to Obtaining Letters**

Results discussed in more detail below suggest that reference letters substantially increase the probability of receiving an employer response. This raises the question of why only about 2% of job seekers in the control group use reference letters in their job search. Experiment 3 tests different barriers to obtaining reference letters.

During follow-up surveys, a significant share of participants could not provide us with a reason why they have not tried to obtain the letter or cited reasons like having “no time” or that they do not need it. This may be a sign that job seekers do not believe they benefit from these reference letters or are in other ways insufficiently incentivized to obtain them. We design two interventions to test potential explanations for low take-up: (i) provide job seekers with information on the effectiveness of letters and (ii) compensate participants with 100 Rand (about half a daily wage) in cell phone airtime if they obtain a letter.

A group of 438 job seekers, previously encouraged to obtain a letter,<sup>13</sup> receives a follow-up text message to their cell phone and (if provided) email address reminding them of how

> 11Table A.4 shows that attrition rate increases from about 6% in wave 1 to 17% in wave 2, likely due to survey fatigue and participants switching phone numbers. Attrition is clearly not random: younger and less educated participants are more likely to attrite, but importantly rates do not differ between treatment and control group.

> 12Participants were informed about a vacancy in a specific sector. Among those with work experience in multiple sectors, we randomly chose which sector we notify them of. For job seekers for whom we do not have information on previous sectors, we send a general notification about a vacancy. Sectoral shares were balanced by treatment status. Applications were submitted to actual vacancies after the end of the last survey wave so that it would not confound employment estimates.

> 13About half of these participants were drawn from the sample in Experiment 1. The other half were participants that were drawn from the ESSA database solely for this experiment. Importantly, there is no overlap with the Experiment 2 sample.

13
to return the completed letter to us. Participants were randomized into three groups.<sup>14</sup> The control group received only this reminder, while the other two groups received one of the following additional messages:

- “ _Research suggests reference letters almost double chances of getting a job interview_ .” (Information)

- “ _To compensate your costs, you get 100 Rand airtime after sending us the completed letter._ ” (Compensation)

# **4 Results**

This section reports and discusses the empirical findings from our three experiments. We begin with the results from the audit study (Exp. 1), where we recover the value of reference letters to both job seekers and hiring firms from a within-subject identification strategy. We then move to Exp. 2, where we can account for changes in job search behavior by study participants and obtain treatment effect estimates for employment. We conclude by presenting the results from Exp. 3 (as well as additional evidence from baseline data and focus groups) to discuss potential reasons behind the low usage of reference letters in equilibrium.

## **4.1 Audit study**

Experiment 1 tests the effect of reference letters on firm demand using within-subject randomization. We use two measures of employer response: (i) a narrow measure of interest that captures interview requests and (ii) a broader measure of interest that captures either an interview request or a different employer response (most commonly, firms asked questions, requested specific documents, or provided more information about the job and asked if job seekers were still interested). Throughout the analysis we will report results for both outcomes.

To estimate the effect of the reference letter, we estimate the following model:

> 14Comparing observable characteristics between the treatment and control group suggests that randomization was successful (Table A.5).

14
Outcome _yis_ is a binary variable measuring whether employers respond to application _i_ of person _s_ . _Refis_ is an indicator variable for whether a reference letter was included with application _i_ . _λs_ and _μk_ capture individual and sector fixed effects, respectively. The error term _eis_ is clustered at the individual level. The coefficient of interest _β_ captures the causal of the reference letters.

# **4.1.1 Employer Responses**

Table 1 reports results from Specification 3. Column 1 to 4 report effects using the broad measure of interest as an outcome, while Column 5 to 8 report effects on interview requests. On a control mean of 4.15 percent, the reference letter significantly increases the chance of getting any employer response by 2.54 percentage points (col. 1) and on getting an interview request by 1.54 percentage points on a control mean of 2.4 percent (col. 4). Coefficient estimates are consistently positive and of similar magnitude when including sector and individual fixed effects (col. 2, 3, 6, 7).

Coefficients on the gender interaction term are statistically significant and large in magnitude: compared to the control mean, both the measure of employer interest and interview requests approximately double for women, whereas they are close to zero for men (col. 4 and 8). Overall, the results in Table 1 show that reference letters have a large positive effect on employer callbacks for women, a result that will be corroborated in Section 4.2 below using a different sample and experimental design .

Table 1: Effect of Reference Letter on Call Back

||y=E|mployer Re|sponse: **Int**|**erest**|y=E|mployer Res|ponse: **Inte**|**rview**|
|---|---|---|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|
|Reference Letter|0.0254<sup>_∗∗_</sup>|0.0251<sup>_∗∗_</sup>|0.0244<sup>_∗∗_</sup>|-0.0019|0.0154<sup>_∗_</sup>|0.0150<sup>_∗_</sup>|0.0143|-0.0039|
||(0.0102)|(0.0102)|(0.0107)|(0.0150)|(0.0087)|(0.0088)|(0.0091)|(0.0114)|
|Female x Letter||||0.0416**||||0.0300*|
|||||(0.0206)||||(0.0171)|
|Sector F.E.|N|Y|Y|Y|N|Y|Y|Y|
|Individual F.E.|N|N|Y|Y|N|N|Y|Y|
|_R_<sup>2</sup>|0.003|0.006|0.077|0.083|0.002|0.008|0.052|0.056|
|N|2050|2050|2050|2050|2050|2050|2050|2050|
|Control mean|0.0415|0.0415|0.0415|0.0415|0.0240|0.0240|0.0240|0.0240|

Notes:<sup>_∗_</sup> _p <_ 0 _._ 10,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01. Standard errors (reported in parentheses) are clustered at the applicant level. Results report OLS estimates. Dependent variables are binary measures of employer response: interview requests (Col. 5-8) and either interview request or a different employer response expressing interest in the job applicant (Col 1-4). Sector fixed effects are included for the six sectors for which we send applications. Since we employ within-subject randomization, the reference letter varies within individual; hence the gender interaction effect can be with individual We also estimate Specification 3 including an interaction term between the reference letter 15
variable and an indicator variable for vacancies receiving multiple applications with letters attached (vacancy 6 in Fig. 2 above). The coefficient on the interaction term is very close to zero indicating that the effect does not differ if the employer receives more than one letter (Table A.6, col. 2 and 5). These results suggest that it is not the novelty of seeing a reference letter that is driving the positive employer response. While it is possible that within a large numbers of applicants a few reference letters may still carry a novelty effect, it is reassuring that tripling the number of candidates that attach the exact same template does not decrease its Next, we test whether there is a negative effect from being in the applicant pool with a job seeker who submits a reference letter. We include a dummy for pure control applications (sent to vacancy 5 in Fig. 2). The estimated coefficients in Columns 3 and 6 of Table A.6 are small in magnitude and not statistically significant. This provides evidence against the possibility of displacement. However, the estimates are relatively imprecise and therefore need to be viewed with caution.

# **4.1.2 Screening Ability**

Information asymmetries may inhibit firms from identifying the most suitable candidates. In order to test whether the letters enable firms to identify applicants of higher ability, we assume that there is an ability parameter _a_ , imperfectly observed by the firm at the time of application, and estimate the following model: where _a_ is proxied by the standardized results on the aptitude test administered as part of the baseline survey. The coefficient _γ_ captures whether employers select higher ability applicants when only the CV is attached, while _γ_ + _δ_ is the effect when the letter is attached.

Results are presented in Table 2. The estimate for _γ_ is small in magnitude and not significant, suggesting that without the reference letter firms are ineffective in selecting the more productive applicants. The estimates for _δ_ , on the other hand, are positive and significant (col. 1 and 3) indicating that reference letters enable firms to identify applicants of higher ability, despite not seeing the aptitude score.

The coefficients are large in magnitude: for applications with reference letters, a one standard deviation higher performance in the aptitude test increases the probability of 16
receiving an employer response and interview request by an additional 2 percentage points and 1.3 percentage points, respectively. Put differently, in control applications the chance of receiving an employer response for job seekers at the 90th ability percentile is 1.8 percentage points higher compared to those at the 10th percentile. Once the reference letter is included, this figure increases to 6.3 percentage points. This is one of our key results, as economic theory predicts that an improvement in screening ability increases labor demand.

Note also that the effect on the firm’s capacity to select candidates with higher aptitude scores does not differ by gender (col. 2 and 4). These results suggest that the letter helps firms to screen higher ability males, despite the evidence in Table 1 showing no increase in employer callback for male applicants with a letter. This is consistent with firms underestimating the ability of female candidates, while being more accurate _on average_ about males (see Malindi, 2016 for evidence of statistical discrimination against black females in South Africa).<sup>15</sup> Table 2: Effect of Reference Letter on Screening Productive Applicants

||Y=In|terest|Y=Int|erview|
|---|---|---|---|---|
||(1)|(2)|(3)|(4)|
|Reference Letter|0.02575<sup>_∗∗_</sup>|0.01669|0.01555*|0.00933|
||(0.010)|(0.019)|(0.009)|(0.013)|
|Aptitude (z-score)|0.00618|0.0074|0.00062|0.00013|
||(0.005)|(0.008)|(0.003)|(0.004)|
|Ref Let x Aptitude (z-score)|0.01999<sup>_∗∗_</sup>|0.01862|0.01305<sup>_∗∗_</sup>|0.01372*|
||(0.008)|(0.011)|(0.006)|(0.008)|
|Ref Let x Female||0.01760||0.01390|
|||(0.022)||(0.018)|
|Ref Let x Female x Aptitude (z-score)||-0.00433||-0.00889|
|||(0.019)||(0.015)|
|_R_<sup>2</sup>|0.003|0.008|0.002|0.004|
|N|2050|2050|2050|2050|
|Control mean|0.0415|0.0415|0.0240|0.0240|

Notes:<sup>_∗_</sup> _p <_ 0 _._ 10,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01. Standard errors (reported in parentheses) are clustered at the applicant level. Results report OLS estimates controlling for sector fixed effects. _Aptitude_ is measuring the standardized English and Math score. For readability reasons, we suppressed coefficients for _Female_ and _Female x Aptitude_ . These coefficients are small in magnitude and insignificant.

> 15Malindi (2016) finds that black females have a much higher returns to job tenure than black males, white females or white males in South Africa. This is consistent with employers initially under-estimating or attaching greater uncertainty to the value of productive attributes possessed by black females.

17
# **4.1.3 Letters are informative**

In order for reference letters to be effective they must (i) be informative of applicants’ skills and (ii) provide information that cannot easily be inferred from other application documents. We test these assumptions by comparing subjective employer ratings to an objective assessment. Specifically, we regress results from the numeracy and literature aptitude test we administer at baseline on the ratings provided by employers on numeracy and literacy. Table 3 shows that employer ratings and test results are highly correlated for both literacy (col. 1) and numeracy (col. 4). This implies that the average letter contains information about the applicant’s skills.

Next, we explore how the correlation changes when we control for additional covariates. In particular, we control for age, education, gender, and school grades in English and math. The results in Column 2 and 5 show that while coefficients decrease in magnitude, they stay highly significant. This suggests that the letter contains information that employers cannot easily infer from the resume.

Finally, Column 3 and 6 show that the coefficients do not differ by gender. This rules out the possibility that any differential employment effect for women is due to references being more informative of females’ aptitudes.

Overall, Table 3 confirms that referee ratings convey additional information to employers, at least for skills captured in the aptitude test. Arguably, it would be even more difficult for firms to learn about other skills from the CV, especially non-cognitive skills like reliability or work ethics (Aamodt, 2015).

18
Table 3: Are Numeracy and Literacy Employer Ratings Correlated with Aptitude?

||Literacy:|Reference Le|tter (z-score)|Numeracy:|Reference L|etter (z-score)|
|---|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|(6)|
|Literacy: Aptitude|0.3645<sup>_∗∗∗_</sup>|0.2274<sup>_∗∗_</sup>|0.2458<sup>_∗∗_</sup>||||
|(z-score)|(0.0935)|(0.1026)|(0.1185)||||
|Female x Literacy Apt|||-0.04907||||
|(z-score)|||(0.2066)||||
|Numeracy: Aptitude||||0.3001<sup>_∗∗∗_</sup>|0.2627<sup>_∗∗∗_</sup>|0.25585<sup>_∗_</sup>|
|(z-score)||||(0.0885)|(0.0966)|(0.1381)|
|Female x Numeracy Apt||||||0.01548|
|(z-score)||||||(0.1788)|
|Covariate|N|Y|Y|N|Y|Y|
|School Grade|N|Y|Y|N|Y|Y|
|_R_<sup>2</sup>|0.136|0.232|0.232|0.093|0.116|0.116|
|N|116|116|116|114|114|114|

Notes:<sup>_∗_</sup> _p <_ 0 _._ 10,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01. The dependent variable is the standardized value of the numeric employer rating (0=below average, 3=very good). _Literacy_ and _Numeracy_ measure the standardized performance in the aptitude test. Control variables include age, gender and education. School grade is measuring the grade (in %) participants achieved in the last math and English class, respectively.

# **4.1.4 The letter’s content matters**

We posit that job seekers with _better_ reference letters are more likely to receive job interviews. An alternative mechanism would be that the mere ability to obtain a reference letter and use it correctly in the job search is the relevant signal for firms.

In order to shed light on this question, we first look at the relationship between referee ratings and employer responses graphically. Figure 3 shows local linear regression estimates of the probability of receiving an interview by employer’s rating. For applications including a reference letter (blue line), the relationship is clearly nonlinear: employer responses increase with higher ratings until the very top, where we observe a large drop in the probability of being interviewed. The figure also shows that when no letter is attached to the application, these _same_ individuals do not experience a discontinuity at the near-perfect scores.

Figure 3 suggests that employers may ignore the reference letter signal if it is perceived to be implausibly positive and thus deemed non-credible.<sup>16</sup> Our data, however, reveal that firms are incorrect in inferring that applicants with perfect scores on their letter

> 16A uniform rating may also indicate that the referee did not take the time to carefully consider each category. However, we do not find that the effect of these uniform assessments differs for letters that include more detailed comments on skill categories, suggesting that the negative effect is not due to a perceived lack of effort of the referee.

19
<!-- Start of picture text --> oo<br>o<br>Le)<br>2°<br>2<br>£<br>E<br>~t<br>o0<br>&<br>oO<br>oO<br>a]<br>Pa<br>ac<br>So<br>3 4 5 6<br>Reference letter score<br>—— No reference letter — Reference letter<br><!-- End of picture text -->
higher demand also receive more positive reference letters, which allows us to rule out an obvious confounding factor.

The additional effect of the referee rating once the letter is revealed to the firm is measured by _δ_ . When we control for applications with perfect scores, the effect of the referee rating on the probability of receiving a response from employers is positive and significant (col. 2, 5). The coefficient estimates in Column 2 and 5 indicate that a one standard deviation higher rating increases employer responses (interviews) by 48% (77%). The coefficient on the interaction term with the all positive dummy is also significant (and negative).

In addition, Column 3 and 6 of Table 4 suggest that the content of the letter may matter more for female applicants: good ratings generally have a larger positive impact and letters with perfect ratings have a more negative effect. This would be consistent with earlier work showing that reducing information asymmetries leads to a larger belief updating among employers for members of disadvantaged groups (Agrawal et al., 2013; Lang and Manove, 2011; Kaas and Manger, 2012).<sup>17</sup> Table 4: Effect of Referee Rating on Call Back

|||Y=Interest|||Y=Interview||
|---|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|(6)|
|Reference Letter|0.0978|0.1213<sup>_∗_</sup>|0.0943|0.0615|0.0902|0.0698|
||(0.0644)|(0.0643)|(0.0702)|(0.0581)|(0.0594)|(0.0615)|
|Referee Rating (z-score)|-0.0032|-0.0066|-0.0001|0.0009|-0.0000|0.0013|
||(0.0060)|(0.0060)|(0.0072)|(0.0039)|(0.0043)|(0.0059)|
|Letter x Rating (z-score)|0.0091|0.0198<sup>_∗∗_</sup>|0.0056|0.0055|0.0185<sup>_∗∗_</sup>|0.0102|
||(0.0093)|(0.0090)|(0.0136)|(0.0086)|(0.0091)|(0.0103)|
|Letter x Rating x Female|||0.0335<sup>_∗_</sup>|||0.0175|
||||(0.0200)|||(0.0166)|
|All positive||0.0206|0.0298||0.0056|-0.0243<sup>_∗_</sup>|
|||(0.0266)|(0.0765)||(0.0139)|(0.0129)|
|Letter x All positive||-0.0632<sup>_∗_</sup>|-0.0895||-0.0762<sup>_∗∗∗_</sup>|-0.0116|
|||(0.0349)|(0.0821)||(0.0239)|(0.0207)|
|Letter x All positive x Female|||0.0110|||-0.1021<sup>_∗∗∗_</sup>|
||||(0.0901)|||(0.0348)|
|_R_<sup>2</sup>|0.014|0.015|0.016|0.014|0.017|0.020|
|N|2050|2050|2050|2050|2050|2050|
|Control content|Y|Y|Y|Y|Y|Y|
|Control mean|0.0415|0.0415|0.0415|0.0240|0.0240|0.0240|

Notes:<sup>_∗_</sup> _p <_ 0 _._ 10,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01. Standard errors (reported in parentheses) are clustered at the applicant level. _Total Score_ measures the average employer rating converted to numeric values (out of 6). _All positive_ is a indicator variable for whether employers give a perfect rating. We estimate the model with all interaction terms but suppress coefficients for readability reasons. All columns control for other content revealed in the reference letter. We include dummy variables for five reference letters that did not include a rating.

> 17This finding is not inconsistent with the results presented in Table 2. Our data shows that math/literacy skills are less correlated with harder-to-observe skills (e.g. team ability, work ethics) for women than they are for men. This would explain why firms pay more attention to the letter content of female applicants. Results (not reported for brevity) confirm that firms pay more attention to these hard-to-observe skills for females.

21
# **4.2 Employment Effects**

The results from the audit experiment reveal three main findings. First, they show that reference letters are valuable in principle: for the _same_ individual, employers more often call back an applicant that attached a reference letter (on average, unconditional on content). Second, this effect is stronger for individuals with higher numeracy and literacy scores on an aptitude test. Third, the content of the letters is informative: employers’ assessment is correlated with an objective measure of ability and higher referee ratings (unless deemed implausibly good) increase the probability of callback from employers.

While novel and informative, the results from Experiment 1 share the main limitations of most audit studies (e.g. Bertrand and Mullainathan, 2004). Specifically, the main outcome is limited to employers’ callback, as opposed to actual employment. Also, the audit framework ignores potential changes in job seekers’ search strategy, given that applications are submitted by the researchers.

## **4.2.1 Effect of reference letters on employment**

Our Experiment 2 allows us to address these limitations and provide a more general contribution. As explained above, Exp. 2 uses a different sample of job seekers and encourages a random half of them to obtain a reference letter. We can then follow their job search behavior and employment outcomes. To test whether the letters increase firm responses and employment when used by job seekers, we estimate the following model: The dependent variable _yij_ is measured for individual _i_ residing in location _j_ . We focus on three key outcomes: (i) number of applications submitted, (ii) job interviews in the last four weeks, and (iii) employment status. _Ti_ captures whether participants were assigned to the treatment group that received the encouragement to obtain a letter. In order to increase precision we control for the baseline value _yij_<sup>_bs_ofoutcomes.Toaccountfor</sup> differences in firm demand across space, we control for location fixed effects _λj_ . Robust standard errors are computed at the individual level. Since the audit study suggest that the effects of reference letters may differ by gender, we also estimate specification 6 separately for women and men.

Table 5 shows the results. Columns 1 to 3 report the intent to treat (ITT) effects after 22
three months. Columns 4 to 6 report the local average treatment effects (LATE) using the random encouragement assignment to instrument for the take-up of reference letters.<sup>18</sup> The coefficient estimates in Table 5 are very much consistent with the main findings from the audit study shown in Table 1. In the pooled sample (Panel A), coefficients on – both the number of applications submitted and on employment outcomes are sizable – LATE estimates range between 20% and 30% of the control mean but not statistically significant. Panel B and C confirm there is important treatment effect heterogeneity by gender: after three months, women in the treatment group submit more applications and are significantly more likely to receive interviews and find employment.

Employment effects are large in magnitude: 5.7 percentage points for ITT estimates (col. 3) and 11.7 p.p. for LATE estimates (col. 6), effectively doubling employment rates for the group of compliers. Coefficients for men are close to zero and insignificant. We can reject that employment effects for women and men are equal at the 10 percent level.

While the magnitudes of the effects are very high, the estimates are relatively imprecise. This warrants some caution in the interpretation of these results. However, it is remarkable that treatment effects estimated from separate experiments, and on different outcomes, point in the same direction. Overall, the combined evidence from the Experiments 1 and 2 show that reference letters can improve employment outcomes. In partic– ular, reducing information asymmetries through reference letters or other interventions – may improve equity by leveling the playing field for women in labor markets.

> 18We do not report results after five weeks as they are generally small and insignificant. This is because the follow-up period is too short for effects to manifest. It takes on average about 3 weeks to obtain reference letters, with males and females equally likely to report having attempted to get a letter. Qualitative evidence also suggests that participants waited for the letter templates to be completed before applying for certain jobs. Finally, a non-negligible number of participants report that it takes them longer than weeks to obtain a reference letter.

23
Table 5: Effect of Reference Letters on Employment (3 months)

||**Intent **|**to Treat **|**Efects**|**Local Ave**|**rage Treat**|**ment Efects**|
|---|---|---|---|---|---|---|
||(1)<br>Application|(2)<br>Interview|(3)<br>Employment|(4)<br>Application|(5)<br>Interview|(6)<br>Employment|
||||**Panel A: **|**POOLED**|||
|Reference Letter|0.660|0.072|0.020|1.336|0.147|0.037|
||(0.426)|(0.046)|(0.022)|(0.857)|(0.092)|(0.046)|
|_R_<sup>2</sup>|0.222|0.051|0.015|0.222|0.046|0.008|
|N|997|996|1033|997|996|1033|
|Control Mean|3.975|0.675|0.130|3.975|0.675|0.130|
||||**Panel B: **|**FEMALE**|||
|Reference Letter|1.051|0.130<sup>_∗∗_</sup>|0.057<sup>_∗_</sup>|2.249|0.280<sup>_∗∗_</sup>|0.117<sup>_∗_</sup>|
||(0.702)|(0.059)|(0.032)|(1.522)|(0.125)|(0.068)|
|_R_<sup>2</sup>|0.267|0.063|0.029|0.242|0.050|0.001|
|N|501|506|528|501|506|528|
|Control Mean|3.842|0.534|0.117|3.842|0.534|0.117|
||||**Panel C**|**: MALE**|||
|Reference Letter|0.118|0.014|-0.015|0.553|0.027|-0.032|
||(0.431)|(0.071)|(0.032)|(0.868)|(0.135)|(0.062)|
|_R_<sup>2</sup>|0.282|0.042|0.021|0.232|0.041|0.020|
|N|491|492|510|491|492|510|
|Control Mean|4.130|0.862|0.157|4.130|0.862|0.157|
|p-value: _βfem_ =_βmale_|0.368|0.241|0.090||||

Notes:<sup>_∗_</sup> _p <_ 0 _._ 10,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01. Results presented in Column 1-3 are intent to treat estimates. Results in Column 4-6 are treatment on the treated estimates, using the encouragement assignment as an instrument for take-up. All regressions control for covariates. Panel A reports estimates from Specification 6 for the full sample. _Application_ and _Interviews_ measures the number of applications submitted and job interviews in the last four weeks, respectively. The number of applications and interviews are winsorized at the 1% level to account for outliers. Employment is an indicator variable denoting if people are in paid employment or self-employed. Panel B and C estimate results separately for women and men.

# **4.2.2 Women use letters more often**

The design of Experiment 2 also allows us to explore job seekers’ search behavior in response to treatment. In particular, we can investigate the usage of reference letters. As mentioned in Section 3.2, participants in Exp. 2 were informed about an open vacancy and asked to submit their application material if interested. We estimate the following specification: 24
(7) where we use two outcome measures: (i) a dummy capturing whether a job seeker _i_ residing in location _j_ submits an application and (ii) a dummy measuring whether they submit a reference letter as part of the application.

Column 1 of Table 6 shows that participants in the treatment group are not more likely to submit applications in response to our email, while Column 2 shows that the effect on applications for women is positive and of relatively large magnitude, although insignificant. For those who did send an application, we can investigate the documents they submitted. Column 3 shows that the share who submits a reference letter is significantly larger in the treatment group. In the control group, only 1.1% submit a letter, confirming that reference letters are nearly absent in the labor market we investigate. This share increases in the treatment group: 8% of all participants submit it as part of the application. This confirms that our intervention has a real impact on job seekers’ behavior, consistent with the results from self-reports.<sup>19</sup> Table 6: Application Material Submitted

||Y=Submi|t Application|Y=Atta|ch Reference|Letter|
|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|
|Reference Letter|-0.001|-0.023|0.069<sup>_∗∗_</sup>|0.007|-0.000|
||(0.022)|(0.033)|(0.029)|(0.030)|(0.006)|
|Female|0.011|-0.017|0.038|-0.018|-0.003|
||(0.023)|(0.032)|(0.029)|(0.021)|(0.004)|
|Ref Let x Female||0.047||0.113<sup>_∗_</sup>|0.021<sup>_∗∗_</sup>|
|||(0.045)||(0.058)|(0.010)|
|_R_<sup>2</sup>|0.006|0.017|0.072|0.091|0.014|
|N|1141|1141|184|184|1141|
|Control Mean|0.163|0.163|0.011|0.011|0.002|
|Sample|full|full|application|application|full|

Notes:<sup>_∗_</sup> _p <_ 0 _._ 10,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01. Standard errors (reported in parentheses) are clustered at the applicant level. Outcomes are binary measures of whether job seekers submit an application (1-2) and whether they attach a reference letter (3-5). Column 3 and 4 restrict the sample to job seekers who submit an application.

19 Slighlty more than 18% of those who obtained a letter attach it to their application. This compares to about 37% of job seekers who report to have used it in the survey. The discrepancy is most likely a result of asking job seekers to submit the material via email, as many job seekers in this market do not have easy access to scanners. In fact, qualitative evidence shows that a larger share of job seekers indeed used the letter in conventional job search channels.

25
Table 6 also reveals a large difference in the usage of reference letters across gender. Women are much more likely than men to attach it as part of the application (Col. 4, 5). Female participants may be more likely to use reference letters if they believe that firms are more uncertain about their skills (a belief that would be consistent with the evidence shown in Section 4 above). This, in turn, can help explain the large and significant employment effects for women reported in Table 5.

# **4.3 Why are Reference Letters Not More Widely Used?**

The previous sections show that both job seekers and firms benefit from reference letters. This raises the question of why reference letters are almost absent in this market. Our analysis rules out two of the most obvious explanations by showing that (i) reference letters contain additional information and (ii) employers use them to update beliefs. This section discusses additional potential explanations on the part of previous employers, hiring firms and job seekers.

## **4.3.1 Previous employers and hiring firms**

We ask job seekers to bring all their application documents to the initial meeting at the labor center. We find that among job seekers with previous work experience, only about 4% have a reference letter from a former employer. When probed, 86.4% of job seekers report that they “Did not ask”, while only 3.1% report that they asked but the employer refused (Table A.7). It is however possible that many job seekers did not ask because they correctly predict that employers would not be willing to write a letter. We can exploit results from our encouragement design to test this hypothesis. Five weeks after the treatment, 56% of job seekers report that they have tried to obtain a letter. Of this group, 73.6% succeeded. Among those that tried, only 4.1% report that they failed to obtain a letter because the employer refused.

Interviews with hiring managers further shows that they recognize that job seekers do not have any bargaining power to request letters. Firms therefore do not require applicants to submit letters.

26
# **4.3.2 Job seekers**

Why do job seekers not request reference letters from employers? We report here the results from Experiment 3, which is designed to test the relative importance of the cost and perceived benefits of obtaining letters. As explained in Section 3, a sub-group of job seekers previously encouraged to obtain a letter receives a reminder on how to return it. Participants receiving this reminder were randomized into three groups. The control group received only the reminder, while the other two groups received either information on the returns to having a reference letter or a monetary incentive. We estimate the following specification: The outcome _yij_ is a binary measure of whether individual _i_ residing in location _j_ returned the reference letter. We report estimates with and without controlling for covariate vector _Xi_ . To account for differences across space, we control for location fixed effects _λj_ .

Table 7 shows the estimated coefficients. Pooling the information and monetary incentive treatment groups, we find a statistically significant increase in the share of people who obtain a letter of 7.4 percentage points (Column 2). When we estimate the effect of each treatment arm separately, we find that the information treatment effect is 12.6 p.p. and statistically significant, while the effect of the monetary incentive is much smaller (4.5 p.p.) and statistically indistinguishable from 0 (Column 4). We can reject that treatment effects are identical at the ten percent significance level. Overall, the results from Experiment 3 show that job seekers may underestimate the potential benefits of reference letters. This, in turn, could help explain the low usage of letters in this market. One of the reasons why job seekers would come to believe that reference letters are not beneficial is that the type of letter in circulation at baseline may in fact be of lower value. Clearly, the effectiveness of the reference letter depends on the noisiness of the signal relative to the resume. Reviewing a total of 30 reference letters collected from job seekers in our sample at baseline reveals that existing letters in the market are generally of low quality. The majority of letters lack information on the workers’ position (48% include this information), responsibilities (38%), skills (28%) duration of employment (48%), and reason for termination of employment (18%). In addition, only 48% of letters are signed and only 56% provide contact information. If job seekers are using reference letters that are both less informative and credible, they may incorrectly infer that all letters are 27
Table 7: Take up Experiment

||(1)|(2)|(3)|(4)|
|---|---|---|---|---|
|Pooled Treatment|0.0753<sup>_∗_</sup>|0.074<sup>_∗∗_</sup>|||
||(0.040)|(0.041)|||
|Information|||0.128<sup>_∗∗_</sup><br>(0.053)|0.126<sup>_∗∗_</sup><br>(0.052)|
|Money|||0.040<br>(0.043)|0.045<br>(0.0433)|
|_R_<sup>2</sup>|0.149|0.169|0.157|0.175|
|N|438|437|438|437|
|Mean Dependable|0.210|0.210|0.210|0.210|
|Control Variables|N|Y|N|Y|
|p-value: _βInf_ = _βMon_|||0.077|0.098|

Notes:<sup>_∗_</sup> _p <_ 0 _._ 10,<sup>_∗∗_</sup> _p <_ 0 _._ 05,<sup>_∗∗∗_</sup> _p <_ 0 _._ 01 Column 1 and 2 pool Information and Money. The control group received message reminding them of how to return the letter.

In-depth interviews with a sample of 28 hiring firms provide further support for this explanation. About 73% of hiring managers report that our reference letter template is more effective than other reference letters they receive. The most frequently cited reasons are that the template provides information on specific skills (55%) and that it is more clearly structured (32%). In addition, the rubric form offers _less ambiguous_ presentation of the assessment than a reference letter in paragraph form. This may particularly benefit women as previous research documents that candidates who are perceived to be similar by the predominantly male hiring managers receive more favorable evaluations (Cardy and Dobbins, 1986).

Asked for reasons that make the template _less_ effective than other letters, managers point to the lack of a firm letter head or stamp (45%) and that letters are too positive (14%). This corroborates our experimental findings documenting the importance of credibility and suggests that modifications in the design of the letter may further increase its effectiveness.

# **5 Conclusion**

Technology has drastically reduced information asymmetries across many markets: online labor market platforms require firms to provide public evaluations of employees’ perfor28
mance and offer workers the option to take tests to certify their skills. Services like LinkedIn offer an easy way to communicate credentials, work experience and even endorsements from former co-workers and employers. These professional network sites also identify common connections than can serve as informal references. Yet, large parts of the global labor force is working in markets that have not been affected by these changes.

Our study investigates the role of information asymmetries in one such market: the lowskill sector in South Africa. We document that information asymmetries are prevalent in this market and employers struggle to identify high-ability job seekers. We find that a simple intervention – encouraging job seekers to obtain a standardized reference letter – from a former employer can lead to improvements in firms’ ability to select job seekers of higher ability from the pool of applicants. Women, who are excluded from many informal referral networks in South Africa, especially benefit from reference letters. Female participants who obtained letters are significantly more likely to receive job interviews and their employment rate doubles after three months. This demonstrates that reducing information asymmetries can improve equity in labor markets.

While our study looks at the effects of reference letters in a static framework, reducing information asymmetries may also have dynamic effects. Similar to other developing countries, South Africa suffers from low quality of education, which limits the use of educational credentials to screen job seekers. This has adverse dynamic effects: if a high school certificate loses its signaling value, youths may be less motivated to study or graduate. Likewise, if workers are employed on temporary contracts and their job performance is not revealed to the market, returns to exerting effort are lower. Our results suggest that reference letters have the potential to provide a powerful incentive to workers. Reducing information asymmetries may therefore have positive effects on productivity beyond the diminished frictions in the matching process explored in this study. Yet, not having a reference letter may also pose a barrier for new labor market entrants as letters enhance firms’ ability to screen applicants with job experience relative to entrants. This could lead to inefficiently low hiring of people without work experience (Pallais, 2014). Quantifying these dynamic effects remains the work of future research.

29