> Using matched sample
analyses and controlling for a rich set of job and applicant attributes found in applicants’ resumes and LinkedIn profiles, we find that women are 9-10% more likely to receive a callback
compared to men, whereas Black, Hispanic, and Asian applicants are 8-13% less likely to receive
a callback compared to White applicants. These outcome gaps do not cancel-out in the later
stages, as female and White applicants are more likely to receive an interview and offer

How accurate this is, however, depends a lot on how well controlling for applicant attributes worked.

> An obvious set of confounders
is the applicant’s objective qualifications such as years of experience, educational attainment, and
field of study, all of which affect the outcome of an application. It is widely known that many of
these attributes differ across demographic groups – for example, women are less likely to major in
STEM subjects, Asian Americans are more likely to have graduate degrees (Camilie Ryan and Kurt
Buaman 2015). To account for these confounds, we control for total years of experience, average
tenure, educational attainment (associate or less, bachelors, masters, and doctorate), field of study,
and rank of the university attended (Top 10, 21-50, 51-10, ). For experience controls, we use the total number of years of experience at the time of application parsed from resume text. For average
tenure, we divide the total years of experience by the number of jobs held. For university rank, we
parse the Education section of the applicant’s LinkedIn profile and join this against the U.S News
Global University Rankings list. If an applicant attended multiple universities, we take the lowest
rank. For the field of study, we parse the Education section of the applicant’s LinkedIn profile
and bucket them into one of the following categories: Technical – mathematics, computer science,
engineering, economics, etc. Business – business administration, finance, accounting, marketing,
etc. Law – law and legal studies. Science – natural sciences such as biology, chemistry, etc. Other
– all other majors.

> An applicant’s professional and social network is another important signal that employers use
to screen applicants (Fernandez and Weinberg 1997; Sterling 2014). Since one’s network tends
to be demographically homogeneous, the effect of gender and race could be confounded by these
affiliations. We control for this in two ways. First, we use a Referral indicator from the ATS,
which indicates whether an applicant has a referral from an existing employee of the firm. Second,
we identify whether an applicant has worked at the company’s talent competitor. We identify a
company’s talent competitors by taking the top 10 companies from which its current employee
pool comes from based on all of LinkedIn data. For example, to identify Company A’s talent
competitors, we first search for all the employees of Company A using all of LinkedIn data. Once
these employees are identified, we look at the previous company these employees worked at before
joining Company A. We then aggregate these previous companies by count, and take the top 10
companies from which Company A’s current employee pool comes from.

> Finally, an applicant’s skills, previous job responsibilities, and fit for the job to which they
applied are perhaps the most important factors in determining the success of an application. We
operationalize this using a text-analytics method called Word2Vec to measure the similarity between
skills and competencies listed in the applicant’s resume and the job description (Mikolov et al.
2013). To do so, we first train a Word2vec model on a corpus of resumes. Using this model, we
transform each document (i.e resumes and job descriptions) into a vector representation based on
skills listed in each document, and measure the cosine similarity between the resume vector vR and
job description vector vJ . The higher the cosine similarity between the job description and resume vector, the better the fit. This type of approach is often used in automatic application screening
tools

I'm skeptical that this actually captures what hiring managers or recruiters care about when looking at resumes, which means I'm not sure I trust the callback numbers. Submitting identical resumes with different demographic characteristics seems like a much more appropriate experimental approach here?

As for whether candidates receive an offer, they don't have anything here where they have actually evaluated the candidates' skills. I've given over 200 technical interviews, and resumes are just not that good a predictor of technical competence. It really doesn't seem to me like they have good enough controls to run this as a correlational experiment.