Attacks against Ranking Algorithms with Text Embeddings: A Case Study on Recruitment Algorithms (2021)

Abstract: Recently, some studies have shown that text classification tasks are vulnerable to poisoning and evasion attacks. However, little work has investigated attacks against decision-making algorithms that use text embeddings, and their output is a ranking. In this paper, we focus on ranking algorithms for the recruitment process that employ text embeddings for ranking applicants’ resumes when compared to a job description. We demonstrate both white-box and black-box attacks that identify text items that, based on their location in embedding space, have a significant contribution in increasing the similarity score between a resume and a job description. The adversary then uses these text items to improve the ranking of their resume among others. We tested recruitment algorithms that use the similarity scores obtained from Universal Sentence Encoder (USE) and Term Frequency–Inverse Document Frequency (TF-IDF) vectors. Our results show that in both adversarial settings, on average the attacker is successful. We also found that attacks against TF-IDF are more successful compared to USE.

## introduction recently some studies have shown that text classification tasks are vulnerable to poisoning and evasion attacks @xcite @xcite . for example, some works have shown that an adversary can fool toxic content detection @xcite , spam detection @xcite and malware detection @xcite by modifying some text items in the adversarial examples. a recent work @xcite showed that applications that rely on word embeddings are vulnerable to poisoning attacks, where an attacker can modify the corpus that the embedding is trained on, i.e., wikipedia and twitter posts, and modify the meaning of new or existing words by changing their locations in the embedding space. in this work, however, we investigate a new type of attack, i.e., rank attack, when the text application utilizes text embedding approaches. in this attack, the adversary does not poison the training corpora but tries to learn about the embedding space, and how adding some keywords to a document can change the representation vector of it, and based on that tries to improve the ranking of the adversarial text document among a collection of documents. as a case study, we focus on ranking algorithms in a recruitment process scenario. recruitment process is a key to finding a suitable candidate for a job application. nowadays, companies use mlbased approaches to rank resumes from a pool of candidates @xcite . one naive approach for boosting the ranking of a resume can be adding the most words and phrases to his/her resume from the job description. however, this is not the best approach all the time, because: first, the attacker must add a specific meaningful word to their resume. for example, they cannot claim they are proficient in some skills while they are not. second, adding any random keyword from the job description to the resume does not always increase the similarity between the resume and job description. instead, we demonstrate that the adversary can learn about the influential words and phrases that can increase the similarity score and use this knowledge and decide about the keywords or phrases to be added to their resume. therefore, in this work, we investigate: (1) how can an adversary utilize the text embedding space to extract words and phrases that have a higher impact on the ranking of a specific document (here resume)? and, (2) how can an adversary, with no knowledge about the ranking algorithm, modify a text document (here resume) to improve its ranking among a set of documents? ## system and threat model the recruitment process helps to find a suitable candidate for a job application. based on glassdoor statistics, the average job opening attracts approximately 250 resumes (team), and a recent survey found that the average cost per hire is just over $4,000 @xcite . to limit work in progress many companies use machine learning to have more efficiency in ranking resumes.a successful approach to rank resumes is calculating similarity among resume and job description leveraging nlp techniques (excellarate). in this study, the recruitment algorithm is based on a ranking algorithm. this algorithm takes input from the resume and the job description and finds similarities based on their matching score. we use universal sentence encoder (use) as the text embedding approach to vectorize each resume and the job descriptions. as it is shown in figure 1 , our algorithm includes three steps: (1) all resumes and the job description are vectorized using use text embedding approach; (2) the similarity between each job description and an individual resume is computed. cosine similarity is used as a metric to compute the similarity score; and (3) the resumes are sorted based on their similarity scores computed in the second step in such a way that the resume with highest similarity score appears at the top of the list and on the other hand the resume with least similarity score appears at bottom. threat model. adversaries have a huge motivation to change the rankings provided by some algorithms, when they are used for decision-making, e.g., for recruitment purposes. adjusting a resume based on the job description is a well-known approach for boosting the chance of being selected for the next rounds of recruitment @xcite . in this work, we show how an adversary can automatically generate adversarial examples specific to a recruitment algorithm and text embeddings. we define this attack as a rank attack, where the adversary adds some words or phrases to its document to improve its ranking among a collection of documents. we consider white-box and black-box settings. in a white-box setting, we assume the attacker has complete knowledge about the ranking algorithm. in a black-box setting, however, the attacker has no knowledge of the recruitment process but has limited access to the recruitment algorithm and can test some resumes against the algorithm. tf-idf. tf-idf or term frequency-inverse document frequency is a widely used approach in information retrieval and text mining. tf-idf is computed as a multiplication of tf, the frequency of a word in a document, and idf, the inverse of the document frequency. ## data collection we collected 100 real public applicant resumes from linkedin public job seeker resumes, github, and personal websites. to have an equal chance for applicants and make our experiences closer to real world recruitment procedures, we only considered resumes related to computer science. resumes are chosen to be in different levels of education (bachelor, master and ph.d. with equal distribution), skills, experiments (entry level, mid level, senior level), and gender (around fifty percent men and fifty percent women). we also developed a web scraper in python to extract computer science jobs from the indeed website. 1 our dataset includes over 10,000 job descriptions, extracted randomly from cities in the usa. we randomly chose 50 job descriptions and used them in our experiments. for black-box settings, our neural network architecture needs a huge amount of data to be trained on. to have enough training sets we augmented our data for our models. for a simple setting model, we created 5,000 records by concatenating 100 resumes to 50 selected job descriptions to augment our data for recruitment algorithms. note that the adversary only needs to obtain or create a set of resumes and they do not need to use the job descriptions. for a more complex setting, we split resumes into half, then joined the upper part of each resume to other resumes' lower parts. with this approach, we could maintain the structure of each resume and each resume could have common resume information, such as skills, education, work experince, etc. we did this procedure for all possible combinations and created a database with 10,000 resumes. ## white-box setting and a recruitment in white-box settings, we assume the adversary has knowledge about the recruitment process and the use of universal sentence encoder (use) or term frequency-inverse document frequency (tf-idf) for obtaining the embedding vectors. we propose a 1 https://www.indeed.com/ white-box setting phase 1: finding the best job description keywords phase 2: sorting keywords based on the resume phase 3: adding n keywords to the adversarial resume novel approach which utilizes the knowledge about the text embedding space, here use, to identify keywords in the job description that can be added to the adversarial resume and increase its similarity score with the job description. our approach as it is shown in figure 2 consists of three phases: in phase 1, the adversary tries to identify the important words in job description; in phase 2, the adversary tries to rank the effectiveness of those words based on its own resume, i.e., identifying those words that can increase the similarity between his/her resume and the job description. after identifying the most effective words, then in phase 3, the adversary modifies its resume based on his/her expertise and skill set and decides to add n of those identified words and phrases. note that in this attack scenario the adversary does not need to have any knowledge about the other candidates' resumes. the details of phase 1 and 2 in this attack are depicted in algorithm 1 and algorithm 2, respectively, and we explain them in the following. ## white-box setting and a recruitment algorithm that employs tf-idf we also investigate the effectiveness of the attacks in white-box setting, when the recruitment algorithm uses tf-idf vectors to compute the similarity between resumes and job descriptions. the attack is the same. the only difference is in the recruitment algorithm. since the adversary has the knowledge that the recruitment algorithm uses tf-idf vectors, then they compute tf-idf vectors instead of use vectors. ## black-box setting in the back-box setting, an adversary does not have access to the model specification but has access to the recruitment algorithm as an oracle machine, which receives an input, e.g., here a resume, and then generates a response, e.g., here accept/ reject, or the ranking among all the resumes in its pool. therefore, the adversary is able to send some resumes to the recruitment algorithm and obtain the output for them, which then can be used to train a neural network model. this model then identifies the features (i.e., keywords) that help resumes to be accepted or obtain better ranking. the adversary then can boost its chance of acceptance by adding some or all of the identified keywords. the input vector of our neural network model is the resumes, and the output is a vector that demonstrates the best keywords. in our experiments, we examine attacks against two recruitment algorithms: a binary classifier, which label a resume as accept or reject, and a ranking algorithm that provides a ranking based on similarity scores between the job description and the resumes, where they are vectorized by use embeddings. the attacks against both algorithms have two phases: phase 1: pre-processing which prepares the data for the neural network, and phase 2: a neural network model. ## limitations and future work this study examines the recruitment algorithms that use text embeddings and a similarity function to rank the resumes. however, examining more complex ranking algorithms is left for future work. in addition, the analysis can be expanded to other types of text embeddings. in the black-box setting, we did not investigate the trade-off between the number of queries that the adversary needs to send to the recruitment algorithm to create the training set, and the performance of the attack. it might be possible that the adversary can obtain good results using even a smaller training set. we will explore this in the future. we also have not studied the types of words that make it easy to game the system. in the future, we will examine the identified words in experiments in terms of their frequency, semantics, and other linguistic properties. moreover, investigating the graybox attack remains for future work, when the attacker has this knowledge that the recruitment algorithm uses some kind of text embedding but does not know its type. this attack setting can help us to investigate the use of one text embedding approach for attacking a ranking algorithm that uses another text embedding approach. in other words, we can examine the transferability of one approach for another approach. ## conclusion in this project we found that an automatic recruitment algorithm, as an example of ranking algorithms, is vulnerable against adversarial examples. we proposed a successful adversarial attack in two settings: white-box and black-box. we proposed a new approach for keyword extraction based on use. we observed the majority of resumes have significant rank improvements by adding more influential keywords. finally, in a black-box setting, we proposed multilabel neural network architecture to predict the proper keyword for each resume and job description.