Retrieving Skills from Job Descriptions: A Language Model Based Extreme Multi-label Classification Framework (2020)

Abstract: We introduce a deep learning model to learn the set of enumerated job skills associated with a job description. In our analysis of a large-scale government job portal mycareersfuture.sg, we observe that as much as 65% of job descriptions miss describing a significant number of relevant skills. Our model addresses this task from the perspective of an extreme multi-label classification (XMLC) problem, where descriptions are the evidence for the binary relevance of thousands of individual skills. Building upon the current state-of-the-art language modeling approaches such as BERT, we show our XMLC method improves on an existing baseline solution by over 9% and 7% absolute improvements in terms of recall and normalized discounted cumulative gain. We further show that our approach effectively addresses the missing skills problem, and helps in recovering relevant skills that were missed out in the job postings by taking into account the structured semantic representation of skills and their co-occurrences through a Correlation Aware Bootstrapping process. We further show that our approach, to ensure the BERT-XMLC model accounts for structured semantic representation of skills and their co-occurrences through a Correlation Aware Bootstrapping process, effectively addresses the missing skills problem, and helps in recovering relevant skills that were missed out in the job postings. To facilitate future research and replication of our work, we have made the dataset and the implementation of our model publicly available.

## introduction finding prospective employees with the correct set of skills required for a job is an important aspect of the recruitment process @xcite . in the current digital age, many recruiters seek to find suitable candidates through multiple channels -e.g., online job portals, professional networks -as well as traditional avenues, such as word of mouth and mass media @xcite . in this work, we focus on online job portals (e.g., linkedin, glassdoor), which have emerged as critical players in the job market with millions of active users. despite the reach of job portals, finding candidates with the right skill fit still remains challenging. these portals often receive thousands of applications, among which fewer than 10% have the appropriate skills @xcite . this bottleneck has attracted both academic and industry researchers from social science and machine learning communities to build automated means to better organize the information present in job descriptions @xcite ). an approach facilitating the matching of jobs to candidates has been to associate a set of enumerated skills from the job descriptions (jds). candidate job-seekers can also list such skills as part of their online profile explicitly, or implicitly via automated extraction from résumés and curriculum vitae (cvs). such categorical skills can then be used to induce a match score between jds and candidate profiles. we examine the challenge of inferring appropriate job skills from jds (cf., fig. 1 ). jds consist of a textual description of the job requirements and a list of required skills. the latter helps in indexing jds to facilitate ease of search through facet navigation. we note that there exist crucial skills that are mentioned in job requirements (in the job description), but are not listed as required skills (skill labels). job requirements: looking for a dynamic individual keen to join a growing organization in the fast-paced and expanding supply chain software industry. as singapore goes through the digital revolution, the individual will play a paramount role in developing and upgrading our saas tools and valued added services for our clients. responsibilities: ## method we introduce a neural xmlc framework, bert-xmlc. the architecture is inspired by the models proposed in @xcite and @xcite . the model comprises two major components: 1. text encoder: the model takes the textual jd (t) as input, consisting of n words (@xmath0, w 2 , ..., w n }) . we use the pre-trained bert base model to encode the text into low-dimensional dense vectors @xcite . internally, it first looks up the wordpiece embeddings @xcite , then applies a pre-trained bi-directional transformer-based language model to yield a hidden representation of the input. we consider the encoding of the [cls] token as the representation of the textual input. ## experiments we have collected data from the singaporean government website, mycareersfuture.sg with permission consisting of over 20, 000 richly-structured job posts having 16 informative fields about the details and the current status of the advertisements. the statistics of the dataset are shown in to the best of our knowledge, there is no relevant large-scale publicly available dataset comprising job descriptions with their corresponding list of required skills at the time of conducting the experiments. for our task, we consider concatenation of "roles & responsibilities" and "job requirements" fields as the textual description, and "required skills" as the set of target discrete labels. we perform the preprocessing operations on the text, inclusive of lower-casing, stopword removal, and rare word removal. we split the dataset into assigned training, validation and testing dataset with an 80:10:10 proportion. to aid the reproduction of our results and to encourage further research in this domain in general, we have made both the code and the dataset publicly available. we might provide updated datasets in the future as more jds are continuously being posted. ## discussion: on implicit versus explicit skills we can think of job descriptions and their required skills as two different forms of the same underlying job. the job's manifestation in both forms ideally should corroborate and support each other, but sometimes maybe incongruous. this results in incomplete ground truth in both forms. our bert-xmlc+cab model is architected to overcome this challenge when the two forms do not reinforce each other as expected. we now analyze how required skills are demographically represented in the jd explicitly (present as substrings in the jd) and implicitly (absent in the jd, but likely inferrable from context). we assess the level of implicitness in the required job skills. as an extreme, if all skills are explicitly mentioned in the jd, our task is easy: it is trivial to list the required skills, as they are all explicit substrings in the jd -a simple string matching algorithm would suffice. at the other extreme, if all required skills are implicit, our task is challenging: every skill needs to be inferred from the context given by the jd. the complexity of the task increases with presence of more implicit skills in the required skill set. figure 4 shows a macroscopic view of this phenomenon in our dataset through a skill histogram. each of the 2,548 skills in the dataset is accounted in the figure. we place the skill in one of 30 bins by the percentage of times it occurs as an implicit skill in the jds where it is required. the average shows that 86.13% of skills are implicit. as the figure is strongly biased towards implicit skills (over 1,500 skills are always implied, never explicitly mentioned), we can see the problem is difficult, and extraction-based models hopelessly fail. abstractive or generalising approaches such as our model are required for implicit skills. to study this in more depth, we also propose and plot (figure 5 ) the following metrics foot_4 that capture model performance in a manner sensitive to the implicit and explicit status of skills: here, again, the results are consistent: increasing model complexity (leftmost model to rightmost model among the eight models compared) improves both instance-level and macro-level recall. more importantly, these results are also consistent for our bert-xmlc+cab model, which performs best in all three metrics. improvement is most pronounced with the implicit (eim and riim) metrics, clearly showing the improved abstractive generalisation capability of our model. note also the eim measure of bert-xmlc+cab tallies to over 100%, suggesting that the model captures a larger number of explicit skills than is actually mentioned in the skill labels (in job descriptions). ## conclusion we address the prediction of required skills from job descriptions (jds). although job descriptions and their associated required skills can be thought of as two views of the same underlying job, our analysis reveals that many required skills are implicitly signaled. we develop an extreme multi-label classification method that utilizes bert base within a transformer model to predict the required skills from a textual job description. importantly, we propose a novel bootstrapping approach that exploits the underlying natural language representation of skills and their co-occurrence relationships with other skills. we perform experiments on a large real-world dataset from a popular singaporean government job portal, mycareersfuture.sg. we show that our model outperforms recent competitive baselines, especially in recalling such implicit skills.