Rethinking Skill Extraction in the Job Market Domain using Large Language Models (2024)

Abstract: Skill Extraction involves identifying skills and qualifications mentioned in documents such as job postings and resumes. The task is commonly tackled by training supervised models using a sequence labeling approach with BIO tags. However, the reliance on manually annotated data limits the generalizability of such approaches. Moreover, the common BIO setting limits the ability of the models to capture complex skill patterns and handle ambiguous mentions. In this paper, we explore the use of in-context learning to overcome these challenges, on a benchmark of 6 uniformized skill extraction datasets. Our approach leverages the few-shot learning capabilities of large language models (LLMs) to identify and extract skills from sentences. We show that LLMs, despite not being on par with traditional supervised models in terms of performance, can better handle syntactically complex skill mentions in skill extraction tasks.

## introduction skill extraction (se) is a challenging task in the job market domain that involves identifying and extracting specific skills mentioned in job postings, resumes, and other job-related documents. se plays a crucial role in various job market applications, such as matching job seekers with relevant job opportunities or analyzing trends in the job market. prior approaches to se rely on rule-based methods or keyword-matching techniques @xcite . more recent methods, which are considered state-of-the-art, propose to fine-tune language models to solve the task @xcite . however, they heavily rely on manually annotated data, which is prohibitively expensive to collect, especially in this application that requires human resource domain experts. in this paper, we investigate the use of large language models (llms) for se in the job market domain. llms have been trained on massive amounts of text data and have shown great potential in capturing the underlying patterns and semantics of language. the se task can be linked with the more generic entity recognition task in natural language processing (nlp). named entity recognition (ner) is a widely studied task in nlp that involves identifying and classifying named entities in text. the typical approach to the ner problem is to formulate it as a sequence labeling task that assigns each token to a predefined entity-related label. recent methods have attempted to tackle this task using llms, with limited success (e.g., @xcite . first, we propose a review of the datasets for se. we uniformize them and release them as a benchmark for future research in the field. it includes six publicly available datasets covering 4 languages (english, french, german and danish) and various domains and skills categories: sayfullina @xcite , skillspan @xcite , @xcite , @xcite , kom-petencer @xcite and fijo @xcite . we test the ability of llms to solve the tasks on these six datasets using various prompting techniques. we compare two types of task formulations, highlighting their impact on various aspects of the model's behavior. through diagnosing the performance of llms, we identify a taxonomy of errors, quantifying their frequency and supporting it with descriptive statistics on the datasets and the model's predictions, comprising multiple skill mentions that are conjoined together. for example, a job posting might require "ability to develop reporting software and statistical softwares". in this case, the phrase "develop reporting software and statistical softwares" represents a conjoined skill, where multiple skills are combined together. however, the current bio annotation scheme does not capture such complex skill mentions effectively, leading to difficulties in accurately extracting each skill. following this, in our error analysis, we highlight llm behavior that is seen as detrimental using common ner evaluation and annotation schemes, but may be more adapted to real-world settings. ## datasets for skill extraction we gather and uniformize six datasets with the b-i-o annotation scheme, where each word in a sentence is associated with one tag. the "b" tag indicates that the associated word marks the beginning of a span; "i" indicates a word inside a span, and "o" marks words outside a span. each dataset is extracted from job ads from various domains and languages, and some are augmented with finegrained annotations. despite the disparity of the train-dev-test split proportion, we kept the original splits from the authors of each dataset to ensure comparability with previously published results. for each dataset, we describe how they were created, and their content. ## method in this work, we formulate the task of skills extraction as a llms generation task, leveraging llms' emergent in-context learning ability. the llm is provided instructions, a set of demonstrations, and a sample to annotate. the instructions define the expected output format precisely. this format is crucial for ner tasks, as the rigid structure of the annotations and metrics poses a challenge in evaluating the generative output. in the following section, we describe our approaches and strategies to design prompt formats. ## conclusion in this paper, we benchmark and uniformize existing datasets for se in job postings. we conduct in-depth experiments and error analysis to evaluate the ability of llms to solve the task, notably implementing two prompting strategies to adapt llms for the task and a dedicated feedback loop. in line with concurrent work @xcite , llms achieve limited performance for skill extraction relative to supervised methods. moreover, we highlight the limitations of the current se task formulation and evaluation, focusing on the adaptation of the ner sequence labeling task, to the token generation task with which llms are pre-trained. in particular, we list the causes of the most frequent errors in se with gpt-3.5. in the absence of training data, llms struggle to understand what skills are and often extract irrelevant information. additionally, gpt-3.5 tends to split conjoined skills into two, leading to less accurate but more granular skill extractions. in a real-world setting, in particular when se is used as a preliminary step for skill classification in a taxonomy (e.g. @xcite , this behavior would be highly beneficial.