ESCOXLM - R : Multilingual Taxonomy-driven Pre-training for the Job Market Domain (2023)

Abstract: The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-R-large, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-R-large on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.

## introduction the dynamic nature of labor markets, driven by technological changes, migration, and digitization, has resulted in a significant amount of job advertisement data (jad) being made available on various platforms to attract qualified candidates @xcite @xcite . this has led to an increase in tasks related to jad, including skill extraction @xcite @xcite @xcite @xcite @xcite @xcite @xcite , skill classification @xcite , job title classification @xcite @xcite , de-identification of entities in job postings @xcite , and multilingual skill entity linking @xcite . while some previous studies have focused on jad in non-english languages @xcite @xcite , their baselines have typically relied on language-specific models, either using domain-adaptive pre-training (dapt; @xcite or off-the-shelf models. the lack of comprehensive, open-source jad data in various languages makes it difficult to fully pre-train a language model (lm) using such data. in this work, we seek external resources that can help improve the multilingual performance on the jad domain. we use the esco taxonomy @xcite , which is a standardized system for describing and categorizing the skills, competences, qualifications, and occupations of workers in the european union. the esco taxonomy, which has been curated by humans, covers over 13,000 skills and 3,000 occupations in 27 languages. therefore, we seek to answer: to what extent can we leverage the esco taxonomy to pre-train a domain-specific and language-agnostic model for the computational job market domain? in this work, we release the first multilingual jad-related model named escoxlm-r, a language model based on xlm-r large that incorporates data from the esco taxonomy through the use of two pre-training objectives (figure 1 ): masked language modeling (mlm) and a novel esco relation prediction task (section 2). we evaluate escoxlm-r on 9 jad-related datasets in 4 different languages covering 2 nlp tasks (section 3). our results show that escoxlm-r outperforms previous state-of-the-art (sota) on 6 out of 9 datasets (section 4). in addition, our fine-grained contributions in this work, we present and release the following: ## escoxlm-r preliminaries in the context of pre-training, an lm is trained using a large number of unlabeled documents, @xmath0, and consists of two main functions: f encoder (.), which maps a sequence of tokens @xmath1, x 2 , ..., x t ) to a contextualized vector representation for each token, represented as (h 1 , h 2 , ..., h t ), and f head (.), the output layer that takes these representations and performs a specific 1 the code for escoxlm-r is available as opensource: https://github.com/mainlp/escoxlmr . we further release escoxlm-r under an apache license 2.0 on huggingface: https://huggingface.co/jjzha/ esco-xlm-roberta-large . multilinguality both bert and roberta have been extended to support multiple languages, resulting in multilingual bert (mbert; @xcite and xlm-roberta (xlm-r; @xcite . xlm-r was found to outperform mbert on many tasks (e.g., @xcite @xcite due to careful tuning, sampling, and scaling to larger amounts of textual data. because of this, our escoxlm-r model is based on xlm-r large . ## experimental setup skillspan @xcite the job posting dataset includes annotations for skills and knowledge, derived from the esco taxonomy. the best model in the relevant paper, jobbert, was retrained using a dapt approach on a dataset of 3.2 million en job posting sentences. this is the bestperforming model which we will compare against. kompetencer @xcite this dataset is used to evaluate models on the task of classifying skills according to their esco taxonomy code. it includes en and da splits, with the en set derived from skillspan. there are three experimental setups for evaluation: fully supervised with en data, zero-shot classification (@xmath2, and few-shot classification (a few da instances). the best-performing model in this work is @xcite , which obtains the highest weighted macro-f1 for both en and da. in this work, we use setup 1 and 3, where all available data is used. bhola @xcite the task of this en job posting dataset is multilabel classification: predicting a list of necessary skills in for a given job description. it was collected from a singaporean government website. it includes job requirements and responsibilities as data fields. pre-processing steps included lowercasing, stopword removal, and rare word removal. their model is bert with a bottleneck layer @xcite . in our work, the bottleneck layer is not used and no additional training data is generated through bootstrapping. to keep comparison fair, we re-train their model without the additional layer and bootstrapping. we use mean reciprocal rank as the main results metric. sayfullina @xcite this dataset is used for soft skill prediction, a sequence labeling problem. soft skills are personal qualities that contribute to success, such as "team working", "being dynamic", and "independent". the models for this dataset include a cnn @xcite , an lstm @xcite , and a hierarchical attention network @xcite . we compare to their best-performing lstm model. jobstack @xcite this corpus is used for de-identifying personal data in job vacancies on stack overflow. the task involves sequence labeling and predicting organization, location, name, profession, and contact details labels. the best-performing model for this task is a transformer-based @xcite model trained in a multi-task learning setting. @xcite propose to use the i2b2/uthealth corpus, which is a medical de-identification task @xcite , as auxiliary data, which showed improvement over their baselines. gnehm @xcite a swiss-german job ad dataset where the task is information and communications technology (ict)-related entity recognition, these could be ict tasks, technology stack, responsibilities, and so forth. the used dataset is a combination of two other swiss datasets namely the swiss job market monitor and an online job ad dataset @xcite . their model is dubbed jobgbert and is based on dapt with german bert base @xcite . fijo @xcite a french job ad dataset with the task of labeling skill types using a sequence labeling approach. the skill groups are based on the aqesss public skills repositories and proprietary skill sets provided by their collaborators. these skill types are divided into four categories: "thoughts", "results", "relational", and "personal". the best-performing model for this task is camembert @xcite . ## results the results of the models are presented in to evaluate the performance, four different models are used in total: escoxlm-r, the best-performing model originally reported in the relevant paper for the downstream task, vanilla xlm-r large , and an xlm-r large model that we continuously pre-trained using only mlm (dapt; excluding the erp objective) using the same pre-training hyperparameters as escoxlm-r. for more information regarding the hyperparameters of fine-tuning, we refer to appendix c ( english escoxlm-r is the best-performing model in 4 out of 6 en datasets. the largest improvement compared to the previous sota is observed in sayfullina and green, with over 19 f1 points. in 3 out of 4 datasets, escoxlm-r has the overall lower standard deviation. for jobstack, the previous sota performs best, and for kompetencer, xlm-r large (+ dapt) has the highest performance. non-english in 2 out of 3 datasets, escoxlm-r improves over the previous sota, with the largest absolute difference on french fijo with 10.3 f1 points. in the danish subset of kompetencer, xlm-r large (+ dapt) has higher performance than escoxlm-r. next, we will discuss potential reasons for these differences. figure 4 : radar charts of span-f1 performance by span token length. we show the performance of xlm-r large and escoxlm-r on different span lengths, we bucketed the performances of both models according to the length of the spans, up to 10 tokens, and presented the average performance over five random seeds. we did not include error bars in these plots. note that in some plots, there are no instances in certain buckets (e.g., sayfullina with 7-8, 9-10). also, some outer rings only go up to 60 span f1, rather than 100. ## related work to the best of our knowledge, we are the first to internalize an lm with esco for job-related nlp tasks. there are, however, several works that integrate factual knowledge (i.e., knowledge graphs/bases) into an lm. @xcite integrates multiple knowledge bases into lms to enhance their representations with structured, human-curated knowledge and improve perplexity, fact recall and downstream performance on various tasks. @xcite ; @xcite ; @xcite combine lm training with knowledge graph embeddings. @xcite introduces k-adapter for injecting knowledge into pre-trained models that adds neural adapters for each kind of knowledge domain. @xcite introduces dict-bert, which incorporates definitions of rare or infrequent words into the input sequence and further pre-trains a bert model. @xcite introduced a multilingual wikipedia hyperlink prediction intermediate task to improve language model pre-training. similarly, @xcite introduced linkbert which leverages links between documents, such as hyperlinks, to capture dependencies and knowledge that span across documents by placing linked documents in the same context and pre-training the lm with mlm and document relation prediction. ## conclusion in this study, we introduce escoxlm-r as a multilingual, domain-adapted lm that has been further pre-trained on the esco taxonomy. we evaluated escoxlm-r, to the best of our knowledge, on the broadest evaluation set in this domain on 4 different languages. the results showed that escoxlm-r outperformed xlm-r large on job-related downstream tasks in 6 out of 9 datasets, particularly when the task was relevant to the esco taxonomy and context was important. it was found that the improvement of escoxlm-r was mainly due to its performance on shorter span lengths, demonstrating the value of pre-training on the esco dataset. escoxlm-r also demonstrated improved performance on both frequent surface spans and a wider range of spans. overall, this work showed the potential of escoxlm-r as an lm for multilingual job-related tasks. we hope that it will encourage further research in this area.