Fine-Grained Extraction and Classification of Skill Requirements in G erman-Speaking Job Ads (2022)

Abstract: Monitoring the development of labor market skill requirements is an information need that is more and more approached by applying text mining methods to job advertisement data. We present an approach for fine-grained extraction and classification of skill requirements from German-speaking job advertisements. We adapt pre-trained transformer-based language models to the domain and task of computing meaningful representations of sentences or spans. By using context from job advertisements and the large ESCO domain ontology we improve our similarity-based unsupervised multi-label classification results. Our best model achieves a mean average precision of 0.969 on the skill class level.

## introduction how skill demand evolves over time in the labor market has always been a main research question in social sciences. research has however been hampered by the following limitations: skills were mostly measured on the supply side (what workers bring, not what employers ask for) and only on an aggregated level (by occupations) and/or crosssectional (one data point in time). furthermore, most data focused on a selection of skills, since defining and measuring skills is difficult (biagi and sebastian, 2020). job advertisement data can help to overcome such shortcomings by providing timeseries measurements on the job level, including all labor market skill requirements (buchmann et al., 2022a). not surprisingly, social science has thus lately shown great interest in applying text mining methods to job advertisements (job ads in the following). our main goals are to, first, extract spans of text in swiss german-speaking job ads that specify workers' skill requirements: specifically, educational requirements, work experiences or skills, and language competences. second, we classify the extracted spans onto the large, fine-grained european skills, competences, qualifications and occupations ontology (esco). @xcite third, we show the value of the data-driven extraction results in evaluations and initial social science analyses. our general idea is to use, in an unsupervised approach, the semantic similarity between ontological concepts and text spans in job ads for fine-grained classification of job ad skills to the esco skill ontology. we rely on state-of-the-art pre-trained transformer-based language models (foundational models) and experiment with adaptations to the job ad domain and to the task of computing the semantic similarity on sentence or span level. additionally, we assess different methods to exploit the textual content and terminological richness of the esco ontology for fine-tuning the foundational language models. and, we show how providing additional textual context from the job ads and/or the ontology improves the similarity scores between skill requirement spans in job ads and their corresponding concepts from the ontology. our contributions include a definition of skill requirement mention types and annotation guidelines for fine-grained extraction, and an exploration of nlp methods for improving semantic similarity measures for matching job ad text snippets with esco terminology. we contribute further sentence-level language representation models that are adapted to the job ad domain and skill-related expressions, and we incorporate terminological variability from a large ontology into the model. section 2 discusses related work. section 3 describes our data. our approaches, experiments, and results for extracting skills are explained in section 4, and for classification in section 5. section 6 shows initial sociological analyses on the extracted data. section 7 summarizes our main findings and directions for future work. ## downstream sociological analysis labor market changes: it is commonly believed that digital technologies have changed the demand for skills to perform tasks in the labor market in the past decades. recent literature points to the importance of new skills entering jobs and altering the required skill combinations (acemoglu et al., 2022). it also emphasizes that most of the changes in skill demand take place within and not across occupations (bisello et al., 2019; freeman et al., 2020). according analyses require time series data on skill demand at the job level that includes valid measures of all skills required in the labor market. such data has been, however, extremely scarce. illustrative analyses: to illustrate the usefulness of our job ads data for social sciences, we present some selected analyses. first, we calculate correlations between occupation-skill matrices that esco provides and those resulting from the sjmm data. at the 1-digit level of the international standard classification of occupations (isco-08) @xcite , for example, the correlation is as high as 0.87, underscoring the validity of our skill extractions. second, we illustrate within-occupation change in skill demand with an example: the evolution of skill requirements in the occupational field of technicians and engineers. to aggregate fine-grained, multi-hierarchical esco skill classes, we used a clustering approach. @xcite the resulting 48 clusters are then applied to the sjmm job level data, generating for each job ad indicators of how strongly the text represents each skill cluster. to keep the picture detailed as well as simple, only three interesting clusters for this occupation are shown in figure 3 . figure 3 confirms -for our example -that .2 .3 .4 .5 .6 mean of summarized term probabilities per job ad 2000 2005 2010 2015 2020 year cluster 16: making patterns, templates and models cluster 41: cad & designing electronic systems cluster 4: teamwork skills required to perform tasks in the occupation 'technicians and engineers' notes: moving averages over 7 years. starting year is 2006. in 2006 the sjmm started to collect job ads from online job boards, what lifted the number of annual cases to a level that is suitable for this type of analysis. the type of required skills changed within occupations over time. skills for making patterns, templates and models were highly required shortly after the turn of the century. across the following 15 years, the demand for these mainly manual and non-digital skills declined. in contrast, the demand for cad and designing electronic systems was nearly nonexistent and then increased sharply. these skills are related to digital technologies and newly entered the occupation. after their entry, also other elements of the required skill combination seem to change, e.g., demand for teamwork skills is increasing (see figure 3 ). this is in line with the literature, which suggests that digital technologies lead to more flexible, team-based settings (autor et al., 2002). ## conclusion our two-step approach of first extracting text spans expressing language skills, experience, and educational requirements, followed by further subdividing these into skill areas, containers, and qualifiers, allowed us to achieve broad coverage of finegrained competency classifications. by grouping skill areas from the same span for transformerbased vector representation, we provide relevant context that helps find appropriate esco ontology concepts for each job ad skill area. for fine-grained classification, our domain and task-specific sbert learning steps boost performance -best models reaching map@10 of 0.969 on class and 0.908 on concept level -and also help deal with more difficult cases encountered in the challenge sample. while infusing terminological variation from the ontology into the model with mnr is by far the most effective, all different pretraining and fine-tuning steps are beneficial to some extent. analyses on low-similarity cases and our gradient-based selection approach showed that similarity values of our best models can be used to select the most relevant ontology concepts and avoid mismatches. in future work, models could be further finetuned with curated task-specific training material (similar to our evaluation data) to improve classification for the most difficult task, experience classification (exp). the next steps in social science analyses could be to assess how required skill combinations evolve within occupations, which occupations shift towards more specialized or diversified requirements, or to which extent the skill requirements of some occupations become more alike.