S kill S pan: Hard and Soft Skill Extraction from E nglish Job Postings (2022)

Abstract: Skill Extraction (SE) is an important and widely-studied task useful to gain insights into labor market dynamics. However, there is a lacuna of datasets and annotation guidelines; available datasets are few and contain crowd-sourced labels on the span-level or labels from a predefined skill inventory. To address this gap, we introduce SKILLSPAN, a novel SE dataset consisting of 14.5K sentences and over 12.5K annotated spans. We release its respective guidelines created over three different sources annotated for hard and soft skills by domain experts. We introduce a BERT baseline (Devlin et al., 2019). To improve upon this baseline, we experiment with language models that are optimized for long spans (Joshi et al., 2020; Beltagy et al., 2020), continuous pre-training on the job posting domain (Han and Eisenstein, 2019; Gururangan et al., 2020), and multi-task learning (Caruana, 1997). Our results show that the domain-adapted models significantly outperform their non-adapted counterparts, and single-task outperforms multi-task learning.

## introduction job markets are under constant developmentoften due to developments in technology, migration, and digitization-so are the skill sets required. consequentially, job vacancy data is emerging on a variety of platforms in big quantities and can provide insights on labor market skill demands or aid job matching @xcite . se is to extract the competences necessary from unstructured text. previous work in se shows promising progress, but is halted by a lack of available datasets and annotation guidelines. two out of 14 studies release their dataset, which limit themselves to crowd-sourced labels @xcite or annotations from a predefined list of skills on the document-level @xcite . additionally, none of the 14 previously mentioned studies release their annotation guidelines, which obscures the meaning of a competence. job markets change, as do the skills in, e.g., the european skills, competences, qualifications and occupations (esco; le @xcite taxonomy (section 3). hence, it is important to cover for possible emerging skills. we propose skillspan, a novel se dataset annotated at the span-level for skill and knowledge components (skcs) in job postings (jps). as illustrated in figure 1 , skcs can be nested inside skills. skillspan allows for extracting possibly undiscovered competences and to diminish the lack of coverage of predefined skill inventories. our analysis (figure 2 ) shows that skcs contain on average longer sequences than typical named entity recognition (ner) tasks. albeit we additionally study models optimized for long spans @xcite , some underperform. overall, we find specialized domain bert models @xcite @xcite perform better than their non-adapted counterparts. we explore the benefits of domain-adaptive pretraining on the jp domain @xcite . last, given the examples from figure 1 , we formulate the task as both as a sequence labeling and a multi-task learning (mtl) problem, i.e., training on both skill and knowledge components jointly @xcite . annotations approach size skill type (baseline) model(s) kivimäki et al. (2013) document-level automatic n/a hard logent., tf-idf, lsa, lda 7 7 zhao et al. (2015) sentence-level automatic n/a hard word2vec 7 7 javed et al. (2017) span-level skill inventory n/a both word2vec 7 7 jia et al. (2018) span-level automatic 21,158 jps* hard lstm 7 7 sayfullina et al. (2018) span-level crowdsourcing 4,863 sent. soft cnn, lstm 3 7 smith et al. (2019) span-level manual 100 jps hard pattern matching 7 7 gugnani and misra (2020) span-level domain experts ⇠200 jps hard word2vec, doc2vec 7 7 li et al. (2020) document-level proprietary n/a hard fasttext 7 7 shi et al. (2020) span-level proprietary n/a hard fasttext, use, bert 7 7 tamburri et al. (2020) sentence-level domain experts ⇠3,000 sent. both bert 7 7 chernova (2020) span-level manual 100 jps both finbert 7 7 bhola et al. (2020) document-level skill inventory 20,298 jps* hard bert 3 7 smith et al. (2021) span-level manual 100 jps hard pattern match., word2vec 7 7 liu et al. (2021) document-level crowdsourcing n/a hard gnn 7 7 this work span-level domain experts 391 jps both (domain-adapted) bert 3 3 with respect to the annotation approach, "manual" indicates uncertainty whether they used domain experts or not. also note that many works do not release their dataset with annotations () nor guidelines (). the list is inspired by @xcite . contributions in this paper: 1 we release skillspan, a novel skill extraction dataset, with annotation guidelines, and our open-source code. 1 2 we present strong baselines for the task including a new spanbert @xcite trained from scratch, and domain-adapted variants (gururangan et al., 2020), which we will release on the huggingface platform @xcite . to the best of our knowledge, we are the first to investigate the extraction of skills and knowledge from job postings with state-of-the-art language models. @xcite we give an analysis on single-task versus multi-task learning in the context of skill extraction, and show that for this particular task single-task learning outperforms multi-task learning. ## related work there is a pool of prior work relating to se. we summarize it in as can be seen in although @xcite annotated on the span-level (thus being useful for se) and release their data, they instead explored several approaches to skill classification. to create the data, they extracted all text snippets containing one soft skill from a predetermined list. crowdworkers then annotated the highlighted skill whether it was a soft skill referring to the candidate or not. they show that an lstm @xcite performs best on classifying the skill in the sentence. in our work, we annotated a dataset three times their size ( once the sentence is identified as containing a skill, the skill cited within is extracted. in contrast, we directly annotate for the span within the sentence. bhola et al. (2020) cast the task of skill extraction as a multi-label skill classification at the document-level. there is a predefined set of unique skills given the job descriptions and they predict multiple skills that are connected to a given job description using bert @xcite . in addition, they experiment with several additional layers for better prediction performance. we instead explore domain-adaptive pre-training for se. the work closest to ours is by chernova (2020), who approach the task similarly with span-level annotations (including longer spans) but approach this for the finnish language. it is unclear whether they annotated by domain experts. also, neither the data nor the annotation guidelines are released. for a comprehensive overview with respect to se, we refer to @xcite . ## skill & knowledge definition there is an abundance of competences and there have been large efforts to categorize them. for example, the the international standard classification of occupations (isco; elias, 1997) is one of the main international classifications of occupations and skills. it belongs to the international family of economic and social classifications. another example, the european skills, competences, qualifications and occupations (esco; le vrang et al., 2014) taxonomy is the european standard terminology linking skills and competences and qualifications to occupations and derived from isco. the esco taxonomy mentions three categories of competences: skill, knowledge, and attitudes. esco defines knowledge as follows: ## skillspan dataset data 5 we continuously collected jps via web data extraction between june 2020-september 2021. our jps come from the three sources: we release the anonymized raw data and annotations of the parts with permissible licenses, i.e., house (from a govermental agency which is our collaborator) and tech. 6 for anonymization, we perform it via manual annotation of job-related sensitive and personal data regarding organization, location, contact, and name following the work by @xcite . ## experimental setup the task of se is formulated as a sequence labeling problem. formally, we consider a set of jps d, where d 2 d is a set of sequences (i.e., entire jps) with the i th input sequence x i @xmath0, x 2 , ..., x t } and a target sequence of biolabels y i @xmath1, y 2 , ..., y t } (e.g., "b-skill", "i-knowledge", "o"). the goal is to use d to train a sequence labeling algorithm h : x 7 ! y to accurately predict entity spans by assigning an output label y t to each token x t . as baseline we consider bert and we investigate more recent variants, and we also train models from scratch. models are chosen due to their stateof-the-art performance, or in particular, for their strong performance on longer spans. bert base @xcite an out-ofthe-box bert base model (bert-base-cased) from the huggingface library @xcite functioning as a baseline. spanbert @xcite a bert-style model that focuses on span representations as opposed to single token representations. spanbert is trained by masking contiguous spans of tokens and optimizing two objectives: (1) masked language modeling, which predicts each masked token from its own vector representation. (2) the span boundary objective, which predicts each masked token from the representations of the unmasked tokens at the start and end of the masked span. we train a spanbert base model from scratch on the bookscorpus @xcite and english wikipedia using cased wordpiece tokens @xcite . we use adamw (kingma and ba, 2015) for figure 3 : performance of models. we test the models on skills, knowledge, and combined. we report the span-f1 and standard deviation (error bars) of runs on five random seeds. note that the y-axis starts from 50 span-f1. stl indicates single-task learning and mtl indicates the multi-task model. differences can be seen on the test set: jobspanbert performs best on skills, jobbert is best on knowledge, and jobbert achieves best in combined. exact numbers of the plots are in jobbert 7 we apply domain-adaptive pre-training @xcite to a bert base model using the 3.2m unlabeled jp sentences ( jobspanbert 8 we apply domain-adaptive pretraining to our spanbert on 3.2m unlabeled jp sentences. we keep parameters identical to the vanilla spanbert, but change the number of steps to 40k to have three passes over the unlabeled data. ## results the results of the experiments are given in figure 3 . we show the average performance of each model in f1 and respective standard deviation over the development and test split. exact scores on each source split and other metric details are provided in appendix e. as mentioned before, we experiment with the following settings: skill, we train and predict only on skills. knowledge, train and only predict for knowledge. combined, we merge the stl predictions of both skills and knowledge. we also train the models in an mtl setting, predicting both skills and knowledge simultaneously. we evaluate the mtl model on both skill and knowledge separately, and also compare it against the aggregated stl predictions. performance on development set in figure 3 , we show the results on the development set in the upper plot. we observe similar performance between the domain-adapted stl models-jobbert and jobspanbert-have similar span-f1 for 9 for reproducibility, we refer to appendix d. skill: 60.05±0.70 vs. 60.07±0.70. in contrast, for knowledge, bert base and jobbert are closest in predictive performance: 60.44±0.58 vs. 60.66±0.43. in the combined setting, jobbert performs highest with a span-f1 of 60.32±0.39. on average, jobbert performs best over all three settings. surprisingly, the models for both skill and knowledge perform similarly (around 60 span-f1), despite the sources' differences in properties and length figure 2 . in addition, we find that mtl is not performing better than stl across sources. for exact numbers and source-level (i.e., big, house, tech), we refer to appendix e. ## discussion what did not work additionally, we experiment whether representing the entire jp for extracting tokens yields better results than the experiments so far, which were sentence-by-sentence processing setups. to handle entire jps and hence much longer sequences we use a pre-trained longformer base @xcite model. the document length we use in the experiments is 4096 tokens. results of the longformer on the test set are lower: for skills, jobspanbert against longformer results in 56.64±0.83 vs. 52.55±2.39. for knowledge, jobbert against longformer shows 63.88±0.28 vs. 57.26±1.05. last, for com-bined, jobbert against longformer results in 59.73±0.38 vs. 55.05±0.71. this drop in performance is difficult to attribute to a concrete reason: e.g., the longformer is trained on more varied sources than bert, but not specifically for jps, which may have contributed to this gap. since the vanilla longformer already performs worse than bert base overall, we did not opt to apply domainadaptive pre-training. overall, we show that representing the full jp is not beneficial for se, at least not in the longformer setup tested here. continuous pretraining helps se as previously mentioned, due to the domain specialization of the domain-adapted pre-trained bert models, they predict more skills and frequently perform better in terms of precision, recall, and f1 as compared to their non-adaptive counterparts. this is especially encouraging as we confirm findings that continuous pre-training helps to adapt models to a specific domain @xcite @xcite . however, there are exceptions. particularly in figure 5 : average length of predictions of single models. we show the average length of the predictions versus the length of our annotated skills and knowledge components on the test set and the total number of predicted skills and knowledge tags in each respective split (#). there is a consistent trend over the three sources. ## conclusion we present a novel dataset for skill extraction on english job postings-skillspan-and domainadapted bert models-jobbert and jobspan-bert. we outline the dataset and annotation guidelines, created for hard and soft skills annotation on the span-level. our analysis shows that domainadaptive pre-training helps to improve performance on the task for both skills and knowledge components. our domain-adapted jobspanbert performs best on skills and jobbert on knowledge. both models achieve almost stochastic dominance over all other models for skills and knowledge extraction, whereas jobbert in the stl setting achieves stochastic dominance over other models. with the rapid emergence of new competences, our new approach of skill extraction has future potential, e.g., to enrich knowledge bases such as esco with unseen skills or knowledge components, and in general, contribute to providing insights into labor market dynamics. we hope our dataset encourages research into this emerging area of computational job market analysis. akshay bhola, kishaloy halder, animesh prasad, and min-yen kan. 2020. retrieving skills from job descriptions: a language model based extreme multilabel classification framework. in proceedings of the 28th international conference on computational linguistics, pages 5832-5842, barcelona, spain (online). international committee on computational linguistics. carlo bonferroni. 1936. teoria statistica delle classi e calcolo delle probabilita. pubblicazioni del r istituto superiore di scienze economiche e commericiali di firenze, 8:3-62. rich caruana. 1997. multitask learning. machine learning, 28(1):41-75. mariia chernova. 2020. occupational skills extraction with finbert. master's thesis. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. rotem dror, segev shlomov, and roi reichart. 2019. deep dominance -how to properly compare deep neural models. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 2773-2785, florence, italy. association for computational linguistics. peter elias. 1997. occupational classification (isco-88): concepts, methods, reliability, validity and crossnational comparability. technical report, oecd publishing. joseph l fleiss. 1971. measuring nominal scale agreement among many raters. psychological bulletin, 76(5):378. joseph l fleiss and jacob cohen. 1973. the equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. educational and psychological measurement, 33(3):613-619. akshay gugnani and hemant misra. 2020. implicit skills extraction using document embedding and its use in job recommendation. in the thirty-fourth aaai conference on artificial intelligence, aaai 2020, the thirty-second innovative applications of artificial intelligence conference, iaai 2020, the tenth aaai symposium on educational advances in artificial intelligence, eaai 2020, new york, ny, usa, february 7-12, 2020, pages 13286-13293. aaai press. suchin gururangan, ana marasović, swabha swayamdipta, kyle lo, iz beltagy, doug downey, and noah a. smith. 2020. don't stop pretraining: adapt language models to domains and tasks. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 8342-8360, online. association for computational linguistics. xiaochuang han and jacob eisenstein. 2019. unsupervised domain adaptation of contextualized embeddings for sequence labeling. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 4238-4248, hong kong, china. association for computational linguistics.