Text Zoning and Classification for Job Advertisements in G erman, F rench and E nglish (2020)

Abstract: We present experiments to structure job ads into text zones and classify them into pro- fessions, industries and management functions, thereby facilitating social science analyses on labor marked demand. Our main contribution are empirical findings on the benefits of contextualized embeddings and the potential of multi-task models for this purpose. With contextualized in-domain embeddings in BiLSTM-CRF models, we reach an accuracy of 91% for token-level text zoning and outperform previous approaches. A multi-tasking BERT model performs well for our classification tasks. We further compare transfer approaches for our multilingual data.

## introduction text mining on job advertisements has become important to analyze labor market demand, since job ads provide unique job-level data on employers' staff needs (atalay et al., 2020; das et al., 2020; calanca et al., 2019; dawson et al., 2019). our proposed techniques will be useful to study how job tasks and skill demand have developed in switzerland in different labor market segments over the last decades. we present preparatory work for precise skill and task extraction: first, we structure job ads into text zones, that is, text parts dedicated to particular topics. second, we classify job ads into professions, industries and management function. by replacing human annotation with scalable nlp, more fine-grained analyses on big data will be feasible. job ads contain information on topics such as the company, the job, or required qualifications. for an accurate extraction of skills and tasks, we need to identify the corresponding text zones, as many key terms are ambiguous, for instance 'dynamic' might refer to a personality trait or to a dynamic crm system. information on different topics can be densely packed in sentences, thus it seems most reasonable to formalize text zoning for job ads as token-level sequence labeling. in addition to this structuring of job ads, we need automatic classifications of job ads to enable detailed analysis, most importantly into professions, but also into industries and management functions. for swiss data in german, french, english and italian we need multilingual approaches. most (labeled) data however is in german. to avoid sparse data problems for minority languages, we thus experiment with transfer approaches. the empirical experiments presented a) investigate the benefit of contextualized embeddings for text zoning, b) compare multilingual modeling with machine translation based approaches, and c) explore the potential of multi-task models for sequence labeling and text classification. classification of professions is often provided by companies, and their methods and performance are not reported (burke et al., 2020; das et al., 2020; calanca et al., 2019). atalay et al. (2020) use embedding similarity measures to match jobs to 110 classes, and reach an accuracy of 53%. ## experimental data we use two job ad data sets, differing in size and data collection method. both have each advantages for experiments here and for future analyses. the swiss job market monitor (sjmm) foot_1 corpus consists of 80,000 job ads in german, french, english and italian, from yearly samples representative for the swiss job market, back to 1950. high-quality human annotations of profession, industry, and management function are available for all job ads. text zones are annotated on german job ads until 2014. the sjmm provides us hence with labeled data for supervised machine learning experiments, and will allow analyses of how job tasks and skills developed over the last decades. the online ads (oa) corpus contains 9 million ads in german, french, english and italian from job portals and company websites in switzerland crawled since 2012 by a private company. this big data set seems valuable for building in-domain embeddings for our experiments, and makes finegrained analyses feasible for future research. text zoning in the sjmm is operationalized as introduced in gnehm (2018). eight zones are distinguished based on their content, and the text is segmented on token level. @xcite token level seems most appropriate, as information on different zones can be densely packed in single sentences. not every ad contains information on every zone (e.g. not every job ad specifies personality traits of the ideal candidate) and zone distribution is strongly skewed: the job description (z6) comprises with more than 30% the largest share of tokens, whereas the least frequent zone, reason of the vacancy (z2), comprises 0.5% of tokens. tokens show high zone ambiguity, with more than 90% of tokens showing up in more than one zone. in text zoning experiments, we use the data split of gnehm (2018), for comparability: aiming for a model optimized for future application, dev and test set (test set a) are restricted to each 10% (@xmath0, the remaining 80% and all data further back to 1970 (@xmath1,700) serve as training data. in pure text classification experiments, we can use all multilingual sjmm data from the time span of interest (1990-2018, @xmath2,600), and take 80% for train, and 10% for dev and test set each (test set b). 4 experiments 4.1 text representation we experiment with different text embeddings: static type-level fasttext (ft) embeddings provide a single vector for all occurrences of a word (bojanowski et al., 2017). contextualized embeddings allow to represent different word senses by capturing the semantics of surrounding text. we contrast bert sub-word embeddings (devlin et al., 2019) with character-based flair embeddings (akbik et al., 2018). given the large amount of in-domain texts, we train flair embeddings on both of our corpora (flairsjmm, flairoa). @xcite we systematically compare the effect of these in-domain vs. general domain embeddings mentioned above. qualitative evidence for the usefulness of contextualized in-domain embeddings for text zoning is provided in figure 1 with a umap (mcinnes et al., 2018) vector visualization of the semantic space. the term 'ansprechpartner' (contact person) in the zone for job description (z6) depicts that part of the job is to serve as contact person, probably for clients or co-workers, and the same term in the zone for the wanted personality (z8) hints furthermore that this person should be approachable or trustworthy. contact person in the residual text (z3) however, simply refers to a contact information for the application procedure. the separation of the respective vectors in the semantic space in figure 1 shows that such zone specific meanings can be recognized with our contextualized in-domain embeddings. ## conclusion contextualized embeddings facilitate precise information extraction. our best single text zoning multilingual experiments showed that our classifiers are affected by mt. utilizing translated material in training, or alternatively multilingual models, are potential strategies, but the question of the best transfer approaches for our multilingual data needs further investigation. the most promising approach for future work seems to be the training of our own domainspecific bert embeddings, both for optimizing classification and for intended subsequent skill and task extraction. this way, we can also exploit the large amount of data in the oa corpus. another direction worthy to explore is multi-tasking, be it by including more variables, or by experimenting with more sophisticated approaches (clark et al., 2019; liu et al., 2019b).