NNOSE : Nearest Neighbor Occupational Skill Extraction (2024)

Abstract: The labor market is changing rapidly, prompting increased interest in the automatic extraction of occupational skills from text. With the advent of English benchmark job description datasets, there is a need for systems that handle their diversity well. We tackle the complexity in occupational skill datasets tasks—combining and leveraging multiple datasets for skill extraction, to identify rarely observed skills within a dataset, and overcoming the scarcity of skills across datasets. In particular, we investigate the retrieval-augmentation of language models, employing an external datastore for retrieving similar skills in a dataset-unifying manner. Our proposed method, N earest N eighbor O ccupational S kill E xtraction (NNOSE) effectively leverages multiple datasets by retrieving neighboring skills from other datasets in the datastore. This improves skill extraction without additional fine-tuning. Crucially, we observe a performance gain in predicting infrequent patterns, with substantial gains of up to 30% span-F1 in cross-dataset settings.

## introduction labor market dynamics, influenced by technological changes, migration, and digitization, have led to the availability of job descriptions (jd) on platforms to attract qualified candidates @xcite @xcite . jds consist of a collection of skills that exhibit a characteristic long-tail pattern, where popular skills are more common while niche expertise appears less frequently across industries @xcite , such as "teamwork" vs. "system design". 1 this pattern poses challenges for skill extraction (se) and analysis, as certain skills may be underrepresented, overlooked, or emerging in jds. this complexity makes the extraction and analysis of skills more difficult, resulting in a 1 examples are from the cedefop skill platform. to address the challenges in se, we explore the potential of nearest neighbors language models (nnlms; @xcite . nnlms calculate the probability of the next token by combining a parametric language model (lm) with a distribution derived from the k-nearest context-token pairs in the datastore. this enables the storage of large amounts of training instances without the need to retrain the lm weights, improving language modeling. however, the extent to which nnlms enhance application-specific end-task performance beyond language modeling remains relatively unexplored. notably, nnlms offer several advantages, as highlighted by @xcite : first, explicit memorization of the training data aids generalization. second, a single lm can adapt to multiple domains without domain-specific training, by incorporating domain-specific data into the datastore (e.g., multiple datasets). third, the nnlm architecture excels at predicting rare patterns, particularly the long-tail. therefore, we seek to answer the question: how effective are nearest neighbors retrieval methods for occupational skill extraction? our contributions are as follows: ## results we evaluate the performance of fine-tuning models enhanced with nnose. we consider different setups: first, we compare using the whitening transformation (+wt) or without. second, we explore two datastore setups: one using an in-dataset datastore ({d}), where each respective training set is stored separately, and another where all datasets are stored in the datastore (∀d). in the latter setup, we encode all three datasets with each fine-tuned model, and each model has its own wt matrix. for example, we fine-tune a model on skillspan and encode the training set tokens of skillspan, say-fullina, and green to populate the datastore. from the results on the development set ( ## analysis as we store training tokens from all datasets in the datastore, we expect the model to recall a greater number of skills based on the current context during inference. in turn, this would lead to improved downstream model performance. we want to address the challenges of se datasets by predicting long-tail patterns, and if we observe improvements in detecting unseen skills in a cross-dataset setting. to investigate in which situations our model improves, we are analyzing the following: 1 the predictive capability of nnose in relation to rarely occurring skills compared to regular fine-tuning (section 5.1). skills exhibit varying frequencies across datasets, we categorize the skill frequencies into buckets and compare the performance between vanilla fine-tuning and the inclusion of knn. 2 if nnose actually retrieves from other datasets when they are combined (section 5.2), and if there is a sign of leveraging multiple datasets, then; 3 how much does nnose enhance performance in a cross-dataset setting (section 5.3)? our results inknn is based on the datastore with all the datasets. we categorize the occurrences of a skill in the test set with respect to the training set. for example, a skill in the test set occurs two times in the training set, we put this in the "low" bin. there are three frequency ranges: high: 10-15, mid-high: 7-10, mid-low: 4-6, low: 0-3. sayfullina does not have any test set skills that occur more than 10 times in the training set. on top of the bars is the number of predicted skills for the test set in each bucket. ## related work skill extraction. the dynamic nature of labor markets has led to an increase in tasks related to jd, including skill extraction @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . ods @xcite . recent methodologies include domain-specific models where lms are continuously pre-trained on unlabeled jd @xcite . however, none of these methodologies introduce a retrieval-augmented model like nnose. general retrieval-augmentation. in retrieval augmentation, lms can utilize external modules to enhance their context-processing ability. two approaches are commonly used: first, using a separately trained model to retrieve relevant documents from a collection. this approach is employed in open-domain question answering tasks @xcite and with specific models such as orqa @xcite , realm @xcite , rag (lewis et al., 2020), fid @xcite , and atlas @xcite . second, previous work on explicit memorization showed promising results with a cache @xcite , which serves as a type of datastore. the cache contains past hidden states of the model as keys and the next word as tokens in key-value pairs. memorization of hidden states in a datastore, involves using the knn algorithm as the retriever. the first work of the knn algorithm as the retrieval component was by @xcite , leading to several lm decoder-based works. ## conclusion we introduce nnose, an lm that incorporates and leverages a non-parametric datastore for nearest neighbor retrieval of skill tokens. to the best of our knowledge, we are the first to introduce the nearest neighbors retrieval component for the extraction of occupational skills. we evaluated nnose on three relevant skill datasets with a wide range of skills and show that nnose enhances the performance of all lms used in this work without additionally tuning the lm parameters. through the combination of train sets in the datastore, our analysis reveals that nnose effectively leverages all the datasets by retrieving from each. moreover, nnose not only performs well on rare skills but also enhances the performance on more frequent patterns. lastly, we observe that our baseline models exhibit poor performance when applied in a cross-dataset setting. however, with the introduc-tion of nnose, the models improve across all settings. overall, our findings indicate that nnose is a promising approach for application-specific skill extraction setups and potentially helps discover skills that were missed in manual annotations.