J ob S kape: A Framework for Generating Synthetic Job Postings to Enhance Skill Matching (2024)

Abstract: Recent approaches in skill matching, employing synthetic training data for classification or similarity model training, have shown promising results, reducing the need for time-consuming and expensive annotations. However, previous synthetic datasets have limitations, such as featuring only one skill per sentence and generally comprising short sentences. In this paper, we introduce JobSkape, a framework to generate synthetic data that tackles these limitations, specifically designed to enhance skill-to-taxonomy matching. Within this framework, we create SkillSkape, a comprehensive open-source synthetic dataset of job postings tailored for skill-matching tasks. We introduce several offline metrics that show that our dataset resembles real-world data. Additionally, we present a multi-step pipeline for skill extraction and matching tasks using large language models (LLMs), benchmarking against known supervised methodologies. We outline that the downstream evaluation results on real-world data can beat baselines, underscoring its efficacy and adaptability.

## introduction in the dynamic modern labor market, understanding job demands at scale is crucial for informed decision-making by policymakers, businesses, and other stakeholders. one way of measuring job market demand lies in skill matching: the extraction and alignment of skills from job descriptions to their disambiguated forms (i.e., a knowledge base or taxonomy). this process facilitates the investigation of current labor market dynamics and the quantification of labor market demands, addressing the occupational skill matching problem. regardless of their predictive effectiveness, supervised learning methods for skill matching re-quire regularly collecting and annotating up-to-date data @xcite , a process that is both expensive and time-consuming. synthetic data circumvents the need for such costly annotations. however, despite efforts in generating synthetic training data @xcite and real-world benchmarks @xcite , challenges like incoherent sentences and over-simplified setups exist in existing datasets. to address these shortcomings, we introduce jobskape, a framework for generating realistic skill matching datasets that can be used for training and benchmarking. jobskape facilitates the creation of diverse labeled textual datasets that align closely with actual job postings, ensuring cleaner and more coherent data. we demonstrate its practical application by generating skillskape, a large-scale dataset linking coherent sets of skills to corresponding job descriptions. jobskape uses generative large language models (llms) to curate meaningful skill combinations and generate appropriate job descriptions containing these combinations. a selfrefinement step using llms @xcite ensures label quality in the refined skillskape dataset, assessed through offline metrics. finally, we challenge traditional supervised skill matching methods with an llm-based, in-context learning (icl) pipeline, to circumvent re-training the supervised model given new data. we evaluate skill matching performance on our synthetic dataset and real-world annotated data @xcite , comparing our proposed extraction and matching pipeline with supervised matching models trained on our dataset as well as previous generation attempts @xcite in a controlled setting. contributions. in this work, we contribute the following: (1) we propose jobskape, a framework for generating a synthetic dataset of job de-scriptions for skill matching with existing skill taxonomies, (2) using our framework, we release a synthetic train and evaluation dataset (skillskape) for skill matching, (3) we show that skillskape has higher textual quality measured in perplexity and implicitness compared to previous synthetic datasets, (4) lastly, we introduce an icl-based approach to extract and match skills from job descriptions to a taxonomy and show that this method can outperform supervised baselines on real-world benchmarks. ## the jobskape framework our goal is to create a synthetic dataset comprising job posting sentences associated with lists of skills from a taxonomy that closely aligns with realworld job posting sentences. we initiate the process by generating combinations of skills, derived from a given taxonomy, that are likely to coexist in a job description. leveraging llms and refinement techniques, we produce diverse, realistic, and accurate job description sentences. to evaluate the quality of our synthetic data generation, we define a set of offline metrics and compare the generated sentences with real job postings. ## experimental setup in this section, we introduce several benchmarks for skill matching tasks. we train a supervised multi-label classifier and present an llm-based approach with in-context learning. ## results and analysis to assess the label refinement method (section 3.4), we apply it to the development set of the skillspan-m benchmark that has annotated skills and associated spans. 40% of our extracted spans match exactly with the annotated span. 60% of our extracted spans are either a perfect match or contain the annotated span. in general, extracted spans have a jaccard similarity of 62% with the annotated spans. ## conclusion we introduce jobskape, a general framework for generating synthetic job posting sentences for skill matching. using our framework, we release skillskape a large dataset of synthetic job posting sentences labeled with esco skills. our analysis shows that skillskape contains more implicit skills, has longer sentences, and is overall closer to real-world data, compared to alternative synthetic dataset from the literature. using our dataset, we conducted several skill matching experiments by training a supervised multi-label classifier and using in-context learning with an llm, and showed that both methods achieved comparable results when evaluated on real-world data (f 1 of 26.1 and 27.3 respectively). furthermore, we note that the potential applications of jobskape extend beyond its current scope. its application in creating synthetic cvs, for instance, can enhance job matching algorithms and facilitate skill-gap analysis in various industries. the framework's adaptability to different skill taxonomies also opens up possibilities for use across multiple sectors. while promising, these extended applications require further exploration to fully assess their impact. english only. our method is limited to processing and understanding english language content. this language-specific focus narrows the scope of our system's applicability, excluding non-english speaking demographics. bias inherited from llms. another significant limitation is the potential bias inherited from the llms. since these models are trained on large datasets that may contain biases, there is a risk that our system may inadvertently perpetuate these biases in its generations. this could manifest in various forms, such as gender, cultural, or industryspecific biases, and could affect the fairness and neutrality of the job postings generated. furthermore, if biased postings are used extensively, they could adversely influence downstream tasks. for example, biased job postings could skew job recommendation algorithms, leading to unfair job suggestions that do not treat all individuals equally. this highlights the need for careful consideration and mitigation of biases in our approach to ensure equitable outcomes in all applications. subset of the taxonomy. due to limited resources, we restricted the generation of our synthetic dataset to ∼8k samples, with a fraction of the esco taxonomy that is also used in the skillspan-m dataset. consequently, the multiclass classifier is also trained to classify with a limited set of skills. scaling up to the full taxonomy might modify the behavior of the supervised classification model, while it should have little to no impact on the icl skill-matching pipeline.