Transfer learning for multilingual vacancy text generation (2022)

Abstract: Writing job vacancies is a repetitive and expensive task for humans. This research focuses on automatically generating the benefit sections of vacancies at redacted from job attributes using mT5, the multilingual version of the state-of-the-art T5 transformer trained on general domains to generate texts in multiple languages. While transformers are accurate at generating coherent text, they are sometimes incorrect at including the structured data (the input) in the generated text. Including the input correctly is crucial for vacancy text generation; otherwise, the candidates may get misled. To evaluate how the model includes the input we developed our own domain-specific metrics (input generation accuracy). This was necessary, because Relation Generation, the pre-existing evaluation metric for data-to-text generation uses only string matching, which was not suitable for our dataset (due to the binary field). With the help of the new evaluation method we were able to measure how well the input is included in the generated text separately for different types of inputs (binary, categorical, numeric), offering another contribution to the field. Additionally, we also evaluated how accurate the mT5 model generates the text in the requested language. The results show that mT5 is very accurate at generating the text in the correct language, at including seen categorical inputs and binary values correctly in the generated text. However, mT5 performed worse when generating text from unseen city names or working with numeric inputs. Furthermore, we found that generating additional synthetic training data for the samples with numeric input can increase the input generation accuracy, however this only works when the numbers are integers and only cover a small range.

## introduction integrating natural language processing (nlp) solutions to improve the human workforce offers great opportunities for process automation @xcite . such process automation can be to automate the writing of parts of the vacancies using the given structured data. this research was carried out with randstad netherlands, a dutch hr services company that provides vacancies in dutch and english. natural language generation (nlg) is the subfield of nlp focusing on automatic text writing. text generation tasks can be categorized depending on the input into text-to-text generation (e.g., text translation) and data-to-text generation (e.g., the generation of wikipedia biographies). generating vacancy text falls under the data-to-text generation category, since particular job-specific information (for example: @xmath0, @xmath1, @xmath2. early data-totext generation approaches used hand-engineered templates @xcite @xcite to generate texts. these templates can be intuitively explained as a series of written text elements, with certain segments serving as the template's 'backbone' and the remaining segments being filled up with information from the structured data @xcite . from the previous example: 'you will receive between 2500 and 2700 euros on a monthly bases' the following template can be created 'you will receive @xmath3. such simple (template-based) method was used as the baseline for this research. while this solution is computationally cheap, fast and the inputs will be correctly included, a lot of manual effort is required to create the templates and the generated texts will be very repetitive. transformer models are a type of neural network architecture that use transfer learning. transfer learning is the technique where these models are initially trained (pre-trained) on a large unlabeled text dataset and some supervised tasks. the 'knowledge' learned through pre-training is then reused on the new task, making transformer methods cur-rently the most effective solution for text generation @xcite . while there is no transformer model available that was pre-trained on data-to-text generation tasks, recent research in @xcite showed that transformer models -that were pre-trained on text-to-text generations tasks -outperform traditional (non-transformer) neural network solutions for data-to-text generation task after fine-tuning. based on this finding and the fact that our dataset is multilingual, we used the multilingual version of t5, mt5 @xcite . in contrast to the template-based method, transformers can provide a wide range of text variation. however, when generating text, transformers pick the next word with some randomness, thus they are less accurate at including the input correctly. this paper investigates how the benefit sections of vacancies (namely: salary, hours, contract and location) can be generated using mt5. our contribution is three-fold. on one hand, we use mt5 in a new domain, with a relatively small and proprietary dataset -that contains many noise and incorrect samples -in a multilingual setting. on the other hand, we confirm the importance of domain specific evaluation metrics in data-to-text generation, by analyzing -with the metrics developed by us -how accurate mt5 is in terms of including the input correctly in the generated text. we also investigate how creating more training data (through translation and generating synthetic samples) can improve this accuracy. focusing on including the input correctly is particularly important in this domain, where incorrect wages, or stating hourly salaries instead of monthly can be misleading to candidates. ## results ## conclusion although the amount and the imbalanced nature of the dataset (in terms of language) prevent us from drawing definite conclusions, our analysis reveals several insights into the behavior of the mt5 model, which we use to answer our three sub-questions. rq1.1 first, we focus on finding how accurately mt5 generates text in the correct language. overall, we found that with the right parameter tuning, prefix, and input, mt5 performs very well in generating text in the correct language. rq1.2 to find how accurate mt5 is at including different types of inputs (numerical, binary, categorical) correctly in the generated text, we used our own new, domain-specific metrics in the generated text. we conclude that while mt5 is highly accurate at generating from binary inputs, and seen categorical inputs, it struggles with unseen categorical and numeric inputs. rq1.3 finally, we explore how fine-tuning mt5 on translated and synthetic data affects the language and input generation accuracy. we found that finetuning mt5 on additional training data (translated and synthetic) can lead to major improvements, however the rate of which strongly depends on the nature of the task and the distribution of numeric samples in the dataset. in conclusion, we evaluated how accurately mt5 includes inputs in the generated text with custom metrics developed by us, on a novel task in the job description domain. we applied transfer learning on a new and unique dataset in terms of volume and language balance. while our findings are mostly in line with earlier studies, we demonstrate that language accuracy, input generation accuracy, and the effectiveness of extra training data are highly dependent on the nature of data and task. overall, even though the model may not be as accurate as a template-based approach, it can be employed under human supervision to generate benefit sections of vacancies, and will yield more diverse outputs. additionally, when using the model, the users can rely on custom evaluation metrics developed by us. promising future work directions include using more languages and other sections of vacancy texts. for example, while the requirements section of the vacancies currently do not have structured input data available; these inputs may be extracted automatically using a skill extraction model, such as linkedin's job2skills model @xcite . extending the study to novel sections of the vacancy could give a great opportunity to observe how transformers work with categorical input with a wider range of values than our salarytype and contracttype, while still being more general than the location input. another intriguing step is to investigate how automatically generated text affects the accessibility of job descriptions. finally, evaluating the models with the recruiters would be a logical next step too.