Template-based Contact Email Generation for Job Recommendation (2022)

Abstract: Text generation has long been a popular research topic in NLP. However, the task of generating contact emails from recruiters to candidates in the job recommendation scenario has received little attention by the research community. This work aims at defining the topic of automatic email generation for job recommendation, identifying the challenges, and providing a baseline template-based solution for Danish jobs. Evaluation by human experts shows that our method is effective. We wrap up by discussing the future research directions for better solving this task.

## introduction and prior work recruitment email generation is a crucial step in the overall job recruitment process. when recruiters find suitable candidates for a job, they need to write emails to contact the candidates, explaining why they are fit for the job and inviting them to apply. the job market produces a huge number of job postings on a daily basis, which results in a significant amount of human labor required to write recruitment emails. the recruiters are in urgent need of an automatic approach to compose these emails, to reduce their efforts and increase productivity. the core challenges facing the task are two-fold: persuasiveness and personalization. first, the emails should contain sufficient reasons to convince the candidate that their qualifications match the requirements of the position, and illustrate that the position meets the candidate's expectations. second, personalized emails should let the job seekers feel that the recruiters pay special attention to them and in turn motivate them to apply. in an interview of recruiters of varying recruitment experience, recruiters believe that proper personalization on the recruitment emails could boost candidate positive response rate, and admit that the current emails, largely composed based on fixed templates, are not personalized and persuasive enough @xcite . recruitment email generation can be seen as a task-oriented text generation problem. one needs to extract information from the input job description and candidate profile, and generate the recruitment email accordingly. a large body of research on emails focuses on the analysis side, typically automatic detection of phishing emails by extracting features @xcite @xcite . in the realm of email synthesis, email subject line generation @xcite has been studied. for generation of the main email body, attempts have been made on fake email generation for malicious purposes @xcite based on a two-step pipeline (chen and rudnicky, 2014a,b) for email synthesis. as far as we know, generating recruitment emails in job recruitment is an unexplored task in literature, despite its practical significance and challenging nature. we therefore formally formulate the task of recruitment email generation and provide a baseline template-based system for it by extending chen and rudnicky (2014a)'s approach. in particular, we randomly generate an email from a library of different pre-defined components, and fill the motivational sentence with certain slots by combining matched skills and occupations extracted from the job and candidate in question. we conduct a user study to evaluate the quality of the generated text and examine if it can save recruiters' time in writing emails. the results show that recruiters under test are overall satisfied with the generated emails, and the time spent on writing emails for each candidate is significantly reduced. the rest of the paper is organized as follows. we define the problem formally in section 2. we run a pilot study to examine the simple end-to-end neural generation algorithm in section 3 and elicit the need for a fine-grained synthesizing approach. our template-based approach is described in section 4, and the user evaluation is reported and discussed in section 5. we conclude and point out future directions in section 6. ## task definition and notations recruitment email generation calls for generating an email based on an input the job and a candidate . it requires us to extract the matched information from different job and candidate fields, and convert it to natural language expressions in the email. typically, a job posting contains its title, company name and a textual description. a candidate profile has a headline, a list of keywords, a list of preferred job titles, the candidate's work experience and educational experience, and the candidate's resume text. the data we work with comes from jobindex foot_0 , one of scandinavia's largest job portals and recruitment agencies. in the jobindex system, the abovementioned fields are the main source of information for jobs and candidates. the majority of the jobs and candidate profiles are in danish. please refer to fig. 1 for an example of a pair of job and candidate, and the real recruitment email written by a human recruiter. the personal information has been removed from the example and all texts are translated to english. ## pilot study: end-to-end neural generation the most intuitive way to handle the recruitment email generation task is to generate the whole email in a fully end-to-end fashion by concatenating the job and company text. we conduct a pilot study to examine whether this setting works in practice. model. we build a transformer encoderdecoder, the state-of-the-art architecture for natural language generation, to support a sequenceto-sequence generation of recruitment email. the job and candidate texts are fed to the encoder side, and the recruitment email is generated in a tokenby-token auto-regressive manner. the transformer structure enables us to load the weights of the pretrained danish language model, danish-bert foot_1 . @xcite conducted a comprehensive comparison of different strategies to use the pretrained bert weights for sequence-to-sequence generation, and found that the bert2bert setting can yield robust performance across different text generation tasks. therefore, we follow this setting and initialize the weights for both encoder and decoder with the danish-bert checkpoint. please refer to @xcite for further details. we claim that this model may not be the state-of-the-art for danish text generation, since danish gpt is publicly available foot_2 . however, it is a strong-performing model due to its transformer architecture, and its generated text is representative of the state-of-theart neural generation models. data preprocessing. we focus on generating danish emails from danish jobs and candidate profiles. we mine (job, candidate, email) triplets from the jobindex database. we filter out samples with non-danish text, too short job descriptions and empty job titles. a total number of 317566 samples are obtained. due to the limitation on input length by the model, we concatenate the summary of job description and the headline, job titles, education experience and work experience of candidate. different fields are split by the periods. for recruitment emails, we only take their main bodies. we replace specific job title and company name with special tokens "[job]" and "[cpy]" respectively. training and evaluation. our transformer encoder-decoder has 12 layers and 12 attention heads and a maximum length of 512 tokens on both encoder and decoder sides. the embedding dimension and hidden dimension are set to 768 and 3072, respectively. we split all samples into training, validation and test set at a 7:2:1 ratio. we perform mini-batch learning to train the model with a batch size of 32. the average cross-entropy loss over all tokens at the decoder side is used as the loss function. we train the model on the training set for a maximum number of 5 epochs with adam optimizer, and stop training when the validation loss stops dropping. we use bilingual evaluation understudy (bleu) as the quantitative metric for the generated message. the bleu scores are calculated for individual translated segments, by comparing them with a set of good quality reference translations. the average scores over the whole test corpus are computed as an estimation of the overall quality. result. we obtained a bleu score of 23.2 on test samples for the generated texts, which is far from the sota neural text generation system for danish (32.5 -33.8) @xcite . this reveals that the generated emails have a limited overall quality. additionally, manual inspection shows that the auto-generated emails have low extent of personalization and persuasiveness. the generated motivational sentences usually do not contain specific reasons of match and are prone to grammatical errors. see fig. 2 for concrete examples. we posit that it is because the motivational sentences are usually located in the middle of email body, embedded in texts primarily generated from templates. also, over 80% of the motivational sentences are devoid of case-specific information such as the matched skills or occupations. therefore, an end-to-end generation system is capable of learning these recurring template-based texts quite well, but poor at learning case-specific motivational sentences. ## proposed methodology the evaluation result of end-to-end neural generation indicates the need for a finer-grained generation system. instead of generating the email as a whole, a special module should be developed to generate case-specific information, such as the motivational sentence, from the input job and candidate texts. we are inspired by the two-step email synthesis @xcite to generate recruitment emails: email structure generation and slot filling. specifically, we randomly generate an email template from a pre-constructed library of email components. the slots of the template indicate case-specific fields, and are then filled by extracting information from the matched job and candidate. the overall process of the system is shown in fig. 3 . potentially, our approach is superior to end-toend generation in three aspects: 1) recruiters follow a similar process to write emails in a real-case scenario, so the generated emails are more likely to be accepted by and benefit recruiters; 2) the algorithm has a better control of the generated content, and ensures a certain extent of grammatical correctness and readability; 3) by explicitly composing case-specific information, the algorithm enhances personalization and persuasiveness of the generated recruitment emails. in the following text, we introduce the implementation details of our template-based generation system. ## evaluation we conduct an offline evaluation with real expert recruiters to evaluate the performance of the templatebased email generation algorithm. the quantitative measures for natural language generation are not employed, since the template-based approach may produce significantly different structures from the real recruitment emails and still be reasonably good. indeed, it remains an open question to design quantitative measures for evaluating the generated recruitment emails, as the judgments on the persuasiveness or personalization of the email vary from person to person. we randomly sample a collection of truly matched (job, candidate) pairs from the jobindex database. they are randomly assigned to each of a group of 10 recruiters. the pairs are randomly split result on text quality. we have collected the results of 224 helped tasks (with generated emails by our algorithm) and 241 unhelped tasks (with our generated templates by our algorithm). the recruiters' annotations on the quality of the generated texts are shown in fig. 8 . in terms of average ratings, we obtained a value of 2.53 for generated emails and 2.46 for generated templates for the evaluation of the new template-based system. over 50% of the generated texts (53.58% for emails and 51.03% for templates) are satisfactory, requiring minor or no edits. it is interesting to see that more recruiters are satisfied with the generated texts than the templates. this implies that the generated motivational sentence is exceptionally helpful for recruiters. in the same time, there is still room for the text quality to improve, and neural method for generating case-specific information is a feasible direction. result on time cost. we computed the average time a recruiter spent on writing the message in the unhelped and helped case, respectively. the time difference between the two scenarios is the reduction of recruiters' time by our template-based automatic text generation system. as a consequence, recruiters spent an average of 178.99s on each matched pair in the helped case, and an average of 235.63s on each matched pair in the unhelped case. on average, a recruiter spends 56.64s shorter on writing an email with automatically generated messages than merely on top of the provided templates. in order to remove the discrepancy across individual recruiters, we zero-averaged the time for each recruiter and recalculated the time difference. a closer but still remarkable gap between the times (51.31s) is observed. it shows that this simple template-based system can significantly save recruiters time on writing recruitment emails. ## conclusion we have proposed recruitment email generation for job recruitment, a novel task in text generation. we demonstrate its challenge and significance in the pilot study and a user study of our templatebased approach. the challenge mainly resides in constructing case-specific components of the email from the input job and candidate to enhance personalization and persuasiveness. quite often, the reasons of a good match are semantically non-explicit and cannot be extracted by common word or phrase matching. at the same time, we have observed that a simple approach can remarkably benefit recruiters by saving around 1/4 of their time. future work could contribute to this task in the following aspects: beyond the real-user evaluation, it remains an open challenge to propose robust metrics that objectively evaluate the generated text, in terms of both language quality and taskrelated aspects.