De-identification of Privacy-related Entities in Job Postings (2021)

Abstract: De-identification is the task of detecting privacy-related entities in text, such as person names, emails and contact data. It has been well-studied within the medical domain. The need for de-identification technology is increasing, as privacy-preserving data handling is in high demand in many domains. In this paper, we focus on job postings. We present JobStack, a new corpus for de-identification of personal data in job vacancies on Stackoverflow. We introduce baselines, comparing Long-Short Term Memory (LSTM) and Transformer models. To improve these baselines, we experiment with BERT representations, and distantly related auxiliary data via multi-task learning. Our results show that auxiliary data helps to improve de-identification performance. While BERT representations improve performance, surprisingly “vanilla” BERT turned out to be more effective than BERT trained on Stackoverflow-related data.

## introduction it is becoming increasingly important to anonymize privacy-related information in text, such as person names and contact details. the task of de-identification is concerned with detecting and anononymizing such information. traditionally, this problem has been studied in the medical domain by e.g., @xcite ; @xcite ; @xcite to anonymize (or pseudo-anonymize) personidentifiable information in electronic health records @xcite . with new privacy-regulations (section 2) de-identification is becoming more important for broader types of text. for example, a company or public institution might seek to ♦ the authors contributed equally to this work. in this paper, we analyze how effectively sequence labeling models are in identifying privacyrelated entities in job posts. to the best of our knowledge, we are the first study that investigates de-identification methods applied to job vacancies. in particular, we examine: how do transformerbased models compare to lstm-based models on this task (rq1)? how does bert compare to bert overflow @xcite to what extent can we use existing medical deidentification data and named entity recognition (ner) data to improve de-identification performance (rq3)? to answer these questions, we put forth a new corpus, jobstack, annotated with around 22,000 sentences in english job postings from stackoverflow for person names, contact details, locations, and information about the profession of the job post itself. contributions we present jobstack, the first job postings dataset with professional and personal entity annotations from stackoverflow. our experiments on entity de-identification with neural methods show that transformers outperform bi-lstms, but surprisingly a bert variant trained on another portion of stackoverflow is less effective. we find auxiliary tasks from both news and the medical domain to help boost performance. ## jobstack dataset in this section, we describe the jobstack dataset. there are two basic approaches to remove privacybearing data from the job postings. first, anonymization identifies instances of personal data (e.g. names, email addresses, phone numbers) and replaces these strings by some placeholder (e.g. {name}, {email}, {phone}). the second approach, pseudonymisation preserves the information of personal data by replacing these privacy-bearing strings with randomly chosen alternative strings from the same privacy type (e.g. replacing a name with "john doe"). the term deidentification subsumes both anonymization and pseudonymisation. in this work, we focus on anonymization. 1 eder et al. (2019) argues that the anonymization approach might be appropriate to eliminate privacy-bearing data in the medical domain, but 1 @xcite notes that de-identification means removing or replacing personal identifiers to make it difficult to reestablish a link between the individual and his or her data, but it does not make this link impossible. if we shift towards pseudonymisation, we argue that there is still the possibility to resurface the original personal data. henceforth, our goal is to anonimize job postings to the extent that one would not be able to easily identify a company from the job posting. however, as job postings are public, we are aware that it would be simple to find the original company that posted it with a search engine. nevertheless, we abide to the gdpr compliance which requires us to protect the personal data and privacy of eu citizens for transactions that occur within eu member states @xcite . in job postings this would be the names of employees, and their corresponding contact information. 2 over a period of time, we scraped 2,755 job postings from stackoverflow and selected 395 documents to annotate, the subset ranges from june 2020 to september 2020. we manually annotated the job postings with the following five entities: organization, location, contact, name, and profession. to make the task as realistic as possible, we kept all sentences in the documents. the statistics pro-... 13. job description: 14. [xxx organization ] is a modern multi tenant, microservices based solution and floor planning is one major functional solution vertical of the [xxx organization ] platform. @xcite . what you'll be doing: 16. as a [xxx profession ] for [xxx organization ], you will be one of the founding members of our [xxx location ] based floor planning development team. 17. you will be in charge for development of future floor planning capabilities on the [xxx organization ] platform and be the software architect for the capability. 18. you will drive the team to improve the coding practices and boost performance. 19. you will also be a member of our [xxx organization ] and have a major influence on feature roadmap and technologies we use. ... vided in the following therefore reflect the natural distribution of entities in the data. a snippet of an example job post can be seen in figure 1 , the full job posting can be found in appendix a. ## methods for entity de-identification we use a classic named entity recognition (ner) approach using a bi-lstm with a crf layer. on top of this we evaluate the performance of transformerbased models with two different pre-trained bert variants. furthermore, we evaluate the helpfulness of auxiliary tasks, both using data close to our domain, such as de-identification of medical notes, and more general ner, which covers only a subset of the entities. further details on the data are given in section 4.3. ## evaluation here we will outline the results of the experiments described in section 4. all results are mean scores across three different runs. 3 the metrics are all calculated using the conlleval script 4 from the original conll-2000 shared task. is a crf layer necessary? in lstm versus transformer initially, lstm networks dominated the de-identification field in the medical domain. up until recently, large-scale pre-trained language models have been ubiquitous in nlp, although rarely used in this field. on both development and test results ( poor performance with bert overflow bert base is the best embedding method among all experiments using bilty, with bert overflow being the worst with a considerable margin. being able to fine-tune bert base does give a good increase in performance overall. the same trend is apparent with fine-tuning bert overflow , but it is not enough to catch up with bert base . we see that overall machamp with bert base and crf is the best model. however, bilty with bert base and crf does have the best precision. we hypothesized the domain-specific bert overflow representations would be beneficial for this task. intuitively, bert overflow would help with detecting profession entities. profession entities contain specific skills related to the it domain, such as python developer, rust developer, scrum master. although the corpus it is trained on is not one-to-one to our vacancy domain, we expected to see at most a slight performance drop. this is not the case, as the drop in performance turned out to be high. it is not fully clear to us why this is the case. it could be the q&a data it is trained on consists of more informal dialogue than in job postings. in the future, we would like to compare these results to training a bert model on job postings data. auxiliary data increases performance looking at the results from the auxiliary experiments in in general, it can be challenging to train multitask networks that outperform or even match their single-task counterparts (alonso and @xcite . @xcite mentions training on a large number of tasks is known to help regularize multi-task models. a related benefit of mtl is the transfer of learned "knowledge" between closely related tasks. in our case, it has been beneficial to add auxiliary tasks to improve our performance on both development and test compared to a single task setting. in particular, it seemed to have helped with pertaining a high recall score. performance on the test set finally, we evaluate the best performing models on our held out test set. the best models are selected based on their performance on f1, precision, and recall. the results are seen in ## conclusions in this work we introduce jobstack, a dataset for de-identification of english stackoverflow job postings. our implementation is publicly available. 5 the dataset is freely available upon request. we present neural baselines based on lstm and transformer models. our experiments show the following: (1) transformer-based models consistently outperform bi-lstm-crf-based models that have been standard for de-identification in the medical domain (rq1). ( 2 ) stackoverflowrelated bert representations are not more effective than regular bert representations on stackoverflow job postings for de-identification (rq2).