Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models (2022)

Abstract: Transformer-based language models trained on large natural language corpora have been very useful in downstream entity extraction tasks. However, they often result in poor performances when applied to domains that are different from those they are pretrained on. Continued pretraining using unlabeled data from target domains can help improve the performances of these language models on the downstream tasks. However, using all of the available unlabeled data for pretraining can be time-intensive; also, it can be detrimental to the performance of the downstream tasks, if the unlabeled data is not aligned with the data distribution for the target tasks. Previous works employed external supervision in the form of ontologies for selecting appropriate data samples for pretraining, but external supervision can be quite hard to obtain in low-resource domains. In this paper, we introduce effective ways to select data from unlabeled corpora of target domains for language model pretraining to improve the performances in target entity extraction tasks. Our data selection strategies do not require any external supervision. We conduct extensive experiments for the task of named entity recognition (NER) on seven different domains and show that language models pretrained on target domain unlabeled data obtained using our data selection strategies achieve better performances compared to those using data selection strategies in previous works that use external supervision. We also show that these pretrained language models using our data selection strategies outperform those pretrained on all of the available unlabeled target domain data.

## introduction named entity recognition (ner) @xcite @xcite is the task of extracting entities from a given piece of text. ner is useful for several + work done while authors were at adobe research. natural language processing (nlp) applications such as information extraction @xcite , retrieval @xcite , and language understanding. over the years, various approaches including rule-based techniques @xcite , unsupervised learning approaches @xcite , feature-based extractions @xcite have been explored for ner. the recent deep learning-based methods are shown to result in state-of-the-art performances for various domains @xcite @xcite . specifically, large pretrained language models such as bert @xcite are widely used for ner, as they mark state-of-the-art performances @xcite . however, these models require large amounts of annotated data which stands as a bottleneck to the training process, particularly when domains having few labeled data are involved. directly fine-tuning bert model on small collection of labeled data can yield sub-optimal results. to address this issue, there have been works on pretraining strategies @xcite to adapt bert-like language models to target domains by further pretraining them on unlabeled target domain data prior to fine-tuning on labeled data to improve performances in tasks like ner. however, further pretraining language models on all of the available unlabeled data from target domains can be sometimes detrimental to the downstream ner performances, if the entire unlabeled data is not aligned with the entity distribution of the labeled data @xcite . specifically, @xcite used two types of unlabeled corpora belonging to the target domain, namely task-level corpus, consisting of unlabeled sentences from the target domain that are highly aligned with the distribution of the labeled downstream ner dataset, and domain-level corpus, consisting of a very large collection of unlabeled sentences from the target domain, excluding sen-tences from the task-level corpus. @xcite showed the advantages of pretraining on both domain-level and task-level corpora for improving ner performance. however, using the task-level corpus only for further pre-training may be inappropriate, as this is usually very small in size. additionally, pretraining on the entire domainlevel corpus can be challenging, particularly in resource-constrained settings * , and can sometimes produce sub-optimal results due to non-alignment and distribution mismatch. @xcite introduced a strategy to select samples from domainlevel corpus to augment the task-level corpus using external supervision in the form of ontologies to improve the downstream ner performance. while this marks the importance of data selection for pretraining using ontology-based filtering, it is constrained by the availability of meta-data and entity information from wikipedia, and thus, is not easily extensible to other low-resource domains † . in this paper, we introduce new methods for data selection via (i) cosine similarity-based retrieval in embedding space of domain-level corpus generated using pretrained and fine-tuned bert (fine-tuned on labeled target domain data), (ii) entity prediction on domain-level corpus using fine-tuned bert. note that both these strategies do not require any additional supervision (e.g., in the form of ontologies) or metadata, thus making them easily extensible to other low-resource domains. similar to our method, @xcite proposes unsupervised methods of computing similarities between different corpora based on word-embedding and perplexity scores of bert-base-cased. however, unlike our method of selecting sentences from the domainlevel corpus based on similarity scores, @xcite uses the similarity scores to select the most appropriate source domain (out of multiple source domains) for a particular target domain (in our setting, both domain-level and task-level corpus belong to the same domain). we evaluate our method on i2b2 @xcite , atticus @xcite , and five other domains curated by @xcite with different amounts of selected sentences for pretraining. experimental results show that our proposed methods for data selection result in better performances in downstream ner tasks, * resource-constrained setting(s) refers to settings with limited availability of compute (cpu or gpu) in terms of time and magnitude. ## task definitions following the notions in @xcite , let d @xmath0, and t @xmath1, y l i )} n l @xmath2, which denotes labeled corpus tagged with bioes for ner, where l c ⊆ t c , and given a large unlabeled d c and relatively small t c , our task involves selectively sampling sentences s @xmath3, where s c ⊂ d c and n d ≫ n s , such that pretraining on t c augmented with s c increases the performance for ner, without having to pretrain on entire d c , which can be very time and resource intensive. for a clearer understanding of d c , t c , and l c we present an example scenario involving these 3 different corpus types: let x be a smallscaled legal firm that has a limited collection of customer contracts. to ease their workflow, they want to automate the process of extracting named entities (like contracting-party, contract-amount, etc.) from contracts using ner models. however, they don't have permission to outsource all the customer contracts to external annotators to create labeled data for ner due to legal restrictions. out of all the customer contracts, only a very small number (without legal restrictions) can be outsourced for annotations. they also have a large collection of unlabeled sec contracts ( https://www.sec.gov/edgar.shtml ) that are publicly available (however, these sec contracts are somewhat different from customer contracts of x in terms of format, layout, etc.). in the context of our setting all the customer contracts to form t c , the small subset of these contracts that can be used for annotations form l c , and the sec contracts constitute d c . ## conclusion we propose novel methods of data selection techniques, which do not require additional external supervision, for pretraining bert for ner. in addition to being illustrated on 7 diverse domains, our method can be easily extended to ner for any new domain with very scarce labeled data and plenty of domain unlabeled data. experiments on multiple domain datasets demonstrate that our selection techniques are better than naive pretraining on the entire domain corpus and also achieve better performance compared to state-of-the-art data selection methods using external knowledge bases. since we may not get domain labels for every corpora, in future we will try to extend our work with corpus consisting of a mixture of different domains.