Improving Few-Shot Domain Transfer for Named Entity Disambiguation with Pattern Exploitation (2022)

Abstract: Named entity disambiguation (NED) is a critical subtask of entity linking, which seeks to connect knowledge base entities with textual mentions of those entities. Naturally, the performance of a model depends on the domain it was trained on; thus, reducing the amount of data required to train models is advantageous. In this work, we leverage recent research on pattern exploitation for NED and explore whether it can reduce the amount of data required for domain adaptation by reformulating the disambiguation task as a masked language modeling problem. Using ADAPET (Tam et al., 2021), which implements a new approach for few-shot learning using fine-tuned transformer-based language models, we produce an NED model which yields, without any sacrifice of in-domain accuracy, a 7% improvement in zero-shot cross-domain performance as evaluated on NEDMed, a new NED dataset of mental health news which we release with this work.

## introduction in order to understand a piece of text, it is often valuable to understand entities which are referred to by that text. while named entity recognition (ner) is an important aspect of this problem, a large variety of applications (e.g. financial credit risk monitoring and open source intelligence gathering) need to further connect these entities to known entities in a knowledge base (kb). this task of linking textual mentions of entities to their kb entries is known as entity linking. entity linking is typically further decomposed into two subtasks: candidate generation and named entity disambiguation (ned). the former is responsible for discovering a set of possible mentions of entities in a given document (for example, producing the candidates for cambridge, ma, usa and cambridge, uk from the surface form "cambridge"). an ned system then takes these lists of candidates and selects which, if any, is the correct referent. named entity disambiguation systems achieve this by utilizing a number of pieces of information, such as related entities, the type of each candidate entity @xcite , and semantic descriptions of each entity (e.g. a snippet from the candidate's wikipedia page). as this association is usually statistically learned from some training dataset, the performance of an ned system on a given document depends on how closely that document's domain is to that of the training data, in terms of vocabulary, syntax, and the types of entities. because these systems are often specialized in specific domains, it is therefore necessary to curate sufficient amounts of training data for each of these applications, which is often costly. our work seeks to reduce the amount of data required via leveraging pretrained language models (lms). lms are often well-suited to assisting lowresource task setups @xcite , for modern language models are sufficiently powerful that their predictive distributions can be interpreted as a basic form of "common-sense reasoning". our contributions are as follows: (1) we take a state-of-the-art baseline ned system @xcite and augment it with an additional signal from an lm fine-tuned with the adapet @xcite procedure, which adapts language models to few-shot learning natural language processing problems. we show that this augmented system, called dca-prompt, achieves similar performance to the baseline in both the same and closely-related domains, but demonstrably outperforms when adapting to a new, dissimilar domain. (2) additionally, we are releasing a new named entity linking dataset, called nedmed, which is based on mental health news data. [cls] <@xmath0. description: <entdesc>. is this a good description @xmath1: pattern used during evaluation. items in square brackets @xcite ) are special tokens, and items in angle brackets (<>) are substituted with information from the problem input. lctx and rctx are the text to the left and right of the entity mention (respectively), mentiontxt is the surface form of the mention, and entdesc is the description of the candidate under consideration. colors represent different logical segments of each piece of the input, and @xcite is what we prompt the language model to substitute into the input. ## methodology in ned (figure 1 ), we presume that, for a given input text containing a set of n mentions (collectively denoted m), which are the textual surface forms of entities, and a set of candidates c i (collectively, c) for each mention m i . an ned system ranks these candidates to select the one most likely to be the referent entity. a full formal description of the task is given in appendix a. adapet solves natural language understanding tasks by "filling-in-the-blank" in natural language patterns. to solve a downstream task, such as classification or question-answering, one formulates the problem instance as some form of prose containing a masked token. a fine-tuned language model is then used to infer which word makes the most sense to substitute this masked token. in order to apply adapet to ned, it was necessary to formulate an appropriate pattern. our system fine-tunes the huggingface (wolf et al., 2020) bert-large-uncased model using the pattern shown in figure 2 , treating the task as a binary classification problem (answering, "does this candidate link to this mention?"). the development process through which we chose this base model and pattern is described in appendix b. as done in the codebase provided by tam et al. ( 2021 ), we fine-tune the model to produce the word "true" for correct mention-candidate pairs and the word "false" for incorrect ones. the entity description ([entdesc]) may be something like a paragraph from an encyclopedia entry or a verbalized form of its knowledge base relationships, as described in @xcite . this representation should capture enough context to uniquely identify the entity. in our work, we use the first paragraph of the candidate's wikipedia page. in order to accurately judge this approach versus state-of-the-art ned systems, we additionally augment an existing state-of-the-art baseline system with an input signal from our adapet-tuned lm. the baseline system we selected is based on dca, and is described by @xcite . in short, this model, is a feedforward neural network which accepts features based on the similarity of the mention context and trained entity embeddings (ψ c ), entity type information (ψ t ), and coherence between the candidates and both previously linked entities and entities linked to those entities (φ and φ ′ ). for further information, readers are referred to @xcite . further technical details regarding our training setup can be found in appendix c. our research focused on the following questions: rq1. does using adapet for ned perform similarly to the baseline cross-domain scenarios for similar domains? rq2. does using adapet for ned reduce the amount of data required to learn the task? rq3. does using adapet for ned improve the ability for a ned system to transfer to another dissimilar domain? to answer our first two research questions, we compared adapet-ned (adapet trained on our ned prompt) against three versions of the baseline dca system: one which operates as normal (etzh-attn+dca-sl from yang et al. ( 2019 )), denoted "dca", one which only has "local" features enabled (i.e. no coherence-based features, which makes it more directly comparable to adapet-ned), denoted "dca (local only)", and a version of the dca system which has been augmented to receive the output from the adapet model's "yes" prediction as an input, denoted "dca-prompt". we model the (2-way k-shot) few-shot learning curves of these models in figures 3(a ) and 3(b), evaluating on in-domain datasets (training on different sized subsets of aida's aida-train training split and evaluating on aida-a and aida-b development and evaluation splits). as expected, this graph shows that adapet-ned greatly underperforms the baseline system. to explore rq1, we measured cross-domain performance on various publicly available datasets and compare to existing benchmarks. specifically, we looked at f1 performance on msnbc, aquaint, ace2004, and cweb (described in section 2), as done in @xcite . these datasets are all general knowledge corpora, based on either news or encyclopedia pages. we train our models on aidatrain and evaluate across all datasets. the results are shown in in order to assess rq3 in a real-world context, we adapted our trained models to a dataset tailored to the medical domain, which is quite different from aida's general news domain. we created a dataset, denoted nedmed 1 , containing 110 internet articles on mental health news, which were partitioned into 66 training documents (nedmedtrain) and 44 evaluation documents (nedmeddev), containing 2,839 and 1,841 mentions, respectively. documents were manually annotated for person, location, and organization types, along with a variety of others. for a full list of types and further details on this dataset, see appendix d. for our experiments, we only utilize entities which have wikipedia links (4,342 mentions, or roughly 92% of the total 4,680). we find that our dca-prompt system yields superior performance in both the zero-shot and transfer learning scenarios. notably, the zero-shot performance of dca-prompt is higher than all three 1 the nedmed dataset is available to download at https: //github.com/basis-technology-corp/ nedmed . metrics of the baseline dca system. ## conclusions and future work these results indicate that pattern exploitation training can effectively be utilized for named entity disambiguation. while results are not state-of-the-art when used in isolation, combining an adapetbased classifier with an existing model which can incorporate global context, such as dca, improves the capacity of that model to flexibly adapt to data from different domains. our new nedmed dataset both provides evidence for this and represents a new domain-specific benchmark which can be used by future ned research. there are a number of ways in which this work could be built upon in the future. this work focused on shifts in domain related to the documents in which mentions are extracted from. another important type of domain shift relates to large changes in the underlying kb. while we expect the system would be able to adapt, this has not been quantified. additionally, the optimal strategy for designing patterns for use with adapet-style techniques is still an open research question @xcite ; as this work relied on human-produced patterns, it is certainly possible that accuracy or data requirements could be improved with more clever pattern design.