Entity Disambiguation on a Tight Labeling Budget (2023)

Abstract: Many real-world NLP applications face the challenge of training an entity disambiguation model for a specific domain with a small labeling budget. In this setting there is often access to a large unlabeled pool of documents. It is then natural to ask the question: which samples should be selected for annotation? In this paper we propose a solution that combines feature diversity with low rank correction. Our sampling strategy is formulated in the context of bilinear tensor models. Our experiments show that the proposed approach can significantly reduce the amount of labeled data necessary to achieve a given performance.

## introduction many real-world nlp applications face the challenge of training an entity disambiguation model for a specific domain with a small labeling budget. for example, a defense research analyst might be interested in mapping mentions of military equipment appearing in military research articles to a target knowledge base describing emergent defense technologies. the time of an expert might be costly and we would like to minimize the amount of annotations required from them. when training an entity disambiguation model one has often access to a large unlabeled pool of documents. we assume that we can recognize mentions of the target class and that we can generate a set of candidate knowledge base (kb) entities for each of them. solving the entity disambiguation task involves training a model that takes a target mention and a set of candidate kb entities as inputs and outputs the correct grounding (i.e. the correct entity in the kb). we assume that we have a small annotation budget and that our goal is to train the best model possible under the budget constraints. one natural approach to tackle this problem is to design sampling strategies that attempt to maximize the diversity of the selected samples. we follow this line of work and develop a method that combines a model correction step designed to improve the generalization performance of models learnt under tight annotation budgets with a diversity sampling strategy. our work is framed in the context of bilinear tensor models for entity disambiguation. these types of models score mention-entity pairs by exploiting a rich representation of the mention context and entity descriptions in the target kb. furthermore, our model has a bilinear lexicalized attention mechanism that assigns a score to every pair in the cartesian product space of context and entity description words. our first observation is that the lexicalized bilinear attention matrix will be typically low-rank. following this insight we develop a model correction technique based on matrix completion that can improve the generalization performance of bilinear tensor models trained on tight annotation budgets. we combine this with feature diversity sampling, and it is by combining these two ideas that we obtain optimal results. the two main contributions of this work are: 1) we develop a novel strategy to improve the generalization performance of bilinear tensor models trained under tight annotation budgets. our strategy combines low-rank matrix completion of a lexical attention matrix and feature diversity sampling. 2) our experiments on entity disambiguation show that the proposed approach can significantly reduce the amount of labeled data necessary to achieve a given entity disambiguation performance. ## related work the sampling problem in learning under a budget is related to active learning @xcite . a few works addressed active learning for entity linking @xcite . different to active learning in the low budget setting, we don't assume the existence of a prior model and thus our setting resembles the cold start prob-lem @xcite . to the best of our knowledge we are the first ones to address cold start for entity disambiguation. as in active learning, the sampling approach can exploit either the informativeness or the representativeness of data points @xcite . @xcite showed that, for low-budget scenarios, representativeness is a better query strategy. we propose two representative-based sampling approaches and study their effectiveness for the case of entity disambiguation. our method combines diversity sampling with low-rank matrix completion, which has been shown to help generalization in low-budget scenarios @xcite . ## diversity sampling and low-rank matrix completion correction in this section we present our approach for learning an entity disambiguation under a budget setting. our approach combines a low-rank model correction step with diversity sampling. ## experiments: learning with a tight budget we run our experiments on aida conll-yago foot_3 @xcite annotated with entities from wikidata foot_4 . we follow van hulst et al. (2020) to select the candidate entities and ignore samples annotated with out-of-kb entities. we produce a learning curve by training models with increasing annotation budgets. each point uses five random seeds, and we report their average performance. we validate all hyper-parameters (both for the sampling strategy and model optimization) on randomly selected 50 samples from each testa and testb partitions. we will use testa* and testb* to refer to the testa and testb set that excludes the 50 samples used for validation. in addition to the cross-product diversity (cpd) and dense diversity (dd) methods, we experiment with two other baseline methods. random (r) sampling and candidate diversity (cd) which tries to cover as many unique candidate entities as possible (see section 4.2 for the approach to the n-set-cover problem.). figure 1 shows performance on testa* and testb* partition. the methods marked with "+" use lowrank matrix completion correction. we can see that cpd and dd sampling combined with low-rank matrix completion help increases model accuracy compared to random and cd sampling, especially with lower budgets. to give some concrete examples, on testa* cpd+ model correction achieves 81% accuracy with 50 labeled mentions, to get the same accuracy with random sampling you need at least 150 mentions, that is 3 times the number of samples. with 200 mentions cpd+ model correction attains 84.4% accuracy, to get that accuracy with random sampling we need more than 500 annotations. in both instances with our approach the amount of supervision required to attain a given performance is reduced by at least half. figure 5 , shows two arbitrary points from the learning curve and decouple the sampling strategies from the low-rank correction. both cpd and dd sampling helps the model's performance even without low-rank matrix completion. moreover, low-rank matrix completion increase models' accuracies regardless of the sampling strategies. ## conclusion we have shown that by combining low-rank matrix completion of a lexicalized attention matrix with diversity sampling we can significantly improve the generalization performance of a bilinear entity disambiguation model under tight annotation budget constraints.