Entity Linking via Explicit Mention-Mention Coreference Modeling
Abstract
Learning representations of entity mentions is a core component of modern entity linking systems for both candidate generation and making linking predictions.
In this paper 1 , we present and empirically analyze a novel training approach for learning mention and entity representations that is based on building minimum spanning arborescences (i.e., directed spanning trees) over mentions and entities across documents to explicitly model mention coreference relationships.
We demonstrate the efficacy of our approach by showing significant improvements in both candidate generation recall and linking accuracy on the Zero-Shot Entity Linking dataset and MedMentions, the largest publicly available biomedical dataset.
In addition, we show that our improvements in candidate generation yield higher quality re-ranking models downstream, setting a new SOTA result in linking accuracy on MedMentions.
Finally, we demonstrate that our improved mention representations are also effective for the discovery of new entities via cross-document coreference.
Introduction
Natural language corpora, such as biomedical research papers (Leaman and Lu, 2016), news articles (Milne and Witten, 2008; Hoffart et al., 2011) and, more generally, web page text (Gabrilovich et al., 2013;
Lazic et al., 2015a), often contain ambiguous mentions of entities.
Resolving this ambiguity requires mentions to either be linked to a knowledge base (KB) of entities or discovered as a new
KB concept if no suitable entry exists.
Grounded entity mentions are beneficial for tasks such as question-answering (Das et al., 2019), semantic search (Leaman and Lu, 2016), recommendation ranking (Noia et al., 2016) and KB construction (Ling et al., 2015).
The task is made particularly challenging in zero-shot settings, where not every entity has labeled training data (Lin et al., 2017;
Logeswaran et al., 2019).
In such settings, a common approach is to make use of entity descriptions, types, and aliases to form entity representations, which can then be used for making predictions.
Learned vector representations of entity mentions are an integral part of modern linking systems (Gillick et al., 2019; Wu et al., 2020, inter alia).
These representations are used for (a) retrieving a short-list of entity candidates for a mention to use with a re-ranker (Wu et al., 2020), (b) making linking predictions directly (Zhang et al., 2021;
Liu et al., 2020; Sung et al., 2020), and (c) performing coreference by clustering mentions to form entities (Logan IV et al., 2020).
In this work, we present a new objective and training procedure for learning mention and entity representations that explicitly model mention coreference relationships.
Our proposed method uses a supervised clustering training objective based on forming a directed minimum spanning tree, or arborescence , over mentions and entities.
We hypothesize that such coreference links provide a useful inductive bias because the two tasks are inherently related (Angell et al., 2021; FitzGerald et al., 2021).
We thoroughly analyze the performance of the proposed procedure in each of the aforementioned use cases on MedMentions (Mohan and Li, 2019) and ZeShEL (Logeswaran et al., 2019), two challenging datasets that require zero-shot generalization at inference.
Retrieving Candidates
We illustrate that our approach yields mention and entity representations useful for candidate retrieval.
We show improvements over baselines that use similarly parameterized models, achieving gains of at least 7.94 and 0.93 points in recall @64 over two standard dualencoder training procedures on MedMentions and
ZeShEL, respectively.
We also consider the linking capacity of our learned embeddings without re-ranking and find that their performance (i.e re- call @1) indeed improves upon our baselines.
Our best performing models show gains of 13.61 & 15.46 points in linking accuracy on MedMentions and 12.06 & 1.52 points on ZeShEL.
Linking Predictions
We further consider the improvement in downstream training of full crossattention re-ranker models using higher quality candidates generated by our approach.
We show consistent gains in linking accuracy on MedMentions, setting a new state-of-the-art with a 1.63 point gain over the previous best model.
We also note that our proposed approach shows mixed results on
ZeShEL, with one variant outperforming all compared models by at least 1.19 points, while the other two underperform the baselines.
We analyze this behavior in a later section and discuss the characteristics of the data distribution sufficient to make our approach effective.
Cross-Document Coreference
Finally, we illustrate that the learned representations can be used to perform coreference of mentions across documents.
This indicates that our approach could be used to discover entities in settings where there is limited or no existing knowledge base of entities.
Arborescence-based Training for Mention & Entity Representations
In this section, we describe our approach for constructing training objectives for dual-encoders that model mention coreference relationships.
2. Problem Definition
Each document d of a corpus D contains a set of entity mention spans M d = { m d 1 , m d 2 , . . .
, m dN } .
All mentions in the corpus are given by M = (cid:83) d ∈D M d .
Following (Logeswaran et al., 2019;
Angell et al., 2021), we assume that these mentions are pre-identified spans of text.
Formally, we define the task of entity linking as follows: given a knowledge base of entities E and a set of mentions M , predict an entity e di ∈ E for each mention m di .
We use e ⋆di to refer to the ground truth entity label for m di .
The zero-shot task refers to the setting where there are entities in the knowledge base that do not have any labeled mentions in the training data.
Linking decisions must instead rely on the provided information for entities, such as descriptions, aliases, and/or entity types.
Coreference We also consider a setting in which the KB of entities is not known in advance and entities must be discovered .
For this task, we map every entity mention m di to a cluster and assign a coreference label c di ∈ C that is independent of the entity labels in the KB.
2. Coreference-based Similarity
In order to jointly train both the mention and entity encoders, we define a similarity measure and an analogous procedure for sampling positive training examples that intersperses the selection of coreferent mentions and gold entities based on a singlelinkage structure formed by the representations generated by the model snapshot.
We construct k -nearest neighbor graphs over coreferent mention and entity clusters, followed by the application of a pruning algorithm to generate arborescence (directed MST) structures rooted at entity nodes.
The resultant edges after pruning the graphs represent the pairs of positive examples used for training.
Let G be a graph with nodes V
= M ∪ E and directed edges
E ⊂ V × V .
Each edge ( x, y ) of the graph has an associated weight w x,y .
We define a dissimilarity function f between two nodes u, v ∈ V to be the weight of the minimax path between the nodes, i.e.
[Equation]
∞ otherwise (
[Equation]
where connected( u, v ) is true if there exists a directed path from node u to v in G , and u ⇝ v is the set of all paths between u and v .
In words, the dissimilarity between u and v is the minimum of the largest edge weights in all paths between the two nodes, and this is often referred to as the "bottleneck edge".
This measure has the property of emitting low dissimilarities between nodes even when their direct edge weight w u,v is high by connecting them through a chain of low-weight edges, providing an inductive bias well-suited for coreference, i.e. not all pairs of points in a cluster are nearby (Figure 1).
This inductive bias is not achieved if we sum edge weights and simply find the minimum path.
With this definition of dissimilarity, we now define how edge weights are calculated.
We use two models: a mention-pair affinity model, ϕ : M × M → R , and a mention-entity affinity model, ψ : E × M → R .
An edge between two
[Table]
</latexit
[Table] [Table]
[Table]
YbqkUvF//zhgkEV27KwjgBqg8sBgUJtyCy8jSsMZOUAJ9rgolkeleLTLHEBHRmNR2Cs3jyMuk1G855o3l3UW9dl3FU0TE6QWfIQZeohW5RB3URQRl6Qi/o1Xg0no034/2ntGKUPYfoD4yPb6IolxI=</latexit>
[Figure]
Enc E <latexit sha1_base64="erSOklk2XvTW/LI9jnRMMyiCjdw=">AAACAHicbVDLSsNAFJ3UV62vqAsXboJFcFWSKuiyKAWXFewD2hAm00k7dCYJMzdCCdn4K25cKOLWz3Dn3ziJWWjrgYEz59zLvff4MWcKbPvLqKysrq1vVDdrW9s7u3vm/kFPRYkktEsiHsmBjxXlLKRdYMDpIJYUC5/Tvj+7yf3+A5WKReE9zGPqCjwJWcAIBi155tFIYJgCpO2QZF7xkSJtZ55Ztxt2AWuZOCWpoxIdz/wcjSOSCBoC4VipoWPH4KZYAiOcZrVRomiMyQxP6FDTEAuq3LQ4ILNOtTK2gkjqF4JVqL87UiyUmgtfV+YbqkUvF//zhgkEV27KwjgBqg8sBgUJtyCy8jSsMZOUAJ9rgolkeleLTLHEBHRmNR2Cs3jyMuk1G855o3l3UW9dl3FU0TE6QWfIQZeohW5RB3URQRl6Qi/o1Xg0no034/2ntGKUPYfoD4yPb6IolxI=</latexit>
[Figure]
mentions m i and m j has weight:
[Equation]
ϕ
[Equation]
m
[Equation] and the weight of the edge from entity e to m i is: [Equation] Each of ϕ ( · , · ) and ψ ( · , · ) are independently parameterized by dual-encoder transformer models (Gillick et al., 2019; Humeau et al., 2019), one for mentions ( Enc M ), and one for entities ( Enc E ).
The affinity models are simply the inner products of the associated encoded representations:
[Equation]
For the mention encoder, Enc M , the transformer input is the surrounding mention context with the mention span marked by special tokens [START] and [END] :
[Equation]
where c left and c right are the left and right contexts of the mention m i in the document.
For the entity encoder, Enc E , the transformer takes as input the title and description of the entity:
[Equation]
In this input, e desc is the token sequence corresponding to the description of the entity, which could include natural language text related to the entity, such as a "wiki" entry, a list of entity aliases, or other available features useful in forming an entity representation. We use a special token [TITLE] to separate the title text from the description.
2. Training Procedure
We now define our approach for training the affinity models, ϕ ( · , · ) and ψ ( · , · ) , and their associated encoders, Enc M and Enc E .
Our objective is to optimize the dissimilarity function f ( · , · ) such that the procedure infers a set of clusters that each contain exactly one entity, and every mention is assigned to the cluster containing its ground truth entity.
We optimize f ( · , · ) using mini-batch gradient descent by sequentially building batches of mentions B ⊂ M over the training data, where each m i ∈ B has its gold entity defined by e ⋆i .
We then build a graph
G B with nodes consisting of (a) each m i ∈ B , (b) each mention coreferent to m i ∈ B , and (c) the set of gold entities for each m i ∈ B .
For every m i , we build a set of directed edges defined by
[Equation]
where M e ⋆i is the set of coreferent mentions with e ⋆i as the ground-truth entity.
The complete set of edges in graph G B for a mini-batch B is then given by E ( G B ) =
(cid:83) m i ∈
B E
m i .
Observe that the resultant edges ensure that each connected component contains exactly one entity (namely, the gold entity for the mentions in that connected component).
Forming Clusters for Positive Sampling
The graph G B is input to a constrained clustering procedure that partitions a graph G into disjoint clusters
C =
{ C 1 , . .
, C M } such that each cluster contains at most one entity.
There are three constraints that every C ∈ C must satisfy:
[Equation]
where λ is a hyperparameter representing the dissimilarity threshold over which edges between nodes are dropped.
We set λ = ∞ during training.
These constraints ensure that (i) there is at most one entity in each cluster, (ii) if u is reachable from v then every edge in the path from v to u has a weight ≤ λ , and (iii) each node in the cluster has a path connecting itself with every other node in the cluster.
We solve this constrained clustering problem, i.e., partition graph G , using a process similar to Angell et al.
Specifically, we first remove all edges in graph G with weight greater than threshold λ .
We then evaluate each edge ( u, v ) ∈ E in descending order of dissimilarity and check if its presence violates any of the three constraints defined above, removing the edge from E if it does.
If not, we evaluate whether there is an entity in the connected component of node u , i.e. | C u ∩ E| = 1 .
If | C u ∩ E| = 1 , we temporarily drop edge ( u, v ) and check whether v can still be reached by an entity node.
If reachable, we permanently drop ( u, v ) , maintaining the validity of constraint (i) as well as our minimax dissimilarity function f ( · , · ) .
If an entity cannot reach v , we retain edge ( u, v ) , preserving the connectivity of the cluster, and iterate further.
Our predicted clusters are the resultant connected components in the partitioned graph G .
Using this procedure on each E m i to generate a pruned set of edges E ⋆ m i , we construct a partitioned target graph G ⋆B =
{ E ⋆m i | m i ∈ B } , which is used to optimize the parametric encoder models.
Note that each mention node in a target edge set E
⋆m
i has only one incoming edge originating from either an entity or a mention, and the selection of E ⋆m i was done in a way to minimize the dissimilarity function f ( · , · ) between mentions and entities with coreferent labels on the subgraph of the mini-batch.
For every cluster with an entity node, the edge structure is a directed analogue of the minimum spanning tree, where there exists a directed path from the entity node to every other node in the cluster.
This structure is often referred to as the minimum spanning arborescence , thus lending its name to our method, i.e. A RBORESCENCE -based linking.
Negative Sampling
Akin to the graph embedding objectives used by Nickel and Kiela (2018) and others, we construct our objective by sampling hard negative edges.
For each mention m i ∈ B , the set of negative edges N ( m i ) is the k/ 2 lowestweight incoming edges from E \ { e ⋆i } and the k/lowest-weight incoming edges from M \ M e ⋆i , where k is a tuned hyperparameter. In other words, we sample negative mention and entity edges that are most similar to the gold edge.
Loss
Function
We define [Equation]
) } to be the set of all neighbors with an outgoing edge to m i in the training graph.
Let I u,m i be the indicator variable such that I u,m i
= 1 if ( u, m i )
∈
E ∗ m i
and I u,m i
= 0 otherwise.
Our loss function with respect to each mention m i ∈ B is then defined as follows:
[Equation]
where σ ( · ) is the softmax function over all edges in Γ( m i ) ×
{ m i } .
The loss for the entire batch B is the mean of losses over all mentions in B .
Optimizing this loss function requires simultaneously increasing the likelihood of the positive edges and decreasing the likelihood of the negative edges.
This objective and training routine are inspired by the supervised single-linkage clustering proposed by
Yadav et al.
but differs in the choice of loss function and selection of negative examples.
We also experimented with the standard cross-entropy formulation, but found its performance subpar.
Experiments
We are interested in investigating the following empirical research questions:
[Table]
MedMentions
ZeShEL
[Table]
|E
[Table]
|E \ E
[Table]
|
Dev
[Table]
7K • Does our proposed approach improve the recall of candidate generators? Do improvements in candidate generation at training lead to improvements in downstream re-ranking models? • Does our approach result in better learned mention embeddings that can be used for coreference or discovering entities when a KB does not exist?
Experiment Details
Our experiments are run on top of BLINK (Wu et al., 2020), a PyTorch (Paszke et al., 2019) implementation of dual- and crossencoder architectures for entity linking, with model fine-tuning performed over only BERT-base, since gains from pre-trained LM size are unrelated to our approach.
More details are provided in Appendix §A.1.
3. Datasets
We run experiments on two entity linking datasets that both require generalization to unseen entities at test time.
Each document in the datasets contain a set of entity mention spans, which are pre-defined using common mention-detection heuristics.
KB entities are composed of two metadata attributes – an entity title and description , which are natural language sequences of text.
ZeShEL, additionally, contains a fine-grained type specification, which is needed due to the diverse disjoint domains contained in the dataset.
The statistics for both datasets are reported in Table 2.
MedMentions (Mohan and Li is a collection of titles and abstractions of bio-medical research papers.
The KB that is used for this dataset is the 2017AA full-version of UMLS.
The validation and test sets contain both entities that are present in the training set as well as entities that are zero-shot (never seen at training time).
We use the author-recommended ST21pv subset.
ZeShEL (Logeswaran et al., 2019) is a collection of crowd-sourced wikis, which are divided into train, validation, and test splits such that no Fandom topic overlaps across the sets.
In this way, all entities that appear at validation and test time are not seen during training.
3. Dual-Encoder Retrieval
In order to robustly evaluate the benefit of modeling coreference relationships for learning representations, we construct three variants of our proposed dual-encoder training objective, which jointly train both the mention-mention similarity function ϕ ( · , · ) and the mention-entity similarity function ψ ( · , · ) .
We compare to baselines that only explicitly train ψ ( · , · ) and rely on the structure of ϕ ( · , · ) sharing representations with ψ ( · , · ) to provide meaningful mention-mention similarities.
Our proposed objectives differ in how the positive training pairs are constructed, thus providing a way to analyze the general idea of using coreference rather than any one specific target structure for training. Our baselines are identical to each other except in how negatives are sampled. [Table]
Arborescence
In the first training variant, for each mention query, we begin by constructing a fully-connected graph of the ground truth coreferent mention cluster along with the gold entity.
We then apply the pruning procedure described in the previous section to compute an arborescence rooted at the entity node.
From the resultant graph, each pair of a mention and its incoming-edge node (which can either be a coreferent mention or the gold entity) is treated as a positive example for training.
Following previous work (Gillick et al., 2019), we use hard negative mining with k =negatives composed of equal number of mention and entities.
1-NN Arborescence
Instead of constructing a fully-connected k -NN graph over the entire gold cluster, in this variant we approximate the arborescence structure by pruning a restricted graph of only the gold entity, the query mention, and the most similar within-cluster mention neighbor of the query.
We keep all other details of the training procedure identical to the first variant.
1-Rand Arborescence
A third training objective we explore modifies the initial k -NN graph construction by restricting the nodes to the gold entity, the query mention, and a random withincluster mention neighbor of the query, instead of the nearest-neighbor.
Baselines
We compare to two baselines following previous work: (a) training ψ ( · , · ) with random negatives (I N - BATCH N EGATIVES ) where each gold entity for a mention in a training batch is treated as a negative example for all other mentions in the batch, and (b) training ψ ( · , · ) with hard negatives ( K -NN N EGATIVES ) similar to the negative mining in our proposed methods albeit with only mention- entity positive selection.
Results
In Table 1, we report the test set recall @64 for each dual-encoder model, where the prediction is evaluated as a hit if the gold entity is retrieved in the top-64 candidates for a mention.
On each dataset, we additionally include the performance of candidate generators used by previous works that we compare to.
We find that models trained with explicit coreference relationships outperform those that incorporate this relationship only indirectly.
For recall @64, our proposed methods improve over the baselines by at least 7.94 percentage points on MedMentions and 0.93 points on ZeShEL.
Even at linking, or recall @1, our proposed methods show similar improvements with gains of 13.61 and 1.52 points over the next best baseline models.
We perform a more comprehensive analysis of the dual-encoder linking performance and describe our inference approach and results in Appendix §A.2 and §A.3.
We posit that much of the observed gains in recall using our methods result from higher quality mention embeddings generated due to a wide array of surface forms available to mention queries at training.
Since each training example evaluates not only the gold entity but also its coreferent mentions, this leads to better generalization of representations.
We evaluate this improvement in representations in the clustering / coreference setting in §3.5.
3. Cross-Encoder Re-ranking
To answer our second research question, we compare five cross-attention models, which are trained using entity candidates generated by the dualencoder variants discussed in the previous experiment.
Training and inference batches are constructed by concatenating each mention with an entity candidate separated by a [SEP] token.
Similar to Wu et al.
(2020), we use the top-64 retrieved entities as hard negatives during training and as linking candidates during inference.
Results
We report the cross-encoder linking accuracy for MedMentions in Table 3.
We additionally
Oracle
Self
Union
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
report the breakdown of accuracy on subsets of test mentions for which the ground truth entities were not evaluated ("unseen") during training, illustrating the zero-shot capability of the models.
We also include the current state-of-the-art results by Angell et al. (2021), which uses an n-gram based model for candidate generation and two crossencoder models, one each for mention-mention and mention-entity scoring, for re-ranking. We observe that each cross-encoder trained with candidates generated by an arborescence-based model outperforms the baselines, including the current SOTA by at least 0.63 points, and the best performing model – A RBORESCENCE – achieves 1.63 point gains.
We note, however, that Angell et al. (2021) does better on unseen entities by 1.91 points compared to A RBORESCENCE , which might be a result of benefiting from a reduced search scope owing to the within-document nature of their TF-IDF retriever.
Table 4 contains linking results for ZeShEL, where each reported model varies only in the method used for retrieving the entity candidates, while the cross-encoder re-ranker training method is held constant ( K -NN N EGATIVES with k = 64 ).
Since ZeShEL is completely zero-shot, we do not include a seen-unseen analysis.
We follow Wu et al.
and report the unnormalized accuracy, which is calculated as the percentage of successes out of the total number of query mentions in the test set, and the macro-averaged unnormalized accuracy, which is a simple average of the unnormalized accuracies over the different "worlds", or domains, in the test set.
We find that the best performing model is 1-R AND A RBORESCENCE , with a 1.19 point difference in macro-averaged accuracy over the next best model (Wu et al., 2020).
We also note that, unlike on MedMentions, not all of our proposed models have higher accuracy than the mention-entity baselines.
Since a key mo- [Figure] tivation for the arborescence-based methods is to explicitly model coreference relationships during training, we expect performance gains to be strongly correlated with the number of coreference links present within the dataset.
We analyze the two datasets in terms of the number of mentions for each KB entity, which can be thought of as how large each cluster of coreferent mentions is.
We report a histogram distribution in Figure 2 and find that the clusters in ZeShEL are typically very small (at most 3), whereas in MedMentions, each cluster has many more mentions with maximum sizes of 1256, 434, and 447 across the train, validation, and test sets. Finally, we also provide representative examples of predictions comparing the link predictions by our best-performing A RBORESCENCE -based method to the baseline of Angell et al. (2021) on
MedMentions and Wu et al.
(2020) on ZeShEL in
Appendix Table 7 and Table 8, respectively.
3. Oracle Inference
In this setting, we isolate the re-ranking capability of the cross-encoder from the quality of the candidates retrieved at inference .
This setting also removes the upper-bound on re-ranking accuracy by artificially injecting the ground-truth entity in the top-64 candidates retrieved at inference for each mention where retrieval failed.
An additional setting we explore holds this oracle candidate set constant across each variant of the cross-encoder by taking a union over all dual-encoder candidate sets and then proceeding to inject the ground-truth.
This construction provides a way to disentangle the factor of candidate retrieval quality at inference, which otherwise conflates the comparison of re-ranking performance.
We refer to these oracle settings as
SELF and UNION , respectively.
Results
As seen in column Oracle of Table 3, the baseline models show higher linking accuracy than our proposed methods when the gold entity is guaranteed to be present in the original candidate set.
However, the performance of the baseline models drops significantly ( ≥ 32 points) when evaluated with the UNION candidate set, while the arborescence-based models show a ± 0.9 point variation.
We believe this discrepancy clearly highlights the poor quality of candidates retrieved by the baseline models compared to our proposed methods.
This also explains the inflation in accuracy of the baselines on the SELF set due to the trivial discrimination task presented to the cross-encoders.
We further point to linking performance on the
UNION set, which provides the more challenging task of differentiating between higher quality candidates that are similar. We argue that the large performance difference ( ≥ 26 .
75 points) is strongly indicative of the greater linking capacity of our proposed methods.
In Table 4, we report both the micro accuracy and macro-averaged accuracy for the two oracle sets.
We observe that 1-R AND A RBO performs the best on the UNION set, but is marginally outperformed by I N -B ATCH on micro accuracy on the SELF set by 0.02 points.
In contrast to the fluctuation on MedMentions, the relative uniformity in results on the oracle candidate sets indicates that the candidates generated by each model have similar quality.
3. Mention Coreference
Next, we evaluate the quality of the learned mention representations for cross-document coreference using the entity labels of each mention as its ground truth cluster assignment.
To form clusters, we build mention-only arborescences using the clustering procedure described in §2.3, tuning [Figure] [Table] [Figure]
[Table] [Figure] [Figure]
[Table] [Figure] the threshold value, λ , based on the validation data. We report the Adjusted Rand Index (ARI) clustering scores in Table 5 using each of our dualencoder representation learning objectives.
For both ZeShEL and MedMentions, we report ARI on all the test mentions (denoted A LL ).
For MedMentions, we report two additional settings: (a) ARI when clustering mentions with ground truth entity not seen at training (denoted U NSEEN O NLY ), and (b) clustering on all mentions but evaluating on the U NSEEN O NLY set (denoted A LL /U NSEEN ).
The results show that representations learned with the A RBORESCENCE objective perform best on each setting, aligning with the inductive bias of its training procedure and indicating its utility in a setting where new entities must be discovered . We further probe the inductive bias of the arborescence-based training procedures by inspecting whether improvements in entity linking recall are accompanied by similar gains in mention coreference performance. In Figure 3, we plot entity and mention recall@64 for each training method on the test set of the two datasets.
Mention recall is calculated by retrieving 64 nearest-neighbors for each mention and counting the number of neighbors that are coreferent as a proportion of the total number of coreferent mentions limited to 64. Entity recall is calculated as defined in §3.2. We find that entity recall indeed demonstrates a positive correlation with mention recall on both datasets when the proposed coreference-based training procedures are compared with entity-only methods. We posit that this demonstrates the efficacy of using explicit mention coreference relationships to learn representations for entity linking.
Entity linking has been widely studied (Milne and Witten, 2008; Cucerzan, 2007;
Lazic et al., 2015b; Gupta et al., 2017; Raiman and Raiman, 2018; Kolitsas et al., 2018; Cao et al., 2021, inter alia).
Dutta and Weikum (2015) combine clustering-based cross-document coreference decisions and linking around sparse bag-of-word representations not well suited for the embeddingbased representations used in this work.
Other works use global or collective models (Kulkarni et al., 2009;
Hoffart et al., 2011; Cheng and Roth, 2013; Ganea and Hofmann, 2017; Le and Titov, 2018, inter alia), which consider the compatibility of entity linking decisions made in the same document(s) rather than making independent predictions.
Zhang and Stratos (2021) use noise contrastive estimation to mine hard negatives for the linking task.
Several previous works have developed models for the crossdocument coreference setting where no entity KB is known in advance (Bagga and Baldwin, 1998;
Gooi and Allan, 2004; Singh et al., 2011; Barhom et al., 2019; Cattan et al., 2020; Caciularu et al., 2021; Ravenscroft et al., 2021; Logan IV et al., inter alia).
Alternatives to Cross-Encoders
Our work demonstrates how clustering-based training improves dual- and cross-encoder models for linking and discovery.
If prediction efficiency, and not training efficiency, was the only concern, one could also use model distillation to improve dualencoder performance (Hinton et al., 2015; Izacard and Grave, 2021, inter alia).
We could also consider models such as poly-encoders as alternatives to dual-encoders (Humeau et al., 2020).
Conclusion
We present a novel approach for learning mention and entity representations for use in entity linking candidate generation and prediction, as well as in the discovery of new entities.
Our method uses an objective that explicitly incorporates mention-tomention coreference relationships.
We demonstrate its empirical effectiveness through analysis on two datasets — MedMentions and the Zero-Shot Entity
Linking dataset.
As future work, we hope to further analyze these objectives with the lens of efficiency, distillation, and domain transfer.
Ethical Considerations
There are several ways in which entity linking / resolution models could be biased and a potential for those biases to have harmful downstream consequences.
There is already a large body of work studying the biases in language models (such as those used for fine-tuning in our work) and coreference models, most notably in understanding when error rates in coreference differ across certain populations (e.g., genders, races, and other entity types, more broadly, that display skewed distributions in the data).
For instance, if entity mentions are author names on citation data and the entities are scientific authors, aggregated statistics like h-index or citation count could be biased if the models used to disambiguate the author names are biased.
If entity linking and discovery systems are used to build or populate knowledge bases, those systems may propagate these biased predictions.
This can be particularly problematic if one used such a biased knowledge base to train future models, thus perpetuating and amplifying the skew.
Lastly, we also note that entity linking and discovery are analogous to surveillance and tracking in computer vision, which should warrant substantial weight of ethical considerations.
Acknowledgements
We thank Nishant Yadav, Sunil Mohan, and members of UMass IESL and NLP groups for helpful discussion and feedback. This work is funded in part by the Center for Data Science, the Center for Intelligent Information Retrieval, in part by the National Science Foundation under Grant No. 1763618, in part by the Chan Zuckerberg Initiative under the project Scientific Knowledge Base
Construction, and in part using high-performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative. Rico Angell is supported by the National Science Foundation Graduate Research Fellowship under Grant No. Any opinions, findings, conclusions, and recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.
A Appendix A. Experiment Details
Each training procedure is run on a single machine using 2 NVIDIA Quadro RTX 8000 GPUs.
Our dual-encoder models for ZeShEL and MedMentions have 218M and 230M parameters, respectively.
Each variant is optimized using mini-batch gradient descent using the Adam optimizer forepochs using a mini-batch size of 128 to accumulate the gradients.
Experiments with batch sizes < 128 performed poorly, possibly due to increased fluctuation of gradients, and sizes > 128 were computationally infeasible to run with our available compute resources.
For ZeShEL, the dual-encoder models are trained using 192 warm-up steps and learning rates of 1e-5, 3e-5, and 3e-5 for I N - BATCH , K -NN, and A RBORESCENCE -based models, respectively.
For MedMentions, each model is trained using 464 warm-up steps and a learning rate of 3e-5.
All cross-encoder models are trained with a minibatch size of 2, learning rate of 2e-5, and an additional linear layer.
Our MedMentions and ZeShEL cross-encoder models have 108M and 109M parameters, respectively.
We use FAISS 2 (Johnson et al., 2017) for fast nearest-neighbor search during graph construction at both training and inference.
For MedMentions, the execution time was 70 mins to embed and index 2M entities and 120K mentions, and 20 mins to perform exact nearestneighbor search for the 120K mentions.
A. Dual-Encoder Inference Procedure Building the Graph The structure of the graph
G impacts the dissimilarity function by changing the paths between pairs of nodes in addition to changing which pairs of nodes are connected.
We advocate for a simple, deterministic approach to construct this graph.
For each mention m , construct
E m by (a) adding edges from m ’s k -nearest neighbor mentions in M to m , and (b) adding an edge from m ’s nearest entity to m :
[Equation]
The complete collection of edges E in G is given by E ( G ) =
(cid:83) m ∈M E m .
There are other ways that one could conceivably pick the pairs of mentions to be connected in the graph.
For example, one could use the minimum spanning tree over the mentions.
This approach, however, has a few drawbacks: (a) the directionality of nearest neighbor relationships is ignored leading to noise in the graph, and (b) the resultant graph includes edges that cross cluster boundaries due to this approach forcing all pairs of mentions to be connected, which is undesirable.
Forming Clusters & Making Predictions
To make linking decisions for each mention m di , we assign the ID of the entity present in the mention’s cluster as the linking label (or NIL if there is no entity in the cluster).
Let C ( m di ) be the predicted cluster of mention m di , then:
[Equation]
Furthermore, the target clusters we aim to predict in the entity discovery setting are exactly C .
A. Experiment: Dual-Encoder Linking
Each model is evaluated using three inference procedures.
"Independent" refers to predictions made using only mention-entity edges.
This method was used by Wu et al.
(2020) to generate candidates for a cross-encoder model trained on ZeShEL.
"Clustering ( UNDIRECTED )" refers to a hierarchical agglomerative clustering (HAC) procedure, following previous work by Angell et al.
(2021), which is akin to the procedure for positive sampling used for training our arborescence models but with no edge directionality.
"Clustering ( DIRECTED )" adds directed edges to the previous method.
For each model, we pick the best performing inference procedure on the validation set and report the test set performance.
We report the linking accuracy in Table 6 and leave out models from previous works since they do not report linking metrics of their candidate generators.
We specify the inference method used in each case, chosen based on the validation set accuracy of the models.
Similar to our cross-encoder results in Table 3, we also report the "seen" and "unseen" performance on MedMentions. A. Qualitative Results
In Table 7 and Table 8, we provide a set of representative examples that demonstrate the improvement in entity linking that our proposed coreferencebased methodology empirically provides on MedMentions and ZeShEL, respectively.
[Table]
[Table]
Angell et
[Table]
This study aimed to determine the methylation phenotype in colorectal cancer for identification of predictive markers for chemotherapy response .
We performed DNA methylation profiling on 43 non - recurrent and five recurrent colorectal cancer patients using the Illumina Infinium HumanMethyla- tion450 Beadchip assay [...] [Table] [Table] [Table]
..] and as an experienced and esteemed paladin comes equipped with perhaps the best items of any NPC; he has both an enchanted sword and plate mail that only he can use.
As an inquisitor paladin [...]
[Table]
Sword
[Table]
): A sword is a melee weapon that in its simplest form consists of a blade and a hilt.
This definition includes those weapons that fit the archetype of blade [...]
[Table]
[Table]
): The warblade (also known as the noble warrior’s blade, or ary’velahr’ke ym in elven), was one of three elfblades created in ancient Cormanthyr to help the elves establish a ruling family [...]
[Table]
The
[Table]
was
[Table]
special two
[Table]
the
[Table]
History
[Table]
He received [
[Table]
[Table]
was just
[Table]
counselor about his disappointment
[Table]
there were no cigars
[Table]
the
[Table]
[Table]
He was
[Table]
a drink and chatting with a bartender in Ten Forward when Montgomery Scott entered the
[Table]
and later
[Table]
" something
[Table]
[Table]
Wallace worked
[Table]
[Table]
[Table]
Green tea (5CCB83C71D089E29): Green tea was a type of tea made with tea leaves that had undergone little oxidation.
It was an exceptional source of antioxidants, alkaloids, and amino acids.
In 2151, sub-commander T’pol came to the mess hall late at night for a cup of hot green tea [...]
[Table]
Unnamed beverages ( 23rd century ) (2A887C680DA92880):
The following is a list of unnamed beverages consumed during the 23rd century.
Green drink: In 2268, Montgomery Scott offered this green drink to the Kelvan Tomar, which he claimed to have found on Ganymede
[Table]
): The "Nenebek" (NAR-21166) was a "Nenebek"-type sublight mining shuttle that was in service in the independent Pentarus system in the mid-24th century.
Constructed sometime in the mid-23rd century, this shuttle was used for ferrying people. "
Nenebek
[Table]