Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference (2021)

Abstract: Streaming cross document entity coreference (CDC) systems disambiguate mentions of named entities in a scalable manner via incremental clustering. Unlike other approaches for named entity disambiguation (e.g., entity linking), streaming CDC allows for the disambiguation of entities that are unknown at inference time. Thus, it is well-suited for processing streams of data where new entities are frequently introduced. Despite these benefits, this task is currently difficult to study, as existing approaches are either evaluated on datasets that are no longer available, or omit other crucial details needed to ensure fair comparison. In this work, we address this issue by compiling a large benchmark adapted from existing free datasets, and performing a comprehensive evaluation of a number of novel and existing baseline models. We investigate: how to best encode mentions, which clustering algorithms are most effective for grouping mentions, how models transfer to different domains, and how bounding the number of mentions tracked during inference impacts performance. Our results show that the relative performance of neural and feature-based mention encoders varies across different domains, and in most cases the best performance is achieved using a combination of both approaches. We also find that performance is minimally impacted by limiting the number of tracked mentions.

## introduction the ability to disambiguate mentions of named entities in text is a central task in the field of information extraction, and is crucial to topic tracking, knowledge base induction and question answering. recent work on this problem has focused almost solely on entity linking-based aprloganiv/streaming-cdc proaches, i.e., models that link mentions to a fixed set of known entities. while significant strides have been made on this front-with systems that can be trained end-to-end @xcite , on millions of entities @xcite , and link to entities using only their textual descriptions @xcite -all entity linking systems suffer from the significant limitation that they are restricted to linking to a curated list of entities that is fixed at inference time. thus they are of limited use when processing data streams where new entities regularly appear, such as research publications, social media feeds, and news articles. in contrast, the alternative approach of crossdocument entity coreference (cdc) @xcite @xcite , which disambiguates mentions via clustering, does not suffer from this shortcoming. instead most cdc algorithms suffer from a different failure mode: lack of scalability. since they run expensive clustering routines over the entire set of mentions, they are not well suited to applications where mentions arrive one at a time. there are, however, a subset of streaming cdc methods that avoid this issue by clustering mentions incrementally (figure 1 ). unfortunately, despite such methods' apparent fitness for streaming data scenarios, this area of research has received little attention from the nlp community. to our knowledge there are only two existing works on the task @xcite , and only the latter evaluates truly streaming systems, i.e., systems that process new mentions in constant time with constant memory. one crucial factor limiting research on this topic is a lack of free, publicly accessible benchmark datasets; datasets used in existing works are either small and impossible to reproduce (e.g., the dataset collected by @xcite only contains a few hundred unique entities, and many of the (b) mentions are encoded as points in a vector space and incrementally clustered. as the space grows some points are removed to ensure that the amount of memory used does not exceed a given threshold. annotated tweets are no longer available for download) or lack the necessary canonical ordering and are expensive to procure (e.g., the ace 2008 and tac-kbp 2009 corpora used by @xcite ). to remedy this, we compile a benchmark of three datasets for evaluating english streaming cdc systems along with a canonical ordering in which evaluation data should be processed. these datasets are derived from existing datasets that cover diverse subject matter: biomedical texts @xcite , news articles @xcite , and wikia fandoms @xcite . we evaluate a number of novel and existing streaming cdc systems on this benchmark. our systems utilize a two step approach where: 1) each mention is encoded using a neural or feature-based model, and 2) the mention is then clustered with existing mentions using an incremental clustering algorithm. we investigate the performance of different mention encoders (existing feature-based methods, pretrained lms, and encoders from entity linkers such as relic @xcite and blink @xcite ), and incremental clustering algorithms (greedy nearest-neighbors clustering, and a recently introduced online agglomerative clustering algorithm, grinch @xcite ). since grinch does not use bounded memory, which is required for scalability in the streaming setting, we introduce a novel bounded memory variant that prunes nodes from the cluster tree when the number of leaves exceeds a given size, and compare its performance to existing bounded memory approaches. our results show that the relative performance of different mention encoders and clustering algorithms varies across different domains. we find that existing approaches for streaming cdc (e.g., feature-based mention encoding with greedy nearest-neighbors clustering) outperform neural approaches on two of three datasets (+1-3% abs. improvement in conll f 1 ), while a relic-based encoder with grinch performs better on the last dataset (+9% abs. improvement in conll f 1 ). in cases where existing approaches perform well, we also find that better performance can be obtained by using a combination of neural and feature-based mention encoders. lastly, we observe that by using relatively simple memory management policies, e.g. removing old and redundant mentions from the mention cache, bounded memory models can achieve performance near on-par with unbounded models while storing only a fraction of the mentions (in one case we observe a 2% abs. drop in conll f 1 caching only 10% of the mentions). ## building streaming cdc systems following previous work, we adopt a two-step approach to performing streaming cross-document coreference. in the first step, an encoder is used to produce a vector representation of the incoming mention m @xmath0. in the second step, these vectors are input into an incremental clustering algorithm to update the predicted clustering c @xmath1, m t ). in the following sections we describe in detail the mention encoders and clustering algorithms used in this work. ## benchmarking streaming cdc in this section, we describe our proposed benchmark for evaluating streaming cdc systems. ## results in this section, we provide a comprehensive evaluation of the design choices that define the existing and proposed approaches for streaming cdc. choice of encoder we include the results for cdc systems with unbounded memory on the benchmark datasets in ## conclusion and future work streaming cross document coreference has a number of compelling applications, especially concerning processing streams of data such as research publications, social media feeds, and news articles where new entities are frequently introduced. despite being well-motivated, this task has received little attention from the nlp community. in order to foster a more welcoming environment for research on this task, we compile a diverse benchmark dataset for evaluating cdc, comprised of existing datasets that are free and publicly available. we additionally evaluate the performance of a collection of existing approaches for cdc, as well as introduce new approaches that leverage modern neural architectures. our results highlight a number of challenges for future cdc research, such as how to better incorporate surface level features into neural mention encoders, as well as alternative policies for memory management that improve upon the naïve baselines studied in this work. benchmark data and materials needed to reproduce our results are provided at: https://github. com/rloganiv/streaming-cdc .