Learning to I gnore: L ong D ocument C oreference with B ounded M emory N eural N etworks (2020)

Abstract: Long document coreference resolution remains a challenging task due to the large memory and runtime requirements of current models. Recent work doing incremental coreference resolution using just the global representation of entities shows practical benefits but requires keeping all entities in memory, which can be impractical for long documents. We argue that keeping all entities in memory is unnecessary, and we propose a memory-augmented neural network that tracks only a small bounded number of entities at a time, thus guaranteeing a linear runtime in length of document. We show that (a) the model remains competitive with models with high memory and computational requirements on OntoNotes and LitBank, and (b) the model learns an efficient memory management strategy easily outperforming a rule-based strategy

## introduction long document coreference resolution poses runtime and memory challenges. current best models for coreference resolution have large memory requirements and quadratic runtime in the document length @xcite , making them impractical for long documents. recent work revisiting the entity-mention paradigm @xcite , which seeks to maintain explicit representations only of entities, rather than all their constituent mentions, has shown practical benefits for memory while being competitive with state-of-theart models @xcite . in particular, unlike other approaches to coreference resolution which maintain representations of both mentions and their corresponding entity clusters @xcite @xcite @xcite , the entity-mention paradigm stores representations only of the entity clusters, which are updated incrementally as coreference predictions are made. while such an approach requires less memory than those that additionally store mention representations, the number of entities can still become impractically large when processing long documents, making the storing of all entity representations problematic. is it necessary to maintain an unbounded number of mentions or entities? psycholinguistic evidence suggests it is not, as human language processing is incremental @xcite and has limited working memory @xcite . in practice, we find that most entities have a small spread (number of tokens from first to last mention of an entity), and thus do not need to be kept persistently in memory. this observation suggests that tracking a limited, small number of entities at any time can resolve the computational issues, albeit at a potential accuracy tradeoff. previous work on finite memory models for coreference resolution has shown potential, but has been tested only on short documents @xcite . moreover, this previous work makes token-level predictions while standard coreference datasets have span-level annotations. we propose a finite memory model that performs quasi-online coreference resolution, foot_0 and test it on litbank @xcite and ontonotes @xcite . the model is trained to manage its limited memory by predicting whether to "forget" an entity already being tracked in exchange for a new (currently untracked) entity. our empirical results show that: (a) the model is competitive with an unbounded memory version, and (b) the model's learned memory management outperforms a strong rule-based baseline. ## entity spread and active entities given input document d, let (x n ) n @xmath0. let start(x i ) and end(x i ) denote the start and end token indices of the mention span x i in document d. let ent(x i ) denote the entity of which x i is a mention. given this notation we next define the following concepts. entity spread entity spread denotes the interval of token indices from the first mention to the last mention of an entity. the entity spread es(e) of entity e is given by: active entity count active entity count ae(t) at token index t denotes the number of unique entities whose spread covers the token t, i.e., ae(t) = |{e | @xmath1. maximum active entity count maximum active entity count mae(d) for a document d denotes the maximum number of active entities at any token index in d, i.e., mae(d) = max @xmath2. this measure can be simply extended to a corpus c as: mae(c) = max @xmath3. ## model based on the preceding finding, we will next describe models that require tracking only a small, bounded number of entities at any time. to make coreference predictions for a document, we first encode the document and propose candilong-doc-coref date mentions. the proposed mentions are then processed sequentially and are either: (a) added to an existing entity cluster, (b) added to a new cluster, (c) ignored due to limited memory capacity (for bounded memory models), or (d) ignored as an invalid mention. document encoding is done using the spanbert large model finetuned for ontonotes and released as part of the coreference model of @xcite . we don't further finetune the spanbert model. to encode long documents, we segment the document using the independent and overlap strategies described in @xcite . 3 in overlap segmentation, for a token present in overlapping bert windows, the token's representation is taken from the bert window with the most neighboring tokens of the concerned token. for both datasets we find that overlap slightly outperforms independent. mention proposal given the encoded document, we next predict the top-scoring mentions which are to be clustered. the goal of this step is to have high recall, and we follow previous work to threshold the number of spans chosen @xcite . given a document d, we choose 0.3×|d| top spans for litbank, and 0.4 × |d| for ontonotes. note that we pretrain the mention proposal model before training the mention proposal and mention clustering pipeline end-to-end, as done by @xcite . the reason is that without pretraining, most of the mentions proposed by the mention proposal model would be invalid mentions, i.e., spans that are not mentions, which would not provide any training signal to the mention clustering stage. for both datasets, we sample invalid spans with 0.2 probability during training, so as to roughly equalize the number of invalid spans and actual mentions, as suggested by @xcite . in the first step, the model decides whether the span x i refers to any of the entities in e as follows: the proposed lb-mem architecture tries to predict a score f r (.) corresponding to the anticipated number of remaining mentions for any entity or 5 we use weighted averaging where the weight for e top corresponds to the number of previous mentions seen for e top . the least recently used (lru) principle is a popular choice among memory models @xcite . while lb-mem considers all potential entities for eviction, with rb-mem this choice is restricted to just the lru entity, i.e., the entity whose mention was least recently seen. the rest of the steps are similar to the lb-mem model. training all the models are trained using teacher forcing. the ground truth decisions for bounded memory models are chosen to maximize the number of mentions tracked by the model (details in appendix a.3). finally, the training loss is calculated via the addition of the cross-entropy losses for the two steps of mention clustering. ## results tables 2 and 3 show results of all the proposed models for litbank and ontonotes respectively. as expected, the bounded memory models improve with increase in memory. for both datasets, the lb-mem model with 20 memory cells is competitive with the u-mem model. the rb-mem model with 20 memory cells is competitive on ontonotes but is significantly worse than the other two on litbank. comparing among the bounded memory models, the lb-mem model is significantly better than rb-mem for lower numbers of memory cells. we analyze the reasons for this in the next section. between the two datasets, we see that the increase in memory results in larger improvement for litbank. we also establish a new state-of-theart for litbank with the u-mem memory model. for ontonotes, our models are competitive with comparable models such as @xcite . the performance difference between the two u-mem models might be because we try to predict invalid mentions which, while beneficial for litbank, can lead to lower mention recall for ontonotes. we expect gains by further finetuning the spanbert model and learning a parameterized global entity representation, but we leave them for future work. ## analysis in this section we analyze the behavior of the three memory models on litbank and ontonotes. ## conclusion and future work we propose a memory model which tracks a small, bounded number of entities. the proposed model guarantees a linear runtime in document length, and in practice significantly reduces peak memory usage during training. empirical results on litbank and ontonotes show that the model is competitive with an unbounded memory version and outperforms a strong rule-based baseline. in particular, we report state of the art results on litbank. in future work we plan to apply our model to longer, book length documents, and plan to add more structure to the memory.