Online Neural Coreference Resolution with Rollback (2022)

Abstract: Humans process natural language online, whether reading a document or participating in multiparty dialogue. Recent advances in neural coreference resolution have focused on offline approaches that assume the full communication history as input. This is neither realistic nor sufficient if we wish to support dialogue understanding in real-time. We benchmark two existing, offline, models and highlight their shortcomings in the online setting. We then modify these models to perform online inference and introduce rollback: a short-term mechanism to correct mistakes. We demonstrate across five English datasets the effectiveness of this approach against an offline and a naive online model in terms of latency, final document-level coreference F1, and average running F1.

## introduction in environments like multiparty spoken dialogue and social media streams, text in the form of tokens and sentences are available in (near) real-time. to promptly make use of this data, nlp systems often need to process text before additional tokens or sentences are available. for example, this could enable interruptions with a response or a clarification question @xcite , make decisions during a social media stream @xcite , or recognize and translate speech live @xcite . while some language technologies operate incrementally in the online setting, many documentlevel understanding models and tasks do not. a core task in language understanding is resolving references. recent work has made significant progress on improving accuracy for single documents @xcite ) and in the cross-document setting @xcite . however, this focus on document-level resolution makes use of global higher order inference and document-level encodings. as interest figure 1 : in this scene from friends, viewers can deduce who "you" refers to at @xmath0, and we want coreference models to be similarly capable. at @xmath1, viewers may need more context, such as the identity of the next speaker, to be certain of who "you" refers to. absent that context for a text-based model, its predictions will be incorrect. our proposed rollback is a cheap and local revision mechanism that corrects these type of mistakes. in coreference resolution is shifting back towards dialogue @xcite , the offline setting is inconsistent with how dialogue is found in the real world. now equipped with neural models and large-scale data, we revisit the online coreference resolution setting @xcite . 1 in this work, we are motivated by the human ability to resolve references without looking into the future (figure 1 ). we simulate the online setting for two offline models @xcite by making full predictions after each sentence and masking the future context. this either leads to significantly increased latency or lowered accuracy. we then modify the latter model to properly perform online inference and show that while accuracy does drop relative to the offline baselines, the latency is substantially lower. finally, we propose rollback, a backtracking method which allows the model to correct recently made decisions. on several coreference datasets, we show that this can recover performance comparable to that of the offline model with the latency of online models. ## task: online coreference resolution in offline (single doc) coreference resolution, the input is a document d, and the output is a set of clusters (or chains) of text mentions, @xmath2, ..., c n } such that any two mentions in a given c i corefer. evaluation can be performed at the document level, s(c pred , c gold ), by comparing the predicted clusters to the gold reference clusters with an average of three corpus-level metrics, muc @xcite , b foot_1 @xcite , and ceaf ϕ 4 @xcite , for the accuracy of mentions, links, and clusters. when each metric is instead computed at the corpus level instead before averaging, we refer to this as final f1 (identical to conll 2012 f1). in the sentence-level online setting, @xmath3, x 2 , ..., x t ] is a stream of sentences or utterances. after time t, we predict clusters c pred,@xmath4,t , ..., c n,t } conditioned on only [x 1 , ...x t ]. for the reference clusters c gold,t , we restrict clusters in c gold,t to contain only mentions up to sentence x t . this may lead to empty clusters which are ignored when calculating the score. 2 to evaluate, we propose additionally using running f1 for each document: these document-level scores are subsequently averaged across the corpus (macro-average), in contrast to the already corpus-level metrics of final f1. we are not the first to observe that references should be resolvable without future context. prior work @xcite @xcite has also emphasized the importance of incremental (online) prediction of reference, especially in the context of dialogue. since most models at that time already operated at the sentence level, their work is at the token-level granularity. our work does not go as far; our goal is to first rein back document level neural models to the sentence level, which is still appropriate in applications where full utterances are available. finally, we would like to compare the latency of different systems. unlike token-level work in speech @xcite or translation @xcite , we are primarily interested in sentences, and we do not have readily available timestamps. furthermore, modern models can process a single sentence in under a second, while sentences take substantially longer to be spoken or typed. therefore, we mainly report document-level latency, which is the wait time between the end of the document and production of predictions. we revisit and discuss sentence-level latency in section 4.4. ## experiments and results we first show that current models rely on future context, which is not readily available in the online setting. we demonstrate the effectiveness of online models under latency and average running f1. in particular, we analyze the benefits of rollback. finally, we verify that for reasonable input stream speeds, online approaches are indeed appropriate. ## conclusion we look at reining back document-level models for neural coreference resolution to the utterance level by proposing a shift towards online inference. we propose a model with the capability for making predictions online, after every sentence. this leads to lower latency than a corresponding offline model, and maintains a consistently high running f1 after each sentence. to edit predictions made without future context, we introduce a rollback mechanism which reverts and corrects recently made predictions, bringing the f1 closer to that of the offline model while maintaining its ability to make online predictions with low latency. future work may consider extensions to this approach by handling online processing at the word-level, revisiting the scenario considered by @xcite . liyan xu and jinho d. choi. 2022. online coreference resolution for dialogue processing: improving mention-linking on real-time conversations. in proceedings of the 11th joint conference on lexical and computational semantics, pages 341-347, seattle, washington. association for computational linguistics. jiahong yuan, mark liberman, and christopher cieri. 2006. towards an integrated understanding of speaking rate in conversation. in ninth international conference on spoken language processing. yu zhang, guoguo chen, dong yu, kaisheng yao, sanjeev khudanpur, and james glass. 2016. highway long short-term memory rnns for distant speech recognition. in 2016 ieee international conference on acoustics, speech and signal processing (icassp), pages 5755-5759. ieee. ethan zhou and jinho d. choi. 2018. they exist! introducing plural mentions to coreference resolution and entity linking. in proceedings of the 27th international conference on computational linguistics, pages 24-34, santa fe, new mexico, usa. association for computational linguistics. a experimental details a.1 datasets preprocessing we use the same preprocessing as joshi et al. (2019) for ontonotes, xia and van durme (2021) for lit-bank (first fold) and qbcoref (first fold), and toshniwal et al. (2021) for ci. for the genre split in ontonotes, we split the full dataset into a conversational and text-based component. some weblog documents are conversations on message boards. we maintain this split because they are less conversational than spoken dialogue, and it is consistent with the split originally used in by icoref. while ontonotes does have non-english splits, we only study english data in this work. to our knowledge, the datasets and codebases were released intended to advance research in coreference resolution, which is aligned with the focus of this work. since icoref does not readily take speaker embeddings, we augment the underlying text of ci with speakers by prepending each utterance with the name of the speaker(s), following the strategy outlined by @xcite , and we only filter out these mentions before evaluation. we note that there could be other ways of representing the speakers, especially in plural situations, which we do not explore as it is beyond the scope of the work. while this follows the same preprocessing as @xcite , we do not do this for c2f, as this model uses the speakers as a feature. we do not evaluate ci following the metrics outlined in zhou and choi (2018) as we are primarily interested in exploring online coreference by using the dialogue and conversational nature of the dataset and not in the plural mentions and multiparty aspect.