CREAD : Combined Resolution of Ellipses and Anaphora in Dialogues (2021)

Abstract: Anaphora and ellipses are two common phenomena in dialogues. Without resolving referring expressions and information omission, dialogue systems may fail to generate consistent and coherent responses. Traditionally, anaphora is resolved by coreference resolution and ellipses by query rewrite. In this work, we propose a novel joint learning framework of modeling coreference resolution and query rewriting for complex, multi-turn dialogue understanding. Given an ongoing dialogue between a user and a dialogue assistant, for the user query, our joint learning model first predicts coreference links between the query and the dialogue context, and then generates a self-contained rewritten user query. To evaluate our model, we annotate a dialogue based coreference resolution dataset, MuDoCo, with rewritten queries. Results show that the performance of query rewrite can be substantially boosted (+2.3% F1) with the aid of coreference modeling. Furthermore, our joint model outperforms the state-of-the-art coreference resolution model (+2% F1) on this dataset.

## introduction in recent years, dialogue systems have attracted growing interest, and been applied to various scenarios, ranging from chatbots to task-oriented dialogues to question answering. despite rapid progress in dialogue systems, several difficulties remain in the understanding of complex, multi-turn dialogues. two major problems are anaphora resolution @xcite and ellipsis @xcite in follow-up turns. take the dialogue in figure 1 as an example: ellipsis happens in user turn 2 where the user is asking for the capital of "costa rica" without explicitly mentioning the country again; coreference happens in user turn 3 where "the capital" refers to ⇤ ⇤ work done while the first author was an intern at apple. figure 1 : an example of a question-answering dialogue where coreference and ellipsis happen in user query, and the corresponding query rewrite annotation. references to the same entity are highlighted in the same color, and can be resolved by coreference resolution modeling. the two system responses in turn 3 indicate two possible interpretations of the city san jose by the system. query rewrite @xcite is an approach that converts a context-dependent user query into a self-contained utterance so that it can be understood and executed independent of previous dialogue context. this technique can solve many cases where coreference or ellipsis happens. for instance, "the capital" in user turn 3 is changed to "san jose" in the rewrite. furthermore, the ellipsis of the country name "costa rica" in user turn 2 can be revealed through rewriting. the rewritten utterance improves multi-turn dialogue understanding @xcite by reducing dependency on the previous turns. although query rewrite implicitly resolves coreference resolution, there is information not contained in a rewrite. first, it does not provide a distinct coreference link between mentions across dialogue turns as in the classic coreference resolution task. this is particularly disadvantageous when there is entity ambiguity in the rewritten sentence. for example, in figure 1 , since "san jose" in rewrite turn 3 can be either san jose in costa rica or san jose in california, it is likely that the system ends up with an incorrect response by generating system (i) instead of system (ii) due to the wrong interpretation of san jose. second, mention detection, an essential step in coreference resolution @xcite , is not involved in query rewrite. by knowing which span in an utterance is a mention, downstream systems like named entity recognition and intent understanding can perform better @xcite . third, if coreference links to dialogue context are available, downstream systems can skip entity linking, which is time-consuming and may introduce noise. to resolve the above issues, we propose a novel joint learning framework that incorporates the benefits of reference resolution into the query rewrite task. to the best of our knowledge, there does not exist, at the time of writing, an english conversational dataset that couples annotations of both query rewrite and coreference resolution (as links or clusters). this motivates us to collect annotations for query rewrite on a recent dialogue dataset -mudoco @xcite , which already has coreference links between user query and dialogue context. compared to existing query rewrite datasets @xcite , rewriting in mudoco is much more challenging since it involves reasoning over multiple turns and spans multiple domains. we design a joint learning model adopting the gpt-2 @xcite architecture that learns both query rewrite and coreference resolution. given an ongoing dialogue, our model first predicts the coreference links, if any, between the latest user query and the dialogue context. then it generates the rewritten query by drawing upon the coreference results. our experiments show that query rewrite performance can be substantially boosted with the aid of coreference training. in addition, our model outperforms strong baselines for the two individual tasks. since both the tasks fundamentally solve reference resolution, the joint training facilitates knowledge sharing. our contributions can be summarized as follows: to the best of our knowledge, our augmented mudoco is the first english dialogue dataset with both coreference resolution and query rewrite annotations. the augmented dataset with our annotations along with the modeling source code are available at https://github.com/apple/ ml-cread. query rewrite the most relevant line of research is the adoption of query rewrite in dialogues to tackle anaphora or ellipses. many prior works employ an lstm-based seq-to-seq model, which takes the dialogue context and user query as input, and generates the rewritten query. @xcite use the pointer-generator model @xcite to rewrite the user query on restaurantdomain task-oriented dialogues. by comparison, query rewrite in mudoco dataset is more challenging as it covers 6 domains and the rewriting patterns are more complex and diverse than in the cam-rest676 dataset @xcite . @xcite introduce an auxiliary objective of copying entity tokens from the delexicalized utterances to augment the learning of pointer network. in @xcite , two separate attention distributions are learned for the dialogue context and the user query respectively with a control gate. this modified copy mechanism shows improvements over the standard pointer-generator on both lstm-based models and transformer-based models @xcite . note that in the dataset used in their work, the dialogue context has only 2 utterances; mudoco, in contrast, has up to 8 utterances, making it much more challenging for query rewrite. coreference resolution research on documentbased coreference resolution has a long history (a detailed survey can be found in @xcite ). various approaches have been proposed, ranging from learning mention-pair classifiers @xcite , latent structuredbased models @xcite @xcite to the more recent neural pipeline based systems that rely on syntactic parsers @xcite and clustering algorithms @xcite . the first neural end-to-end coreference resolution model was proposed in @xcite and achieved better results without external resources. an improved version was proposed in @xcite , which considers higher-order structures by iteratively refining span representations. recently, powerful pre-trained models have been used to extract representations for these endto-end models using bert @xcite or spanbert @xcite . @xcite approach the problem in a question answering framework. for each detected mention candidate, the sentence it resides in serves as the query and is used to predict the referent in the passage. different from these works, we focus on coreference resolution in dialogues with the following main distinctions: 1) the speaker information in dialogues is clear; 2) less descriptive content may cause the pronoun mention to be more ambiguous; and 3) coreference resolution is conducted only between the latest user query and the previous dialogue context -unlike in document-based coreference resolution where a model can look ahead for the resolution, future turns are not available to a dialogue agent. we encourage the reader to refer to @xcite for more details. joint learning in contrast to prior works that focus solely on either query rewrite or coreference resolution, we present a novel joint learning approach to tackle both the tasks using one single model. we hope that this work serves as a first step towards this new, challenging and practical problem for dialogue understanding. ## dataset and task the mudoco dataset contains 7.5k task-oriented multi-turn dialogues across 6 domains. a dialogue has an average of 2.6 turns and a maximum of 5 (a turn includes a user query and a system response). figure 2 shows an example. for each partial dialogue, the coreference links, if existing, are annotated between the latest user query and its dialogue context. for example, when we consider the partial dialogue preceding up to user turn 2, there is a coreference link between the anaphora "this" in user turn 2 and the antecedent "song" in user turn 1. when an anaphora has multiple antecedents in the context, e.g., "song" in user turn 1 and "yellow submarine" in system turn 1, only one of them is annotated as its referent in the coreference link. on top of the existing coreference labels, we annotate the rewrite for each utterance. the goal is to rewrite the query into a self-contained query independent of the dialogue context. 30 annotators are recruited for the data collection. each of them is shown a partial dialogue, and is asked (1) to decide if the query needs to be rewritten due to coreference or ellipsis; and (2) to provide the rewritten query, when rewriting is required. we notice that there can be various ways of rewriting an utterance. for example, some annotators might include every detail of the rewritten entity, while others might choose a precise term; some might paraphrase the rewritten utterance, while others keep the same expression. to ensure data consistency and high annotation quality, we designed a comprehensive guideline for the annotators to follow and undertook a two-stage collection process: 1) we organized two training sessions with annotators. in each session, 50 representative examples were selected and assigned to each annotator. an author inspected these training results individually and provided feedback to the annotators. 2) 5% of the grading results were manually evaluated by an author for quality assurance. detailed annotation guidelines can be found in the appendix. the joint learning task requires the machine to predict both coreference links and the rewritten query for the latest user query given an ongoing dialogue. the outputs of the two individual tasks complement each other and provide more comprehensive information for dialogue understanding. for instance, the "yellow submarine" in figure 2 can be either a song name or an album name. explicit coreference resolution helps to disambiguate between various possibilities by linking entities to previously resolved ones. more importantly, the supervision of coreference resolution can be beneficial to rewriting the anaphora to its antecedent. ## modeling our proposed model for jointly learning coreference resolution and query rewrite is designed based on the gpt-2 architecture, presented in figure 3 . the input to the model is the concatenation of the dialogue context and the latest user query, where special tokens are used to separate utterances and indicate speaker information. passing through the standard decoder layers, the hidden state h l t 2 r d and attention score a l,j t 2 r t at each position of the input sequence are calculated, where l, j and t denote the index of the decoder layer, that of the attention head, and the input token position respectively; d and t denote the embedding size and the length of the input sequence respectively. inspired by the end-to-end coreference resolution model @xcite , our model first predicts mentions in the user query and grounds them to their corresponding referent in the dialogue context using attention heads. the model then generates the rewritten query conditioned on the resolved coreference links. the prediction process has four main steps, described in detail below: step 1: mention detection first, the model detects any possible referring expressions in the user query. here we use the term mention to include all those expressions that require reference resolution (e.g., pronouns or partial entity names). we formulate mention detection as a sequence labeling problem: each token in a query is labelled as one of three classes {s, e, n }, referring to start of mention, end of mention and none respectively. this sequence tagger in the mention detector, parameterized by a feed-forward network, takes the hidden states of the query from the last decoder layer as input, and predicts the sequence of class labels. then the mention spans in the query can be determined by a pair of mention start s and end e tags. for instance, in figure 3 the label of position "this" is class s and that of "from" is class e, while the rest of the positions in the query are labelled as class n . we use m s and m e to respectively denote the start and end position index of a predicted mention m. step 2: reference resolution for each detected mention m, the model resolves it to the antecedent (or referent) in the dialogue context by predicting the span boundaries: the position index of the referent start r s and end r e . essentially, the distributions of the boundaries (r s and r e ) are learned by supervising multiple attention heads associated with the target mention m. in other words, the at-tention distribution a m s (the attention score of each position associated with the mention start m s ) is supervised to focus on the referent start r s . similarly, attention scores a m e associated with the mention end m e are used to learn the boundary of referent end r e . concretely: our design of reference resolution effectively leverages the powerful attention mechanism in gpt-2 without adding any extra components for reference resolution. ## conclusion and future work we propose a novel joint learning framework for coreference resolution and query rewrite in dialogues. modeling coreference resolution not only complements the missing information in query rewrite, but is also beneficial to rewriting anaphoric expressions. our joint learning model can predict coreference links between the user query and dialogue context, and generate the rewritten query. we show that with the aid of coreference resolution, the performance of query rewrite can be substantially boosted. furthermore, our model produces competitive results in coreference resolution when compared to state-of-the-art bert-based systems. we hope that the presented joint learning task with the release of our query rewrite annotations on the mudoco dataset provides a promising research direction in multi-turn dialogue understanding. one restriction of our model is that by virtue of the model being designed to predict the boundaries of a reference, our model is only able to handle cases involving continuous spans of words. in addition, the influence of query rewrite on coreference resolution is limited due to the nature of the information flow in our current model design. future work will focus on these perspectives.