Incomplete Utterance Rewriting by A Two-Phase Locate-and-Fill Regime (2023)

Abstract: Rewriting incomplete and ambiguous utterances can improve dialogue models’ understanding of the context and help them generate better results. However, the existing end-to-end models will have the problem of too large search space, resulting in poor quality of rewriting results. We propose a 2-phase rewriting framework which first predicts the empty slots in the utterance that need to be completed, and then generate the part to be filled into each positions. Our framework is simple to implement, fast to run, and achieves the state-of-the-art results on several public rewriting datasets.

## introduction in multi-turn dialogues, speakers naturally tend to make heavy use of references or omit complex discourses to save the efforts. thus natural language understanding models usually need the dialogue history to understand the true meaning of the current utterance. the existence of such incomplete utterances increases the difficulty of modeling dialogues. a: i heard that j.k. rowling's new book has been published. b: great. i'm going to the bookstore in town. the sources of incompleteness of an utterance can be divided into two categories: coreference and ellipsis. the task for solving these two kinds of incompleteness is called incomplete utterance * the corresponding author. rewriting (iur). as shown in figure 1 , the third utterance of this multi-turn dialogue is incomplete. if this utterance is taken out alone without a context, we will not be able to understand what "one" means and where to buy it. the fourth utterance is a rewriting of the third one. we can see that "one" in the third utterance is replaced by "j.k. rowling's new book". in addition, the place adverbial "from the book store in town" is inserted after "for me". in today's industry strength dialogue systems and applications, due to stringent requirements on running time and maintenance cost, single-turn models are much more preferred than multi-turn models. if an incomplete single-turn utterance can be completed, it will be more understandable without the context, and the cost of downstream nlp tasks, such as intention extraction and response generation, will be reduced. figure 1 shows that that all the words added in the rewritten utterance except "from" come from the context. inspired by this, many early rewriting works used pointer networks @xcite or sequence to sequence models with copy mechanism @xcite to directly copy parts from the context into the target utterance. more recently, pre-trained language models such as t5 @xcite succeeds in many nlp tasks, and it appears that t5 is a plausible choice for utterance rewriting as well. however, iur task is different from other generation tasks in that new parts typically only need to be added in one or two specific locations in the original utterance. that is, the changes to the utterance are localized. for example, a typical operation is adding modifiers before or after a noun. on the contrary, end-toend text generation models such as t5 may not preserve the syntactic structure of the input, which may cause the loss of important information and the introduction of wrong information into the output, which is illustrated as below (two examples are generated by t5.). another problem of the end-to-end pre-trained models, which generate the rewritten utterances from scratch, is that they generally incur a large search space and are therefore not only imprecise but also inefficient. in order to solve the large search space issue, @xcite treated utterance rewriting as a sequence tagging task. for each input word, they predict whether it should be deleted and the span that needs to be replaced with. @xcite formulated iur as a syntactic segmentation task. they predict segmentation operations required on the utterance to be rewritten. however, they still did not take the important step of predicting the site of rewrite, particularly the position within the syntactic structure of the input utterance. if the model can learn the syntactic structure information in the target sentence, it can predict which part of the sentence needs to be modified, i.e., which words need to be replaced and where new words need to be inserted. after that, the model only needs to fill in these predicted positions. these two tasks are relatively simple to perform, and they collectively avoid the above problems. our approach is based on the above intuition. in order to effectively utilize the syntactic structure of the sentence to be rewritten, we divide the iur task into two phases. the first phase is to predict which positions in the utterance need to be rewritten (including coreference and ellipsis). the second phase is to fill in the predicted positions. in the first phase, we use the sequence annotation method to predict the locations of coreference and ellipsis in the utterance. in the second phase, we take the utterances with blanks as input and directly predict the words required for the blank position. by seperating the original rewriting task into two relatively simple phases, our results show that our model performs the best among recent state-of-theart rewriting models 1 . our main contributions are as follows. ## experiment in this section, we will introduce our experiment setup and results. ## conclusion in this work, we present a new 2-phase framework which includes locating positions to rewrite and filling the blanks for solving incomplete utterance rewriting (iur) task. we also propose an lcs based method to align the original incomplete sentence with the ground truth utterance to obtain the positions of coreference and ellipsis. results show that our model performs the best in several metrics. we also recognize two directions for further research. first, as the performance of our 2-phase framework is often limited by the first phase, we will try to improve the accuracy of locating rewriting positions. second, it will be useful to study the best way for applying our rewriting model to other downstream nlp tasks.