Incomplete Utterance Rewriting as Sequential Greedy Tagging (2023)

Abstract: The task of incomplete utterance rewriting has recently gotten much attention. Previous models struggled to extract information from the dialogue context, as evidenced by the low restoration scores. To address this issue, we propose a novel sequence tagging-based model, which is more adept at extracting information from context. Meanwhile, we introduce speaker-aware embedding to model speaker variation. Experiments on multiple public datasets show that our model achieves optimal results on all nine restoration scores while having other metric scores comparable to previous state-of-the-art models. Furthermore, benefitting from the model’s simplicity, our approach outperforms most previous models on inference speed.

## introduction recent years have witnessed increasing attention in dialogue systems mainly due to its promising potential for applications like virtual assistants or customer support systems @xcite . however, studies @xcite show that users of dialogue systems tend to use incomplete utterances which usually omit (a.k.a. ellipsis) or refer back (a.k.a. co-reference) to the concepts that appeared in previous dialogue contexts. (also known as non-sentential utterances, @xcite . thus, dialogue systems must understand these incomplete utterances to make appropriate responses. to tackle the problem, the task of incomplete utterance rewriting(iur, also known as context rewriting) @xcite @xcite , which aims to rewrite an incomplete utterance into an utterance that is semantically equivalent but self-contained to be understood without context, has recently become an increasing focus of nlp research. as depicted in turn utterance with translation u 1 (a) 深圳最近天气怎么样？ (how is the recent weather in shenzhen?) u 2 (b) 最近经常阴天下雨。 (it is always raining recently.) u 3 (a) 冬天就是这样的。 (winter is like this.) u ′ 3 in order to break through those limitations, tagging-based approach @xcite @xcite @xcite was proposed. for specifically, here we consider models like run @xcite as a tagging-based method. its semantic segmentation figure 1 : the illustration of the main learning task of the sequence tagging model sgt(sequential greedy tagging). we adopt the same example dialogue from the previous top tagging-based approach generally formalizes iur as learning the edit operation and corresponding location. the tagging-based approach enjoys a smaller search space than the generation-based approach and can better utilize the information that the main structure of a rewritten utterance is always the same as the incomplete utterance. despite their success, existing approach that learning edit operation and the corresponding location has difficulty handling situations where multiple inserts correspond to one position. moreover, models like run adopt a heavy model that takes ten convolution layers in addition to the bert encoder, which will increase its training time and slows down its infer speed. more critically, although bert @xcite has shown to be powerful in extracting information, the generally low restoration scores prove that previous bertbased models are ineffective in extracting the information needed for iur from historical utterances. finally, the experimental results of sa-bert @xcite demonstrate that explicitly modeling speaker changes has a specific enhancement effect on modeling multi-turn dialogue tasks. the previous approach did not model this critical information. to address these issues, we propose a novel sequence tagging model named sgt(sequential greedy tagging), which is not based on learning editing operations and can significantly improve the restoration score and inference speed. our solution was derived from the following thinking: first, we consider that in the dialogue process, any complete utterance is composed by only of a few fragments. for example, "i love you" includes three components: subject, verb, and object. even if it is expanded with modifications and qualifications, its composition is still minimal. based on this insight, we thought it would be possible to build a model to identify the fragments and their order from dialogue history to form a target completed utterance. and then, splice those fragments together in sequence and get the complete utterance. meanwhile, in order to keep the number of fragments constituting the target rewritten utterance relatively small, we adopt the greedy tagging strategy. our model will identify all the fragments and their order required to form a completed utterance; each fragment is the longest fragment found in the given order. we might as well call this fragment glcs (greedy longest common subsequence). specifically, we use the tag type to represent the order of glcs for composing the target rewrite utterance, for example, the first glcs that constitutes a rewritten utterance would be tagged as a, and the second is b, the third is c, and so on. in the above manner, we converted iur into a simple sequence tagging task, as illustrated in figure 1 . after the model has identified all glcss from the dialogue history through this strategy, the target rewritten utterance can be obtained by splicing each glcs in alphabetical order according to its tag. furthermore, we introduce speaker-aware embedding to model the speaker changes in different rounds. finally, to better perceive the boundaries of each tagging mention, we add two simple losses in addition to the sequence labeling loss. in summary, our contributions are as follows: ## related work earlier efforts @xcite treated dialogue utterance rewriting as a common text generation problem and integrated seq-toseq models with copy mechanisms to model this task. later work @xcite @xcite explore task-specific features for additional gains in performance. for example, @xcite adopt a pipeline-based method. the idea is to detect keywords first and then append those words to the context and adopt a pointer generator that takes the output of the first step to produce the output. however, this two-step method inevitably accumulates errors. srl @xcite trains a semantic labeling model to highlight the central meaning of keywords in the dialogue as a sort of prior knowledge for the model. to obtain an accurate srl model for dialogues, they manually annotate srl information for more than 27,000 dialogue turns, which is costly and time-consuming. run @xcite convert this task into a semantic segmentation problem, a significant task in computer vision. in particular, their model generates a word-level edit matrix, which contains the operations of insertion and substitution for each original utterance. rather than word embeddings, rau @xcite directly extracts ellipsis and co-reference relationships from transformer's self-attention weighting matrix and edits the original text accordingly to generate complete utterances. run++ @xcite introduce contrastive learning and keyword detection tasks to model the problem jointly. both rau and run++ make significant improvements in most metrics on several datasets. although some additional effective strategies exist. it is still in the same paradigm as run, learning edit matrix by cast iur as a semantic segmentation task. rast @xcite is the first work to convert dialogue utterance rewriting into a sequence tagging task. it takes experimentation to prove that most models for this task suffer from the robustness issue, i.e., performance drops when testing on a different dataset. by contrast, rast is more robust than the previous works on cross-domain situations. moreover, this work design additional reinforcement learning task to improve fluency. despite all these efforts, its overall in-domain performance still lags behind methods that learn edit operation matric @xcite . to better enhance pre-trained language models for multi-turn response selection in retrieval-based chatbots. a model named speaker-aware bert (sa-bert) @xcite proposed to make the model aware of the speaker's changed information, which is an essential and intrinsic property of multiturn dialogues. although rast has a different learning paradigm from works that learn edit matrix, it still tries to learn the edit operation and corresponding location by sequence tagging. as mentioned before, our method is sequence tagging-based but takes an entirely new paradigm that would not learn edit operations. besides, inspired by sa-bert @xcite , we introduce speaker embedding to this task. finally, we introduce two simple sequence labeling tasks to model this problem jointly. ## experiments in this section, we conduct through experiments to demonstrate the superiority of our approach. datasets we conduct experiments on three public datasets across different domains: chinese datasets in open-domain dialogues: multi @xcite and rewrite @xcite , english task-oriented dialogue task @xcite . for a fair comparison, we adopt the same data split for these datasets as our baselines. the statistics of these datasets are displayed in baselines to prove the effectiveness of our approach, we take the state-of-the-art models as strong baselines including srl @xcite , sarg @xcite , pac @xcite , rast @xcite , t-ptr-λ @xcite , run @xcite and run++ @xcite . evaluation we employ credible automatic metrics to evaluate our approach. as in literature @xcite , we examine sgt using the widely used automatic metrics bleu, rouge, em and restoration f-score. (i) bleu n (b n ) evaluates how similar the rewritten utterances are to the golden ones via the cumulative n-gram bleu score @xcite . (ii) rouge n (r n ) measures the n-gram overlapping between the rewritten utterances and the golden ones, while rouge l (r l ) measures the longest matching sequence between them @xcite . (iii) em stands for the exact match accuracy, which is the strictest evaluation metric. (iv) restoration precision n , restoration recall n and restoration f-score n (p n , r n , f n ) emphasize more on words from dialogue context which are argued to be harder to copy @xcite . therefore, they are calculated on the collection of n-grams that contain at least one word from context utterance. as validated by @xcite , above automatic metrics are credible indicators to reflect the rewrite quality. implementation our implementation was based on pytorch @xcite and fastnlp @xcite . in practice, we adopt the exact connection words setting with run and append the list of connection words to the head of h, as part of it. considering that only two speakers are in the datasets related to our experiments, we set the hidden_size of sa to 1. for encoding different tagging types, we choose io encoding, the simplest tag encoding schema, which tags each token as either being in (i-x) a particular type of named entity type x or in no entity (o). since the distribution of tag types is severely unbalanced (e.g. (o) accounts for more than 81% on multi), we employed weighted cross-entropy loss and tuned the weight on development sets. we used adam @xcite to optimize our model and set the learning rate as 2e-5. we set the dropout rate as 0.3 for the dropout operation on the equation 2. for a fair comparison, the bert used in our model is bert-base which is the same as our baselines. ## conclusion in this paper, we convert the iur problem into a simple sequence tagging task, sgt. the simplicity and effectiveness of the modeling paradigm not only improve the inference speed and allow the pretrained bert encoder to fully exploit its widely validated information extraction ability which can significantly improve the restoration score and ensure that other metrics are competitive. we also introduced speaker-aware embedding to explicitly model speaker changes and verified that it has some improvement effect on the iur task. in the future, we will explore the following directions: