RAST : Domain-Robust Dialogue Rewriting as Sequence Tagging (2021)

Abstract: The task of dialogue rewriting aims to reconstruct the latest dialogue utterance by copying the missing content from the dialogue context. Until now, the existing models for this task suffer from the robustness issue, i.e., performances drop dramatically when testing on a different dataset. We address this robustness issue by proposing a novel sequence-tagging-based model so that the search space is significantly reduced, yet the core of this task is still well covered. As a common issue of most tagging models for text generation, the model’s outputs may lack fluency. To alleviate this issue, we inject the loss signal from BLEU or GPT-2 under a REINFORCE framework. Experiments show huge improvements of our model over the current state-of-the-art systems when transferring to another dataset.

## introduction recent years have witnessed increasing attention in conversation-based tasks, such as conversational question answering @xcite @xcite , dialogue response generation @xcite @xcite , dialogue state tracking @xcite and dialogue understanding @xcite @xcite , mainly due to increasing commercial demands. however, current models still face tremendous challenges in representing multi-turn dialogues, due to the frequent omission (a.k.a. ellipsis) and coreference that people naturally use in conversations for brevity. specifically, recent work @xcite has shown that ellipsis and coreference can exist in more than 70% of dialogue utterances. to tackle this problem, people have proposed coreference resolution and zero pronoun recovery. but, the state-of-the-art performances turn utterance with translation u1 上海最近天气怎么样？ (how is the recent weather in shanghai?) u2 最近经常阴天下雨。 (it is always raining recently.) u3 冬天就是这样。 (winter is like this.) u 3 recently, the task of dialogue utterance rewriting @xcite @xcite was proposed as for explicitly representing multi-turn dialogues. the task aims to reconstruct the latest dialogue utterance into a new utterance that is semantically equivalent to the original one and can be understood without referring to the context. in another point of view, it integrates the recovering of both coreference and omission. as shown in most previous efforts @xcite @xcite consider this task as a standard text-generation problem, adopting a sequence-to-sequence model with a copy mechanism @xcite @xcite . they have demonstrated almost ready-to-use performances on the test set from the same data source as the training set. however, they are not robust, as our experiments show that their performances can drop dramatically (by roughly 33 bleu4 @xcite points and 44 percent of exact match) on another test set created from a different data source (not necessarily from a totally different domain). we argue that it may not be the best practice to model utterance rewriting as standard text generation. one main reason is that text generation introduces an overly large search space, while a rewriting output (e.g., u 3 in in this paper, we propose a novel solution that treats utterance rewriting as multi-task sequence tagging. in particular, for each input word, we decide whether to delete it or not, and at the same time, we choose what span from the dialogue context need to be inserted to the front of the current word. in this way, our solution enjoys a far smaller search space than the generation based approaches. since our model does not directly take features from the word-to-word interactions of its output utterances, this may cause the lack of fluency. to encourage more fluent outputs, we propose to inject additional supervisions from two popular metrics, i.e., sentence-level bleu @xcite and the perplexity of a pretrained gpt-2 @xcite model, using the framework of "reinforce with a baseline" @xcite . sentence-level bleu is computationally efficient, but it requires references and thus may only provide domain-specific knowledge. conversely, the perplexity by gpt-2 is reference-free, giving more guidance on open-domain scenarios benefiting from the large-scale pretraining. experiments on two dialogue rewriting benchmarks show that our model can give huge improvements (14.6 in bleu4 score and 18.9 percent of exact match) over the current state-of-the-art model for cross-dataset evaluation. more analysis shows that the outputs of our model keep more semantic information from the inputs. our code is available at https://github.com/ freesunshine0316/rast-plus . ## related work initial efforts @xcite treat dialogue utterance rewriting as a stan-dard text generation problem, adopting sequenceto-sequence models with copy mechanism to tackle this problem. later work @xcite @xcite explores taskspecific features for additional gains in performance. for instance, @xcite adopts a pipeline-based method, where all context words that need to be inserted during rewriting are identified in the first step. the second step adopts a pointer generator that takes the outputs of the first step as additional features to produce the output. @xcite train a model of semantic role labeling (srl) to highlight the core meaning (e.g., who did what to whom) of each input dialogue to prevent their rewriter from violating this information. to obtain an accurate srl model on dialogues, they manually annotate srl information for more than 27,000 dialogue turns, which is timeconsuming and costly. @xcite casts this task into a semantic segmentation problem, a major task in computer vision. in particular, their model generates a word-level matrix, which contains the operations of substitution and insertion, for each original utterance. they adopt a heavy model that takes 10 convolution layers in addition to the bert encoder. none of the existing efforts mention the robustness issue, a critical aspect for the usability of this task. besides, they only compare performances under automatic metrics (e.g., bleu). we take the first step to address this severe robustness issue, and we adopt multiple measures for comprehensive evaluation. besides, we propose a novel model based on sequence tagging for solving this task, and our model takes a much smaller search space than previous models. ## baseline: trans-pg+bert our baseline consists of a bert @xcite encoder and a transformer @xcite decoder with a copy mechanism. given input tokens @xmath0, . . . , x n ) that is the concatenation of the current dialogue context @xmath1, . . . , u i-1 ) and the latest utterance u i , the bert encoder is firstly adopted to represent the input with contextualized embeddings: next, the transformer decoder with copy mechanism is adopted to generate a rewriting output @xmath2, . . . , y m ) one token at a time: ## rast: rewriting as sequence tagging in this section, we describe how to convert the dialogue rewriting task into a multi-task sequence tagging problem. task description our analysis shows that dialogue rewriting mainly handles two linguistic phenomena: coreference and omission. to recover a coreference, it has to replace a pronoun in the current utterance with the phrase it refers to in the dialogue context. to recall an omission, it needs to insert the corresponding phrase into the omission position. accordingly, we cast the dialogue rewriting as a sequence tagging task by introducing two types of tags for each word x n : recovering a coreference corresponds to the operation {deletion:1, insertion:[start, end]}, and recalling an omission corresponds to the operation {deletion:0, insertion:[start, end]}, where [start, end] denotes the corresponding phrase in the dialogue context. for the other words without any change, the operation is {deletion:0, insertion:[-1, -1]}. figure 1 shows an example, where the word "这样 (like this)" corresponds to a coreference, and the word "冬天 (winter)" corresponds to an omission in front of it. foot_0 constructing annotated data the gold tags for dialogue utterance rewriting are not naturally available. in response to this problem, we construct the annotated data based on the alignment between the input and reference utterances. specifically, we employ the longest common sub-sequence (lcs) 2 algorithm to generate the word alignments between the input utterance u i and the reference utterance u i for each instance (the black lines in figure 1 ). the lcs algorithm is based on dynamic programming, which takes a time complexity of o( for the words in the reference utterance that are not aligned, we search them from the dialogue context and obtain their span (e.g. the words in color highlighting). given the aligned instance, we construct the annotation tags by traversing the alignments in a left-to-right manner and comparing each alignment with its preceding one 3 under the following rules: r1. if the two alignments are adjacent in both utterances (e.g. "就是 (is)"), there is no change for the current word, which is assigned the tags {deletion:0, insertion:[-1, -1]}. r2. if the two alignments are only adjacent in the input utterance (e.g. "冬天 (winter)"), this generally corresponds to an omission . we insert the reference words between the two alignments (i.e. "上海 (shanghai)") in front of the current input word. accordingly, we assign the current word "冬天 (winter)" the tags {deletion:0, insertion:[1, 1]}. r3. if the two alignments are only adjacent in the reference utterance, we simply delete the input words between the two alignments, and assign them the tags {deletion:1, insertion:[-1, -1]}. this situation is rare in the task of dialogue utterance rewriting. r4. if the two alignments are not adjacent in either utterance (e.g. "。 (.)"), this generally corresponds to a coreference that requires a replacement . we first delete the input words between the two alignments (i.e. "这样 (like this)"), then insert the corresponding target phrase (i.e. "经常 阴天 下雨 (always raining)") in front of the left-most deleted input word 2 https://en.wikipedia.org/wiki/ longest_common_subsequence_problem 3 we insert a special flag "<@xmath3. ## experiments we study the robustness of our tagging-based model on two benchmarks for dialogue rewriting. ## conclusion in this paper, we addressed the robustness issue of dialogue utterance rewriting, which is crucial for its usability on real applications. we proposed a novel tagging-based approach that results in a significantly smaller search space than the existing methods on this task, and we introduced additional supervision (e.g. by gpt-2) to improve the fluency of model outputs. experiments with automatic metrics, human evaluation and semantic matching show that our model is much more robust than the previous state-of-the-art system without sacrificing its in-domain performances. future work includes evaluating this tagging framework on other english benchmarks, such as smcalflow @xcite and treedst @xcite .