S emantic R ole L abeling G uided M ulti-turn D ialogue R e W riter (2020)

Abstract: For multi-turn dialogue rewriting, the capacity of effectively modeling the linguistic knowledge in dialog context and getting ride of the noises is essential to improve its performance. Existing attentive models attend to all words without prior focus, which results in inaccurate concentration on some dispensable words. In this paper, we propose to use semantic role labeling (SRL), which highlights the core semantic information of who did what to whom, to provide additional guidance for the rewriter model. Experiments show that this information significantly improves a RoBERTa-based model that already outperforms previous state-of-the-art systems.

## introduction recent research @xcite @xcite @xcite on dialogue generation has been achieving impressive progress for making singleturn responses, while producing coherent multiturn replies still remains extremely challenging. one important factor that contributes to this difficulty is coreference and information omission, where mention is dropped or replaced by a pronoun for simplicity. these phenomena dramatically introduce the requirements for long-distance reasoning, as they frequently occurred in our daily conversations, especially in pro-drop languages like chinese and japanese. to tackle these problems, sentence rewriting was introduced to ease the burden of dialogue models by simplifying the multi-turn dialogue modeling into a single-turn problem. several approaches @xcite @xcite have been proposed to address the rewriting task. conceptually, these models follow the conventional encoder-decoder architecture that first encodes the dialogue context into a distributional representation and then decodes it to the rewritten utterance. their decoders mainly use global attention methods that attends to all words in the dialogue context without prior focus, which may result in inaccurate concentration on some dispensable words. we also observe that the accuracy of their models significantly decreases when working on long dialogue contexts. this observation is expected since if the text is lengthy, it would be quite difficult for deep learning models to understand as it suffers from noise and pays vague attention to the text components. motivated by these observations, we propose to incorporate the information of semantic role labeling (srl) @xcite to improve sentence rewriting. srl is broadly used to identify the predicate-argument structures of a sentence, where these structures could capture the main semantic information of who did what to whom. as a result, we believe that it can pick out the important words, which are semantically most related to the utterance that needs to be rewritten. as shown in for more details, we first take an srl parser to recognize the predicate-argument (pa) structures from dialog contexts, before encoding that semantic information into our model. since conventional srl benchmarks only contain sentencelevel annotations, existing pretrained srl parsers @xcite can fail to extract the cross-turn pa structures in dialogues. to address this problem, we extend the traditional srl to the conversational scenario by additionally annotating a dialogue dataset with standard srl labels. our rewriting model is based on a pre-trained roberta model @xcite that takes the outputs of srl parsing and dialogue history as its inputs, before generating rewriting outputs word by word. experimental results show that even without the srl information, our model already outperforms previous state-of-the-art models by a large margin. augmenting the srl information, the model performance is further improved significantly without adding any new parameters. ## task definition formally, an input for dialogue rewriting is a dialogue session @xmath0, ..., u n ) of n utterances, and u n is the most recent utterance that needs to be revised. the output is r, the resulting utterance after recovering all coreference and omitted information in u n . our goal is to learn a model that can automatically rewrite u n based on the dialogue context. ## model given a dialogue context c, we first apply an srl parser to identify the predicate-argument structures z; then conditioned on c and z, the rewritten utterance is generated as p(r|c, z). the backbone of our infrastructure is similar to the transformer blocks in @xcite , which supports both bi-directional encoding and uni-directional decoding flexibly via specific self-attention masks. specifically, we concatenate z, c and r as a sequence, feeding them into our model for training; during decoding, our model takes the z and c before generating the rewritten utterance word by word. our model uses a pre-trained chinese roberta @xcite for rich features. ## experiments we evaluate our model on two rewrite datasets, which are built by @xcite and @xcite . match score (em) (the percentage of decoded sequences that exactly match the human references). we implemented three baselines that use the same transformer-based encoder but differ in the choice of the decoder. specifically, trans-gen uses a pure generation decoder which generates words from a fixed vocabulary; trans-pointer applies a pure pointer-based decoder @xcite which can only copy the word from the input; trans-hybrid uses a hybrid pointer+generation decoder as in @xcite , which can either copy the words from the input or generate words from a fixed vocabulary. we can see that even without the srl information, our model still significantly outperforms these baselines on two datasets, indicating that adapting a pre-trained language model could greatly improve the performance of such a generation task. we can also see that the model with the pointer-based decoder achieves better performance than the generation-based and the hybrid one, which is similar to the observation as in @xcite . this result is expected since there is a high chance the coreference or omission could be perfectly resolved by only using previous dialogue turns. in addition, we find that incorporating the srl information can further improve the performance by at 1.45 bleu-1 and 1.6 bleu-2 points, achieving the state-of-the-art performances on the dataset of @xcite . let us first look at the impact of attention mask design on our model. to incorporate the srl information into our model, we view the linearized predicate-argument structures as a regular utterance (say u pa ) and append it in the front of the input. we experimented with two choices of attention masks. specifically, the first one is a bidirectional mask (referred as bi-mask), that is, words in u pa could attend each other; the second one (referred as triple-mask) only allows words to attend its neighbors in the same triple, i.e., words in different triples are not visible to each other. from since our framework works in a pipeline fashion, one bottleneck of our system can lie in the performance of the srl parser. one natural question is how accurate our srl parser can be and how much performance improvement for the rewriter model we could have by introducing the srl information. to investigate this, we employ a conventional srl parser foot_1 to analyze the gold rewritten utterance. these extracted pa structures are considered as gold srl annotations to measure the accuracy of our conversational srl parser. in particular, we evaluate our srl parser on the micro-averaged f1 over the (predicate, argument, label) tuples. we find our srl parser achieves 75.66 precision, 74.47 recall, and 75.06 f 1 . on the other hand, we use the gold srl results instead of our srl parsing results to train and test the model (referred as bert+gold-srl). from we also investigate which type of dialogues our model could benefit from incorporating srl information? by analyzing the dialogues and our predicted rewritten utterances, we find that the srl information mainly improves the performance on the dialogues that require information completion. one omitted information is considered as properly completed if the rewritten utterance recovers the omitted words. we find the srl parser naturally offers important guidance into the selection of omitted words. examples of rewritten utterances are shown in the appendix. recall that, there is one additional scope option to apply the srl parser to extract pa structures, i.e., only working on the last utterance that needs to be rewritten. we evaluate this option on our dataset (referred as bert+partial-srl) and results are shown in ## conclusions in this paper, we introduce a novel srl-guided framework for enhancing dialogue rewriting. for this purpose, we adapted traditional srl to the conversational scenario by annotating cross-turn annotations on 3,000 dialogues. experimental results showed that introducing srl could significantly improve the rewriting performance without adding extra model parameters.