Enhance Incomplete Utterance Restoration by Joint Learning Token Extraction and Text Generation (2022)

Abstract: This paper introduces a model for incomplete utterance restoration (IUR) called JET (Joint learning token Extraction and Text generation). Different from prior studies that only work on extraction or abstraction datasets, we design a simple but effective model, working for both scenarios of IUR. Our design simulates the nature of IUR, where omitted tokens from the context contribute to restoration. From this, we construct a Picker that identifies the omitted tokens. To support the picker, we design two label creation methods (soft and hard labels), which can work in cases of no annotation data for the omitted tokens. The restoration is done by using a Generator with the help of the Picker on joint learning. Promising results on four benchmark datasets in extraction and abstraction scenarios show that our model is better than the pretrained T5 and non-generative language model methods in both rich and limited training data settings.

## introduction understanding conversational interactions through nlp has become important with increasing connectivity and range of capabilities. the applications using natural conversations cover a wide range of solutions including dialogue systems, information extraction, and summarization. for example, @xcite ; @xcite aimed to build the dialogue system where an intelligent virtual agent answers human conversations and makes suggestions in an open/closed domain. @xcite ; @xcite attempted to detect decision-related utterances from multi-party meeting recordings, while @xcite applied extractive summarization for online forum discussions. these features allow users to to quickly catch up with the current situation, decisions and next-action without having to follow a lengthy or comprehensive dialogue. however, utterances, the components of a conversation, are generally not self-contained and are difficult to understand by their own. this comes from the nature of multi-turn dialogue where each utterance contains co-references, rephrases, and ellipses (figure 1 ). @xcite also showed that co-references and ellipses occur in over 70% of utterances in conversations. this is a ubiquitous problem in conversational ai, making the challenge for building practical systems with conversations. incomplete utterance restoration (iur) @xcite is one solution to restore semantically underspecified utterances (i.e., incomplete utterances) in conversations. figure 1 shows an example of iur, in which the model rewrites the incomplete utterance to the reference. iur is a challenging task due to two reasons. firstly, the gold utterance (the reference) overlaps a lot of tokens with the pre-restored, incomplete utterance, while it overlaps only a few tokens with utterances in the context. we observed that for canard @xcite , 85% of tokens in incomplete utterances were directly cited for rewriting, while only 17% of tokens in context was cited for rewriting. secondly, it is important to detect omitted tokens in incomplete utterances and to include them in the restoration process. in actual cases of iur, no matter how fluent and grammatically correct the machine's generation is, it is useless as long as important tokens are left out. recent studies used several methods for iur. it includes the extraction of omitted tokens for restoration (pac) @xcite , two-stage learning @xcite , seq2seq fine-tuning @xcite , semantic segmentation (run-bert) @xcite , or the tagger to detect which tokens in incomplete utterances should be kept, deleted or changed for restoration (sarg) figure 1 : the sample data from canard. iur models rewrite the incomplete utterance to be as similar as possible to the reference. the blue tokens are omitted tokens (excluding stop words) in the incomplete utterance. the red tokens are defined by our hard labeling approach as an important token. @xcite . however, we argue that these methods can only work on neither extractive nor abstractive iur datasets. for example, sarg and seq2seq achieve promising results on restoration 200k @xcite where omitted tokens can be directly extracted from the context (extraction). but they are not the best on canard @xcite , which requires more abstraction for restoration. in figure 1 , foot_0 we can observe that the output of sarg and seq2seq are worse than that of our jet. text editing strategy by sarg is limited in its ability to generate abstractive rewriting while seq2seq has the problem in picking omitted tokens. as the result, the generality of these methods is still an open question. we introduce a simple but effective model to deal with the generality of iur methods named jet (joint learning token extraction and text generation). the model is designed to work widely from extractive to abstractive scenarios. to do that, we first address the problem of identifying omitted tokens from the dialogue context by introducing a picker. the picker uses a new matching method for dealing with various forms of tokens (figure 1 ) in the extraction style. we next consider the abstraction aspect of restoration by offering a generator. the generator utilizes the power of the pre-trained t5 model to rewrite incomplete utterances. the picker and generator share the t5's encoder and are jointly trained in a unified model for iur. this paper makes three main contributions: ## related work sentence rewriting iur can be considered to be similar to the sentence rewriting task @xcite @xcite . recent studies have been addressed the iur task with various sophisticated methods. for example, pan et al. 2019 introduced a pick-then-combine model for iur. the model picks up omitted tokens which are combined with incomplete utterances for restoration. @xcite proposed a semantic segmentation method that segments tokens in an edit matrix then applied an edit operation to generate utterances. huang et al. 2021 presented a complicated model which uses a tagger for detecting kept, deleted, or changed tokens for restoration. we share the idea of using a tagger with huang et al. 2021 for iur. however, we design a more simple but effective model which includes a picker (picking omitted tokens) and a generator for the restoration of incomplete utterances. text generation iur can be formulated as text generation by using the seq2seq model @xcite . for the generation, several well-known pre-trained models have been applied @xcite @xcite with promising results. we employ the t5 model @xcite as the main component to rewrite utterances. to address the problem of missing important tokens in model's rewriting, we enhance t5 by introducing a picker and two labeling methods (section 3.2). ## settings and evaluation metrics data we conducted all experiments on four wellknown datasets of utterance rewriting in settings for running jet, we used adamw with β 1 = 0.9, β 2 = 0.999 and a weight decay of 0.01 with a batch size of 12 and learning rate of 5e -5 . we used 3 ffn layers (dimension as 768, 256, 64) with relu as the activation function. the final dimension is 1 for soft labeling and 3 for hard labeling. we set α = 1 for the loss function. we applied beam search with the beam size of 8. for picker's label creation, we used stop words from nltk for english and from stopwordsiso foot_1 for chinese. for lemmatization and stemming, nltk's word-netlemmatizer and porterstemmer were adopted for english, while lemmatization and stemming were skipped for chinese. the pre-trained model was t5-base (english foot_2 and chinese foot_3 ). in the full training data setting (section 5.1), the epoch size of 6 was used for restoration200k and canard and 20 for rewrite and task. in the limited training data setting (section 5.2), the epoch size of 20 was used for all four datasets ( ## conclusion this paper introduces a simple but effective model for incomplete utterance restoration. the model is designed based on the nature of conversational utterances, where important omitted tokens should be included in restored utterances. to do that, we introduce a picker with two labeling methods for supporting a generator for restoration. we found that the picker contributes to improve the generality of the model on four benchmark datasets. the model works well in english and chinese, from extractive to abstractive scenario in both full and limited training data settings. the future work will investigate the behavior of the model in other domains and the potential application of jet, e.g. combining utterance extraction and utterance restoration for information extraction from dialogue.