Pretraining the Noisy Channel Model for Task-Oriented Dialogue (2021)

Abstract: Direct decoding for task-oriented dialogue is known to suffer from the explaining-away effect, manifested in models that prefer short and generic responses. Here we argue for the use of Bayes’ theorem to factorize the dialogue task into two models, the distribution of the context given the response, and the prior for the response itself. This approach, an instantiation of the noisy channel model, both mitigates the explaining-away effect and allows the principled incorporation of large pretrained models for the response prior. We present extensive experiments showing that a noisy channel model decodes better responses compared to direct decoding and that a two-stage pretraining strategy, employing both open-domain and task-oriented dialogue data, improves over randomly initialized models.

## introduction task-oriented dialogue agents provide a conversational interface to assist users in accomplishing specific goals, such as finding a restaurant or booking a hotel @xcite @xcite . increasing demand from industry for natural language assistants and scalable customer service solutions has recently been driving a renaissance in the development of task-oriented dialogue models. in addition, the specification of explicit dialogue agent goals, afforded by the task-oriented paradigm, makes such research easier to ground and evaluate than open-domain chatbots. current research on task-oriented dialogue is dominated by monolithic sequence-to-sequence models that directly parameterize the conditional distribution of the response given the prior dia-logue context. however, this monolithic approach conflates the task-specific and language-general aspects of dialogue, and adversely favors short and generic responses @xcite due to the explaining-away effect @xcite . here we pursue an alternative to the direct model. using bayes' rule allows us to factorize the probability of the response given the context p(r| c) into a language model p(r) and a context model p(c| r). 1 within natural language processing (nlp), this approach is traditionally known as the noisy channel model @xcite , and has recently seen renewed interest with its successful application to neural machine translation @xcite @xcite . we hypothesize that the noisy channel reformulation is advantageous for dialogue because the factorization enables each sub-module to specialize in a dialogue sub-task. in particular, the context conditional model can help to discount short and generic responses and mitigate the explainingaway effect, while the language model helps ensure that responses are natural. we find that a noisy channel model with the same number of parameters as a direct model achieves better accuracy on three task-oriented dialogue datasets. moreover, a larger noisy channel model can be trained with the same hardware, by training the sub-modules separately, yielding additional improvements. it has become common in recent years to pretrain dialogue models on large text data, either general text @xcite @xcite or dialoguestructured data @xcite , such as tweets and reddit posts. we utilise a similar strategy with reddit data and find that the benefits of pretraining to the noisy channel model are similar to those for the direct model. further, we evaluate transfer across task-oriented dialogue datasets by implementing a second pretraining stage using taskmaster @xcite and schema-guided dialogue @xcite as training data, before fine-tuning on our final tasks. we evaluate the algorithm on three datasets, multiwoz 2.0 @xcite , cam-rest676 @xcite , and smcalflow @xcite , demonstrating that the noisy channel approach is robust to different dialogue schema annotations used across datasets. further analysis demonstrates that the noisy channel models can decode responses with similar lengths and zipf scores compared to ground-truth responses and reduce the likelihood of falling into repetition loops @xcite . ## a seq-to-seq dialogue model in this section, we introduce a discriminative sequence-to-sequence model for task-oriented dialogue. the traditional sequence of steps needed to produce a system turn in a task-directed dialogue is shown in figure 1 , with an example from multiwoz 2.0 @xcite . given a dialogue context containing previous user and system utterances, the dialogue system first predicts a belief state, consisting of a set of slotvalue pairs (e.g., destination: cambridge), to capture user intent. to ground the system with external information, the belief state can be converted into a database query in order to retrieve relevant information, such as the number of matches and booking information. next, the system predicts a set of dialogue acts, representing the abstract meaning of the proposed dialogue response @xcite . finally, a delexicalized dialogue response is generated, where slot values are replaced by generic placeholders, such as value time for a train departure time, in order to reduce lexical variation. the delexicalized response can be converted to a lexicalized response in post-processing by filling in the slot values based on belief states and database information. we use the multiwoz schema for illustration in sections 2 and 3, but our models easily generalize to different schema annotations (e.g., datasets without annotated dialogue acts @xcite ). because it is well known that pipelined models tend to suffer from error propagation, many nlp tasks have been reformulated in recent years as end-to-end text-to-text transformations @xcite . stateof-the-art task-oriented dialogue systems have followed this approach @xcite ). we represent the example from figure 1 as follows, serializing turns and using special start and end tokens to encapsulate each data field: context: [c] i am looking to . . . [/u] what is your . . . [/r] i'll be leaving . . . [/u] [/c] belief: [b] [train] destination cambridge, day tuesday, arrive 12:30, departure london [/b] database: [db] [train] match 1, status not booked [/db] act: [a] [train] inform arrive, inform leave, offer reservation [/a] response: [r] there is a train that leaves at [value time] and arrives at [value time]. should i book it? [/r] given this text representation, the direct discriminative approach models p(b, a, r| c), where c, b, a, and r represent dialogue context, belief state, dialogue act, and delexicalized response, respectively. 2 we use the serialized text of the dialogue context as input, and the concatenation of belief state, dialogue act, and response as target output, making the task amenable to the application of an autoregressive sequence-to-sequence model. b, a, and r can be generated sequentially with direct decoding methods, such as greedy decoding and beam search. we use a sequenceto-sequence transformer @xcite to implement p(b, a, r| c). this distribution will also be used to build the noisy channel model in section 3. ## noisy channel model for dialogue while direct decoding is an effective approach for decoding belief states @xcite , it may be sub-optimal for generating responses. first, it favors short and generic responses @xcite . as a result, the decoded responses are bland and lack diversity @xcite . second, it suffers from the explaining-away effect @xcite , where inputs are ''explained-away'' by highly predictive output prefixes. for example, if there is one hotel matching the user's intent as encoded in the belief state, the model is nevertheless prone to decoding ''no'' given the output prefix ''there is'', ignoring the input information. in this work, we propose using the neural noisy channel model @xcite to mitigate the above problems for response generation. given an input sequence x and output sequence y, the noisy channel formulation @xcite uses bayes' rule to rewrite the model p(y|x) as for the full task, we develop a noisy channel model for p(b, a, r| c). using the chain rule, p(b, a, r| c) = p(b| c) • p(a, r| c, b). following hosseini-asl et al. (2020), we use the direct model described in section 2 to parameterize p(b| c) and decode b, which our preliminary experiments confirmed to be advantageous. we use the noisy channel formulation to parameterize p(a, r| c, b). using bayes' rule, p(a, r| c, b) ∝ p(c, b| a, r) • p(a, r). the channel model p(c, b| a, r) and source model p(a, r) are implemented as transformers. we choose to use the noisy channel formulation for decoding a based on preliminary experiments that showed improved overall accuracy over direct decoding, possibly because poor dialogue act prediction by the direct model led to worse quality responses. the serialized text of a and r are concatenated during training, and the decoded sequence is split into a and r with the special start/end tokens during decoding. we suggest that the noisy channel model has three advantages over the direct model for response generation: (1) the channel model can penalize short and generic responses. such responses can be mapped to a large number of contexts, resulting in a flat distribution over contexts. this leads to a lower channel model score for short and generic responses @xcite . ## model and pretraining we use three transformer @xcite networks to parameterize the direct model p(b, a, r| c), the channel model p(c, b| a, r) and the source model p(a, r), respectively. the input to each transformer is the sum of four embeddings: word embeddings, position embeddings, role embeddings (user/system), and turn embeddings (each word corresponds to a turn number). cross entropy is used as the loss function. given training samples (c, b, a, r), if we train the channel model using complete (a, r) pairs as input, a significant discrepancy arises between training and decoding for noisy channel online decoding. since the channel model is used to score partial act and response pairs, that is, p(c, b| o) in algorithm 1, the channel model trained with complete (a, r) pairs is unsuited to scoring partial sequences. in order to manually create partial sequences during training that are better matched for online decoding, we truncate the (a, r) pairs with a truncation length uniformly sampled from 1 to the sequence length (inclusive). the direct model and the source model are trained with complete sequences, as partial sequences occur naturally in their standard autoregressive training procedure. as in-domain dialogue data are usually scarce, we use a two-stage pretraining strategy to enhance the noisy channel model. although the effectiveness of pretraining with reddit data has been validated for open-domain dialogue @xcite @xcite , relatively little work has applied such data to task-oriented dialogue. 4 in the first stage, we explore reddit pretraining (where the reddit data is pre-processed into (c, r), i.e., contextresponse, pairs as described below). in the second stage, we use two task-oriented dialogue datasets, taskmaster foot_3 @xcite and schema-guided dialogue foot_4 @xcite , to specialize the reddit-pretrained models. because the reddit data consists of open-domain-style dialogues (where belief states and dialogue acts are missing), pretraining on these datasets can familiarize the models with the sequence-to-sequence representation of task-oriented dialogue. three models, a context-to-response model, a responseto-context model and a response language model, are pretrained to initialize the direct model, the channel model and the source model, respectively. ## experiments we fine-tune and evaluate the pretrained models on three dialogue datasets: multiwoz 2.0, cam-rest676 and smcalflow @xcite . in this section we describe the datasets (section 5.1), fine-tuning (section 5.2), decoding (section 5.3), and evaluation metrics (section 5.4). results are presented in section 6, and analysis and ablation studies in section 7. ## analysis in this section, we use multiwoz and cam-rest676 to perform ablation studies on the effects of model combination, large-scale pretraining, and sample efficiency; as well as analyzing the runtime requirements of our model and the reasons for its success.