Probing Task-Oriented Dialogue Representation from Language Models (2020)

Abstract: This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks. We approach the problem from two aspects: supervised classifier probe and unsupervised mutual information probe. We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way. Meanwhile, we propose an unsupervised mutual information probe to evaluate the mutual dependence between a real clustering and a representation clustering. The goals of this empirical paper are to 1) investigate probing techniques, especially from the unsupervised mutual information aspect, 2) provide guidelines of pre-trained language model selection for the dialogue research community, 3) find insights of pre-training factors for dialogue application that may be the key to success.

## introduction task-oriented dialogue systems achieve specific user goals within a limited number of dialogue turns via natural language. they have been used in a wide range of applications, such as booking restaurants @xcite , providing tourist information @xcite , ordering tickets @xcite , and healthcare consultation @xcite . they are also crucial components of intelligent virtual assistants like siri, alexa, and google assistant. most of the task-oriented dialogue systems nowadays, are benefited from transfer learning @xcite , especially pre-trained language models trained on general text, such as bert @xcite and gpt2 @xcite . however, previous work claims that linguistic patterns could differ between writing text and human conversation, resulting in a large gap of data distributions @xcite . recently, several approaches are leveraging open-domain data @xcite , or aggregating task-oriented data @xcite to pre-train language models. in this paper, we are interested in answering these questions: which language model has the most informative representations that is better for what task-oriented dialogue task? does pretraining with dialogue-specific data or different objectives make any difference? we investigate how good these pre-trained representations are for a task-oriented dialogue system, ignoring the model architectures and training strategies by only probing their final representations with fine-tuning models. a good representation implies better knowledge transferring and domain generalization ability, making downstream applications easier and cheaper to be improved. we tackle this problem with two probing solutions: supervised classifier probe and unsupervised mutual information probe. classifier probe is commonly used in different nlp tasks such as morphology @xcite , sentence length @xcite , or linguistic structure @xcite . in this setting, we fine-tune a simple classifier for a specific task (e.g., intent identification) on a fixed pre-trained language model. the probe uses supervision to find the best transformation for each sub-task. in addition, we present mutual information probe to investigate these language models by directly clustering their output representations, as recent study @xcite suggests that a simple classifier may not be able to achieve the best estimate of mutual information between features and the downstream task. we apply two clustering techniques, k-means @xcite and gaussian mixture model @xcite , to calculate its adjusted normalized mutual information (anmi) @xcite between the predicted clustering and the true task-specific clustering. we investigate 12 language models, as shown in ## pre-trained language models w can roughly divide pre-trained language models into two categories: uni-directional and bidirectional. bert-based systems are bi-directional language models and usually trained with the masked language modeling (mlm) objective, i.e., given the left and right context to predict the current masked token. gpt-based models, on the other hand, are uni-directional language models trained always to predict the next token in an autoregressive way. for a bert-based model, we use the final-layer hidden state of its first token, @xcite , to represent an input sequence. this built-in token is originally designed to aggregate the information. since gptbased models are uni-directional and do not have a similar design as the [cls] token, we use the mean pooling of its output hidden states to represent the input sequence, which is better than only using the last hidden state in our experiments. bert-based bert is a transformer @xcite encoder with a self-attention mechanism, which is trained on wikipedia and bookcorpus using the mlm and next sentence prediction objectives. @xcite proposed a robustly optimized approach for bert, call roberta, where they improved it by training the model longer with bigger batches over more data and longer sequences, and removing the next sentence prediction objective. @xcite proposed a lite bert (albert) that trained with mlm and inter-sentence coherence losses, and aimed to lower memory consumption and increase the training speed. with similar motivation, @xcite trained a distilbert that reduce 40% of parameters with a triple loss, including mlm, distillation, and cosine-distance losses. @xcite proposed electra using a sample-efficient pretraining task called replaced token detection. they used a generator network (electra-gen) to replace tokens with plausible alternative tokens and trained a discriminative model (electra-dis) to predict whether the generator replaced each token in the input. most of the pre-trained models above are trained on general text corpora with language modeling objectives. @xcite , on the other hand, used social media conversational data to train the convert model. it is a transformer-based dualencoder model pre-trained on a dialogue response selection task using 727m reddit (input, response) pairs. very recently, @xcite proposed taskoriented dialogue bert (tod-bert), which is initialized by bert and further pre-trained on nine publicly available task-oriented dialogue corpora. they have one version with only mlm objective (tod-bert-mlm) and another with both mlm and contrastive learning objectives of response selection (tod-bert-jnt). tod-bert has shown good performance on several task-oriented downstream tasks, especially in the few-shot setting. gpt-based gpt2 @xcite is the representative of uni-directional language models using a transformer decoder, where the objective is to maximize left-to-right generation likelihood. to ensure diverse and nearly unlimited text sources, they use common crawl to obtain 8m documents as its training data. @xcite trained gpt2 on task-oriented response generation task, taking system belief, database result, and last dialogue turn as inputs. it only uses one dataset to train its model because few public datasets have database information available for pre-training. @xcite pre-trained gpt2 on 147m open-domain reddit data for response generation and called it dialogpt. it aims to generate more relevant, contentful, and consistent responses for chit-chat dialogue systems. in this paper, following tod-bert's idea, we train a task-oriented gpt2 model (tod-gpt2) built on the gpt2 model and further pre-trained with task-oriented datasets. we use the same dataset collection, which contains nine datasets in total, as shown in @xcite , to pre-train the model as a reference. ## method we define a dialogue corpus @xmath0, . . . , d m } has m dialogue samples, and each dialogue sample d m has t turns of conversational exchange {u 1 , s 1 . . . , u t , s t } between a user and a system. for every utterance u t or s t , we have human-annotated domain, user intent, slot, and dialogue act labels. we first feed all the utterances to a pre-trained model and obtain user and system representations. in this section, we first discuss how we design our classifier probe and then introduce our mutual information probe's background and usage. ## more analysis difference between probes ideally, both probes should distinguish the goodness of different pretrained language models, i.e., features that can be easily classified or features with high correlation with true distributions are preferred. however, we found that although, in general, the trends we observe from two probing methods are similar, they are not the same in terms of the ranking. when comparing the ranking of gpt2 and dialogpt models in figure 1 and figure 2 , we found that they obtain almost the worse anmi scores but work quite good in classification accuracy. this observation means that their representations of different classes are "close" to each other as a low anmi score suggesting a more noisy clustering. still, at the same time, it is not hard to find a hyperplane that can well discriminate those features. we discuss some possible reasons for this interesting observation in the following. the first guess is that these features may not follow a gaussian distribution, as we assume during clustering, suggesting that more advanced clustering techniques can be investigated in future work. the second guess is that these features have an unavoidable clustering noise that can be denoised or debiased easily by a strong supervision signal. the third guess, which may be a possible reason, is that these features are clustered by some other factors that are convert cluster 1 (failed booking) i am sorry but dojo noodle bar is solidly booked at that time . i can try a different time or day for you . 1 moment while i try to make the reservation of table for 8 , friday at 16:30 . booking was unsuccessful . can you try another time slot ? i am very sorry i was unable to book at acorn guest house for 5 nights , would you like to try for a shorter stay ? i am afraid that booking is unsuccessful . would you like a different day or amount of days ? cluster 2 (train time) there are 5 trains available , may i book 1 for you that leaves at 7:40 and arrives at 10:23 ? tr0330 departs at 14:09 and arrives by 15:54 . would you like a ticket ? the tr2141 arrives by 15:27 . would you like me to reserve some seats for you ? i have train tr4283 that leaves cambridge at 5:29 and arrives in bishops stortford at 6:07 . would you like to make reservations ? i have a train that leaves cambridge 14:01 arriving in birmingham new street at 16:44 . would that work ? cluster 3 (restaurant request) there are 21 restaurant -s available in the centre of town . how about a specific type of cuisine ? there are 9 indian restaurant -s in centre what price range do you want ? i am sorry , there are no catalan dining establishments in the city centre . would you like to look for a different cuisine or area ? i found 4 restaurant -s with the name tandoori that serve indian food on the south , west , and east . do you have a location preference ? there are no singaporean restaurant -s , but there are cheap ones offering several different cuisines . cluster 4 (confirm booking) all set . your reference number is k2bo09vq . i have got you booked for 16:30 . the reference number is eq0yaq1g . your reservation was a success and the reference number is jtwxfm7m . i have got your booking set , the reference number is 9rmfgjma . i booked tr3932 , reference number is fiw5abo2 . cluster 5 (hotel request) what part of town there are none in the west . i can help you with that . do you have any special area you would like to stay ? or possibly a star request ? there are no colleges close to the area you are requesting , would you like to chose another destination ? sure , what area are you thinking of staying ? i would be happy to help . may i ask what price range and area of town you are looking for ? as a result, we suggest a simple rule of thumb regarding which probing results. in short, the results of the classifier probe could be useful if a supervised approach for a downstream task is designed, e.g., user dialogue act prediction and dialogue state tracking. on the other hand, the mutual information probe is more effective for an unsupervised problem, e.g., utterance clustering and dialogue parsing tasks. visualization in figure 3 and figure 4 , we visualize the embeddings of tod-bert-jnt and gpt2 given the same system responses from the mwoz test set. each point is reduced from its highdimension features to a two-dimension point using the t-distributed stochastic neighbor embedding (tsne). we use different colors to represent different domains (left), dialogue acts (middle), and turn slots (right). as one can observe, tod-bert-jnt has more clear group boundaries and better clustering results than gpt2. visualization plots for other pre-trained models are shown in the appendix. what utterances are clustered together? in ## conclusion we investigate representations from pre-trained language models for task-oriented dialogue tasks, including domain identification, intent detection, slot tagging, and dialogue act prediction. we use a supervised classifier probe and a proposed unsupervised mutual information probe. from the ranking results of two different probings, we show a list of interesting observations to provide model selection guidelines and shed light on future research towards a more advanced language modeling learning for dialogue applications.