Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning (2021)

Abstract: We use dialogue act recognition (DAR) to investigate how well BERT represents utterances in dialogue, and how fine-tuning and large-scale pre-training contribute to its performance. We find that while both the standard BERT pre-training and pretraining on dialogue-like data are useful, task-specific fine-tuning is essential for good performance.

## data we perform experiments on the switchboard dialogue act corpus (swda), which is a subset of the larger switchboard corpus, and the dialogue acttagged portion of the ami meeting corpus (ami-da). swda is tagged with a set of 220 dialogue act tags which, following @xcite , we cluster into a smaller set of 42 tags. ami uses a smaller tagset of 16 dialogue acts @xcite . see preprocessing we make an effort to normalize transcription conventions across swda and ami. we remove disfluency annotations and slashes from the end of utterances in swda. in both corpora, acronyms are tokenized as individual letters. all utterances are lower-cased. utterances are tokenized with bert's word piece tokenizer with a vocabulary of 30,000. to this vocabulary we added five speaker tokens and prepend each utterance with a speaker token that uniquely identifies the corresponding speaker within that dialogue. ## model we use a simple neural architecture with two components: an encoder that vectorizes utterances (bert), and single-layer rnn sequence model that takes the utterance representations as input. 2 at each time step, the rnn takes the encoded utterance as input and its hidden state is passed to a linear layer with softmax over dialogue act tags. 3 conceptually, the encoded utterance represents the context-agnostic features of the utterance, and the hidden state of the rnn represents the full discourse context. for the bert utterance encoder, we use the bert base model with hidden size of 768 and 12 transformer layers and self-attention heads @xcite . in our implementation, we use the un-cased model provided by @xcite . the rnn has a hidden layer size of 100. ## discussion a key aspiration of transfer learning is to expose the model to phenomena that are too infrequent to learn from labeled training data alone. we show some evidence of that here. pre-trained bert-ft performs better on infrequent dialogue acts than bert-ri, suggesting it draws on the extensive pre-training to represent infrequent features of those utterances. indeed, a simple lexical probe supports this explanation: in utterances where the pre-trained model is correct and the randomly initialized model is not, the rarest word is 1.9 times rarer on average than is typical of corpus as a whole. in spite of that, the representations learned through pre-training are simply not performant without task-specific fine-tuning, suggesting that they are fundamentally lacking in information that is important for the dialogue context. we should note that this is in stark contrast to many other non-dialogical semantic tasks, where frozen bert performs on par or better than the fine-tuned model @xcite . by performing additional pre-training on a large dialogue-like corpus (opensubtitles), we were able to raise the performance of the frozen encoder by a small amount. this deserves further investigation. @xcite find that further pre-training bert on a large-scale reddit and twitter corpus is helpful for response selection, but given the unimpressive results with subtitles, it remains an open question how well the text chat and social media domains transfer to natural dialogue. there is also abundant room to investigate how speech-related information, such as laughter, prosody, and disfluencies can be incorporated into a dar model that uses pre-trained features. @xcite showed, for example, that dialogue acts can have specific prosodic manifestations that can be used to improve dialogue act classification. incorporating such information is crucial if models pre-trained on large-scale text corpora are to be adapted for use in dialogue applications.