Hierarchical Pre-training for Sequence Labelling in Spoken Dialog (2020)

Abstract: Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. In this work, we propose a new approach to learn generic representations adapted to spoken dialog, which we evaluate on a new benchmark we call Sequence labellIng evaLuatIon benChmark fOr spoken laNguagE benchmark (SILICONE). SILICONE is model-agnostic and contains 10 different datasets of various sizes. We obtain our representations with a hierarchical encoder based on transformer architectures, for which we extend two well-known pre-training objectives. Pre-training is performed on OpenSubtitles: a large corpus of spoken dialog containing over 2.3 billion of tokens. We demonstrate how hierarchical encoders achieve competitive results with consistently fewer parameters compared to state-of-the-art models and we show their importance for both pre-training and fine-tuning.

## introduction the identification of both dialog acts (da) and emotion/sentiment (e/s) in spoken language is an important step toward improving model performances on spontaneous dialogue task. especially, it is essential to avoid the generic response problem, i.e., having an automatic dialog system generate an unspecific response -that can be an answer to a very large number of user utterances @xcite . da and emotion identification @xcite are done through sequence labelling systems that are usually trained on large corpora (with over 100k labelled utterances) such as switchboard @xcite ⇤ equal contribution 1992), mrda @xcite or daily dialog act @xcite . even though large corpora enable learning complex models from scratch (e.g., seq2seq @xcite ), those models are very specific to the labelling scheme employed. adapting them to di↵erent sets of emotions or dialog acts would require more annotated data. generic representations @xcite @xcite @xcite have been shown to be an e↵ective way to adapt models across di↵erent sets of labels. those representations are usually trained on large written corpora such as oscar @xcite , book corpus @xcite or wikipedia @xcite . although achieving state-of-the-art (sota) results on written benchmarks @xcite , they are not tailored to spoken dialog (sd). indeed, @xcite have suggested that training a parser on conversational speech data can improve results, due to the discrepancy between spoken and written language (e.g., disfluencies @xcite , fillers @xcite , different data distribution). furthermore, capturing discourse-level features, which distinguish dialog from other types of text @xcite , e.g., capturing multi-utterance dependencies, is key to embed dialog that is not explicitly present in pre-training objectives @xcite @xcite , as they often treat sentences as a simple stream of tokens. the goal of this work is to train on sd data a generic dialog encoder capturing discourse-level features that produce representations adapted to spoken dialog. we evaluate these representations on both da and e/s labelling through a new benchmark silicone (sequence labelling evaluation benchmark for spoken language) composed of datasets of varying sizes using di↵erent sets of labels. we place ourselves in the general trend of using smaller models to obtain lightweight representations @xcite that can be trained without a costly computation infrastructure while achieving good performance on several downstream tasks @xcite . concretely, since hierarchy is an inherent characteristic of dialog @xcite , we propose the first hierarchical generic multi-utterance encoder based on a hierarchy of transformers. this allows us to factorise the model parameters, getting rid of long term dependencies and enabling training on a reduced number of gpus. based on this hierarchical structure, we generalise two existing pre-training objectives. as embeddings highly depend on data quality @xcite and volume @xcite , we preprocess opensubtitles @xcite : a large corpus of spoken dialog from movies. this corpora is an order of magnitude bigger than corpora @xcite @xcite used in previous works @xcite . lastly, we evaluate our encoder along with other baselines on silicone, which lets us draw finer conclusions of the generalisation capability of our models foot_0 . ## method we start by formally defining the sequence labelling problem. at the highest level, we have a set d of conversations composed of utterances, i.e., @xmath0, c 2 , . . . , c |d| ) with @xmath1, y 2 , . . . , y |d| ) being the corresponding set of labels (e.g., da, e/s). at a lower level each conversation c i is composed of utterances u, i.e c @xmath2, u 2 , . . . , u |c i | ) with y @xmath3, y 2 , . . . , y |c i | ) being the corresponding sequence of labels: each u i is associated with a unique label y i . at the lowest level, each utterance u i can be seen as a sequence of words, i.e u @xmath4 concrete examples with dialog act can be found in utterances da how long does that take you to get to work? qw uh, about forty-five, fifty minutes. ## results on silicone this section gathers experiments performed on the silicone benchmark. we first analyse an appropriate choice for the decoder, which is selected over a set of experiments on our baseline encoders: a pre-trained bert model and a hierarchical rnn-based encoder (hr). since we focus on small-sized pre-trained representations, we limit the sizes of our pre-trained models to tiny and small (see ## model analysis in this section, we dissect our hierarchical pretrained models in order to better understand the relative importance of each component. we show how a hierarchical encoder allows us to obtain a light and e cient model. additional experiments can be found in appendix c. ## conclusions in this paper, we propose a hierarchical transformer-based encoder tailored for spoken dialog. we extend two well-known pre-training objectives to adapt them to a hierarchical setting and use opensubtitles, the largest spoken language dataset available, for encoder pretraining. additionally, we provide an evaluation benchmark dedicated to comparing sequence labelling systems for the nlp community, silicone, on which we compare our models and pre-training procedures with previous approaches. by conducting ablation studies, we demonstrate the importance of using a hierarchical structure for the encoder, both for pre-training and fine-tuning. finally, we find that our approach is a powerful method to learn generic representations on spoken dialog, with less parameters than state-of-the-art transformer models. these results open new future research directions: (1) to investigate new pre-training objectives leveraging the hierarchical framework in order to achieve better results on silicone while keeping light models (2) to provide multilingual models using the whole pre-training corpus (opensubtitles) available in 62 languages, (3) investigate robust methods @xcite and the application of our embedding to di↵erent anomaly detection settings @xcite . we hope that the silicone benchmark, experimental results, and publicly available code encourage further research to build stronger sequence labelling systems for nlp.