What BERT Based Language Model Learns in Spoken Transcripts: An Empirical Study (2021)

Abstract: Language Models (LMs) have been ubiquitously leveraged in various tasks including spoken language understanding (SLU). Spoken language requires careful understanding of speaker interactions, dialog states and speech induced multimodal behaviors to generate a meaningful representation of the conversation. In this work, we propose to dissect SLU into three representative properties: conversational (disfluency, pause, overtalk), channel (speaker-type, turn-tasks) and ASR (insertion, deletion, substitution). We probe BERT based language models (BERT, RoBERTa) trained on spoken transcripts to investigate its ability to understand multifarious properties in absence of any speech cues. Empirical results indicate that LM is surprisingly good at capturing conversational properties such as pause prediction and overtalk detection from lexical tokens. On the downsides, the LM scores low on turn-tasks and ASR errors predictions. Additionally, pre-training the LM on spoken transcripts restrain its linguistic understanding. Finally, we establish the efficacy and transferability of the mentioned properties on two benchmark datasets: Switchboard Dialog Act and Disfluency datasets.

## introduction language models (lm) such as bert, gpt etc., have led to significant advancements in the field of natural language processing (nlp) by learning representations pre-trained on a vast amount of text data. these lms have shown to perform well on a multitude of downstream tasks such as question answering, intent, entity and sentiment detection, natural language inference etc. @xcite . along with nlp, the proliferation of voice-enabled technologies has resulted in the significance of spoken language understanding (slu) tasks. the general modus-operandi of slu systems is to convert voice into text using an asr engine and use natural language understanding (nlu) on the transcribed text by modelling conversational and channel properties while being robust to asr errors. since spoken conversation boasts of amalgamation of spontaneous speaker interactions, it has become imperative for model architectures to capture multimodal features from text and speech modalities @xcite . the aim of these multimodal systems is to capture acoustic information such as pitch, intonation, rate of speech, etc. along with properties such as pause, disfluency, overtalk and turn shifts to synthesize a complete understanding of what is being spoken and how it is being spoken. while some of these properties rely on speech signals and can only be captured by acoustic systems, properties such as pauses, disfluency, overtalk, turn shifts, speech recognition errors etc. get passed onto the lexical domain in the form of spoken transcripts and metainformation. few works suggest that the auxiliary tasks such as speaker intent detection @xcite , filler prediction @xcite , pause and speaking-rate @xcite help to train a robust model for downstream tasks. thus, there is a need for a vigilant language model capable of encoding the multi-dimensional properties of a slu dataset to reduce dependence on auxiliary information and/or multimodal inputs. however, in order to come up with such lm for slu domain, a pre-requisite is to investigate the extent to which an lm already encodes these properties. in this work, we propose to categorize the spoken language understanding into distinctive properties: 1) conversational: encompass spontaneous speech properties such as pauses, disfluency, overtalk; 2) channel: properties that capture speaker turns such as speaker-type, turn-taking; 3) asr: properties such as error type @xcite . while core focus in this work is to not propose a state-of-the-art language model, we believe that unveiling the behavior of lm on these properties will not only help us to understand the encoded information in a pre-trained model, but will also influence architectural designs to build upon the findings. thus, we limit our experiments to utilize only lexical tokens present in the spoken transcripts. specifically, we aim to answer the following research questions (rq) in our work: to answer rq1, we frame a set of probing tasks ( in this section, we define probing tasks with respective examples. we also describe the dataset creation methodology in detail in section a.1. ## properties infused lm: a multi-task learning setup (rq3) we hypothesize that an lm that better encodes the conversational, channel and asr properties in its representation performs better on downstream slu tasks. in order to verify the hypothesis, we follow a two step pipeline: 1. in the first step, we infuse lm with these properties by fine-tuning both lms on all probe tasks simultaneously using a multi-task learning (mtl) framework. 2. next, we freeze the weights of the lms and evaluate the performance of the probe classifier on a couple of external datasets: switchboard dialog act and switchboard disfluency for dialog act classification and disfluency detection respectively. it should be noted that none of these datasets are utilized in the pretraining step of the two lms. since mtl-trained lms get explicit supervision to capture properties needed by probing tasks, their performance on downstream tasks would also indicate the relevance of the proposed probing tasks. ## results and analysis the results for experiments are presented in ## related works recent years have seen a surge of works on the theme of neural network interpretability and understandability. a group of work has focused on unveiling secrets of language models @xcite . @xcite probe the surface and linguistic patterns in the attention head of bert through a set of probing tasks, @xcite throws light on structural understanding of the language captured by phrase and span representations. other research works show limitations of lm such as ignoring negation and getting confused by simple distractors @xcite in addition to bert being inexact in encoding numeracy in its representations @xcite . there has also been research on probing lm on application specific representations such as question answering @xcite , information retrieval @xcite , recommendation systems @xcite , dialog systems @xcite etc. the entire spectrum of these works aims to understand the learning capability and properties encoded in the lm along with discovering their shortcomings. the findings in these works have been utilized by the scientific community to create more robust language models @xcite propose semantics-aware bert that performs better than vanilla bert, while @xcite demonstrate that incorporating sen-tence structure in pre-training lm pushes the results on downstream tasks. however, a major section of work have primarily focused on interpreting lms trained on clean text. slu tasks come with their distinctive properties of asr errors and non-grammatical, illpunctuated texts. a spoken interaction additionally requires conversational and channel understandability as highlighted in this work. while works have been carried out in understanding disfluency @xcite and turn-taking @xcite , the authors narrowly aim at improving the task specific results by modelling acoustic cues @xcite or training with auxiliary tasks @xcite @xcite . the effort in our work is orthogonal to what has been carried out in the past research. we investigate the representations of language model on real-life and benchmark datasets to identify the strength, limitations and possibility of a generic lm under the lens of conversational, channel and asr properties. ## conclusions and future work we investigate the representations of language model across conversational, channel and asr properties with probing tasks such as pause, disfluency, overtalk identification, speaker role prediction, turn-tasks and asr error type. empirical analysis shows that lm encodes conversational and speaker-type properties to a large extent without external supervision while it has lower performance on turn-tasks and asr error prediction. experiments also show that a language model trained on spoken transcripts loses the linguistic understanding of dependency relations. a set of mtl experiments demonstrate the efficacy and transferability of the probe tasks to an unseen slu dataset that advocates a need and possibility of a domain specific lm. in future, we would like to research along a few directions that originates from this work: benchmarking properties-infused lm against multimodal counterparts and revisiting the pre-training setup to identify advanced mechanisms to infuse such properties. additionally, we would also like to evaluate on other language models such as xl-net, gpt etc. to understand the generalizability of the results.