The importance of fillers for text representations of speech transcripts (2020)

Abstract: While being an essential component of spoken language, fillers (e.g. “um” or “uh”) often remain overlooked in Spoken Language Understanding (SLU) tasks. We explore the possibility of representing them with deep contextualised embeddings, showing improvements on modelling spoken language and two downstream tasks — predicting a speaker’s stance and expressed confidence.

## introduction disfluencies are interruptions in the regular flow of speech, such as pausing silently, repeating words, or interrupting oneself to correct something said previously @xcite . they commonly occur in spoken language, as spoken language is rarely fluent. fillers are a type of disfluency that can be a sound ("um" or "uh") filling a pause in an utterance or conversation. recent work has shown that contextualised embeddings pre-trained on large written corpora can be fine-tuned on smaller spoken language corpora to learn structures of spoken language @xcite . however, for nlp tasks, fillers and all disfluencies are typically removed in pre-processing, as nlp models achieve highest accuracy on syntactically correct utterances. this contradicts linguistic studies, which show that fillers are an essential and informative part of spoken language @xcite @xcite @xcite . so far, the information carried by fillers has only been studied using hand crafted features, for example in le @xcite ; @xcite ; @xcite . besides, @xcite show that pre-trained word embeddings such as word2vec @xcite , have poor representation of spontaneous speech words such as "uh", as they are trained on written text and do not carry the same meaning as when used in speech. we address the matter of representing fillers with deep contextualised word representations @xcite , and investigate their usefulness in nlp tasks for spoken language, without handcrafting features. hence, the present work is motivated by the following observations: (1) fillers play an important role in spoken language. for example, a speaker can use fillers to inform the listener about the linguistic structure of their utterance, such as in their (difficulties of) selection of appropriate vocabulary while informing the listener about a pause in their upcoming speech stream @xcite . (2) fillers and prosodic cues have also been linked to a speaker's feeling of knowing (fok) or expressed confidence, that is, a speaker's certainty or commitment to a statement @xcite . @xcite observed that fillers and prosodic cues contribute to the listener's perception of the speaker's expressed confidence in their utterance, which they refer to as the feeling of another's knowing (foak), also observed by @xcite . (3) recent work has shown that fillers have been successful in stance prediction (stance referring to the subjective spoken attitude towards something) @xcite . aim of this work: we want to verify that these observations are still valid when we represent fillers in an automatic and efficient way. hence, our contributions are as follows: (1) fillers contain useful information that can be leveraged by deep contextualised embeddings to better model spoken language and thus should not be removed. in addition, we study which filler representation strategies are best suited to our task of spoken language modelling (slm) and investigate the learnt positional distribution of fillers. (2) we show that in a spontaneous speech corpus of spoken monologues, fillers are a discriminative feature in predicting the perception of expressed confidence of the speaker, and perception of a speaker's stance (which we measure by sentiment). ## conclusion when working with deep contextualised representations of transcribed spoken language, we showed that retaining fillers can improve results, both when modelling language and on a downstream task (foak and stance prediction). besides, we propose and compare various token representation and pre-processing strategies in order to integrate fillers. we plan to extend these results by studying the mixing of such textual filler-oriented representations with acoustic representations, and further investigate the representation of fillers learnt during pre-training.