Quantifying the redundancy between prosody and text (2023)

Abstract: Prosody—the suprasegmental component of speech, including pitch, loudness, and tempo—carries critical aspects of meaning. However, the relationship between the information conveyed by prosody vs. by the words themselves remains poorly understood. We use large language models (LLMs) to estimate how much information is redundant between prosody and the words themselves. Using a large spoken corpus of English audiobooks, we extract prosodic features aligned to individual words and test how well they can be predicted from LLM embeddings, compared to non-contextual word embeddings. We find a high degree of redundancy between the information carried by the words and prosodic information across several prosodic features, including intensity, duration, pauses, and pitch contours. Furthermore, a word’s prosodic information is redundant with both the word itself and the context preceding as well as following it. Still, we observe that prosodic features can not be fully predicted from text, suggesting that prosody carries information above and beyond the words. Along with this paper, we release a general-purpose data processing pipeline for quantifying the relationship between linguistic information and extra-linguistic features.

## introduction in human communication, information flows between interlocutors along multiple channels. apart from linguistic information conveyed by the text foot_0 itself, meaningful content is also transmitted through facial, hand, and body gestures as well as through prosody. in spoken language, prosody consists of the acoustic features of the speech signal such as energy, duration, and fundamental frequency (f 0 ). 2 it carries both linguistic information, such as lexical stress, and discourse function, e.g., extra-linguistic information, such as social and emotional cues @xcite @xcite . previous studies have suggested partial redundancy between the acoustic features of prosody and linguistic information. for instance, several prosodic features, e.g., pitch and duration, correlate with the predictability of a word in context @xcite @xcite @xcite and can mark the location and type of focus in a sentence, emphasizing which information is novel @xcite . these studies suggest that information signaled by prosody is redundant with information from the preceding text, arguably implying that prosody is largely used to direct the interlocutor's attention. however, in some cases, prosody seems essential to transmit unique information, like marking irony, resolving ambiguous syntactic attachments, or transforming a statement into a question. therefore, characterizing the relationship between the information conveyed by prosody and by words is essential for understanding the role of prosody in human communication. our paper quantifies the redundancy between prosody and text using information-theoretic techniques. we operationalize redundancy as mutual information @xcite , an information-theoretic measure that quantifies the amount of information (in units such as bits) obtained about one random variable (e.g., prosody) by observing the other random variable (e.g., text). we build upon existing work in the emerging field of information-theoretic linguistics @xcite @xcite . previous research has applied information theory to analyze various facets of language such as word lengths, ambiguity, and syntax @xcite @xcite ., f0 contours (a.u) 0 0.2 0.4 0.6 0.8 1 1.2 1.4 time (seconds) pauses (sec) duration (sec/syllable) energy (a.u) what shall i do without you? frequency (hz) 0 10k 5k prominence (a.u) relative prominence (a.u) extracted prosodic features current word past context future context non-contextual word embeddings (e.g., fasttext) bidirectional llm (e.g., bert) current prosodic feature autoregressive llm (e.g., gpt-2) bidirectional context ## redundancy as mutual information our objective is to estimate the redundancy between a text and its accompanying prosodic features. as a first step, we define the notation we will use in this section. let σ be an alphabet, a finite, non-empty set of symbols. we will consider a text to be a string, i.e., an element of σ * , and we will denote it as w. with w, we denote a text-valued random variable. in the experimental section of this work, we will encounter text that is either one word or several sentences totaling no more than 100 words. we further consider prosody to be a real-valued vector p @xmath0, and we denote a prosody-valued random variable as p t . the subscript t marks the distinguished position of the word in a text whose prosody we wish to investigate. in contrast to p t , the portion of text that w corresponds to is currently under-specified. as detailed later in §3, we consider text either before, after, or at position t (see figure 1 ). we now offer a quantitative treatment of the redundancy between prosody and text, operationalizing it as the mutual information: foot_3 we next discuss a few properties of this mutual information, and how we estimate it. ## the role of context while in §2 we deliberately kept the text-valued random variable w underspecified, we now introduce it more precisely to quantify the redundancy of prosody and different parts of the text. we measure the redundancy of prosody at a word t, which we write as p t , with three different textvalued random variables: a single word w t , all past words w ← , and all other words in a sentence w ↔ . we then estimate these measures using uncontextualized word embeddings, as well as autoregressive and bidirectional language models. we discuss each of these setups, briefly, below. ## our prosody pipeline in this section, we describe the data processing pipeline we use to extract the prosodic features energy, duration, pause, and pitch curve from wordlevel text-speech alignments. ## discussion we studied the interaction between speech prosody and linguistic meaning conveyed by the words themselves via computing the mutual information (mi) between prosodic features extracted from a corpus of spoken english audiobooks and its associated text. we found that, consistently across a variety of prosodic features, there exists non-trivial mi between the prosody of a given word and the word itself. for all prosodic features, there is an increase of the mi between prosody and text when considering the preceding context as well. furthermore, for most prosodic features, considering the words following the target word further increases the mi with the prosody. our results quantitatively reveal the partial redundancy between prosodic features and linguistic content extracted from text alone. our large-scale approach to prosody. much previous research on prosody concentrated on characterizing specific prosodic features and identifying their contributions to specific linguistic functions. in contrast to these previous studies, our methods facilitated a large-scale systematic examination of the statistical properties of various prosodic features and their mi with abstract text representations extracted using language models. as such, our approach lacks some interpretability relative to previous studies, as we do not know what are the specific linguistic functions associated with the information that we found to be redundant with certain prosodic features. the latter could be further explored in future work by comparing text where prosody predictions were especially high or low. however, a big advantage of our approach compared to previous studies is the quantitative large-scale examination of a spoken linguistic corpus, thus potentially revealing more mutual information between prosody and linguistic information than was previously possible to show, because the models may extract abstract linguistic representations that were not specifically targeted by previous studies. the role of context. the positive mi of the non-contextualized estimators indicates that there is already information in word identity about prosodic features. this indicated that some words tend to have different prototypical prosodic values than other words. this is consistent with past studies suggesting that the cognitive representation of words interacts with their typical prosody @xcite . comparing the mi of prosody with past vs. bidirectional context can give insight into real-time language production and comprehension. for production, the gain in mi for both context types indicates that language production systems (whether in humans or machines) would be well-served by having a representation of the full utterance in order to effectively plan and execute the prosodic features of their speech. as far as online comprehension, the results for bidirectional contexts suggest two things: first, it is not just the case that linguistic content can be used to predict prosody, but that prosodic information could be used to predict future textual information, insofar as there is redundant information between the two. second, they suggest that during comprehension, the meaning of a prosodic event may only be fully perceived at the end of the utterance. differences between prosodic features. although our results suggest that most prosodic features gain a significant amount of information from past context as well as future context (figure 2 ), there are some interesting details and exceptions. f0 contours as well as relative prominence gain most of their information from past and much less from future context. in contrast, pauses gain most of their information with text from future context and almost none from past context. the relatively larger importance of future context for pauses we interpret as confirming previous work on the relationship between pauses and syntactic boundaries @xcite , where the following words can help reveal whether a current word marks the end of a syntactic unit. one potential confounding factor for all these points, however, is the effect of punctuation marks, which are considered in the bidirectional but not the autoregressive estimators. in future research, we plan to disentangle these two to better understand the effect of the following words factoring out punctuation marks, which are not present in spoken communication. it is important to mention that the order of magnitude difference in the mutual information scales between the f0 contours and the rest of the prosodic features (figure 2 ) is due to their representation as multidimentional features. we represented the full f0 curve with a dimensionality reduction achieved by a dct decomposition and found that 8 dct components were favorable in terms of mi maximization. future work can use our methodology to further explore the prosodic functions of f0 curves. future work. exploring the interaction between linguistic communication channels holds a deep significance in understanding the multidimensional distribution of meaning in language, which extends beyond prosody to include facets like gestures and facial expressions. the implications of this understanding are profound. for example, the excess information contained in prosody could set a theoretical performance ceiling for textto-speech systems, which are tasked with generating speech prosody from text alone. the integration of prosodic information could aid neural language model training, which, despite excelling in machine translation @xcite and language generation @xcite , predominantly rely on text data. as prosody is fundamental to spoken language, incorporating it may address the current endeavor to boost language model performance using developmentally realistic data volumes @xcite . ## conclusion we presented a systematic large-scale analysis of the mutual information between a variety of suprasegmental prosodic features and linguistic information extracted from text alone, using the most powerful openly available language models. the theoretical framework we developed to estimate mutual information between these two communication channels, as well as our openly available data processing implementation, can be used to investigate interactions between other communication channels. these tools enabled us to quantitatively demonstrate that information conveyed by speech prosody is partly redundant with information that is contained in the words alone. our results demonstrate a relationship between the prosody of a single word and the word's identity, as well as with both past and future words.