Analyzing discourse functions with acoustic features and phone embeddings: non-lexical items in T aiwan M andarin (2022)

Abstract: Non-lexical items are expressive devices used in conversations that are not words but are nevertheless meaningful. These items play crucial roles, such as signaling turn-taking or marking stances in interactions. However, as the non-lexical items do not stably correspond to written or phonological forms, past studies tend to focus on studying their acoustic properties, such as pitches and durations. In this paper, we investigate the discourse functions of non-lexical items through their acoustic properties and the phone embeddings extracted from a deep learning model. Firstly, we create a non-lexical item dataset based on the interpellation video clips from Taiwan’s Legislative Yuan. Then, we manually identify the non-lexical items and their discourse functions in the videos. Next, we analyze the acoustic properties of those items through statistical modeling and building classifiers based on phone embeddings extracted from a phone recognition model. We show that (1) the discourse functions have significant effects on the acoustic features; and (2) the classifiers built on phone embeddings perform better than the ones on conventional acoustic properties. These results suggest that phone embeddings may reflect the phonetic variations crucial in differentiating the discourse functions of non-lexical items.

## introduction people's everyday interactions include sounds that are not verbal words in the traditional sense. these sounds, such as sighs, sniffs, and grunts, are used in indexing the turn-taking in dialogues, marking stance, showing affections, and expressing roles and meanings in conversations @xcite . examples of these non-lexical items are un-huh in english as a marker showing understanding and attentiveness, while the single syllable uh and um act as fillers and disfluency markers @xcite . while these non-lexical items are important linguistically, they pose an interesting challenge to linguistic inquiry. non-lexical items do not belong to a major word class, and some do not conform to the language's phonological requirements @xcite . moreover, while the phonetic properties of non-lexical items could be generally described, they are nevertheless "phonetically underspecified." @xcite for example, in the study of "moan" in board game interactions, hofstetter (2020) found "moans" involve phonetic properties related to open vowels, irrespective of their frontness, backness, or roundedness. the study suggests that a non-lexical item can not be represented as a single phonetic symbol; instead, it may refer to the vowel space for which we do not have a general phonetic symbol. some studies, therefore, analyze these items in terms of their acoustic properties: the components' sound @xcite , the fundamental frequencies, durations, and intensities. @xcite . in contrast to the conventional acoustic property analysis, an alternative approach to analyzing non-lexical items is through the acoustic representations learned by data- the 34th conference on computational linguistics and speech processing (rocling 2022) @xcite . the association for computational linguistics and chinese language processing driven methods. these methods include deep learning models mapping the audio segments to the latent embedding space from acoustic data in a (self-)supervised fashion @xcite @xcite . although the models are not explicitly trained to represent the similarities among phonetic features, studies nonetheless find the audio segments with similar linguistic properties are closer together in the embedding space @xcite @xcite . therefore, these phonetic representations may already encode the phonetic variability of non-lexical items to reflect their different discourse functions. this study thus aims to investigate how the acoustic properties contribute to the nonlexical items' discourse functions and how the phone embeddings extracted from the deep learning model help differentiate those functions. the rest of the paper is organized as follows. we first review related works on discourse markers and how they are analyzed with acoustic properties (sec. 2). next, we describe our dataset on non-lexical items (sec. 3) in taiwan mandarin, in which we manually identify the items and annotate their discourse functions in interpellation video clips of taiwan's legislative yuan. finally, based on the dataset, we conduct the acoustic property analysis (sec. 4) and build classifiers based on the phone embeddings extracted from a deep learning model (sec. 5). finally, section 6 concludes the paper. ## non-lexical items dataset first, we used four interpellation video clips from taiwan's legislative yuan. 3 . audio tracks were then extracted from the clips, converted into 16 bit wav format, and resampled with 22khz sampling rates. the overall data comprise separate interpellation of two male and two female legislators, each ranging 6-8 minutes. the equal number of genders was to balance potential gender differences in the utterances. secondly, the audio segments of non-lexical items (e.g., uh, em, and ho) were annotated by three native speakers via praat 6.2.03 @xcite . each nonlexical item acquired two tags, one for functional role and one for pragmatic meaning. referring to @xcite , we defined the six candidates of role as follows: similarly, we summarized the following eight candidates for meaning. it is noted that certain non-lexical items may carry multiple pragmatic meanings, and that the candidates below are not mutually exclusive. thus, one non-lexical item is allowed to be annotated with multiple meaning tags. in sum, a total of 143 non-lexical items produced by the legislators were manually annotated. we then moved on to extract the acoustic properties for the dataset. ## acoustic property analysis with the assumption that the discourse functions may encode phonological variations, we illustrated our data collection and the annotation for non-lexical items in sec. 3. the following sections (4.1 and 4.2) then present the analyses and results of acoustic properties. ## phone embeddings instead of the common end-to-end models trained on waveforms and language-specific transcriptions in asr tasks, we chose the allosaurus model by @xcite foot_7 for retrieving the phone embeddings. specifically, the allosaurus is an universal phone recognizer integrating an asr encoder with an allophone layer, in which language-independent phone distributions are directly recognized and mapped into language-dependent phoneme distributions. we first examine the phone embeddings learned by the phone recognition model. in the video clips collected in section 3, the model automatically identifies 29,218 phones in the conversations. to investigate the phone organizations in the embedding space, we then extract the bi-lstm representations 9 with which model predicts the phones as phone embeddings. next, we average these embeddings by their predicted phones and obtain 34 phone centroids in the embedding space. we follow the literature @xcite and conduct hierarchical clustering with ward linkage based on the euclidean distances between the centroids. the clustering results are shown in figure 1a and figure 1b . we not only observe clear clusters of vowels and consonants but observe that the fricatives and stops tend to be close to each other with similar phonetic properties. the patterns suggest that the phone embeddings might reflect the phonetic variations in our conversation data. moreover, we inspect the clustering structure of recognized phones that occurred in the non-lexical items. figure 1c shows the twodimensional t-sne @xcite visualization of the 640-dimension phone embeddings obtained from allosaurus. the same phones tend to form distinct clusters, and the general distinction between vowels and consonants is still observed in the figure. it indicates that the embeddings may represent their corresponding phonetic properties. as @xcite have shown in their studies, allosaurus has the advantage of multilingual phone recog9 referring to the comments from the reviewers, the bi-lstm representations are used as the phone embeddings considering their better performance than the other representations (i.e., the 40-dimension mfccs and the phone logits) generated by allosaurus. nition and involves more phonological knowledge. it is thus appropriate for us to leverage these phone embeddings, by which the discourse functions of non-lexical items may be encoded. ## conclusion this paper focuses on the phonetic-pragmatic interrelationship of non-lexical discourse markers in taiwan mandarin. as we assume that linear svm model and the model baseline are adopted to not only display the data distributions but highlight the results of allosaurus, as we mainly focus on whether the phone representations really help us explore non-lexical items. based on the results, we did find the the model using phonetic realizations performs better in predicting the discourse functions, and we expect future research to develop better representations and state-of-the-art models that allow us to describe non-lexical items more appropriately. the discourse functions may be captured by the phonological variations, we firstly analyzed on the common acoustic properties (i.e., duration, nasality, mean pitch, f1, f2, and f3), followed by the classification tasks considering the 640d-phone embeddings. in comparison with the conventional acoustic properties, the model using phonetic realizations performs better in prediction of the functional role and pragmatic meaning of the non-lexical items. the result is consistent with our hypotheses that the phonetic realizations, embeddings via deep learning, encode certain phonological variations of non-lexical items and correlate with their discourse functions.