Paralinguistic Attitude Recognition for Spoken Dialogue Systems (2025)

Abstract: Although paralinguistic information is critical for human communication, most spoken dialogue systems ignore such information, hindering natural communication between humans and machines. This study addresses the recognition of paralinguistic attitudes in user speech. Specifically, we focus on four essential attitudes for generating an appropriate system response, namely agreement, disagreement, questions, and stalling. The proposed model can help a dialogue system better understand what the user is trying to convey. In our experiments, we trained and evaluated a model that classified paralinguistic attitudes on a reading-speech dataset without using linguistic information. The proposed model outperformed human perception. Furthermore, experimental results indicate that speech enhancement alleviates the degradation of model performance caused by background noise, whereas reverberation remains a challenge.

## introduction in human dialogue, people communicate various messages through paralinguistic features of speech, such as prosody and voice quality. speech can convey emotions and attitudes through paralinguistic features regardless of linguistic information. humans can recognize four intentions, namely affirm, deny, ask for repetition, and filler, with high accuracy using only paralinguistic features @xcite . moreover, humans can convey six intentions, namely criticism, doubt, naming, suggestion, warning, and wish, through prosodic patterns irrespective of lexical meaning @xcite . the paralinguistic information transmitted in this manner can affect listener behavior. by contrast, paralinguistic information is ignored by most spoken dialogue systems, which adopt a cascaded pipeline of automatic speech recognition (asr) and a linguistic dialogue model. this restriction requires users to convey their messages using only linguistic information; otherwise, miscommunication can occur. the limited paralinguistic ability in spoken dialogue systems impedes natural communication with humans. in this study, we address the challenge of enabling a spoken dialogue system to recognize attitudes expressed through paralinguistic features in user speech. specifically, we focus on four attitude classes, namely agreement, disagreement, questions, and stalling. only one of the four attitudes is deemed to accompany a single utterance. this is understood by the fact that boundary pitch movement at the end of an utterance substantially affects the attitude. therefore, articulating multiple attitudes in a single utterance is challenging for most users of spoken dialogue systems. in other words, the paralinguistic attitudes investigated in this study are mutually exclusive and evoked in units of utterances. we introduce one of the expected uses of our model, as illustrated in fig. 1 . an input user utterance is processed in parallel using an asr model and a paralinguistic attitude recognition model. the transcription and the inferred attitude are subsequently passed on to a large language model (llm). finally, a text-to-speech (tts) model synthesizes a system utterance according to the output generated by the llm. previous studies have explored methods to process paralinguistic cues in conjunction with transcripts by employing llms @xcite @xcite . a simple approach is to concatenate the transcript and class label in a prompt, for example: "@xmath0." ## model the network structure of the proposed model is listed in it is known that speech ssl models embed prosodic information in their hidden representations @xcite . moreover, the explicit incorporation of pitch into the input features in our preliminary experiments did not enhance the model performance. hence, we chose to use only the hidden representation of the hubert model. we note that linguistic information was not used as an input feature. one reason for this choice is that a cascaded pipeline of asr and the attitude recognition model cause considerable latency in generating a system response. to use linguistic information, the paralinguistic attitude recognition model should wait until the asr model yields a transcription, inevitably causing additional latency. thus, we made the model recognize attitude using only acoustic features to avoid hindering smooth communication. another reason is that a spoken phrase can be accompanied by distinct intentions depending on its paralinguistic features regardless of linguistic information @xcite @xcite . therefore, linguistic features were not significant in recognizing the four paralinguistic attitudes. another motivation was to avoid the domain dependence of linguistic features. linguistic choices are affected by situations where dialogue occurs and the relationship between participants. previous studies on paralinguistic information employing linguistic features focused on a specific domain, such as meetings @xcite and news delivery @xcite . we used only acoustic features so that the model is useful in various domains. ## data in this study, speeches read in japanese by crowd workers and actors were used. ## experiments we trained and evaluated a paralinguistic attitude recognition model using a speech dataset of crowd workers and actors. all speech data were processed at a sampling rate of 16 khz on a single channel. the hubert model was frozen, and the head layers were fine-tuned during training. we measured the performance in terms of the macro-f 1 score using six-fold cross-validation. for each fold, the entire dataset was split into six sets, namely four for training, one for validation, and one for testing. we augmented the training data four-fold by adding background noise and reverberation to improve model robustness. noise signals were randomly selected from the demand @xcite , musan @xcite , and fsd50k @xcite datasets. the signalto-noise ratio was randomly chosen from the uniform distribution from -10 to 10 db. room impulse responses were randomly sampled from the bird database @xcite . whether noise or reverberation was added during the test depended on the evaluation settings, as explained below. we set the layer depth at which the hubertlarge model yielded an embedding vector to 12 based on the validation data. this result is consistent with the findings of previous studies in which paralinguistic information was incorporated into the middle layers of the ssl models @xcite @xcite . ## conclusion this study addressed paralinguistic attitude recognition in user speech for spoken dialogue systems. specifically, we focused on four essential attitudes for determining a system reaction, namely agreement, disagreement, questions, and stalling. we trained and evaluated the model using a reading-speech dataset of actors and crowd workers. the proposed model outperformed human perception when evaluating the actors' speech under a clean condition. furthermore, the proposed model achieved almost the same performance on the crowd workers' speech after filtering by quality. noise and reverberation degraded the model performance. speech enhancement can alleviate the degradation caused by noise. however, the influence of reverberation remains a challenge. the use of paralinguistic attitude recognition enables spoken dialogue systems to understand what users convey through speech. finally, we discuss future research directions. we used a reading-speech dataset in this study. however, the manner in which attitudes are expressed through paralinguistic features varies depending on the situation in which speech utterances occur. human speech directed to spoken dialogue systems is more diverse than reading speech but more controlled than casual everyday conversation. therefore, we should investigate paralinguistic attitude recognition for speech directed to dialogue systems. another direction is to clarify how to determine a system reaction, given an inferred user's paralinguistic attitude. moreover, joint models of speech enhancement and paralinguistic attitude recognition should be examined to alleviate the degradation caused by reverberation.