Hierarchical Fusion for Online Multimodal Dialog Act Classification (2023)

Abstract: We propose a framework for online multimodal dialog act (DA) classification based on raw audio and ASR-generated transcriptions of current and past utterances. Existing multimodal DA classification approaches are limited by ineffective audio modeling and late-stage fusion. We showcase significant improvements in multimodal DA classification by integrating modalities at a more granular level and incorporating recent advancements in large language and audio models for audio feature extraction. We further investigate the effectiveness of self-attention and cross-attention mechanisms in modeling utterances and dialogs for DA classification. We achieve a substantial increase of 3 percentage points in the F1 score relative to current state-of-the-art models on two prominent DA classification datasets, MRDA and EMOTyDA.

## introduction dialog acts (das), as described by @xcite , are the minimal units of linguistic communication that represent the speaker's intention behind an utterance in a conversation. successful da classification facilitates a range of downstream tasks including task-oriented dialog systems @xcite , conversational agents @xcite , and dialog summarization @xcite . since da classification is crucial for understanding spontaneous dialog @xcite , a significant amount of effort has been put into modeling das computationally. however, most of the research efforts on da classification heavily rely on oracle transcriptions @xcite @xcite @xcite , resulting in several limitations. first, oracle transcriptions are typically inaccessible during real-time dialog processing, such as in conversational agent systems. these systems speaker utterance dialog act me018 i guess when sunil gets here he can do his last or something. ## proposed method in this section, we will describe the components of the proposed model in detail (see figure 1 ) ## conclusion we have put forward a hierarchical fusion model for online dialogue act (da) classification. the early-stage fusion encoder component of the proposed model generates multimodal utterance representations more effectively. we also propose a novel audio feature extraction method using whisper. our detailed examination of different parts of our model confirms their relative importance and explains the effectiveness of our appraoch. the proposed hierarchical fusion model yields a remarkable 3% improvement in the f1 score over the related baseline, and a noteworthy 5% improvement over using a text based model for online da classification. inspired by our exciting results from early fusion, in future we want to investigate even more granular fusion of the modalities at word level from unaligned data.