Who is Speaking? Speaker-Aware Multiparty Dialogue Act Classification (2023)

Abstract: Utterances do not occur in isolation in dialogues; it is essential to have the information of who the speaker of an utterance is to be able to recover the speaker’s intention with respect to the surrounding context. Beyond simply capturing speaker switches, identifying how speakers interact with each other in a dialogue is crucial to understanding conversational flow. This becomes increasingly important and simultaneously difficult to model when more than two interlocutors take part in a conversation. To overcome this challenge, we propose to explicitly add speaker awareness to each utterance representation. To that end, we use a graph neural network to model how each speaker is behaving within the local context of a conversation. The speaker representations learned this way are then used to update their respective utterance representations. We experiment with both multiparticipant and dyadic conversations on the MRDA and SwDA datasets and show the effectiveness of our approach.

## introduction dialogue acts (das) describe the intention behind a speaker's utterance in a conversational context @xcite . recognizing das is helpful for several downstream tasks such as question-answering (goo and chen, 2018a), summarization @xcite , and conversational ai @xcite . utterances are the building blocks of a conversation, but they do not occur in isolation. to recover the communicative function behind an utterance, we need to consider its context. the context here refers not just to previous utterances, but also to dialogue structure, speaker behavior, etc. utterances are multifunctional in the sense that they encode many roles-e.g., turn management, relaying communicative intentions, etc. @xcite . speaker utterance label a and i've been sharing them with u u w folks also. statement b i'm sorry you've been what? understanding check c okay. backchannel b showing them? understanding check a sharing them with the u w folks. correct misspeaking b sharing them. mimic b okay. acknowledgement b okay. backchannel c okay. floor grabber one example of this is the broader category of the question da, which, along with seeking information, also implicitly gives the floor to the addressee. therefore, interlocutor information is needed to instill situational awareness of speakers in a conversation. in a dialogue, speakers influence each other which in turn dictates how they behave and convey their intentions. different speakers can also fulfill different roles, for example, within the local context of a conversation, a speaker can be leading the conversation by providing information while others could be active listeners and signal they are following along by means of backchannels as shown in there has not been much prior work on incorporating speaker awareness for da classification, particularly in the case of multiparty conversations (i.e., conversations involving more than two interlocutors). while there is some research on modeling speakers for da classification, it is mostly focused on two-party conversations. @xcite use a separate bilstm to model each speaker in a dyadic setting. although this is one way to capture speaker behavior, it does not scale to the case of multiparty dialogues and real-world situations where the number of speakers can vary at test time. recently, @xcite introduced a way to capture speaker turns by learning turn embeddings. for this to work in a multiparty setting, the dialogue needs to be reduced to a dyadic one. this is not ideal since doing this would take away crucial information on different participants' contributions. in the related field of emotion recognition in dialogues (erc), some work has been done to encode speakers @xcite @xcite . these works primarily use graph neural networks (gnns) to enhance the representation of utterances. speaker information is injected either through edges or by adding speaker nodes to the graph. since the main objective of these methods is to encode utterances better, often different types of edges are introduced to better capture the different relationships between utterances. for example, different types of edges for past and future connections @xcite or learning dedicated edge types for each individual speaker @xcite . this translates to learning a bigger model with more parameters as well as a denser graph with many edges. due to the large memory requirements of such models, training them can be hard in a resource-scarce situation. we show in our work that instead of using the speaker as a means to capture utterance context better, directly learning the speaker representations is more efficient. this results in a simpler graph with fewer edges. in this work, we present a gnn-based framework that takes contextual utterance vectors as input and encodes conversation dynamics by connecting the node of each speaker with their utterances. the learned speaker representations are then concatenated with the utterance representations to get speaker-enriched utterance representations which are then used for da classification. we conduct extensive analysis on the icsi meeting recorder dialog act (mrda) corpus @xcite to understand how the fine-grained da classes are affected by incorporating speaker representations. to make utterances speaker turn-aware, @xcite learn two turn embeddings to capture turns. the embedding vector based on the speaker label is added to the utterance vector and then passed to another rnn to capture context. whereas shang et al. (2020) use a crf layer to model turn-taking. while this is an effective way to inject speaker information, multiparty dialogues need to be converted to dyadic dialogues for these approaches to work. @xcite work in the setting of conversational agents and aim to capture the speaker roles of the agent and the user using a separate bilstm for each. this technique cannot be applied to natural conversations where the number of participants is not fixed. @xcite use three rnn encoders: word-level, speaker-level, and utterance-level. the speaker-level rnn is fed utterances grouped by speaker to capture speaker personas. the representations from this encoder are then fed to the utterance-level encoder for wider context modeling. their work differs from ours in the way that they model each speaker. while they encode utterance context at the speaker level, we learn an explicit speaker representation. ## model in this section, we describe the components of our model, an overview of which is presented in figure 1 . ## results and discussion we compare the speaker graph model against the stronger of the two baselines and consequently also use the turn-aware input for the graph model. all the results are an average of 5 runs to account for the fluctuation introduced by randomness. the speaker graph model's performance is statistically significant with @xmath0.0001 using student's t-test. ## conclusion we propose a graph-based approach to learn speaker-informed utterance representations for da classification. we show that directly learning speaker representations with a simple graph is both effective and efficient. instilling speaker information this way helps disambiguate da labels in the case of multiparty dialogues. the learned speaker representations can be used on top of any utterance encoding scheme to include speaker information. future work can also look into incorporating audio features to encode a speaker since prosody plays an important part in signaling intention.