Transformer-based Multi-Party Conversation Generation using Dialogue Discourse Acts Planning (2023)

Abstract: Recent transformer-based approaches to multi-party conversation generation may produce syntactically coherent but discursively inconsistent dialogues in some cases. To address this issue, we propose an approach to integrate a dialogue act planning stage into the end-to-end transformer-based generation pipeline. This approach consists of a transformer fine-tuning procedure based on linearized dialogue representations that include special discourse tokens. The obtained results demonstrate that incorporating discourse tokens into training sequences is sufficient to significantly improve dialogue consistency and overall generation quality. The suggested approach performs well, including for automatically annotated data. Apart from that, it is observed that increasing the weight of the discourse planning task in the loss function accelerates learning convergence.

## introduction the popularity of dialogue systems has resulted in an increased demand for their utilization in various applications. existing approaches are largely focused on two-party conversations, which is applicable in chat-bots and assistance systems @xcite @xcite . at the same time, there is another type of dialogue, known as multi-party conversations @xcite @xcite . in this case, several interlocutors are involved in the dialogue, and the dialogue tree, consisting of successive utterances, is wide enough. this type of dialogue can be observed in internet forum discussion threads. due to the complexity of the structure of mpc dialogues, it becomes more challenging for the base seq2seq models to generate response texts. multi-task learning and external knowledge can be considered to simplify the utterance generation task. to this end, we additionally leverage the theory of dialogue acts @xcite whatever you do, never remove dinner time. that's the only time of the day i can take a shower. you should ask them to also expand dt to 30 minutes again so you can catch a few minutes sleep. sleep? why sleep when you have coffee 2017), which shows by which discourse rhetorical relations (more precisely, dialogue acts) the individual utterances of the dialogue are connected. figure 1 shows an example of such a structure. our major idea is that the use of dedicated discourse tokens to both input and target texts will enhance the coherence of discourse and consequently the overall quality of generation. to illustrate, let us examine the following example of a help request forum thread. initially, the user describes a problem and seeks advice. subsequently, multiple dialogue turns occur, culminating in the following phrases: generally speaking, we suggest multi-task learning consisting of dialogue acts planning and response generation joined in the single pipeline. we integrate discourse tokens into a two-stage pipeline for mpc generation (see section 3.1 for details). its first stage is used to identify a speaker and an addressee at the current step, whereas the second stage is used to generate the current response text. the first part is quite challenging, but recent studies allow one to solve it qualitatively @xcite . at the same time, only base models were researched for the second stage, leaving the relevance of discourse usage in dialogue generation unexplored. therefore, we mainly focus on the second stage. the task can be formalized as a graph2text, within which the bart and t5 models have already been partially investigated. key part here is a linearization technique that was used for some graph structures @xcite , but not for the discourse structure and dialogue generation yet. thus, we suggest integrating dialogue acts into the linearization of mpc graphs. our contributions can be summarized as follows: the code is available at https://github.com/ alchernyavskiy/discourse_mpc_generation . ## related work in this paper, we consider multi-party conversations (mpcs). the process of generation is generally split into two stages since it consists of several entire tasks. @xcite presented a task of identifying the speaker and the addressee of an utterance (first stage), and recent approaches have been aimed at improving results in this task @xcite . at the second stage, associated with the response generation task, some approaches use gcn to encode the complex mpc structure @xcite . it intended to improve prior recurrent neural network-based sequence-to-sequence (seq2seq) generation models @xcite . at the same time, it was shown that recent transformer-based approaches, such as bart @xcite and t5 @xcite , achieve top results in various generation tasks, including dialogue generation. therefore, bart and t5 are commonly used as the base generation models in recent approaches. for instance, @xcite uses the bart as a backbone to train the model that considers long-range contextual emotional relationships. moreover, recent transformer-based approaches effectively solve graph-to-text generation tasks. @xcite demonstrated that bart and t5 outperform various gnns trained to encode amr graphs in the amr-to-text task. similarly, @xcite indicated that pre-training in the form of t5 enables simple, end-to-end models to outperform pipelined neural architectures tailored for data-to-text generation. the key factor here is that any graph can be linearized, and @xcite showed that transformers are invariant to the method by which graphs are linearized. thus, we do not explore ways of linearizing dialogue graphs augmented by discourse relations, but choose one of the most reasonable ones. discourse parsing of multi-party conversations is an adjacent direction that is gaining popularity. there are several works where the dialogues were analyzed in terms of discourse structure ad discourse relations parsing @xcite @xcite . despite this, there are not a large number of publicly available datasets with discursivelyannotated dialogues. basically, all comparisons are conducted for the stac dataset @xcite , which is quite small. as far as we know, the only large publicly available dataset is the cdsc dataset @xcite . the main difference in our research from this direction is that we do not aim to suggest a novel discourse parser. at the same time, we use existing parsers and explore the importance of using discourse in the applied generative task. our idea of generating discourse relations in dialogues comes from the story-telling task. to ## datasets this section presents the discursively-annotated datasets used for evaluation. ## experiments this section discusses implementation details and experiment results, including human evaluation. ## discussion in this section, we partially explain how discourse improves generation quality for the model trained using discourse planning and demonstrate differences using concrete examples. ## conclusion and future work in this paper, we explored the effectiveness of transformer fine-tuning based on discourse dialogue acts planning for the multi-party conversation generation task. we evaluated our approach on the largest manually and automatically annotated datasets of dialogues from reddit. the evaluation including automatic and human assessments revealed that incorporating special discourse tokens into the linearized training sequences could significantly improve the generation metrics and is an important step towards coherent generation. the proposed approach performed well even on the automatically annotated dataset, and increasing the weight of discourse tokens in the loss function further accelerated learning convergence. future work includes the analysis of other types of linguistic information (such as syntactic and semantic relations), other ways of integrating them into the training process, as well as experiments for alternative mpc generation pipelines.