G round H og: Dialogue Generation using Multi-Grained Linguistic Input (2024)

Abstract: Recent language models have significantly boosted conversational AI by enabling fast and cost-effective response generation in dialogue systems. However, dialogue systems based on neural generative approaches often lack truthfulness, reliability, and the ability to analyze the dialogue flow needed for smooth and consistent conversations with users. To address these issues, we introduce GroundHog, a modified BART architecture, to capture long multi-grained inputs gathered from various factual and linguistic sources, such as Abstract Meaning Representation, discourse relations, sentiment, and grounding information. For experiments, we present an automatically collected dataset from Reddit that includes multi-party conversations devoted to movies and TV series. The evaluation encompasses both automatic evaluation metrics and human evaluation. The obtained results demonstrate that using several linguistic inputs has the potential to enhance dialogue consistency, meaningfulness, and overall generation quality, even for automatically annotated data. We also provide an analysis that highlights the importance of individual linguistic features in interpreting the observed enhancements.

## introduction text generation methods, particularly for conversational systems, have become increasingly popular in recent years. the conversational systems play a crucial role in enhancing the effectiveness of useragent interactions @xcite @xcite . dialogue systems are used for human-machine conversations on various topics. some systems are built as question-answering systems or personal assistants, focusing on specific domains or general inquiries. despite showing impressive response generation capabilities, language models, even ones like gpt-4, have shortcomings in terms of truthfulness @xcite . consequently, researchers are exploring methods to combine generative and extractive approaches in order to make the responses of dialogue systems more logical and reliable. here, the primary objective is to incorporate external knowledge, resources, or databases into the response generation process. the previous studies have demonstrated a substantial enhancement in the quality of generation by incorporating grounding, which improves the factual accuracy of the responses @xcite . grounding input is commonly integrated into dialogue generation models along with the context of a particular utterance @xcite or a preceding part of the dialogue that represents the conversational history @xcite . furthermore, previous works have explored leveraging grounding in combination with other features, including commonsense and named entities @xcite , dialogue acts @xcite , topic shifts @xcite , discourse annotation @xcite , to improve dialogue generation. despite the fact that additional linguistic features are frequently used to improve the consistency of generated dialogues @xcite , previous studies focused on individual and superficial examination of linguistic features. in our research, we conducted a more comprehensive analysis, evaluating the relative significance of each of them and the overall contribution. we primarily investigate the impact of various linguistic features on response generation in a multi-grained input framework. specifically, we analyze the effects of semantic relations derived from abstract meaning representation (amr) @xcite , dialogue acts extracted from dialogue discourse trees @xcite , and utterance-based sentiment representation. experiments on response generation are generally conducted using open-source sequence-tosequence models @xcite . among these models, the bart architecture @xcite has gained significant popularity due to its state-of-the-art performance in various text generation tasks. due to its efficiency in processing linearized inputs, it is often utilized in graph2text tasks @xcite . moreover, this capability can be further extended to analyze conversation graphs. however, the length of input texts can often present challenges for transformerbased models. in this study, we introduce ground-hog, an approach that uses multiple input encoders to preserve input information effectively. our contributions can be summarized as follows: ## dataset the most popular datasets, including open-domain conversations grounded in wikipedia information, are wizard of wikipedia @xcite and cmu dog @xcite . to narrow the scope of this study and facilitate the language model training, cmu dog was used as a starting point. this dataset contains 4112 grounded conversations devoted to the discussion of wikipedia articles about popular movies. to extend the dataset, we collected reddit foot_0 conversations on the same topic in english. specifically, we parsed conversations from the 25 most popular subreddits related to films, series, and tv shows. these subreddits provided discussions that were tied to specific topics or comments. additionally, we gathered comments that mentioned key phrases such as "movie" and "film". the dataset preprocessing stage involved removing images, extra symbols, and emojis, as these were not considered in our research. in total, our collected dataset consists of approximately 62,500 multi-party dialogues, with an average of 5 turns per conversation (see figure1). the length of extracted reddit conversations is significantly shorter compared to cmu dog dialogues, which have an average of 21.43 turns per conversation. the dataset contains conversations collected along with linguistic annotations, grounding, and meta information related to each extracted dialogue. specifically, automatically retrieved linguistic features for each turn in the dataset are presented in the following format: the process of annotating data is described in detail in section 3.1. our final dataset is publicly available at the link: https://huggingface.co/ datasets/alexchern5757/groundhog_reddit . it should be emphasized that all datasets containing open-domain dialogues share the same limitations related to grounding. casual conversations are distinguished by the absence of rigid topic boundaries, stylistic ambiguity, and a strong reliance on context. evaluative information in these dialogues is often presented as facts, which can result in inaccurate grounding extraction. ## discussion in this section, our major objective is to gain a deeper understanding of the linguistic features that contribute to the improved quality of groundhog. to this end, we conduct a comparative analysis of the texts generated by bart (b) and the texts generated by groundhog (g 5 ). regarding the interpretation of grounding, its incorporation enhances the factual component of generation. however, the qualitative aspects of grounding in our dataset are not very robust, and it can be a direction for further research. sentiment we started our investigation with the analysis of sentiment due to its ease of interpretation. to assess the sentiment in the generated texts, we utilized the same classifier that was applied to the training dataset. the results yielded an overall accuracy of 0.43 for the bart model and 0.44 for the groundhog model, with no sizeable difference observed. it is worth noting that the majority of texts in the dataset were negative or neutral, as users generally tend to criticize films or actors. specifically, there were 1525 negative utterances, 1374 neutral utterances, and 841 positive utterances within the test dataset. consequently, the focus should be shifted to generating more accurate negative responses. in terms of these responses, the base bart model achieved an f1-macro score of 0.461, while the groundhog model achieved an f1-macro score of 0.487. this improvement is particularly noteworthy as it leads to an overall enhancement in language modeling. figure 4 (a) presents an illustrative mpc example. it is observed that all input utterances within the dialogue are negative in nature. consequently, the subsequent utterance should also embody a negative sentiment, either by aligning with the general criticism of the film director or by critiquing the statements expressed by other participants. in this context, it can be inferred that groundhog has produced an appropriate response. conversely, the response generated by the base bart model is positive in sentiment and considered inappropriate. discourse dialogue acts contribute to dialoguebased consistency and, to some extent, utterancebased coherence. since existing models do not explicitly generate dialogue acts, we utilized a trained discourse parser to label these acts for comparison with the original responses. the groundhog model had a higher accuracy score of 0.551 compared to 0.538 for the base model. the confusion matrices showed similar patterns, but there was a slight difference in the distribution of dialogue acts (see in the conversation depicted in figure 4 (b), it can be observed that the custom model response exhibits better consistency. the most correct target response should include the answer or agreement relations rather than elaboration. unlike the bart model, which lacks information about the previous response being a question, the groundhog model incorporates this knowledge in order to generate a response that is discursively consistent. moreover, groundhog aims to incorporate the main amr entities, such as the concept of "controversial." amr interpreting the impact of amr representations is challenging due to their inherent complexity. generally, amr has a direct influence on the semantic aspect, specifically the representation of entities and their relations. in this regard, human evaluation has shown that the scores for the criterion of "meaningfulness" are higher for groundhog texts compared to bart texts. figure 4 (c) provides a concrete example illustrating a discussion where each participant expresses their opinion about some statement. here, both generative models produced thematically correct answers. however, the groundhog model used more appropriate words, resulting in a response that was more consistent with the dialogue history. we hypothesize that this can be attributed primarily to the amr input. for the first utterance, the amr representation is as follows: ## conclusion and future work in this paper, we investigated the efficacy of incorporating grounding and multi-grained linguistic information for multi-party conversation generation. to address the challenge of handling lengthy input texts, we proposed the groundhog model, which leverages both grounding and linguistic features. for evaluation, we collected a novel redditbased dataset designed for training dialogue systems. this dataset was augmented with linguistic features, including semantic and discourse information, as well as sentiment. experiments involving both automatic metrics and human evaluation have shown that generated texts using linguistic inputs were more preferable. in our supplementary analysis, we interpreted the obtained results. further research directions include the investigation of other linguistic inputs as well as other representations of inputs. also, we plan to experiment with the recent llms to analyze their possibilities of leveraging linguistic features.