Syndicom: Improving Conversational Commonsense with Error-Injection and Natural Language Feedback (2023)

Abstract: Commonsense reasoning is a critical aspect of human communication. Despite recent advances in conversational AI driven by large language models, commonsense reasoning remains a challenging task. In this work, we introduce Syndicom - a method for improving commonsense in dialogue response generation. Syndicom consists of two components. The first component is a dataset composed of commonsense dialogues created from a knowledge graph and synthesized into natural language. This dataset includes both valid and invalid responses to dialogue contexts, along with natural language feedback (NLF) for the invalid responses. The second contribution is a two-step procedure: training a model to predict natural language feedback (NLF) for invalid responses, and then training a response generation model conditioned on the predicted NLF, the invalid response, and the dialogue. Syndicom is scalable and does not require reinforcement learning. Empirical results on three tasks are evaluated using a broad range of metrics. Syndicom achieves a relative improvement of 53% over ChatGPT on ROUGE-1, and human evaluators prefer Syndicom over ChatGPT 57% of the time. We will publicly release the code and the full dataset.

## introduction conversational ai has witnessed rapid advancements in recent years, largely due to the success of large language models (llms) such as gpt-3 @xcite . these advancements have been driven by the notable achievements of models like chatgpt, which is built upon instructgpt @xcite . instructgpt was trained on an extensive dataset of instructions for various language tasks and was further enhanced using human feedback and reinforcement learning (rl). consequently, research in conversational ai has shifted towards leveraging large models trained on extensive datasets, supplemented by human feedback. while these models have consistently demonstrated significant improvements in reasoning and problem-solving capabilities, they still exhibit flaws and issues. in many critical applications of llms, the tolerance for errors in dialogue responses is exceedingly low. addressing these problems remains challenging, primarily due to the scarcity of data and the high cost associated with human feedback. recent research has started exploring alternative techniques beyond human feedback and rl, such as natural language feedback (nlf) and self-correction @xcite @xcite . furthermore, even with the progress made, large models often generate hallucinations, underscoring the ongoing importance of knowledge grounding. one of the most demanding aspects of knowledge grounding is commonsense knowledge. recent advancements in incorporating commonsense into llms have utilized resources such as conceptnet @xcite or atomic @xcite . this paper presents a method for improving commonsense dialogue responses by (1) replacing human feedback and rl with natural language responses and (2) leveraging recent knowledge graph techniques to ground responses in commonsense knowledge derived from atomic. to address the scarcity of data and the high cost of human feedback, the natural language feedback is elicited in a manner that specifically targets the chosen error types determined by the designer. this approach significantly enhances the speed and quality of model learning and refinement. the contributions of this paper are as follows: ## recent work the field of conversational ai has experienced a surge of interest in commonsense reasoning in recent years, with a significant focus on curating datasets @xcite . concept-net @xcite and atomic @xcite have emerged as widely used resources for dataset curation, establishing a de facto standard. several datasets serve as sources for the dialogues, including dailydialogue @xcite , mutual @xcite , dream @xcite , and the ubuntu dialogue corpus @xcite . our research lies at the intersection of two crit-ical areas in conversational ai: the synthesis of commonsense datasets and the training of models using natural language feedback. these areas have recently garnered significant research attention due to their potential to enhance the ability of conversational agents to understand and respond to complex human interactions with greater accuracy and consistency. by leveraging the synergies between these domains, our work aims to address the existing limitations in conversational agents and pave the way for more robust and effective conversational systems. ## the syndicom method taking inspiration from recent nlf methods, this paper presents a new approach called syndicom. this new approach combines the synthesis of commonsense dialogue data from a grounded knowledge graph (atomic) with an nlf response improvement approach to improve dialogue responses. figure 1 illustrates the two phase process. ## experiments in this section, we provide a detailed description of the experiments conducted to evaluate our proposed method, syndicom. the experiments aim to compare the direct prediction of the improved response in equation 2 (syndicom-direct) with the response prediction when conditioned on natural language human feedback (nlhf) that explains why the initial response is invalid (syndicom-nlhf). additionally, we explore a multistep implementation of nlhf (syndicom-multistep). we compare the performance of our method against a chatgpt baseline (gpt-3.5-turbo) using various text generation metrics, such as rouge, bleu, sacrebleu, bertscore, and meteor. ## discussion in the discussion section, we analyze the performance of our proposed syndicom method in conversational ai compared to the baseline model chatgpt. the results are summarized in tables 4 and 5 , where we observe that syndicom outperforms chatgpt on all automatic metrics for the feedback and dialogue response improvement tasks. specifically, one contributing factor to the superior performance of the multi-step method is the additional information encoded in the feedback model. the feedback model is trained on human feedback, providing it with more contextual information compared to the direct model, which is solely trained on valid and invalid responses. even in cases where the direct model achieves slightly higher scores in certain metrics, the differences are negligible. notably, bertscore, which represents the most comprehensive model-based metric utilized in our when examining the nlhf columns in overall, syndicom exhibits strong performance compared to the state-of-the-art large language model chatgpt, despite both models being based on the same underlying architecture (gpt-3.5). it is worth noting that chatgpt underwent substantial reinforcement learning through human feedback during its refinement process, making the success of syndicom even more noteworthy. ## conclusion in this paper, we introduced syndicom, a novel method for enhancing commonsense reasoning in dialogue response generation. by integrating a commonsense dialogue synthesis approach with targeted error injection, we tackled the challenge of incorporating commonsense knowledge into conversational ai systems. our method comprised two key components: (1) a dataset consisting of valid and invalid responses to dialogue contexts, along with natural language feedback (nlf) for the invalid responses, and (2) a two-step procedure in-volving training a model to predict nlf for invalid responses, followed by training a response generation model conditioned on the predicted nlf, the invalid response, and the dialogue. a notable advantage of syndicom is its scalability and independence from reinforcement learning techniques, which are commonly employed in previous methods utilizing human feedback. through comprehensive empirical evaluations across three tasks, we demonstrated the effectiveness of our approach using a diverse range of metrics. notably, syndicom outperformed chat-gpt on all metrics for both the dialogue improvement tasks, with and without human feedback. to facilitate further research and practical adoption, we plan to release the code implementation of syndicom as well as the complete dataset utilized in this work. by making these resources openly accessible, we aim to encourage collaboration and promote advancements in commonsense reasoning for dialogue systems.