C os M o: Conditional S eq2 S eq-based Mixture Model for Zero-Shot Commonsense Question Answering (2020)

Abstract: Commonsense reasoning refers to the ability of evaluating a social situation and acting accordingly. Identification of the implicit causes and effects of a social context is the driving capability which can enable machines to perform commonsense reasoning. The dynamic world of social interactions requires context-dependent on-demand systems to infer such underlying information. However, current approaches in this realm lack the ability to perform commonsense reasoning upon facing an unseen situation, mostly due to incapability of identifying a diverse range of implicit social relations. Hence they fail to estimate the correct reasoning path. In this paper, we present Conditional Seq2Seq-based Mixture model (CosMo), which provides us with the capabilities of dynamic and diverse content generation. We use CosMo to generate context-dependent clauses, which form a dynamic Knowledge Graph (KG) on-the-fly for commonsense reasoning. To show the adaptability of our model to context-dependant knowledge generation, we address the task of zero-shot commonsense question answering. The empirical results indicate an improvement of up to +5.2% over the state-of-the-art models.

## introduction people understand narratives of everyday life by capitalising on their commonsense knowledge. they can easily reason about unobserved causes and effects in relation to the events described in narratives, as well as plausible characteristics and mental states of the involved persons. although this kind of reasoning seems trivial for humans, it is still out of reach for current natural language understanding (nlu) systems. recently, there have been fast-growing interests in building ai systems with such human-like reasoning capabilities based on inferential commonsense knowledge @xcite . such systems are often evaluated by answering questions based on narratives. as illustrated in figure 1 , given a narrative "austin often spends her weekend at the lake fishing with friends" and a question regarding the intention of austin, an ai system is supposed to associate this event to relevant events in an inferential knowledge base or a web-scale corpus, find plausible reasons of those events, and conclude that "wanted to relax" is the most probable answer. knowledge-based approaches to such commonsense question-answering (qa) require an inferential knowledge base. atomic @xcite is the largest commonsense knowledge base of this kind, which contains 300,000 short textual description of events and 877,000 typed if-then relations between events, categorised into 9 dimensions. for instance, if the event "x puts trust in y" occurs and the target relation is "xwant", then "x wants to develop a relationship". prior work utilizes knowledge in atomic by formulating the learning problem as event prediction in if-then relations @xcite . in particular, they encode the textual description of an event and a relation into an embedding, and maximise the probability of predicting the description of the associated event or characteristic of an involved person. however, due to the nature of commonsense knowledge, given an event and a relation, there are multiple plausible associated events. moreover, these models fail to predict all associated events context: austin often spends her weekend at the lake fishing with friends. question: why did austin do this? to a given event and relation, hence fail to identify every implicit reasoning path to address the task of question answering. to address the above challenges, we propose a conditional seq2seq-based mixture model, named cosmo, to answer questions on everyday inferential knowledge in a zero-shot setting. as the events in our target commonsense kgs are described in text, such as atomic, we distil the knowledge into a sequenceto-sequence (seq2seq) model. the distilled model memorises knowledge of the kg and generalise it to new events for a given context and specific relation, on-demand. however, a direct application of maximum-likelihood training on seq2seq models leads to deteriorated performance, because the underlying distribution over diverse outputs is inherently multi-modal @xcite . to address the above challenge, we incorporate a latent variable into a pre-trained seq2seq transformers @xcite . each value of the variable corresponds to an embedding, which indicates the hidden factors explaining different hypothesis. unlike the existing training methods of such models that cannot guarantee a definitive alignment of different hypothesis with different components during training, our proposed constrained-em (expectation-maximisation) training procedure enforces that for the same input, different model outputs align with different latent embedding. to tackle the task of commonsense qa, we use cosmo to generate context-dependent information, for desired relations. the new generated events form a dynamic knowledge graph, which is reasoned over, until the correct answer is chosen by the model. the ubiquitous nature of everyday commonsense knowledge and lack of training data for each situation motivate addressing the task in zero-shot setting. it is unrealistic to expect the presence of manually constructed training datasets in each commonsense qa domain. with the lack of training data in such question-answering settings, we devise a bespoken answer scoring module to assess the likelihood of each answer. given a narrative and a question, the model generates sequences of plausible fact descriptions, as reasoning paths, by choosing different values of the latent variable. for each answer and each reasoning path, the scoring module computes a score based on similarity between the answer and the last fact description in the reasoning path. the most probable answer is determined by its answer score and the probability of the associated reasoning path jointly. to sum up, our contributions are three-folds: commonsense knowledge base completion as an essential way of enabling machines to perform commonsense reasoning, the methods for automatic kg construction and completion have been studied recently. @xcite used lstm as a generator for commonsense knowledge about social situations. @xcite proposed an unsupervised method to extract commonsense knowledge from pretrained models, to complete kg. @xcite developed a model which takes both structural and semantic characteristics of the nodes in a kg to address this task. on the other hand, some works have developed generative models on top of pre-trained language models to extract new commonsense information @xcite . however, when adapting to the kg, the previously acquired knowledge of these models is forgotten. unlike these approaches, our neural model ensures both diversity and memorisation. commonsense question answering recent surge of commonsense question answering dataset @xcite has led to many supervised approaches to address this task. most of these approaches are based on transfer learning, where a large scale pre-trained model @xcite is finetuned on the target task @xcite . on the other hand, with the release of new commonsense kgs, such as atomic @xcite and conceptnet @xcite , the possibility of enriching language models with these kgs has been investigated vastly @xcite @xcite . most of these approaches map the context of a question to an entity/event in kg and perform reasoning on the kg @xcite @xcite . while these methods enable the system to perform multi-hop reasoning, they are limited to the set of entities in kg. other works deployed generative models to generate context-aware information to answer the questions @xcite @xcite . these approaches overcome the limitation of static kgs, but they lack diversity in generating new relations, which makes their inference path limited. furthermore, for scoring the answer choices they have solely relied on the conditional likelihood of the generative model, which lack to perform when the distribution of kg and qa differ. ## our approach in this section, we present our model cosmo for question answering in the zero-shot setting. in this setting, there is no training data for the qa task, thus we train our model only on atomic to acquire inferential knowledge. we apply the trained model to answer multi-choice questions on everyday inferential knowledge by augmenting the trained model with a non-parametric answer scoring module. formally, given a narrative describing a context c, a commonsense-required question q, and a set of m candidate answers @xmath0, ..., a m }, the task is to choose the most plausible answer from the set a. in order to measure the plausibility of answer candidates, the required commonsense inferential knowledge is from a large external knowledge base b, which is a set of typed if-then relations. each if-then relation takes the form of "if z i and r then z j ", where z i denotes a word sequence describing that the event i and r is a relation between events from the pre-defined set r. such a relation is also referred to as inference dimension in atomic. to this end, we define the target task as finding the most plausible answer by applying inferential reasoning over the knowledge base b. the reasoning process is characterised by finding a plausible reasoning path z, which starts from a given context c, along the target relation determined by a question q, to reach an answer a. a reasoning path is a sequence of events correlated by if-then relations derived from b. therefore, for a given context c and a question q, we find the most likely answer by solving the following optimisation problem: in the following, we will detail the kb module as well as its training procedure on atomic, followed by presenting the answer scoring module and how to apply them together for the target task. ## experiments in this section, we report the evaluation of our model on zero-shot question answering. to this end, we use test set and development set of socialiqa @xcite . we evaluate the performance of our model with two variations. in zero-hop setup, we only consider generating clauses using cosmo based on the context of the question, and the answers are then scored against the generated clauses. in one-hop setting, we take a step further and generate more clauses using the generated clauses in the previous step. this approach helps us uncover more implicit context-dependent information. the final answer is chosen against the combination of all generated clauses. to further analyse the capability of our proposed model in terms of clause generation, we compare our model to other approaches on atomic @xcite , and test and development set of socialiqa. ## conclusion in this paper, we propose a novel approach for neuralising large-scale commonsense knowledge graph, conditional seq2seq-based mixture model, cosmo. our proposed model provides diverse clause generation, to ensure coverage for the target task. we use the proposed model to generate diverse context related clauses, alongside with our proposed answer classifier model, to address the task of zero-shot commonsense question answering task. empirical results on zero-shot commonsense question answering dataset show the superiority of our model over the state-of-the-art methods, by up to 5.2%. furthermore, our model outperforms baselines in terms of diversity of clause generations.