C o S e-Co: Text Conditioned Generative C ommon S ense Contextualizer (2022)

Abstract: Pre-trained Language Models (PTLMs) have been shown to perform well on natural language tasks. Many prior works have leveraged structured commonsense present in the form of entities linked through labeled relations in Knowledge Graphs (KGs) to assist PTLMs. Retrieval approaches use KG as a separate static module which limits coverage since KGs contain finite knowledge. Generative methods train PTLMs on KG triples to improve the scale at which knowledge can be obtained. However, training on symbolic KG entities limits their applicability in tasks involving natural language text where they ignore overall context. To mitigate this, we propose a CommonSense Contextualizer (CoSe-Co) conditioned on sentences as input to make it generically usable in tasks for generating knowledge relevant to the overall context of input text. To train CoSe-Co, we propose a novel dataset comprising of sentence and commonsense knowledge pairs. The knowledge inferred by CoSe-Co is diverse and contain novel entities not present in the underlying KG. We augment generated knowledge in Multi-Choice QA and Open-ended CommonSense Reasoning tasks leading to improvements over current best methods on CSQA, ARC, QASC and OBQA datasets. We also demonstrate its applicability in improving performance of a baseline model for paraphrase generation task.

## introduction while dealing with natural language text, commonsense allows humans to expand salient concepts and infer additional information. for example, by reading a sign like men at work on a road, we implicitly know to slow down our vehicles, look carefully for workers. this implicit process of using common sense to make logical inferences is critical to natural language understanding @xcite . a natural question to ask then is how we can incorporate common sense in now-ubiquitous language models (lms) @xcite @xcite . there have been various efforts @xcite @xcite to leverage structured knowledge present in commonsense knowledge graphs -kgs (we use kg as a shorthand for commonsense knowledge graph) @xcite . such works have primarily focused on either retrieving or generating required knowledge. retrieval methods rely heavily on structure of downstream task like multi-choice question answering (qa) to leverage knowledge in a kg @xcite and are not applicable beyond a specific task. further, retrieval can restrict total knowledge that can be garnered since static kgs lack coverage due to sparsity @xcite . the other body of work addresses this comprising of generative methods that learn commonsense through training a lm on symbolic entities and relations between them in a kg. they have either been designed for kg completion @xcite , i.e. generate tail entity of a kg triple given head entity and relation, or to generate commonsense paths connecting a pair of entities which suffer from two shortcomings. firstly, applying such methods in downstream tasks require entity extraction from text as a prerequisite step and secondly, they generate knowledge between entity pairs ignoring overall context of sentence @xcite . hence, applying such methods is sub-optimal since most nlp tasks comprise of sentences. further, being trained on entities, applying them directly on sentences is infeasible and lead to train-inference input type mismatch. to address these limitations, we propose commonsense contextualizer -cose-co, a generative framework which generates relevant commonsense knowledge given natural language sentence as input. we condition it on sentences to make it learn to incorporate overall text context and enable it to dynamically select entities/phrases from an input sentence as well as output novel yet relevant entities as part of commonsense inferences generated. we consider commonsense knowledge in the form of paths, i.e., sequence of entities connected through relations. we first create sentencepath paired dataset by -1) sampling paths from an underlying kg; 2) sampling a subset of entities from a path; and 3) retrieving & filtering sentences (from a sentence corpus) that are semantically similar to the path. the paired data is then used to train a generative language model to generate a path given a sentence as input. to analyse the usefulness of generated commonsense, we augment it in various downstream tasks. the reasoning ability of nlp systems is commonly analysed using qa. hence, we choose two such tasks: 1) multi-choice qa, where given a question and set of choices, the model has to identify the most appropriate answer choice. however, often more than one choice is a suitable answer. to mitigate this, 2) opencsr (open-ended common-sense reasoning) @xcite was proposed, where each question is labeled with a set of answers which have to be generated without choices. we also show applicability of cose-co in improving performance on paraphrase generation task ( §4.5). our contributions can be summarised as: ## related work commonsense knowledge graphs (kgs) are structured knowledge sources comprising of entity nodes in the form of symbolic natural language phrases connected through relations @xcite @xcite . the knowledge in kgs is leveraged to provide additional context in nlp tasks @xcite @xcite and perform explainable structured reasoning @xcite . additionally, a variety of natural language inference (nli) and generation tasks requiring commonsense reasoning have been proposed over the years @xcite @xcite @xcite . pre-trained language models (ptlms) @xcite trained over large text corpus have been shown to posses textual knowledge @xcite @xcite and semantic understanding @xcite . consequently, they have been used for reasoning where they perform well to some extent @xcite . however, it remains unclear whether this performance can be genuinely attributed to reasoning capability or if it is due to unknown data correlation @xcite @xcite . due to this, various lm + kg systems have been explored @xcite @xcite to combine broad textual coverage of lms with kg's structured reasoning capability. early works on kg guided qa retrieve sub-graph relevant to question entities but suffer noise due to irrelevant nodes @xcite . hybrid graph network based methods generate missing edges in the retrieved sub-graph while filtering out irrelevant edges @xcite . graph neural networks (gnns) have been used to model embeddings of kg nodes @xcite . more recently, @xcite proposed an improved framework (qa-gnn) leveraging a static kg by unifying gnn based kg entity embeddings with lm based qa representations. although, such frameworks extract relevant evidence from a kg, it undesirably restricts knowledge that can be garnered since knowledge source is static and might lack coverage due to sparsity @xcite . contrarily, we train a generative model on a given kg to enable it to dynamically generate relevant commonsense inferences making it more generalizable and scalable. bosselut et al. (2019) cast commonsense acquisition by lms as kg completion. they propose comet, a gpt @xcite based framework to generate tail entity given head and relation in a kg triple as input. owing to training on symbolic kg nodes, using comet in downstream tasks involving natural language text is not straightforward. specifically, it requires extracting entities from text as a prerequisite @xcite . further, training on single triples makes its application in tasks requiring multi-hop reasoning challenging due to large relation search space @xcite . to address this, path generator (pgqa) was proposed to generate commonsense paths between entities pair @xcite . designed for multi-choice qa, they extract question entities and generate paths between each question entity and answer choice pair. even though generated paths are multi-hop, training on entities limits applying it directly on sentences due to train-inference input type mismatch. further, being conditioned only on question-choice entity pairs, paths are generated ignoring overall question context. to mitigate this, we design cose-co as a generic framework to dynamically generate multi-hop commonsense inference given natural language sentence as input. separately, retrieval methods have been explored to search relevant sentences to generate text corresponding to concepts @xcite . different from this task, we retrieve sentences relevant to paths in a kg to create paired sentence-path data. ## proposed cose-co framework problem setting given a commonsense knowledge graph @xmath0, r), where e is the set of entity nodes and r is the set of labeled directed rela-tional edges between entities, we aim to model a commonsense contextualizer (cose-co) which generates a set of commonsense inferences in the form of paths derived using g, that are relevant to a natural language text given as input. it is desirable that such a generative commonsense knowledge model should be generic, task agnostic, and takes into account the overall context of language input while generating commonsense. since most tasks comprise of text in the form of sentences, we model the input to cose-co as a sentence. in order to train such a model, a dataset is required which comprises of mappings of the form {(s 1 , p 1 ), (s 2 , p 2 ), ..., (s n , p n )}, where s j and p j are relevant sentence-commonsense inference path pair. however, no existing dataset consists of such mappings. to bridge this gap, we first devise a methodology to create a dataset d comprising of sentences paired with relevant commonsense inference paths. broadly, we first extract a large corpus c constituting sentences {s 1 , s 2 , ..., s |c| }. subsequently, we sample a set of paths @xmath1, p 2 , ..., p |p| } from g such that each @xmath2, r 1 , e 2 , r 2 , ..., e |p|+1 }, where e @xmath3. for each @xmath4, a set of contextually and semantically relevant sentences s ⊂ c is retrieved and mapped to p. we then train a generative lm based commonsense knowledge model using d. during inference, given a sentence s ′ , it generates commonsense paths of the form ## conclusion we presented cose-co, a framework to generate commonsense inferences that are relevant to the overall context of a given natural language text. we created a novel dataset @xmath5, commonsense @xmath6. empirical evaluation shows that commonsense inferences generated by cose-co are relevant, diverse, and also contain novel entities not present in kg. we augment knowledge generated by cose-co in commonsense tasks such as multi-choice qa and open-ended commonsense reasoning, achieving sota results for these tasks. further, we also used cose-co for nlp tasks such as paraphrase generation achieving improved performance. while, using conceptnet as our base kg allowed us to perform an exhaustive fair comparison with a variety of benchmark methods-where the motivation is to provide more relevant knowledge (in symbolic form as in kg) to tasks-cose-co can further be enhanced by utilizing other commonsense kgs. our work can be extended to explore better ways of integrating the generated knowledge generically across a variety of kgs and lms, and is a potential direction for future work.