kogito: A Commonsense Knowledge Inference Toolkit (2023)

Abstract: In this paper, we present kogito, an open-source tool for generating commonsense inferences about situations described in text. kogito provides an intuitive and extensible interface to interact with natural language generation models that can be used for hypothesizing commonsense knowledge inference from a textual input. In particular, kogito offers several features for targeted, multi-granularity knowledge generation. These include a standardized API for training and evaluating knowledge models, and generating and filtering inferences from them. We also include helper functions for converting natural language texts into a format ingestible by knowledge models — intermediate pipeline stages such as knowledge head extraction from text, heuristic and model-based knowledge head-relation matching, and an ability to define and use custom knowledge relations. We make the code for kogito available at https://github.com/epfl-nlp/kogito along with thorough documentation at https://kogito.readthedocs.io .

## introduction in recent years, large-scale language models @xcite @xcite trained on massive amounts of text have been conceptualized as implicit knowledge bases that encode knowledge about the world @xcite . as they are trained to receive natural language inputs, these models can be prompted to generate text that expresses a fact. leveraging this property, knowlege models train on knowledge graph tuples (triplets of head entity, relation, tail entity) and learn to express knowledge encoded in the parameters of language models when provided with a head entity and relation @xcite @xcite . the success of these knowledge models has inspired the field to deploy them in various down-stream use-cases such as generating figurative language @xcite , producing sarcastic responses @xcite , designing plots for stories @xcite ) and text-based games @xcite , and developing persona-grounded dialogue agents @xcite . given the prevalence of applications that benefit from augmenting nlp systems with commonsense inferences, we present a novel commonsense knowledge inference toolkit, kogito, that standardizes commonsense inference generation from knowledge models. to the best of our knowledge, kogito is the first library that facilitates access to knowledge models through an easy-to-use, customizable interface. in particular, we make the following contributions: ## challenges of commonsense inference while many works use knowledge models as commonsense inference engines to augment natural language inputs, no work has formalized the pipeline for producing such inferences. here, we outline the challenges of effectively setting up this pipeline. ## data representation to allow for standardization and ease of maintenance, kogito defines an interface to represent core concepts such as a knowledge tuple, a commonsense knowledge graph, and a knowledge model. commonsense knowledge tuple commonsense knowledge graphs @xcite and knowledge models @xcite typically represent instances of knowledge as tuples of 3 elements: (head, relation, tail). the head entity refers to the subject of a piece of knowledge (e.g., objects such as hammer; events such as "personx becomes a great basketball player"). a relation provides an implicit question about the head (e.g., @xmath0; @xmath1. finally, tail entities provide an answer option (among potentially many) to the relation with respect to the head (e.g., put nail in wood; to practice hard). we often refer to the tail as the commonsense inference about the head. following this convention, we define a class with these elements and an additional two classes for knowledge head and relation representation. while knowledge heads and tails can be arbitrary text, we use predefined relations from the atomic2020 @xcite and conceptnet @xcite knowledge graphs. 5 below is an example of defining a knowledge tuple in kogito: from kogito.core.head import knowledgehead from kogito.core.knowledge import knowledge from kogito.core.relation import x_need @xmath2, @xmath3, @xmath4 knowledge graph in addition to individual knowledge tuples, we also define a knowledge 5 kogito also supports defining new custom relations and using them to generate commonsense inferences ( §8) graph as a collection of knowledge tuples. in kogito, a knowledge graph serves as the standardized input object to (and output from) knowledge models, and has a simple api to manipulate knowledge instances collectively. in particular, a knowledge graph can be used to easily iterate over, read, and write a collection of knowledge instances to and from various files, and perform set-like operations on multiple knowledge graphs. these operations require a notion of equality, so we define two knowledge instances to be equal if they have the same head, relation and tail. below is an example of defining and manipulating knowledge graphs: # read from jsonl (list of json objects) @xmath5.from_jsonl("sample_graph2.jsonl", @xmath6, @xmath7, @xmath8 knowledge model knowledge models conceptually accept as input a (head, relation) pair and output an inferred knowledge tail. however, these models can sometimes expect a subtly different formats for these inputs and outputs. to increase the extensibility and interoperability of our tool, so that users can easily substitute one knowledge model for another, we define a model-agnostic abstraction over possible types of knowledge models. consequently, knowledge models inherit from an abstract interface that defines core abstract methods, which users can implement to port new knowledge models into kogito. the knowledge model interface provides methods to train, generate from, evaluate these models, as well as save and load them. the input dataset for training, generating, or evaluating is given as a knowledge graph and the output dataset (in the case of generation) is returned as a knowledge graph. kogito currently offers the following knowledge models: comet-bart and comet-gpt2 from @xcite , gpt2-zeroshot (radford et al., 2019), and gpt3-zeroshot @xcite . pre-trained comet models can be loaded either from huggingface foot_4 by name or from local disk by model path. the gpt-3 model requires an api key. each model method supports customization of various model-specific hyperparameters. kogito currently evaluates models using the following metrics: bleu @xcite , rouge @xcite , meteor @xcite , cider @xcite and bertscore @xcite . pipeline design in the following sections, we discuss the head extraction, relation matching, and inference filtering components of this pipeline. we note that we provide a "dry-run" mode which allows for faster iteration on head extraction and relation matching by skipping the inference generation portion of kogito's pipeline. more details on these configuration options can be found in the kogito docs. ## head extraction head extraction refers to finding relevant chunks of a text in a sequence that can serve as knowledge heads (i.e., the concepts commonsense inferences should be generated about). for example, given a text input "personx becomes a great basketball player", we might be interested in generating inferences for the full sentence, but also about entities such as "basketball player", "basketball", or potentially "become player". for different applications, different sets of head entities might be appropriate for generating inferences. consequently, kogito allows the user to customize this behaviour and define arbitrary head extraction methods. 7 at the same time, by default, kogito comes with a few standard head extraction methods. these built-in methods segment sentences, and then extract noun phrases (np) and verb phrases (vp) using dependency parses produced from spacy. 8 extracted heads are deduplicated using string matching and passed onto the next stage of the pipeline, relation matching. we note that the head extraction stage itself is optional and the user can also provide a dedicated list of heads to kogito, which would replace the pre-processing of head entities. ## relation matching not all relations that a knowledge model is trained with will be relevant to each extracted head. for example, a head entity, "hammer", would ideally be match to a relation such as atlocation, while a relation such as xwants (i.e., what does this head entity want) would not be matched. similarly, "personx becomes a basketball player" might be matched to a relation such as xintent (i.e., what is the intent of the main persona in the head entity), while a relation such as usedfor (i.e., what is the head entity used for) would yield an incoherent inference. in this next stage, kogito matches relations to the given head input so that the resulting (head, relation) pair constitutes a sensible and plausible prompt for the knowledge model. base matcher every relation defined for a knowledge graph is matched to the head entities. this matcher is particularly useful if the user predefines a set of acceptable known relations or if they define new relations for their use case ( §8). ## inference filtering by default, the commonsense inference module returns all generated tails without any filtering applied. however, many of these resulting inferences may be irrelevant to the initial context, particularly for extracted heads that have been decontextualized. given most users may only be interested in relevant subsets of these commonsense inferences, kogito provides a separate module to determine the relevance of the given knowledge tuples with respect to the initial context from which it was extracted. in our running example, "per-sonx becomes a great basketball player", an extracted head entity "player" may yield contextuallyirrelevant inferences such as "player plays video games" and "player is at a soccer match", which to evaluate how well our method transfers to this new setting, we perform an expert study on the performance of the inference filtering model with respect to the knowledge generated from a knowledge model such as comet. we randomly select 50 instances from the test split of roc-atomic dataset @xcite where each instance is composed of a context and a fact as a knowledge tuple @xcite . we then run the default kogito inference pipeline (with full head extraction and heuristic relation matching) on the heads which produces several inferences per head instance. we select 100 results randomly from the output of the previous step and apply our inference filtering model. finally, we ask a human expert to annotate each instance with the true relevance label of the fact and find that our model achieves a 75% f1 on the knowledge model generated inferences. we also offer a modular interface to define and plug in new filtering models in the future. ## defining new relations in previous knowledge modeling papers @xcite , the set of rela-tions that can be used in prompts is limited by the knowledge graph used to to train the knowledge model (e.g., atomic2020). however, a user may want to generate inferences for new dimensions of knowledge, define their own custom relations for them, and produce commonsense inferences based on these new properties. however, if there are no large kgs that use this schema, training a suitable knowledge model would pose a challenge. kogito provides this functionality by implementing the approach of @xcite , which allows a user to prompt large language models for knowledge using custom relations and has been shown to generate high-quality knowledge. specifically, a user defines an instance of a knowledge relation class, a verbalizer function that describes how to convert the new relation into a natural language prompt (with a head and tail), and an instruction prompt to gpt-3. at inference time, the user provides a list of sample knowledge tuples that use the new relation. these tuples are verbalized using the verbalizer function and provided to the gpt-3 model along with the instruction prompt. below, we illustrate this process with an example where a new relation, xwishes, which describes person's wishes, is defined using the sample code: def x_wishes_verbalizer(head, ** kwargs): # index will be passed from the model # so that we can enumerate samples # which helps with inference @xmath9.get("index") @xmath10: {head}." then, to use this new relation for inference, the user can provide a sample knowledge graph (i.e., a prompt filled with example tuples using this relation), and a head such as "personx makes a huge mistake" to generate inferences about. below, we show how such a sample knowledge graph could be verbalized into a prompt for gpt-3: how does the situation affect the character's wishes? situation 1: john is at a party. as a result, john wishes to drink beer and dance situation 2: terry bleeds a lot. as a result, terry wishes to see a doctor situation 3: eileen works as a cashier. as a result, eileen wishes to be a store manager situation 4: james gets dirty. as a result, james wishes to clean up situation 5: janice stays up all night studying. as a result, janice wishes to sleep all day situation 6: isaac makes a huge mistake. as a result, isaac wishes... the result of prompting gpt-3 with the above text is returned as the generated tail inference for the given head. using this approach, users can instantiate a prompt defining a new relation, and use large language models to produce inferences for it. ## conclusion & future work in this system description, we presented kogito, a toolkit for generating commonsense inferences for open-world text using knowledge models. kogito provides a foundational, customizable, and extensible interface for inference generation from knowledge models, and supports preprocessing and manipulation utilities such as head extraction, relation matching, and relation definition. future work may include improved head extraction, such as semantic head extraction (e.g., paraphrased noun phrase extraction, etc.), new relation matching methods that more rigorously trade off performance and latency, support for new knowledge models trained on other knowledge graphs (e.g., @xcite , and multimodal inputs such as images. imaging, sony group corporation, and the allen institute for ai.