Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering (2020)

Abstract: The aim of all Question Answering (QA) systems is to generalize to unseen questions. Current supervised methods are reliant on expensive data annotation. Moreover, such annotations can introduce unintended annotator bias, making systems focus more on the bias than the actual task. This work proposes Knowledge Triplet Learning (KTL), a self-supervised task over knowledge graphs. We propose heuristics to create synthetic graphs for commonsense and scientific knowledge. We propose using KTL to perform zero-shot question answering, and our experiments show considerable improvements over large pre-trained transformer language models.

## introduction the ability to understand natural language and answer questions is one of the core focuses in the field of natural language processing. to measure and study the different aspects of question answering, several datasets are developed, such as squad @xcite , hotpotqa @xcite , and natural questions @xcite which require systems to perform extractive question answering. on the other hand, datasets such as socialiqa @xcite , common-senseqa @xcite , swag @xcite and winogrande @xcite require systems to choose the correct answer from a given set. these multiple-choice question answering datasets are very challenging, but recent large pre-trained language models such as bert @xcite , xlnet @xcite and roberta @xcite have shown very strong performance on them. moreover, as shown in winogrande @xcite , acquiring unbiased labels requires a "carefully designed crowdsourcing procedure", which adds to the cost of data annotation. this is also quantified in other natural language tasks such as natural language inference @xcite and argument reasoning comprehension (niven and kao, 2019), where such annotation artifacts lead to "clever hans effect" in the models @xcite . one way to resolve this is to design and create datasets in a clever way, such as in winogrande @xcite , another way is to ignore the data annotations and to build systems to perform unsupervised question answering @xcite . in this paper, we focus on building unsupervised zero-shot multiple-choice qa systems. recent work @xcite try to generate a synthetic dataset using a text corpus such as wikipedia, to solve extractive qa. other works @xcite use large pre-trained generative language models such as gpt-2 @xcite to generate knowledge, questions, and answers and compare against the given answer choices. in this work, we utilize the information present in knowledge graphs such as atomic @xcite . we define a new task of knowledge triplet learning (ktl) over these knowledge graphs. for tasks which do not have appropriate knowledge graphs, we propose heuristics to create synthetic knowledge graphs. knowledge triplet learning is like knowledge representation learning and knowledge graph completion but not limited to it. knowledge representation learning @xcite learns the low-dimensional projected and distributed representations of entities and relations defined in a knowledge graph. knowledge graph completion @xcite aims to identify new relations and entities to expand an incomplete input knowledge graph. in ktl, as shown in figure 1 , we define a triplet (h, r, t), and given any two as input, we learn to generate the third. this tri-directional reasoning forces the system to learn all the possible relations between the three inputs. we map the question answering task to ktl, by mapping the context, question and answer to (h, r, t) respectively. we define two different ways to perform self-supervised ktl. this task can be designed as a representation generation task or a masked language modeling task. we compare both the strategies in this work. we show how to use models trained on this task to perform zero-shot question answering without any additional supervision. we also show how models pre-trained on this task perform considerably well compared to strong pre-trained language models on few-shot learning. we evaluate our approach on the three commonsense and three science multiplechoice qa datasets. the contributions of this paper are summarized as follows: ## knowledge triplet learning we define the task of knowledge triplet learning (ktl) in this section. we define @xmath0, e) as a knowledge graph, where v is the set of vertices, e is the set of edges. v consists of entities which can be phrases or named-entities depending on the given input knowledge graph. let s be a set of fact triples, s ⊆ v ×e×v with the format (h, r, t), we define the task of ktl as follows: given input a triple (h, r, t), we learn the following three functions. that is, each function learns to generate one component of the triple given the other two. the intuition behind learning these three functions is as follows. let us take the above example: (personx puts per-sonx's trust in persony, how is personx seen as?, faithful). the first function f t (h, r) learns to generate the answer t given the context and the question. the second function f h (r, t) learns to generate one context where the question and the answer may be valid. the final function f r (h, t) is a jeopardystyle generating the question which connects the context and the answer. in multiple-choice qa, given the context, two choices may be true for two different questions. similarly, given the question, two answer choices may be true for two different contexts. for example, given the context: personx puts personx's trust in persony, the answers personx is considered trustworthy by others and personx is polite are true for two different questions how does this affect others? and how is personx seen as?. learning these three functions enables us to score these relations between the context, question, and answers. ## synthetic graph construction this section describes our method to create a synthetic knowledge graph from a text corpus containing sentences. not all types of knowledge are present in a structured knowledge graph, such as atomic, which might help answer questions. for example, the questions in qasc dataset @xcite ) require knowledge about scientific concepts, such as, "clouds regulate the global engine of atmosphere and ocean.". the qasc dataset contains a textual knowledge corpus containing science facts. similarly, the open mind commonsense (omcs) knowledge corpus contains knowledge about different commonsense facts, such as, "you are likely to find a jellyfish in a book". another kind of knowledge about social interactions and story progression is present in several story understanding datasets, such as rocstories and the story cloze test @xcite . to perform question answering using this knowledge and ktl, we create the following two graphs: the common concept graph and the directed story graph. common concept graph to create the common concept graph, we extract noun-chunks and verb-chunks from each of the sentences using the spacy part-of-speech tagger @xcite . we assign all the extracted chunks as the graph's vertices and the sentences as the graph's edges. to generate training samples for ktl, we assign triples (h, r, t) as (e 1 , e 2 , v i ) where v i is the common concept present in both the sentences e 1 and e 2 . for example, in the sentence clouds regulate the global engine of atmosphere and ocean., the extracted concepts are clouds, global engine, atmosphere, ocean and regulate. the triplet assignment will be, [warm moist air from the pacific ocean brings fog and low stratus clouds to the maritime zone., clouds regulate the global engine of atmosphere and ocean., clouds]. we create two such synthetic graphs using the qasc science corpus and the omcs concept corpus. our hypothesis is this graph, and the ktl framework will allow the model to understand the concepts common in two facts, which allows question answering. directed story graph this graph is created using short stories from the rocstories and story cloze test datasets. this graph is different from the above graph as this graph has a directional property, and each story graph is disconnected. to create this graph, we take each short story with k sentences, [s 1 , s 2 , s 3 .., s k ] and create a directed graph such that all sentences are vertices and each sentence is connected with a directed edge only to sentences that occur after it. for example, s 1 is connected to s 2 with a directed edge but not vice versa. we generate triples (h, r, t) by sampling vertices (s i , s j , s k ) such that there is a directed path between the sentences s i and s k through s j . this format captures a smaller story where the head is an event that occurs before the relation and the tail. this graph is designed for story understanding and abductive reasoning using the ktl framework. random sampling there are around 17m sentences in the qasc text corpus; similarly, there are 640k sentences in the omcs text corpus. our synthetic triple generation leads to a significantly large set of triples in order of 10 12 and more. to restrict the train dataset size for our ktl framework, we randomly sample triples and limit the train dataset size to be at max 1m samples; we refer to this as random sampling. curriculum filtering here, we extract the noun and verb chunks from the context, question, and answer options present in the question answering datasets. we filter triples from the generated dataset and keep only those triples where at least one of the entities is present in the extracted noun and verb chunks set. this filtering is analogous to a reallife human examination setting where a teacher provides the set of concepts upon which questions would be asked, and the students can learn the concepts. we perform the sampling and filtering only on the huge common concept graphs generated from qasc and omcs corpus. ## datasets we evaluate our framework on the following six datasets: socialiqa @xcite , anli @xcite , commonsenseqa @xcite , qasc @xcite , open-bookqa @xcite and arc @xcite . socialiqa, anli, and common-senseqa require commonsense reasoning and external knowledge to answer the questions. similarly, qasc, openbookqa, and arc require scientific knowledge. ## conclusion this work proposes a new framework of knowledge triplet learning over knowledge graph entities and relations. we show learning all three possible functions, f r , f h , and f t help the model perform zero-shot multiple-choice question answering, where we do not use question-answering annotations. we learn from both human-annotated and synthetic knowledge graphs and evaluate our framework on the six question-answering datasets. our framework achieves state-of-the-art in the zero-shot question answering task achieving performance like prior supervised work and sets a strong baseline in the few-shot question answering task.