Enhancing Question Answering by Injecting Ontological Knowledge through Regularization (2020)

Abstract: Deep neural networks have demonstrated high performance on many natural language processing (NLP) tasks that can be answered directly from text, and have struggled to solve NLP tasks requiring external (e.g., world) knowledge. In this paper, we present OSCR (Ontology-based Semantic Composition Regularization), a method for injecting task-agnostic knowledge from an Ontology or knowledge graph into a neural network during pre-training. We evaluated the performance of BERT pre-trained on Wikipedia with and without OSCR by measuring the performance when fine-tuning on two question answering tasks involving world knowledge and causal reasoning and one requiring domain (healthcare) knowledge and obtained 33.3%, 18.6%, and 4% improved accuracy compared to pre-training BERT without OSCR.

## the problem as shown in figure 1 , suppose we were interested in determining whether, upon showing the policeman his badge, it is more plausible that the detective would be let into the crime scene or that the police officer would confiscate the detective's badge? to answer this question, we would need premise: the detective flashed his badge to the police officer. question: what is the most likely effect? a: the police officer confiscated the detective's badge. ## background and related work the idea of training a model on a related problem before training on the problem of interest has been shown to be effective for many natural language processing tasks @xcite @xcite . more recent uses of pre-training adapt transfer learning by first training a network on a language modeling task and then fine-tuning (retraining) that model for a supervised problem of interest @xcite @xcite . pre-training, in this way, has the advantage that the model can build on previous parameters to reduce the amount of information it needs to learn for a specific downstream task. conceptually, the model can be viewed as applying what it has already learned from the language model task when learning the downstream task. bert (bidirectional encoder representations from transformers) is a pre-trained neural network that has been shown to obtain state-of-the-art results on eleven natural language processing tasks after fine-tuning @xcite . bert relies on two pre-training objectives: (1) a variant of language modeling called cloze (originally proposed in @xcite where-in 20 % of the words in a sentence are masked, and the model must unmask them and (2) a next sentence prediction task where-in the model is given two pairs of sentences and must decide if the second sentence immediately follows the first. despite its strong empirical performance, the architecture of bert is relatively simple: four layers of transformers @xcite are stacked to process each sentence. in terms of injecting knowledge into pre-training, @xcite explored injecting entity information into bert using multi-head attention. however, their approach requires explicitly indicating entity boundaries or relation constituents with special input tokens for down-stream fine-tuning. by contrast, oscr requires no modification of input formats in the host network. @xcite explored modifying bert's pre-training by masking entire entities and phrases extracted from external knowledge. meanwhile, @xcite explored projecting propositional knowledge using graph convolutional networks (gcns). oscr, instead, introduces a regularization term that can be added to any natural language pre-training objectives, without modifying the architecture of the network or the pre-training objectives themselves. ## the data incorporating oscr into pre-training requires an embedded ontology (or knowledge) graph, and one or more natural language pre-training objectives to regularize -in our case, bert's cloze and nextsentence prediction tasks. these objectives, in turn, require a document collection. ## the approach virtually all neural networks designed for natural language processing represent language as a sequence of words, subwords, or characters. by contrast, ontologies and knowledge bases encode semantic information about entities, which may correspond to individual nouns (e.g., "badge") or multiword phrases ("police officer"). consequently, injecting world and domain knowledge from a knowledge base into the network requires semantically decomposing the information about an entity into the supporting information about its constituent words. for example, injecting the semantics of "spanish civil war" into the network requires learning what information the word "spanish" introduces to the nominal "civil war" and what information "civil" adds to the word "war". to do this, oscr is implemented using a three-step approach illustrated in figure 2 entity encoding and the pre-trained entity encoding from the ontology is used as a regularization term in the pre-training loss function. by training the model to compose sequences of subwords into entities, during back-propagation, the semantics of each entity are decomposed and injected into the network based on the neural activations associated with its constituent words. ## conclusions in this paper we presented oscr (ontology-based semantic composition regularization), a learned regularization method for injecting task-agnostic knowledge from an ontology or knowledge graph into a neural network during pretraining. we evaluated the impact of including oscr when pretraining bert with wikipedia articles by measuring the performance when fine-tuning on two question answering tasks involving world knowledge and causal reasoning and one requiring domain (healthcare) knowledge and obtained 33.3 %, 18.6 %, and 4 % improved accuracy compared to pre-training bert without oscr.