Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (2020)

Abstract: In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition, we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective that is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmarks, including question answering and knowledge base completion.

## introduction self-supervised pre-trained language models (lms) like elmo @xcite and bert @xcite learn powerful contextualized representations. with task-specific modules and finetuning, they have achieved state-of-the-art results on a wide range of natural language processing tasks. nevertheless, open questions remain about what these models have learned and improvements can be made along several directions. one such direction is, when downstream task performance depends on relational knowledge -the kind modeled by knowledge graphs foot_0 (kgs) -directly finetuning a pre-trained lm often yields sub-optimal results, even though some works @xcite show pre-trained lms have been partially equipped with such knowledge. to address this shortcoming, several recent works attempt to integrate kgs into pre-trained lms. these approaches can be coarsely categorized into two classes, as shown in figure 1 . the first line of methods retrieves a kg subgraph @xcite @xcite and/or pre-trained graph embeddings @xcite via entity linking during both training and inference on downstream tasks. while these methods inject domain-specific knowledge directly into language representations, they rely heavily on the performance of the linking algorithm and/or the quality of graph embeddings. graph embeddings, to be tractable over large-scale kgs, are often learned using shallow models (e.g., transe @xcite , tucker @xcite and conve @xcite ) with limited expressive power. besides, the linking and retrieval invoked during both finetuning and inference are costly, hence limiting these methods' practicality. the second class of methods @xcite @xcite uses contextualized representations from pre-trained lms to enrich graph embeddings and thus alleviates graph sparsity issues. this is especially helpful in the case of commonsense kgs (e.g., conceptnet @xcite ) that consist of non-canonicalized text and hence suffer from severe sparsity @xcite . specifically, these methods usually feed concatenated triples (e.g., [head, relation, tail]) into lms for training or finetuning. the drawback is that focusing on knowledge base completion tends to over-adapt the models to this specific task, which comes at the cost of generalization to text-based tasks, e.g., qa. in this work, we equip masked language models (mlms), e.g., bert @xcite , with structured knowledge via self-supervised pretraining on raw text. compared to the first class, we expose lms to structured information only during pre-training, thus circumventing costly knowledge retrieval and integration in both finetuning and inference. also the dependency on the performance of linking algorithm is greatly reduced. compared to the second class, we learn from free-form text through mlms rather than triples, which fosters generalization on other downstream tasks. specifically, given a corpus of raw text and a kg, two kg-guided self-supervision tasks are formulated to inject structured knowledge into mlms. first, taking inspiration from baidu-ernie @xcite , we reformulate the masked language modeling objective to an entity-level masking strategy, where entities are identified by linking their text mentions to either concepts in a commonsense kg or named entities in an ontological kg @xcite . the role of kg here is to provide a "vocabulary" of entities to be masked. to further exploit implicit relational information underlying raw text, we design a kg-guided masking scheme that selects informative entities by considering both document frequency and mutual reachability of the entities detected in the text. in addition to the new entity-level mlm task above, a novel distractor-suppressed ranking task is proposed. negative entity samples are derived from the kg and used as distractors for the masked entities to make the learning more effective. note that our approach never observes the kg directly, through triples or other forms. rather, the kg plays a guiding role in the proposed selfsupervised tasks. its guidance helps the model exploit the corpus more effectively as verified in the experiments. if a downstream task can benefit from explicit exposure to kg, a method by @xcite can be used to transform kg triples into natural grammatical texts for our model. we evaluate our method on five benchmarks (including question answering and knowledge base completion) and one zero-shot testing. results show our method achieves state-of-the-art or competitive performance on all benchmarks. ## vanilla masked language model to ground our approach, this section summarizes mlms for pre-training bidirectional transformers @xcite . compared to causal lms @xcite trained unidirectionally, mlms randomly mask some tokens and predict the masked tokens by considering their context on both sides. formally, given a piece of text u , a tokenizer, e.g., bpe @xcite , is used to produce a sequence of tokens [w 1 , . . . , w n ]. a certain percentage of the original tokens are then masked and replaced: of those, 80% with the special token [mask], 10% with a token sampled from the vocabulary v, and the remaining kept unchanged. the masked sequence, denoted as [w (m) 1 , . . . , w (m) n ], is passed into a transformer encoder to produce contextualized representations for the sequence: ## proposed method this section begins with a description of entitylevel masked language modeling. section 3.2 proposes a kg-guided entity masking scheme for the entity-level mlm task. a novel distractorsuppressed ranking task is presented in section 3.3, which operates on the masked entities and their negative entity samples from the kg. we use a multitask learning objective combining the two tasks above to jointly train our proposed graph-guided masked language model (glm). an illustration of the glm is shown in figure 2 . ## experiments in this section we demonstrate the effectiveness of glm on multiple benchmarks. additional insights are gained with ablation study and other analyses. setups. we focus on non-canonicalized commonsense kgs in this work, specifically conceptnet, and the proposed approach is also applicable to ontological kgs such as freebase. for training efficiency we use two relatively small free-form corpora. one is the open mind common sense (omcs) raw corpus foot_1 consisting of 800k short sentences. the other is the arc corpus @xcite containing 14m unordered, sciencerelated sentences. both corpora are parsed to have their entities linked to conceptnet by using an inverted index built with fuzzy matching. in addition, we initialize glm with either bert or roberta rather than training from scratch. we choose to match the corresponding baseline model (whether it uses bert or roberta) in each downstream task. in practice, we can initialize glm with any state-of-the-art pre-trained bidirectional lm. experiment code is available at https://github.com/taoshen58/glm-codes , and detailed settings about training and testing are provided in appendix a. ## conclusion in this work, we aim at equipping pre-trained lms with structured knowledge via self-supervised tasks. building on entity-level mlms, we propose an entity masking scheme under kg's guidance. it masks informative mentions and facilitates learning structured knowledge in free-form text. moreover, we propose a distractor-suppressed ranking objective to utilize negative samples from kg as distractors for effective training. experiments show finetuning our kg-guided pre-trained mlms yields improved performance on related downstream tasks. in the future, instead of pre-training on sentences, we will leverage raw text at passage or document level to alleviate the performance degeneration brought by short context. moreover, we will use a combination of commonsense and ontological kgs, and large-scale corpora (e.g., common crawl) to pre-train an mlm from scratch, which we expect to benefit a wide range of tasks. finetuning hyperparameters. during finetuning on downstream tasks, we conduct grid search for hyperparameters, including batch size, number of epochs/steps, learning rate, which are summarized in details about kg-guided entity masking. in addition to finding informative and non-trivial masks, the kg-guided entity masking method can be used to filter the corpus, since if a piece of text (e.g., sentence or passage) contains only trivial and undeducible entities, the text may hardly contribute to model learning. with this strategy, most (∼ 90%) sentences in arc corpus were filtered out, which leads to a very efficient training for our models. training our model based on bert-large or roberta-large only costs about 1 day on single v100 gpu with mixed float precision, in total about 70k steps.