C o LAKE : Contextualized Language and Knowledge Embedding (2020)

Abstract: With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models. Few works explore the potential of deep contextualized knowledge representation when injecting knowledge. In this paper, we propose the Contextualized Language and Knowledge Embedding (CoLAKE), which jointly learns contextualized representation for both language and knowledge with the extended MLM objective. Instead of injecting only entity embeddings, CoLAKE extracts the knowledge context of an entity from large-scale knowledge bases. To handle the heterogeneity of knowledge context and language context, we integrate them in a unified data structure, word-knowledge graph (WK graph). CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. Experimental results show that CoLAKE outperforms previous counterparts on most of the tasks. Besides, CoLAKE achieves surprisingly high performance on our synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language and knowledge representation.

## introduction deep contextualized language models pre-trained on large-scale unlabeled corpora have achieved significant improvement on a wide range of nlp tasks @xcite @xcite . however, they are shown to have difficulty capturing factual knowledge @xcite . recently, there is a growing interest in combining pre-trained language models (plms) with structured knowledge. a popular approach is to inject pre-trained entity embeddings into plms to better capture factual knowledge, such as ernie @xcite and knowbert @xcite . the shortcomings of these models can be summarized as follows: (1) the entity embeddings are separately pre-trained with some knowledge embedding (ke) models (e.g., transe @xcite ), and fixed during training plms. thus they are not real joint models to learn the knowledge embedding and language embedding simultaneously. (2) the previous models only take entity embeddings to enhance plms, which are hard to fully capture the rich contextual information of an entity in the knowledge graph (kg). thus their performance gains are limited by the quality of pre-trained entity embeddings. (3) the pre-trained entity embeddings are static and need to be re-trained when the kg is slightly changed. in this paper, we propose the contextualized language and knowledge embedding (colake), which jointly learns language representation and knowledge representation in a common representation space. different from the previous models, colake dynamically represents an entity according to its knowledge context and language context. for each entity, colake considers a sub-graph surrounding it as its knowledge context that contains the facts (triplets) about the entity. in this way, colake can dynamically access different facts as background knowledge to help understand the current text. as shown in figure 1 (a), to understand different sentences, colake can utilize different facts about the linked entity harry potter. the knowledge context of harry potter is a sub-graph containing the triplets about it. according to whether or not to utilize entities' knowledge context, our proposed colake can be distinguished from previous models, which we call semi-contextualized joint models since they only contextualize language representation. to deal with the heterogeneous structure of language and kg, we build a graph to integrate them into a unified data structure, called word-knowledge graph (wk graph). most recent successful plms use transformer architecture @xcite , which treats input sequences as fully-connected word graphs. wk graph is knowledge-augmented word graph. using entities mentioned in the sentence, we extract sub-graphs centered on those mentioned entities from kgs. then we mosaic such sub-graphs and the word graph in a unified heterogeneous graph, i.e. wk graph. an instance of the wk graph can be found in figure 1(b ). the constructed wk graph is fed into colake along with its adjacency matrix to control the information flow to reflect the graph structure. colake is based on the transformer encoder, with the embedding layer and the encoder layers slightly modified to adapt to input in the form of wk graph. besides, we extend the masked language model (mlm) objective @xcite to the whole input graph. that is, apply the same masking strategy to word, entity, and relation nodes and training the model to predict the masked nodes based on the rest of the graph. we evaluate colake on several knowledge-required tasks and glue @xcite . experimental results demonstrate that colake outperforms previous semi-contextualized counterparts on most of the tasks. to explore potential applications of colake, we design a synthetic task called wordknowledge graph completion. our evaluation on this task shows that colake outperforms several ke models by a large margin, in transductive setting and inductive setting. in summary, colake can be characterized in three-fold: (1) colake learns contextualized language representation and contextualized knowledge representation simultaneously with the extended mlm objective. (2) colake adopts the wk graph to integrate the heterogeneous input for language and knowledge. (3) colake is essentially a pre-trained graph neural network (gnn), thereby being structure-aware and easy to extend. knowledge representation learning. knowledge representation learning (krl) is also termed as knowledge embedding (ke), which is to map entities and relations into low-dimensional continuous vectors. most existing methods use triplets as training samples to learn static, non-contextual embeddings for entities and relations @xcite @xcite . recent advances focusing on contextualized representation, which use subgraphs or paths as training samples, have achieved new state-of-the-art results on kg tasks @xcite . ## colake colake jointly learns contextualized representation for language and knowledge by pre-training on structured, unlabeled word-knowledge graphs (wk graphs). we first introduce how to construct such wk graphs, then we describe the model architecture and the implementation details. ## experiments in this section, we present the details of pre-training and fine-tuning colake, and its experimental results on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. ## conclusion in this paper, we propose colake to jointly learn contextualized representation for language and knowledge. we integrate the language context and knowledge context in a unified data structure, wordknowledge graph. the experimental results show the effectiveness of colake on knowledge-required tasks. besides, to explore the potential application of the wk graph, we design a task named wk graph completion, which shows that colake is essentially a powerful gnn that is structure-aware and inductive. the surprisingly high performance on wk graph completion inspires the potential applications of wk graph, for example, (a) colake may help to denoise distantly annotated samples of relation extraction @xcite , (b) colake can be used to measure the quality of graph-to-text templates @xcite due to its capability of preserving the original graph structure. we leave these applications as future work.