U NDERSTANDING
K NOWLEDGE
I
NTEGRATION
IN
L ANGUAGE M ODELS WITH G RAPH C ONVOLUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Pretrained language models (LMs) are not very good at robustly capturing factual knowledge.
This has led to the development of a number of knowledge integration (KI) methods which aim to incorporate external knowledge into pretrained LMs.
Even though KI methods show some performance gains over vanilla LMs, the efﬁcacy and limitations of these methods are not well-understood.
For instance, it is unclear how and what kind of knowledge is effectively integrated into LMs and if such integration may lead to catastrophic forgetting of already learned knowledge.
In this paper, we revisit the KI process in an information-theoretic view and show that KI could be interpreted using a graph convolution operation.
We propose a simple probe model called Graph Convolution Simulator (GCS) for interpreting knowledge-enhanced LMs and exposing what kind of knowledge is integrated into these models.
We conduct experiments to verify that our GCS model can indeed be used to correctly interpret the KI process, and we use it to analyze two typical knowledge-enhanced LMs: K-Adapter and ERNIE.
We ﬁnd that only a small amount of factual knowledge is captured in these models during integration.
While K-Adapter is better at integrating simple relational knowledge, complex relational knowledge is integrated better in ERNIE.
We further ﬁnd that while K-Adapter struggles to integrate time-related knowledge, it successfully integrates knowledge of unpopular entities and relations.
Our analysis also show some challenges in KI.
In particular, we ﬁnd simply increasing the size of the KI corpus may not lead to better KI and more fundamental advances may be needed.
I NTRODUCTION
Pretrained language models (LMs) have achieved state-of-the-art performance across various natural language processing (NLP) tasks.
Previous works have shown that linguistic knowledge is captured quite well by LMs and it plays a vital role in their success (Liu et al., 2019a; Jawahar et al., 2019).
However, factual knowledge is sparse and is expressed in varied ways in text.
Thus, LMs are much worse in capturing factual knowledge about the world (Petroni et al., 2019; Wang et al., 2021b).
This has led to the development of a variety of knowledge integration (KI) methods which aim to integrate external knowledge into LMs (Colon-Hernandez et al., 2021;
Wang et al., 2021a; Zhang et al., 2019).
Even though knowledge-enhanced LMs perform better on knowledge-related tasks, we lack a deep understanding about the inner workings of these models.
Better downstream performance indicates that some new knowledge has been integrated, but how much knowledge has been successfully integrated, which type of knowledge is integrated is not well-understood.
To understand what knowledge is learned in LMs, many model-agnostic methods have been proposed. Previous works have focused on designing simple classiﬁers as probe models (Hewitt &
Manning, 2019; Ribeiro et al., 2016). To get more reliable interpretation, information-theoretic approaches have also been introduced (Guan et al., 2019; Pimentel et al., 2020; Hou & Sachan, 2021).
However, factual knowledge is typically organized as large-scale sparse knowledge graphs (KGs). Previous interpretation methods while being suitable for small linguistic graphs, cannot provide reasonable interpretations for large sparse KGs. Prompting is yet another way to understand what factual knowledge do these models learn. Prompts can be designed to let LMs solve ﬁll-in-the-blanks problems, and the prompt performance can be interpreted as a probe (Petroni et al., 2019; Shin et al., 2020; Zhong et al., 2021).
However, these methods rely on manually constructed templates which is very time-consuming.
There are two other common challenges to understand KI.
First, there are a large number of approaches for KI.
KI in LMs can be implemented by matching sentences to entities or triples in knowledge graphs – called entity-wise integration (Peters et al., 2019; Zhang et al., 2019) and triplewise integration (Liu et al., 2020;
Wang et al., 2021a). There are several modeling choices (ColonHernandez et al., 2021) for KI, including proposing some modiﬁcations to the Transformer architecture (Peters et al., 2019; Zhang et al., 2019; Liu et al., 2020), verbalizing knowledge triples and using data augmentation for ﬁnetuning (Agarwal et al., 2021), and designing objective functions that predict the factual knowledge (Yao et al., 2019; Wang et al., 2021a).
How to design a general method to understand KI in both entity-wise and triple-wise manner is challenging.
Furthermore, KI is typically implemented in a continual learning setup (Parisi et al., 2019) – KI is usually a secondary pretraining or ﬁnetuning step (Lu et al., 2021).
As new knowledge is integrated, old knowledge could be catastrophically forgotten (CF; Kirkpatrick et al., 2016).
KI could also lead to a situation called catastrophic remembering (CR; Kaushik et al., 2021), where the old knowledge may prevent the integration of new knowledge.
Our understanding of these issues is limited.
In this paper, we ﬁrst revisit the KI process (§2). We formulate KI in an information-theoretic view (§2.1), and construct a transformation to approximate general KI process (§2.2).
Then, we prove that the KI process can be simulated by graph convolution operations (§2.3).
Second, we introduce how to analyze KI, CR, and CF.
Speciﬁcally, we prove that the KI process can be interpreted by graph attention mechanism (§3.1). Based on that, we propose Graph Convolution Simulator (GCS) model to simulate and interpret the KI process (§3.2), and introduce the way to analyze its interpretation results (§3.3).
We show that our designed probe model GCS can correctly simulate and interpret the KI process for two popular knowledge-enhanced LMs: K-Adapter (Wang et al., 2021a) and ERNIE (Zhang et al., 2019).
We ﬁnd that K-Adapter integrates simple relational knowledge (i.e., entities are leaf nodes in KGs) well, while ERNIE is better at integrating complex relational knowledge (i.e., entities are center nodes in KGs).
In our qualitative study, we ﬁnd that K-Adapter does not learn temporal knowledge at all.
We further break down our analysis of KI in terms of type of relations and the popularity of entities.
We ﬁnd that catastrophic remembering often happens to simple relational knowledge (i.e. simple relational knowledge is harder to edit), while complex relational knowledge is often catastrophically forgotten.
We also ﬁnd that catastrophic forgetting easily happens to popular entities, while catastrophic remembering often happens when the entities are not very common.
Finally, we investigate the correlation between the size of the KI corpus and
KI quality.
We ﬁnd that there is no apparent positive relationship between them, suggesting that merely building larger KI datasets may not be enough and we may need to make more fundamental advances to build better knowledge-enhanced language models.
S IMULATING K NOWLEDGE I NTEGRATION
In this section, we revisit KI in an information-theoretic view, and prove that KI can be simulated by graph convolutions.
Speciﬁcally, we ﬁrst formulate the KI process.
We use MI to measure the knowledge learned in LMs, and use the change of MI to deﬁne KI, catastrophic remembering (CR), and catastrophic forgetting (CF).
Then, based on the deﬁnition, we construct a multistep transformation to approximate the KI process with arbitrary accuracy.
We show that KI can only happen in certain steps of on the transformation, which are graph convolutions.
2. K NOWLEDGE I NTEGRATION D EFINITION
Before presenting a formal deﬁnition of KI based on MI, we introduce some basic concepts.
Knowledge graphs.
We assume that factual knowledge can be formulated as a knowledge graph
G =
( V , E ) , where nodes v i ∈ V represent entities, and edges in E represent relations between them.
Let N v i denote the set of neighbors of node v i , and t i denote the entity label corresponding to the node v i .
Further, let x i =
LM( t i ) denote the entity (label) representations of t
i given by a LM 1 .
Let
X ∈ R |V|× d denote a matrix formed by stacking all entity representations x i ∈ R d .
In this paper, we only consider nodes and relations in the KG and ignore other kinds of KG information such as edge weights, edge directions and multi-edges (multi-relations).
Information-theoretic probe.
We follow theoretical settings of Hou & Sachan (2021) that measures the knowledge captured in LMs by MI.
We assume that the local graph structure G ( v i ) contains all the factual information regarding v i .
In this work, we consider factual knowledge in the form of triples ( v i , r, v j ) .
Since a triple only contains entities within one-hop, it sufﬁces to set G ( v i ) =
N v i .
Factual knowledge that has been successfully integrated should be reﬂected in entity representations.
Let x be a random variable that takes values ranging over all possible entity representations of a
LM 2 , and g be a random variable that ranges over all possible corresponding local structures
G ( v i ) .
Intuitively, MI( x ; g ) measures the amount of information in g that is contained in x .
Deﬁnition 1 (Knowledge Integration) .
Given G , let entity representation matrices given by a LM before and after KI be X and H .
Corresponding random variables are g , x , and h .
We formulate KI process f ( x , g ) =
h as the change of MI: MI( x ; g ) → MI( h ; g ) , i.e., knowledge of g is integrated into x to get h .
If MI( h ; g ) ≈ MI( x ; g ) , it means that CR happens, i.e., there is a failure to integrate new knowledge.
If CF happens, we have MI( h ; x ) ≈ 0 . [Figure]
The deﬁnition can be intuitively visualized by Figure 1.
The MI change can be represented by regions 1 and 4 in the Venn diagram.
Ideally, if we have MI( h ; g ) − MI( x ; g ) ≈ MI( g ; g ) − MI( x ; g ) , i.e., the region 5 is small, we say most knowledge is successfully integrated.
If little new knowledge has been integrated, the region 4 is very small.
Then, we say that CR has happened.
If CF happens, most knowledge in x is forgotten in h after KI, and region 1 is large.
Successful KI happens when much new knowledge is integrated (i.e., MI( h ; g ) is large) and little old knowledge is forgotten (i.e., MI( h ; x ) is large).
2. A PPROXIMATED T RANSFORMATION C ONSTRUCTION
In this subsection, we show that we can construct a transformation to approximate the KI process with arbitrary accuracy.
We begin by introducing the concept of Graph Fourier transforms.
Graph Fourier transformation.
Graph Fourier transform (GFT) can be used to transform the entity representation matrix X in the Euclidean space to the graph spectral domain (i.e., KG space ).
Speciﬁcally, let A ∈ R |V|×|V| be the symmetric adjacency matrix corresponding to G .
Let L n =
I − D − 1 / 2 AD − 1 / 2 denote the normalized Laplacian matrix for G , where D denotes the degree matrix of G .
We do the eigendecomposition for L n as L n = U Λ U T , where U is the matrix of eigenvectors ordered by eigenvalues and Λ = diag ( λ 1 , λ 2 , ..., λ N ) is the diagonal matrix of eigenvalues.
Based on the GFT (Sandryhaila & Moura, 2014), formally, the transformation of X to the KG space can be written as GFT( X ) =
U T X , and its inverse transformation can be written as RGFT(GFT( X ))
=
U GFT( X ) =
X .
It is intractable to directly simulate the KI process (with simple probe models), i.e., integrating knowledge information from g into x to get h (Deﬁnition 1), since g is graph data that are not in the
Euclidean space.
Fortunately, with the help of GFT, we can transform x and h into the KG space, and design transformation there to simulate the KI process. Below theorem provides a feasible way to construct a transformation that can simulate the KI process with arbitrary accuracy.
Theorem 2 (Transformation Existence) .
Denote the graph Fourier transformation and its inverse transformation in terms of G as GFT( · ) and RGFT( · ) .
Given a LM and its knowledge-enhanced version, suppose that MI( h ; g ) − MI( x ; g ) > 0 , and there exists a mapping that satisﬁes f
( x , g ) = h .
Then, for any (cid:15) > 0 , there exists a neural network NN( · ) such that
[Equation]
The proof can be found in Appendix B. Theorem 2 shows that there exists an approximated transformation composed of GFT and a neural network that can simulate the KI process (i.e., f ( x , g ) = h ) with arbitrary accuracy.
In practice, the mapping f ( x , g ) is realized by complex KI training process on large LMs with new datasets, objectives, or even new LM parameters. Here, we can use a simple transformation RGFT(NN(GFT( x ))) to generally approximate and simulate the complex KI process.
However, KGs are normally very large, and computing the eigendecomposition of the Laplacian matrix for GFT is prohibitively expensive. Besides, the transformation is not interpretable, which can not be used as a probe model directly.
Thus, we take a deeper look into this transformation.
We ﬁrst show that it can be further simpliﬁed as graph convolutions.
2. K NOWLEDGE I NTEGRATION S IMULATION WITH G RAPH C ONVOLUTIONS
In this subsection, we introduce details of the transformation in Theorem 2.
We prove that even if there are multiple steps of the transformation, MI change only happens in certain steps.
We show that these steps are equivalent to graph convolutions.
Graph convolutions.
Convolution operations on graphs are often used to model relational information of KGs, where entities aggregate information from their neighbors and pass the information along based on the graph structure.
Graph convolutions can be implemented by ﬁlters g Θ in the graph spectral domain (i.e., KG space).
As the GFT of the convolution of g Θ and X is the pointwise product of their GFT (Bracewell & Bracewell, 1986), the convolution can be written as g Θ (cid:63) X
= RGFT( g Θ [Equation] GFT( X ))
(Bruna et al., 2014).
In the below proposition, we show that the multistep transformation for KI simulation (Theorem 2) can be simpliﬁed as graph convolutions.
Proposition 3 (Graph Convolutions for Simulation) .
Suppose MI( x ; g ) <
MI( h ; g ) , MI( h ; x ) < MI( x ; x ) , and the mapping f ( x , g ) =
h can be well approximated by the transformation RGFT(NN(GFT( x ))) where the neural network NN( · ) has n layers.
We can only use n linear functions in the neural network (i.e., graph convolutions) to well simulate the MI change. Proposition 3 indicates that if some new knowledge is integrated and some old knowledge is forgotten during KI, we can only use n linear functions instead of the whole transformation in Theoremto simulate the MI change (i.e., formulated KI process). These linear functions in the KG space are actually graph convolution operations. And thus, we can circumvent the expensive eigendecomposition in GFT with fast and well-approximated graph ﬁlters (Defferrard et al., 2016; Kipf & Welling, 2017).
The formal proof is in Appendix C. Figure 2 brieﬂy illustrates the overall idea.
[Figure]
The transformation proposed in Theoremcan simulate KI with arbitrary accuracy.
Even if the transformation is simple, it is still not very efﬁcient with large KGs and not interpretable.
The red solid lines represent the simulation of KI using graph convolutions (Proposition 3), which are much more fast and interpretable.
According to the invariance property of MI (Kraskov et al., 2004), the introduction of bijective functions does not introduce any new information – MI remains unchanged upon the introduction of bijective functions.
We know that GFT and RGFT are both bijective (Appendix C.1).
We show that nonlinear activation functions in a neural network (e.g., sigmoid( · ) ) are bijective as well (proof in Appendix C.1).
Thus, the MI change in the KI process can only happen in the linear function (proof in Appendix C.2).
Based on the convolution theorem (Bracewell & Bracewell, 1986), linear functions in graph space are graph convolutions (Sandryhaila & Moura, 2014; Bruna et al., 2014; Kipf & Welling, 2017) (Appendix C.3).
Thus, we can simply use graph convolution operations instead to simulate the KI process.  I NTERPRETING K NOWLEDGE I NTEGRATION
In the last section, we showed that KI can be simulated by graph convolutions. In this section, we introduce the mechanism to interpret the KI process. Speciﬁcally, we ﬁrst illustrate that the graph attention mechanism can be used to interpret graph convolutions. Then, we introduce an implementation of the probe model proposed in our work: Graph Convolution Simulator (GCS), as well as a method to analyze our interpretation results.
Following Proposition 3, we can select powerful graph ﬁlters to simulate and interpret the KI process. Velickovic et al. (2018) and Thekumparampil et al.
(2018) introduce the attention mechanism to graph ﬁlters, where the contribution of each edge to the convolution can be shown explicitly.
Graph attention makes ﬁlters more powerful and convolutions more interpretable (Fu et al., 2020).
Proposition 4 (Graph Attention for Interpretation) .
Suppose MI( x ; g ) <
MI( h ; g ) , MI( h ; x ) <
MI( x ; x ) , and the KI process MI( x ; g ) → MI( h ; g ) can be well simulated by n graph convolution operations. Then, we can use the attention coefﬁcients on edges and self-loops to interpret the KI, CR, and CF. The formal proof can be found in Appendix D. Note that we have n graph convolutions, and they function differently (Bruna et al., 2014).
In a multi-layer graph convolution network, the k -th graph convolution step aggregates information from k -hop neighbors 3 .
In this work, we consider integration of knowledge in the form of knowledge triples. Knowledge triples link entities within 1-hop.
Thus, in practice, we set n = 1 for simplicity in our work.
According to Fu et al.
(2020), graph attention can also be seen as edge denoising.
This provides an alternative explanation of our GCS probe from the denoising view.
More details can be found in Appendix D.
3. GCS ARCHITECTURE .
Based on Proposition 3 and Proposition 4, we design GCS with two bijective functions and one graph convolution function in between.
To implement a bijective function in practice, we show that special MLP layers can be bijective if the weight matrix is a square matrix (plus a small noise).
The formal description and the proof are in Appendix E. We design our GCS model with one graph convolutional layer and two bijective MLP layers as:
[Equation]
where MLP n ( · ) is the bijective MLP layer and GC( · , G ) is the graph convolutional layer on the KG
G which is used to simulate and interpret the KI process.
Given an entity v i and its set of neighbors
N v i , we can write the graph convolutional layer as:
[Equation]
Here, x i is the entity representation of v i before knowledge integration, the activation function σ ( · ) is ELU( · ) function, and W V is a weight matrix.
a i,j is the attention coefﬁcient on the edge that connects v i and v j .
W Q and W K are two parameter matrices in the graph attention.
d k is the dimension of vector W K x j , and softmax ( · ) is the edge-wise softmax function with respect to node v i .
Temperature t is a hyperparameter that controls the attention distribution to be hard or soft.
As the multiplicative attention mechanism (Vaswani et al., 2017) is broadly used in LMs, we also select multiplicative attention in the graph attention.
We optimize GCS by letting its outputs be as close to h as possible. This can be achieved by using a reconstruction loss minimization or maximizing
MI between the outputs of GCS and h . We use MI maximization in our implementation. More implementation details can be found in Appendix H. A NALYZING I NTERPRETATION R ESULTS
We implement the GCS model in Equation 2 and use attention coefﬁcients in Equation 3 for all relations and entities in the KG for interpretation. Then we analyze these interpretation results to get conclusions for the KI process.
As introduced in §3.1, large edge attention coefﬁcients mean that the triples corresponding to the edge are integrated well. To understand CR and CF, we add self-loops on entities in the KG, and use attention coefﬁcients on the self-loops to show how much of the original information is remembered/forgotten for entities. In particular, large self-loop attention coefﬁcients mean that the original entity information is kept well. Thus, we introduce thresholds to analyze our interpretation results as follows 4 .
We simply regard triples with attention coefﬁcients a i,j > 0 . 1 on edges as integrated triples. Entities with attention coefﬁcients 0 . 4 < a i,i < 0 .
6 on self-loops as well-learned entities.
Entities with a i,i < 0 . 1 on self-loops means CF has happened, where much new factual knowledge information is integrated and original entity information is forgotten. Correspondingly, a i,i > 0 .means CR has happened. For interpretation, attention coefﬁcients on edges are used for triple-wise integration, and those on self-loops are for entity-wise integration.
E XPERIMENTS
We ﬁrst introduce two knowledge-enhanced LMs considered in this work: K-Adapter (Wang et al., 2021a) and ERNIE (Zhang et al., 2019).
KG is integrated in a triple-wise manner in K-Adapter, and entity-wise manner in ERNIE.
Then, we move on to our experiments. First, we verify our GCS model. We prove that GCS can correctly interpret how much KG information is integrated, as well as which set of entities and triples are integrated.
After that, we use GCS to interpret the KI process for K-Adapter and ERNIE.
We present the interpretation results and ﬁnd that both K-Adapter and ERNIE have only integrated few triples, but they integrate many entities.
Finally, we use our probe to understand which kinds of knowledge is integrated well in these models. In order to do this, we stratify knowledge in terms of various relation type and ﬁnd that K-Adapter and ERNIE integrate different kinds of factual knowledge to different extents.
4. K NOWLEDGE -E NHANCED M ODELS
Below, we describe the two knowledge-enhanced LMs considered in this work:
K-Adapter.
K-Adapter takes RoBERTa (Liu et al., 2019b) as the backbone model and inserts three new layers into RoBERTa to learn new knowledge. The ﬁnal output is concatenated with the output of RoBERTa 5 .
During the integration, parameters of RoBERTa are frozen, only parameters of the newly inserted layers are updated.
K-Adapter uses the T-REx-rc (ElSahar et al., 2018) dataset for KI, which has an alignment of natural sentences with knowledge triples in Wikidata.
For the KI objective, K-Adapter decides whether certain relations exist or not, and classiﬁes relation labels given the aligned sentence.
As knowledge is integrated in newly inserted layers, the model no longer needs to use the T-REx-rc dataset for inference, and it can be ﬁnetuned like any other pretrained LMs on downstream tasks.
ERNIE.
ERNIE integrates factual knowledge into BERT (Devlin et al., 2019) directly without introducing extra parameters.
The Wikipedia corpus and Wikidata knowledge triples are selected for integration.
As there is no provided alignment between natural sentences and knowledge entities, ERNIE uses TAGME (Ferragina & Scaiella, 2010) to extract entity mentions in sentences and aligns them with corresponding entities in KGs.
A new objective is designed for KI in addition to the standard MLM and NSP objectives: alignments in the input text are randomly masked, and the model is asked to select aligned entities from KGs.
Different from K-Adapter that stores factual knowledge in newly introduced parameters, when ERNIE ﬁnds the aligned entity, its embedding obtained from Bordes et al.
(2013) is integrated into the output representations.
Thus, during inference, the KG and its embeddings are still required to be fed into ERNIE.
We design a set of experiments to verify that GCS can indeed correctly interpret the KI process. We ﬁrst verify that GCS can correctly interpret how much knowledge is integrated using synthetic experiments (§4.2.1). Then, we verify that GCS can correctly interpret which type of knowledge is integrated based on the KI dataset (§4.2.2) and downstream task performance (§4.2.3).
4.2. S YNTHETIC EXPERIMENT
Setting. We create a synthetic KI scenario where different amounts of knowledge is integrated into the LM. Speciﬁcally, we ﬁrst use DeepWalk (Perozzi et al., 2014) to obtain entity embeddings (i.e., KG embeddings) in the KG used for KI.
We regard entity embeddings as the entity representations of knowledge-enhanced LMs, and we add Gaussian noise with different noise ratios on entity embeddings and regard it as the entity representations of vanilla LMs.
Then, we simulate the KI processes from vanilla LMs (noisy entity embeddings) to knowledge-enhanced LMs (entity embeddings).
Large noise ratio means much information of KG is integrated.
[Figure] Baselines. We select several baselines for comparison. Representation analysis : we calculate the similarity (e.g., Cosine, Euclidean) between two connected entities to estimate how much knowledge about the corresponding triple is contained in the
LM.
Finally, the similarity gap between vanilla LMs and knowledge-enhanced LMs can be used to interpret how much knowledge is integrated.
Linear probe : we design a linear classiﬁer to do link prediction in the KG based on entity representations.
The performance (i.e., AUC score) can be used to show how much knowledge is contained in the LM, and its gap shows how much knowledge is integrated.
As for GCS, we use the mean value of self-loop attention coefﬁcients to show how much knowledge is integrated: large values mean that little knowledge is integrated. For the convenience of observation, we report one minus its mean value as the integration score. Implementation details are in Appendix F. Results.
Figure 3 shows the interpretation results. Solid lines show the results of synthetic KI processes.
We ﬁnd that GCS and representation analysis methods provide correct results: the similarity gap/score increases as the noise ratio increases. However, linear classiﬁer probe can only differentiate whether new knowledge is integrated or not, and fails to tell how much knowledge is integrated (its curve is not very monotone.).
In practice, dotted lines and dashed lines show interpretation results of K-Adapter and ERNIE.
We ﬁnd that only GCS provides reasonable results: they integrate little knowledge (equivalent to noise ratio ≈ 5% ).
All baselines fail in interpretation: K-Adapter and ERNIE do not integrate any factual knowledge, or they even forget learned factual knowledge. 4.2. V ERIFICATION USING THE KI DATASET
Setting.
This experiment is composed of three steps. First, we use GCS to interpret the KI process in K-Adapter and ERNIE, and identify triples and entities that are integrated successfully.
Second, we retrain BERT/RoBERTa to get K-Adapter (dropped)/ERNIE (dropped) only using the triples/entities that are identiﬁed as successfully integrated.
Third, we ﬁnetune K-Adapter/ERNIE and their dropped versions on downstream tasks.
If GCS correctly interpreted the KI process, the performance of their dropped versions on downstream tasks should be roughly the same as that of
K-Adapter/ERNIE.
As introduced in §4.1, K-Adapter and ERNIE integrate knowledge by aligned natural sentences instead of using triples/entities directly.
Thus, after we get the interpretation results, i.e., triples with edge attention coefﬁcients larger than 0 . 1 or entities with self-loop attention coefﬁcients smaller than 0 . 9 , we only keep data aligned with the integrated knowledge. Speciﬁcally, we drop sentences for
K-Adapter and entity embeddings (obtained by Bordes et al. for ERNIE. Then, we ﬁnetune them on two downstream tasks (entity typing) that K-Adapter and ERNIE outperform RoBERTa and
BERT most signiﬁcantly: OpenEntity (Choi et al., 2018) and FIGER (Ling et al., 2015).
Implementation details of reproduction and GCS can be found in Appendix G and Appendix H. [Table] Results. We only keep 10 .
09% natural sentences that aligned with successfully integrated triples for
K-Adapter (dropped), and 61 . 72% entity embeddings (obtained by Bordes et al. for ERNIE (dropped).
More details are in Appendix I. From Table 1, we can ﬁnd that even if we drop large amount of KI data in this way, the performance of K-Adapter (dropped) and ERNIE (dropped) on entity typing task is roughly the same as original (reproduced) versions, and they are obviously better than that of BERT and RoBERTa.
We further verify GCS by introducing a random dropping strategy as comparison on the OpenEntity dataset.
Detailed results can be found in Appendix J.
4.2. V ERIFICATION WITH A DOWNSTREAM TASK
Settings.
In this experiment, we combine the interpretation results of GCS with a downstream task. Speciﬁcally, we align entities in the KI dataset and the OpenEntity dataset based on their Wikidata Q identiﬁer 6 . For the entity typing task (OpenEntity dataset), we drop the ﬁnetuning test data samples that aligs with the integrated knowledge and non-integrated entities (called drop-IE test set and dropUE test set), and test K-Adapter and ERNIE on the two dropped test sets. If GCS correctly interpret the KI process, knowledge-enhanced LMs should perform better on the drop-UE test set and worse on the drop-IE test set.
[Table] Results. Table 2 presents detailed results. We can ﬁnd that for K-Adapter, the gap is not very obvious.
We hypothesize that this may be because of the differences in the ﬁnetuning objective and the KI objective, and because the knowledge integrated in K-Adapter may change during ﬁnetuning. As for ERNIE, the gaps are signiﬁcant. The performance (F1-Micro) on the test set (drop-IE) is 20 F1 points worse than that on the complete test set. These results also verify that GCS can correctly interpret which set of knowledge is integrated.
After verifying GCS with three groups of experiments, we analyze the interpretation results. From a macro view, we ﬁnd that both K-Adapter and ERNIE integrate few knowledge triples ( ≈ 20% − 30%) and some knowledge entities ( ≈ 60% − 70%) .
Detailed results can be found in Appendix K.
From a micro view, we classify knowledge based on relation types (in terms of their topology type and Wiki data type) and analyze how K-Adapter and ERNIE integrate them respectively. KI analysis for K-Adapter and ERNIE in terms of relation topology. We classify relations into three types based on their topology features. Speciﬁcally, relations that connect two leaf nodes (entities) in the KG are 1 − 1 relations, and relations that connect two center nodes (entities) in the KG are N − M relations.
Others are N − 1 relations. For the analysis results in terms of different types of relations, we report the percentage of successfully integrated triples and entities for K-Adapter and ERNIE. Besides, we also present the percentage of catastrophic remembered (CR) entities and catastrophic forgotten entities (CF).
Table 3 presents speciﬁc results. We ﬁnd that for K-Adapter, triples with N − M relations are not captured well.
However, K-Adapter integrates triples with 1 − 1 relations well.
This phenomenon is common for Transformer encoders, where knowledge with complex structure cannot be captured well (Petroni et al., 2019).
ERNIE shows different behaviors.
We ﬁnd that entities connected with 1 − 1 and N − M relations are captured well.
But for entities connected to N − 1 relations, they [Table] are not integrated well.
Since ERNIE relies on KG embedding to learn structure knowledge [Table] KI is highly consistent with the quality of the KG embedding provided in Bordes et al.
Regarding
CR and CF
[Table] we ﬁnd that for both K-Adapter and ERNIE [Table] CR happens more often to entities in simple structures (i
[Table]
e., connected to 1 − 1 or 1 − N relations), while CF is more common for entities in complex structures (i
[Table] e., connected to N − M relations) [Table] We list 6 relations and classify them into three types based on the Wiki
[Table] [Table]
KI analysis for K-Adapter and ERNIE in terms of relation’s
Wiki features. We select six relations aligned with roughly the same number of sentences in the T-REx-rc dataset (see Appendix L for statistics) and categorize them into three groups based on the Wiki Count and Wiki data type 7 : low-frequency (LF) relations, time-related (TR) relations, and high-frequency (HF) relations.
From Table 4, we can ﬁnd that even if LF relations has roughly the same Wiki Count as TR relations.
However, since the Wiki data type of the latter set is “Time”, triples with those relations cannot be integrated by K-Adapter.
We speculate that this is because Transformer encoders do not capture information about time well (Dhingra et al., 2021;
Zhou et al., 2021).
When comparing
LF relations and HF relations, we ﬁnd that if relations have small Wiki Count, knowledge triples are easier to be captured during KI.
Above experiments show that interpretation results of GCS are consistent with existing analysis works, which also verify GCS indirectly.
Besides, we design a case study experiment in Appendix M, where we ﬁnd that CR often happens to rare entities (small Google Ngrams), while CF often happens to popular entities (large Google Ngrams).
Moreover, we study the correlation between the edge attention coefﬁcient and the number of aligned sentences for K-Adapter in Appendix N, and ﬁnd that the Pearson Correlation Coefﬁcient is -0.0055.
It implies that simply increasing the KI dataset may not help LMs integrate unlearn knowledge.
L IMITATIONS AND F UTURE W ORK
In this paper, we illustrate that the graph attention can be used to interpret the KI process for knowledge-enhanced LMs, and thus propose a Graph Convolutional Simulator (GCS) that is capable of correctly interpreting existing knowledge-enhanced LMs.
In our experiments, we verify GCS and use it to obtain interesting ﬁndings.
There are some limitations of our work.
We simplify the KG without considering edge direction, labels, multi-edges, entity descriptions, and timestamps. These can be considered by future works.
Besides, GCS only provides a way to interpret the knowledge integration.
Once we have an understanding of the KI process, improving the integration quality still remains challenging.
R EPRODUCIBILITY S TATEMENT
We will publish the code, as well as the interpretation results after the review process.
The design of GCS can be found in §3.
The implementation details about the KI for K-Adapter and ERNIE can be found in Appendix G. And details about GCS can be found in Appendix H
E THICS S TATEMENT
While our probe models are not tuned for any speciﬁc real-world application, our methods could be used in sensitive contexts such as legal or health-care settings; and it is essential that any work that builds on our approaches undertake extensive quality-assurance and robustness testing before using it in their setting.
A N OTATIONS
[Table]
B P ROOF OF T HEOREM
Proof.
As aforementioned, the graph Fourier transformation GFT( · ) and its inverse transformation
RGFT( · ) in terms of the KG G can be written as
[Equation]
The second equation can be derived since U is the set of eigenvectors of the normalized Laplacian matrix in terms of G , which is orthogonal.
According to the universal approximation theorem (Cybenko, 1992), in general, we can use onelayer neural networks (arbitrary width) with the sigmoid activation function to ﬁt any functions.
Ohn & Kim (2019) bound the approximation with both the width and depth, and supports more activation functions.
Based on the conclusion of Ohn & Kim (2019), we know that given a mapping f (cid:48) ( · ) , for any (cid:15) (cid:48) > 0 , there exists a neural network parameterized by θ (cid:48)
s.t.
[Equation]
Note that there are some constraints about the input and the model architecture, i.e., width and depth.
We leave out those details for simplicity.
More details can be found in Ohn & Kim (2019).
Since h is obtained by integrating g into x , we can simplify the mapping in the graph space by researching on the transformation from GFT( x ) to GFT( h ) 8 .
Assume the mapping satisﬁes f
[Equation]
(GFT( x ))
=
GFT( h ) .
Then we have
[Equation]
Consider that we have f ( x , g )
=
h = RGFT( f (cid:48)
(GFT( x ))) .
If we assign (cid:15) (cid:48) =
(cid:15) | U | > 0 , we have
[Equation]
Since we know that
[Equation]
we have
[Equation]
where NN( · ) is parameterized by θ (cid:48) with activation function σ as NN σ ( ·| θ (cid:48) ) .
And without loss of generality, we assume it is composed of n layers.
C G RAPH C ONVOLUTIONS FOR KI S IMULATION (P ROOF )
Proof.
The basic idea of this proof can be found in Figure 2.
We ﬁrst simply prove that the graph
Fourier transformation is bijective.
Similarly, the nonlinear activation function can be proved bijective.
Then, we show that information gain and loss can only happen in the linear function in graph space.
After that, we brieﬂy illustrate that linear function in graph space is graph convolution operation.
Finally, we prove that graph attention works as edge denoising, and we can use it to interpret the KI.
GFT( · ) , RGFT( · ) , and sigmoid( · ) are bijective.
Given two entity representations x i , x j and the matrix of eigenvectors of the KG as U , suppose that GFT( x i ) =
GFT( x j ) .
Then, we have
U T x i =
U T x j .
Since U T are set of eigenvectors and are by deﬁnition nonzero, we have x i = x j .
If x i = x j , it is easy to get GFT( x i ) =
GFT( x j ) .
Thus, graph Fourier transformation is bijective.
As for the nonlinear activation function, since we consider neural networks composed of MLP layers, the activation function is sigmoid( · ) function.
It is easy to ﬁnd that its inverse function is f ( y ) =
ln(1 − 1 y ) .
Similarly, we can prove that it is bijective as well. C. S TEP
Information gain and loss can only happen in the linear function in graph space.
Based on the invariance of MI (Kraskov et al., 2004), we have
[Equation]
Since we know that
MI
[Equation]
, and the neural network can well approximate the mapping, we have
[Equation]
If we write NN( · ) with n MLP layers as n × σ (Linear( · )) , we have
[Equation]
( x ) , g ) > 0 .
Recursively with equations 4, it is easy to get that MI only changes in the Linear( · ) functions.
C. S TEP
The linear function in the KG space (i.e., graph spectral domain) is the graph convolution operation.
Even if many existing works (Sandryhaila & Moura, 2014; Bruna et al., 2014; Kipf & Welling, 2017) have provided clear descriptions, we simply re-illustrate it under the multi-channel setting.
Consider the graph ﬁlter in Bruna et al.
(2014) as an exmaple.
For a linear function f ( x ) =
W × x , its weight matrix W ∈ R F × C is parameterized by Θ ∈ R F × C .
If the parameters are not shared for all nodes, the input X ∈ R |V|× C can be rescaled in R |V|× C × 1 , and the weight matrix is W ∈ R |V|× F × C parameterized by Θ ∈ R F × C ×|V| .
The output of this linear function is mapped in R |V|× F .
Consider the signal in graph convolution, i.e., all x in X ∈ R |V|× C .
Since parameters are not shared (Bruna et al., 2014), for one graph ﬁlter, the parameters in g Θ is in R C ×|V|×|V| that is parameterized by Θ ∈ R C ×|V| with simple diagonalization.
If we have F different graph ﬁlters for the convolution, g Θ is in R F × C ×|V|×|V| that is parameterized by Θ ∈ R F × C ×|V| .
Here, the graph
Fourier transformation of X is GFT( X ) ∈ R |V|× C , which can be rescaled in R 1 ×|V|× C × 1 with simple diagonalization.
The output is in R F ×|V|×|V|× 1 .
Note that since the parameters in the graph ﬁlter is diagonalized, we can rescale the output in R |V|× F .
If we regard the weight matrix W as the parameters in the graph ﬁlter g Θ , the input matrix X as the signal, obviously, the linear function in the graph space is the graph convolution operation. D G RAPH A TTENTION FOR KI I NTERPRETATION (P ROOF )
Proof. Graph attention works as edge denoising 9 .
Graph attention works as a better graph convolution ﬁlter, since it can adaptively learn the optimal convolution weights (i.e., attention coefﬁcients).
Consider a graph signal denoising problem that we aim to extract the ground-truth node features ˆ X and edge weights ˆ A n from a graph G =
( V , E , A n ) with noise in both node features X and edge weights A n .
Here, A n is the normalized adjacency matrix A n = D − 1 / 2 AD − 1 / 2 .
To this end, we formulate the optimization problem under the assumption that the ground-truth node features ˆ X are smooth w.r.t the ground-truth adjacency matrix ˆ A n and the noise in the graph can be upper-bounded:
[Equation]
where ˆ L = I − ˆ A , (cid:15) 1 , (cid:15) 2 ∈ R , are the level of noise in node features and edge weights, respectively.
Tr( · ) indicates the trace of a matrix.
By Lagrange multipliers methods, we can obtain the solution as following:
[Equation]
where γ > 0 is the Lagrangian multiplier.
Note that the attention coefﬁcients of GAT (Velickovic et al., 2018) and AGNN (Thekumparampil et al., 2018) are obtained by (without less of generality, we show the results in the ﬁrst-layer) equation 8 and equation 9, respectively:
[Equation]
where H = ReLU ( XW ) , a , W in equation 8, and β , W in equation 9 are learnable parameters.
The attention coefﬁcents of GAT and AGNN are then used as the weights of aggregating the neighbohood information of nodes.
As we can see that equation 7, equation 8, and equation 9 are in a form of measuring the similarity between paired node features.
Similar to the denoised edge weights obtained in equation 7, the attention coefﬁcents (i.e. the aggregation weights) between a node and its neighborhoods are proportional to the similarity of their node embeddings.
Therefore, the attention coefﬁcients of GAT and AGNN can be regarded as the results of denoised weights on the existing edges in a graph, i.e., the graph attentions are implicitly denoising the edge weights. In general case, graph attention functions as denoising edge weights. The input is noisy representations and the output is the groundtruth.
Attention coefﬁcients show how much distortion is corrected during the convolution operation.
For example, if the input representations are also groundtruth, there is no need to fetch information from neighbors to get output. And edge weights will be reduced to 0 , i.e., attention coefﬁcients on edges are calculated as 0 . If the input representations are very noisy, i.e., much noise are removed, attention coefﬁcients on edges should be large to restore the groundtruth signal. Therefore, in the KI scenario, we can use attention coefﬁcients in graph attention in graph convolution layer to interpret the KI process such as how much triple information is integrated.
As for the CR and CF, equally, we can use the attention coefﬁcients on the self-loop edges for interpretation, such as how much original information is remembered/forgotten.
E MLP C AN BE B IJECTIVE (P ROOF )
Theorem 5.
Give an MLP layer denoted as MLP( x ) = sigmoid( W x + b ) .
If W is a square matrix, there exist a constant λ (cid:48) 0 > 0 that for any 0 < (cid:15) < λ (cid:48) 0 , the function below is bijective:
[Equation]
Proof.
We ﬁrst prove that two bijective function compositions are still bijective.
Then, we prove that adding a small noise on MLP weight matrix can make it bijective.
Give two function f 1 ( · ) and f 2 ( · ) .
Suppose they are injective and suppose f 1 ( f 2 ( x ))
=
f 1 ( f 2 ( y )) .
Since we know that f 1 ( · ) is injective, we have f 2 ( x ) =
f 2 ( y ) .
Similarly, since f 2 ( · ) is injective, we have x = y .
Thus f 1 ( f 2 ( · )) is injective.
Suppose f 1 ( · ) and f 2 ( · ) are surjective and z ∈ C .
Since we know that f 1 ( · ) is surjective, there exists a set of y ∈ B with f 1 ( y )
=
z .
Similarly, since f 2 ( · ) is surjective, there exists a set of x ∈ A with f 2 ( x ) =
y .
Then, we have z = f 1 ( f 2 ( x )) and so z is onto f 1 ( f 2 ( · )) .
Thus, f 1 ( f 2 ( · )) is surjective.
Therefore, if f 1 ( · ) and f 2 ( · ) are bijective, f 1 ( f 2 ( · )) is also bijective.
To prove that the special MLP is bijective, consider an MLP function as
[Equation]
where W ∈ R C × C is the weight matrix and b ∈ R C is the bias.
Let
[Equation]
i [Equation] be the characteristic polynomial for weight matrix W .
Here λ (cid:48)
i are eigenvalues of matrix W .
Without loss of generality, let | λ (cid:48)
| =
min i | λ (cid:48)
i | .
Then, we know that for any constant 0 < (cid:15) < | λ (cid:48)
| , we have
[Equation]
Thus, if the perturbation (cid:15) is small enough, the perturbed matrix W (cid:48)
=
W − (cid:15) I is nonsingular.
Consider the fact that the nonlinear activation function σ ( · ) is sigmoid( · ) function, which is bijective.
Therefore, the special MLP function MLP n ( · ) is bijective.
And there is no information loss.
F I MPLEMENTATION D ETAILS OF THE S IMULATION E XPERIMENT
DeepWalk. We implement DeepWalk to get KG embeddings. The hyperparameters are set as the same as its default values 10 : the number of walks is set as 10 , and the walk length is set as 40 .
The dimension of embeddings is set as 128 .
Note that since knowledge triples only contain neighbors within one-hop, we select the window size as 1 .
Noisy KG embeddings .
We add Gaussian noisy on the KG embeddings with different ratios.
For example, if we add 10% ratio of noise, it can be written as 10% noisy KGE = 0 .
9 ∗ KGE
+ 0 .
1 ∗ noise .
Similarity .
The Cosine similarity is implemented as
[Equation]
distance
[Equation]
, ) , and the Euclidean similarity is implemented as
[Equation]
Linear probe .
We design a linear classiﬁer to do link prediction for the KG.
Note that we cannot implement the linear classiﬁer on the large and sparse KG to do link prediction directly.
To get meaningful results, we do negative sampling with sample number as 5 .
Then, we select to report the
AUC ( Area Under the Curve ) score since other metrics cannot be differentiable under this condition.
We randomly split the edge set of the KG into two even sets for training and test. To reduce variance, we repeat the experiment for 10 times. GCS . We train our GCS model with input as noisy KGE and output as KGE.
Note that here noisy KGE and KGE are in the same space, and the task is simpler compared to practical KI interpretation. Thus, we set the epoch number as 10 , and use the reconstruction loss (i.e., mean absolute error) for efﬁciency.
Other hyperparameters are set as the same as introduced in Appendix H. G I MPLEMENTATION D ETAILS OF LM S KI. To ensure that the experiment settings are fair, we set hyperparameters as the default values.
For K-Adapter , the code and hyperparameters for KI that we use are from the ofﬁcial projectspublished by the authors (Wang et al., 2021a).
The only two differences are that: we use PyTorch ﬂoat 32 instead of ﬂoat 16 since BERT and RoBERTa that we use are ﬂoat32 , and we use 4 NVIDIA Tesla V100 GPUs for KI training.
For ERNIE , things are the same.
All hyperparameters for KI are set as their default values 12 .
Similarly, ﬂoat 16 of PyTorch is changed to ﬂoat 32 , and we do the integration with 4 NVIDIA Tesla V100 GPUs.
Note that the dataset that ERNIE used for KI is Wikipedia, since the code is to fetch latest version of it, the data that we use could be slightly different.
Therefore, for both ERNIE and K-Adapter, to ensure the fairness, we reproduce their KI, and report the results of reproduced models instead of results provided in their papers. Finetuning.
As for the downstream tasks, all the hyperparameters are consistent with the ofﬁcial project: either they are given in the project or in the README. In the same way, ﬂoat 32 and NVIDIA Tesla V100 GPUs are chosen to make sure that the comparison is fair. Note that for KAdapter and ERNIE, the best performance for different datasets is achieved in different settings.
For example, the best performance for K-Adapter on the OpenEntity dataset is achieved with single
GPU, but on the TACRED dataset is achieved with four GPUs.
Since we focus on the relative performance instead of the best one, we run ﬁnetuning on 4 NVIDIA Tesla V100 GPUs for all downstream tasks and all LMs (as well as BERT and RoBERTa).
[Table] The datasets that K-Adapter and ERNIE use are T-REx-rc and Wikidata, some statistics of them are given in Table 6.
H I MPLEMENTATION D ETAILS OF GCS
In this section, we introduce details of implementing GCS.
In practice, GCS is composed of 3 layers: bijective MLP layer, graph convolutional layer, and another bijective MLP layer.
As for bijective
MLP layers, since weight matrices in them are square matrices, the dimension would remain unchanged: 1024 for K-Adapter and 768 for ERNIE.
The nonlinear activation functions are set as ELU( · ) function, which is also bijective.
The learning rate is set as 1 e − 3 , and the dropout rate of the ﬁrst two MLP layers is 0 .
Regarding the graph attention, to make sure interpretation results are stable, we apply multi-head attention mechanism, where the number of attention head is set as 8 .
Entity representations are ﬁrst embedded into a space with the dimension as 64 .
Then, the embedded representations are used to calculate the attention coefﬁcients.
Note that since the purpose is to interpret and analyze the KI process, we do not split datasets for KI.
Considering that GCS model is very simple for large KGs, overﬁtting is unlikely to happen.
Thus, we optimize GCS for the whole datasets.
Speciﬁcally, for K-Adapter, the whole KG is used for optimization, and results are used for interpretation.
And for ERNIE, since the KG is very large, we sample a small subgraph with 1 , 344 , 393 entities and 3 , 240 , 272 triples for optimization (see Table 6), and then implement the optimized GCS on the whole KG for interpretation. The objective function of training GCS can be reconstruction loss minimization or MI maximization.
In this paper, except the Simulation Experiment, we all select MI maximization as the objective. For reconstruction loss minimization, we use the mean absolute error (MAE) between our GCS outputs and entity representations of the knowledge-enhanced LMs. Regarding the MI maximization, we deﬁne the objective function by maximizing the MI as [Equation]
We optimize MI equation 11 by maximizing the compression lemma lower bound (Banerjee, 2006) as in Belghazi et al.
The inputs of GCS are X , and let the output be denoted by Z .
We can regard Z and H as empirical samples of random variables z and h .
Thus, we have:
[Equation]
Here, F can be any class of functions T : Ω → R satisfying certain integrability constraints (Belghazi et al., 2018).
P zh represents the joint distribution of z and h , and P z ⊗ P h represents the product of their marginal distributions.
In practice, we let F = { T θ 2 } be the set of functions parameterized by a neural network, and optimize it by stochastic gradient descent.
Then, the objective function can be rephrased as
[Equation]
In equation 13, P |V| z represents the empirical distribution of z , i.e., Z . If the KG is very large, we can optimize the network by sampling a small subgraph of the KG. In practice, we simply add two MLPs layers to GCS for MI maximization.
The added two MLP layers may not be bijective, where the dimension would be ﬁrst reduced to 64 , then to 1 for MI maximization.
The nonlinear activation functions are all set as ELU( · ) function, which is also bijective.
For interpretation, we use the attention coefﬁcients on edges and self-loops to analyze the KI in terms of triples and entities.
Different from Schlichtkrull et al.
(2020) that specially designs a discrete function to mask edges that are not important, we simply introduce a temperature hyperparameter t and set it as t = 0 .
1 to make the attention coefﬁcient distribution hard 13 .
Thus, knowledge can be well clustered into learned and unlearned.
I A DDITIONAL S TATISTICS FOR I NTEGRATION E XPERIMENT
[Table]
J A DDITIONAL R ESULTS FOR GCS VERIFICATION
We gradually drop out sentences aligned to knowledge triples whose attention coefﬁcients are small for KI.
For comparison, we randomly drop same number of sentences named the random strategy.
We can see that dropping out KI datasets based on the interpretation results of GCS signiﬁcantly outperforms the random strategy.
It further supports that GCS can effectively interpret the KI situation for the K-Adapter. [Figure]
[Figure]
K K NOWLEDGE I NTEGRATION O VERVIEW
Figure 5 presents the empirical distributions of attention coefﬁcients for K-Adapter and ERNIE.
The above two subﬁgures show distributions on edges, interpreting the KI from triples.
While the bottom two subﬁgures show distributions of self-loops, illustrating the KI from entities.
We can ﬁnd that most knowledge triples are not integrated well for both K-Adapter and ERNIE (i
[Table]
e., a i,j < 0 . 1 ), while K-Adapter performs slightly better.
When it comes to entity-wise integration, in general, entity knowledge is also not integrated well.
We ﬁnd that both CR (i.e., a i,i > 0 . 9 ) and CF (i.e., a i,i < 0 . 1 ) happens for many entities, especially for ERNIE.
K-Adapter outperforms ERNIE since some entities are integrated well (i
[Table]
e., 0 .
4 < a i,i < 0 . 6 )
[Table]
L A DDITIONAL S TATISTICS FOR T ABLE
[Table]
M A DDITIONAL E XPERIMENT : C ASE S TUDY
[Table]
license
[Table]
95 , 2 . 47 ) × 10 −
This article incorporates public domain material from the United States Geological Survey document ”Wasson Rock” (content from the Geographic Names Information System ).
Ofﬁce held by head of government 0.46% 99.54% 0.00% 14,
CF, ( 0 . 91 ,
[Table]
33 ) ×
[Table]
Seven people served as governor of Colorado Territory over eight terms, appointed by the President of the United States .
Opposite of
[Table]
[Table]
It is well covered with deciduous and evergreen forests.
[Table]
The prison’s north wing is
[Table]
[Table]
[Table]
Knowledge of different entities (sorted by their popularity) has different CR and CF ratios (case study).
Table 9 reports the KI for K-Adapter in terms of entities.
We select three relations { “ Copyright license ”, “ Ofﬁce held by head of government ”, “ Opposite of ” } .
CR happens to most connected entities for relation “ Copyright license ”.
CF happens to most connected entities for relation “ Ofﬁce held by head of government ”.
Some connected entities are learned well for relation “ Opposite of ”.
We list 5 aligned sentences for the three relations, and report the Google Ngrams (year 2019) 14 of entities to show their popularity.
We ﬁnd that CF often happens to entities with large Google Ngrams such as “ left ” and “ President ”.
And CR often happens to entities with small
Google Ngrams.
Well-learned entities are also not very popular.
N A DDITIONAL E XPERIMENT : C ORRELATION
[Figure]
Can we improve the KI quality by simply increasing the size of our dataset?
Above results analyze KI in LMs in different ways.
However, there is a key question: can we simply improve the quality of KI by increasing the amount of our aligned training corpora?
We try to answer this question in Figure 6 which plots correlation between the attention coefﬁcients and the number of aligned sentences for knowledge triples in the dataset.
We ﬁnd that the Pearson correlation between the two is − 0 .
There is no positive correlation between the two variables.
This implies there is no apparent positive relationship between the KI quality and the size of the KI dataset.
It suggests that simply increasing the size of the aligned dataset alone may not improve KI, but, we might need more fundamental advances to push the state-of-the-art in KI forward.