KALA: Knowledge-Augmented Language Model Adaptation
Abstract
Pre-trained language models (PLMs) have achieved remarkable success on various natural language understanding tasks.
Simple ﬁne-tuning of PLMs, on the other hand, might be suboptimal for domain-speciﬁc tasks because they cannot possibly cover knowledge from all domains.
While adaptive pre-training of PLMs can help them obtain domain-speciﬁc knowledge, it requires a large training cost.
Moreover, adaptive pre-training can harm the PLM’s performance on the downstream task by causing catastrophic forgetting of its general knowledge.
To overcome such limitations of adaptive pre-training for PLM adaption, we propose a novel domain adaption framework for PLMs coined as K nowledge-
A ugmented L anguage model
A daptation ( KALA ), which modulates the intermediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts.
We validate the performance of our KALA on question answering and named entity recognition tasks on multiple datasets across various domains.
The results show that, despite being computationally efﬁcient, our KALA largely outperforms adaptive pre-training.
Code is available at: https://github.com/Nardien/KALA.
Introduction
Pre-trained Language Models (PLMs) (Devlin et al., 2019; Brown et al., 2020) have shown to be effective on various Natural Language Understanding (NLU) tasks.
Although PLMs aim to address diverse downstream tasks from various data sources, there have been considerable efforts to adapt the PLMs to speciﬁc domains —distributions over the language characterizing a given topic or genre (Gururangan et al., 2020)— for which the acquisition of domain knowledge is required to accurately solve the downstream tasks (e.g., Biomedical
Named Entity Recognition (Dogan et al., 2014)).
[Figure]
This problem, known as Language Model Adaptation , can be viewed as a transfer learning problem (Yosinski et al., 2014; Ruder, 2019) under domain shift, where the model is pre-trained on the general domain and the labeled distribution is available for the target domain-speciﬁc task.
The most prevalent approach to this problem is adaptive pre-training (Figure 2a) which further updates all parameters of the PLM on a large domain-speciﬁc or curated task-speciﬁc corpus, with the same pretraining strategy (e.g., masked language modeling) before ﬁne-tuning it on the downstream task (Beltagy et al., 2019; Lee et al., 2020; Gururangan et al., 2020).
This continual pre-training of a PLM on the target domain corpus allows it to learn the distribution of the target domain, resulting in improved performance on domain-speciﬁc tasks (Howard and
Ruder, 2018; Han and Eisenstein, 2019).
While it has shown to be effective, adaptive pretraining has obvious drawbacks.
First, it is computationally inefﬁcient.
Although a PLM becomes more powerful with the increasing amount of pretraining data (Gururangan et al., 2020), further pre-training on the additional data requires larger memory and computational cost as the dataset size grows (Bai et al., 2021).
Besides, it is difﬁcult to adapt the PLM to a new domain without forgetting the general knowledge it obtained from the initial pretraining step, since all pre-trained parameters are continually updated to ﬁt the domain-speciﬁc corpus during adaptive pre-training (Chen et al., 2020).
This catastrophic forgetting of the task-
[Figure] Medical Textbook)
[Figure]
KALA
[Figure]
DAPT
[Figure]
KALA
[Figure]
general knowledge may lead to the performance degradation on the downstream tasks.
In Figure 1, we show that adaptive pre-training with more training steps could lead to performance degeneration.
Thus, it would be preferable if we could adapt the PLM to the domain-speciﬁc task without costly adaptive pre-training.
To this end, we aim to integrate the domain-speciﬁc knowledge into the PLM directly during the task-speciﬁc ﬁne-tuning step, as shown in Figure 2b, eliminating the adaptive pre-training stage.
Speciﬁcally, we ﬁrst note that entities and relations are core building blocks of the domain-speciﬁc knowledge that are required to solve for the domain-speciﬁc downstream tasks.
Clinical domain experts, for example, are familiar with medical terminologies and their complex relations.
Then, to represent the domain knowledge consisting of entities and relations, we introduce the Entity Memory , which is the source of entity embeddings but independent of the PLM parameters (See Entity Memory in Figure 2b).
Then, we further exploit the relational structures of the entities by utilizing a Knowledge Graph (KG), which denotes the factual relationships between entities, as shown in Knowledge Graph of Figure 2b.
The remaining step is how to integrate the knowledge into the PLM during ﬁne-tuning.
To this end, we propose a novel layer named Knowledgeconditioned Feature Modulation (KFM, §3.2), which scales and shifts the intermediate hidden representations of PLMs by conditioning them with retrieved knowledge representations.
This knowledge integration scheme has several advantages.
First, it does not modify the original PLM architecture, and thus could be integrated into any PLMs regardless of their architectures.
Also, it only requires marginal computational and memory overhead, while eliminating the need of excessive further pre-training (Figure 1).
Finally, it can effectively handle unseen entities with relational knowledge from the KG, which are suboptimally embedded by adaptive pre-training.
For example, as shown in Figure 2, an entity restenosis does not appear in the training dataset for ﬁne-tuning, thus adaptive pre-training only implicitly infers them within the context from the broad domain corpus.
However, we can explicitly represent the unknown entity by aggregating the representations of known entities in the entity memory (i.e., in Figure 2, neighboring entities, such as asthma and pethidine , are used to represent the unseen entity restenosis ).
We combine all the previously described components into a novel language model adaptation framework, coined as K nowledge-
A ugmented
L anguage model A daptation (KALA) (Figure 3).
We empirically verify that KALA improves the performance of the PLM over adaptive pre-training on various domains with two knowledge-intensive tasks: Question Answering (QA) and Named Entity
Recognition (NER).
Our contribution is threefold:
• We propose a novel LM adaptation framework, which augments PLMs with entities and their relations from the target domain, during ﬁne-tuning without any further pre-training.
To our knowledge, this is the ﬁrst work that utilizes the structured knowledge for language model adaptation.
• To reﬂect structural knowledge into the PLM, we introduce a novel layer which scales and shifts the intermediate PLM representations with the entity representations contextualized by their related entities according to the KG.
• We show that our KALA signiﬁcantly enhances the model’s performance on domain-speciﬁc QA and NER tasks, while being signiﬁcantly more efﬁcient over existing LM adaptation methods.
Related Work Language Model Adaptation
Nowadays, transfer learning (Howard and Ruder, 2018) is a dominant approach for solving Natural Language Understanding (NLU) tasks.
This strategy ﬁrst pretrains a Language Model (LM) on a large and unlabeled corpus, then ﬁne-tunes it on downstream tasks with labeled data (Devlin et al., 2019).
While this scheme alone achieves impressive performance on various NLU tasks, adaptive pre-training of the
PLM on a domain-speciﬁc corpus helps the PLM achieve better performance on the domain-speciﬁc tasks.
For example, Lee et al.
(2020) demonstrated that a further pre-trained LM on biomedical documents outperforms the original LM on biomedical NLU tasks.
Also, Gururangan et al.
(2020) showed that adaptive pre-training of the PLM on the corpus of a target domain (Domain-adaptive
Pre-training; DAPT) or a target task (Task-adaptive
Pre-training; TAPT) improves its performance on domain-speciﬁc tasks.
However, above approaches generally require a large amount of computational costs for pre-training.
Knowledge-aware LM
Accompanied with increasing sources of knowledge (Vrandecic and
Krötzsch, 2014), some prior works have proposed to integrate external knowledge into PLMs, to enhance their performance on tasks that require structured knowledge.
For instance, ERNIE (Zhang et al., 2019) and KnowBERT (Peters et al., 2019) incorporate entities as additional inputs in the pretraining stage to obtain a knowledge-aware LM, wherein a pre-trained knowledge graph embedding from Wikidata (Vrandecic and Krötzsch, 2014) is used to represent entities.
Entity-as-Experts (Févry et al., 2020) and LUKE (Yamada et al., 2020) use the entity memory that is pre-trained along with the LMs from scratch.
ERICA (Qin et al., 2021) further uses the fact consisting of entities and their relations in the pre-training stage of LMs from scratch.
Previous works aim to integrate external knowledge into the LMs during the pre-training step to obtain a universal knowledge-aware LM that requires additional parameters for millions of entities.
In contrast to this, our framework aims to efﬁciently modify a general PLM for the domainspeciﬁc task with a linear modulation layer scheme discussed in Section 3.2, during ﬁne-tuning.
Method 3. Problem Statement
Our goal is to solve Natural Language Understanding (NLU) tasks for a speciﬁc domain, with a knowledge-augmented Language Model (LM).
We ﬁrst introduce the NLU tasks we target, followed by the descriptions of the proposed knowledgeaugmented LM.
After that, we formally deﬁne the ingredients for structured knowledge integration.
NLU tasks
The goal of an NLU task is to predict the label y of the given input instance x , where the input x contains the sequence of tokens (Devlin et al., 2019):
x = [ w 1 , w 2 , . .
, w | x | ] .
Then, given a training dataset D = { ( x ( i ) , y ( i ) ) } Ni =1 , the objective is to maximize the log-likelihood as follows:
[Equation]
where f is an encoder of the PLM which outputs contextualized representation H from x , and g is a decoder which models the probability distribution p of the label y , with trainable parameters θ
= ( θ f , θ g ) .
If the LM is composed of L-layers of transformer blocks (Devlin et al., 2019), the function f is decomposed to multiple functions f = [ f 0 , . . .
, f L ] , where each block gets the output of the previous block as the input: H l = f l ( H l − 1 ) .
Knowledge-Augmented Language Model
The conventional learning objective deﬁned above might be sufﬁcient for understanding the texts if the tasks require only the general knowledge stored in PLMs.
However, it is suboptimal for tackling domain-speciﬁc tasks since the general knowledge captured by the parameters θ f may not include the knowledge required for solving the domain-speciﬁc tasks.
Thus, contextualizing the texts by the domain knowledge, captured by the domain-speciﬁc entities and their relations, is more appropriate for handling such domain-speciﬁc problems.
To this end, we propose a function h ( · ; φ ) which augments PLMs conditioned on the domain knowledge.
Formally, the objective for a NLU task with our knowledge-augmented LM is given as follows:
[Equation]
where φ is parameters for the function h , E is the set of entities, M is the set of corresponding mentions, and G is a knowledge graph.
In the following, we will describe the deﬁnition of the knowledgerelated inputs E , M , G , and the details of h ( · , φ ) .
Deﬁnition 1 (Entity and Mention).
Given a sequence of tokens x = [ w 1 , . . .
, w | x | ] , let E be a set of entities in x .
Then an entity e ∈ E is composed of one or multiple adjacent tokens within the input text: [ w m α , . .
. , w m ω ] (cid:118) x 2 .
Here, m = ( m α , m ω ) is a mention that denotes the start and end locations for the entity within the input tokens x , which term is commonly used for deﬁning entities (Févry et al., 2020).
Consequently, for each given input x ( i ) , there are a set of entities
E ( i ) = { e 1 , . . .
, e K } and their corresponding mentions M ( i ) = { m 1 , . . .
, m K } .
For example, given an input x = [ New, York, is, a, city ] , we have two entities E
= New_York , city } and their associated mentions M =
We further construct the entity vocabulary
E train =
(cid:83) Ni =1 E
( i ) , which consists of all entities appearing in the training dataset.
However, at test time, we may encounter unseen entities that are not in E train .
To tackle this, we regard unknown entities as the null entity e ∅ , so that ∀ e ∈ E train ∪ { e ∅ } .
Deﬁnition 2 (Entity Memory).
Given a set of all entities E train ∪ { e ∅ } , we represent them in the continuous vector (feature) space to learn meaningful entity embeddings.
In order to implement this, we deﬁne the entity memory E ∈ R ( |E train | +1) ×
d that comprises of an entity e ∈ R as a key and its embedding e ∈ R d as its value.
Also, to access the value in the entity memory, we deﬁne the point-wise memory access function EntEmbed which takes an entity as an input.
For instance, e =
EntEmbed ( New_York ) returns the embedding of the New_York entity, and e = EntEmbed ( e ∅ ) returns the zero embedding.
This entity memory E is the part of the parameter φ used in function h .
Deﬁnition 3 (Knowledge Graph).
Since the entity memory alone cannot represent relational information between entities, we further deﬁne a
Knowledge Graph (KG) G that consists of a set of factual triplets { ( h, r, t ) } , where the head and the tail entities, h and t , are the elements of E , and a relation r is an element of a set of relations
R : h, t ∈ E and r ∈ R .
We assume that a preconstructed KG G ( i ) is given for each input x ( i ) , and provide the details of the KGs and how to construct them in Appendix A.
3. Knowledge-conditioned Feature Modulation on Transformer
The remaining problem is how to augment a PLM by conditioning it on the domain-speciﬁc knowledge, through the function h .
An effective approach to do so without stacking additional layers on top of the LM is to interleave the knowledge from h with the pre-trained parameters of the language model (Devlin et al., 2019) consisting of transformer layers (Vaswani et al., 2017).
Before describing our interleaving method in detail, we ﬁrst describe the Transformer architecture.
Transformer
Given | x | token representations
[Equation]
] ∈ R |
[Equation]
| [Equation] d from the layer l − 1 where d is the embedding size, each transformer block outputs the contextualized representations for all tokens.
In detail, the l -th block consists of the multi-head self-attention ( Attn ) layer and the residual feed-forward ( FF ) layer as follows:
[Equation]
where LN is a layer normalization (Ba et al., 2016), σ is an activation function (Hendrycks and Gimpel, 2016), W 2 ∈ R d (cid:48)
× d and W 1 ∈ R d × d (cid:48) are weight matrices, and d (cid:48) is an intermediate hidden size.
We omit the bias term for brevity.
Linear Modulation on Transformer
An effective yet efﬁcient way to fuse knowledge from different sources without modifying the original model architecture is to scale and shift the features of one source with respect to the data from another source (Dumoulin et al., 2018).
This scheme of feature-wise afﬁne transformation is effective on various tasks, such as language-conditioned image reasoning (Perez et al., 2018) or style-transfer in image generation (Huang and Belongie, 2017).
[Figure]
Entity
Memory
[Figure]
Relational Retrieval from Entity Memory
[Figure]
Motivated by them, we propose to linearly transform the intermediate features after the layer normalization of the transformer-based PLM, conditioned on the knowledge sources E , M , G .
We term this method as the Knowledge-conditioned Feature Modulation (KFM), described as follows:
[Equation] where H l − 1 ∈ R | x |× d is the matrix of hidden representations from the previous layer, ◦ denotes the hadamard (element-wise) product, and Γ =
[ γ 1 , . . . , γ | x | ] ∈ R | x |× d , B = [ β 1 , . . .
, β | x | ] ∈
R | x |× d . Γ and B are learnable modulation parameters from the function h , which are conditioned by the entity representation.
For instance, in Figure 3, γ and β for token ‘New’ are conditioned on the corresponding entity New_York .
However, if tokens are not part of any entity (e.g., ‘is’), γ and β for such tokens are ﬁxed to 1 and 0 , respectively.
One notable advantage of our KFM is that multiple tokens associated to the identical entity are affected by the same modulation (e.g., ‘New’ and ‘York’ in Figure 3), which allows the PLM to know which adjacent tokens are in the same entity.
This is important for representing the tokens of the domain entity (e.g., ‘cod’ and ‘on’), since the original
PLM might regard them as separate, unrelated tokens (See analysis in §5.5 with Figure 5).
However, with our KFM, the PLM can identify associated tokens and embed them to be close to each other.
Then, how can we design such functional operations in h ?
The easiest way is to retrieve the entity embedding of e , associated to the typical token, from the entity memory E , and then use the retrieved entity embedding as the input to obtain γ and β for every entity (See Figure 3).
Formally, for each entity e ∈ E and its mention ( m α , m ω ) ∈ M , v
[Equation]
EntEmbed
[Equation]
where v is the retrieved entity embedding from the entity memory, h 1 , h 2 , h 3 , and h 4 are mutually independent Multi-Layer Perceptrons (MLPs) which return a zero vector 0 if e = e ∅ .
3. Relational Retrieval from Entity Memory
Although the simple access to the entity memory can retrieve the necessary entity embeddings for the modulation, this approach has obvious drawbacks as it not only fails to reﬂect the relations with other entities, but also regards unseen entities as the same null entity e ∅ .
If so, all unseen entities are inevitably modulated by the same parameters even if they have essentially different meaning.
To tackle these limitations, we further consider the relational information between two entities that are linked with a particular relation.
For example, the entity New_York alone will not give meaningful information.
However, with two associated facts ( New_York , instance of , city ) and ( New_York , country , USA ), it is clear that New_York is a city in the
USA .
Motivated by this observation, we propose
Relational Retrieval which leverages a KG G to retrieve entity embeddings from the memory, according to the relations deﬁned in the given KG (See Figure 3, right).
More speciﬁcally, our goal is to effectively utilize the relations among entities in G , to improve the EntEmbed function in equation 2.
We tackle this objective by utilizing a Graph Neural Network (GNN) which learns feature representations of each node using a neighborhood aggregation scheme (Hamilton et al., 2017), as follows:
[Equation]
where N ( e ; G ) is a set of neighboring entities of the entity e , AGG is the function that aggregates embeddings of neighboring entities of e , and UPDATE is the function that updates the representation of e with the aggregated messages from AGG .
However, simple aggregation (e.g., mean) cannot reﬂect the relative importance on neighboring nodes, thus we consider the attentive scheme (Velickovic et al., 2018; Brody et al., 2021) for neighborhood aggregation, to allocate weights to the target entity’s neighbors by their importance.
This scheme is helpful in ﬁltering out less useful relations.
Formally, we ﬁrst deﬁne a scoring function ψ that calculates a score for every triplet ( e i , r ij , e j ) , which is then used to weigh each node during aggregation:
[Equation]
where σ is a nonlinear activation, e ∗ ∈ R 4 d is concatenated vector where (cid:107) denotes the concatenation,
a ∈ R d and W ∈ R d × 4 d are learnable parameters, r ij ∈ R
d is a embedding of the relation, and h e i ∈ R
d is a context representation of the entity e i obtained from the intermediate hidden states of the LM 3 .
The scores obtained from ψ are normalized across all neighbors e j ∈ N
( e i ; G ) with softmax:
[Equation]
Then, we update the entity embedding with a weighted average of the neighboring nodes with α as an attention coefﬁcient, denoted as follows:
[Equation]
By replacing the EntEmbed function in equation 2 with the above GNN in equation 3, we now represent each entity with its relational information in KG.
This relational retrieval has several advantages over simple retrieval of a single entity from the entity memory.
First, the relational retrieval with KG can consider richer interactions among entities, as described in Figure 3.
In addition, we can naturally represent an unseen entity – which is not seen during training but appears at test time – through neighboring aggregation, which is impossible only with the entity memory.
In Figure 2, we provide an illustrative example of the unseen entity representation, where the unseen entity restenosis is represented with a weighted sum of representations of its neighboring entities myocardial_infarction , asthma , and pethidine , which is beneﬁcial when the set of entities for training and test datasets have small overlaps.
Experiment 4. Tasks and Datasets
We evaluate our model on two NLU tasks: Question Answering (QA) and Named Entity Recognition (NER).
For QA , we use three domain-speciﬁc datasets: NewsQA (News, Trischler et al., 2017) and two subsets (Relation, Medication) of EMRQA (Clinical, Pampari et al., 2018).
We use the ExactMatch (EM) and the F1 score as evaluation metrics.
For NER , we use three datasets from different domains, namely CoNLL-2003 (News, Sang and
Meulder, 2003), WNUT-17 (Social Media, Derczynski et al., 2017) and NCBI-Disease (Biomedical, Dogan et al., 2014).
We use the F1 score as the evaluation metric.
We report statistics and detailed descriptions of each dataset in Appendix B.2.
4. Baselines
A direct baseline of our KALA is the adaptive pre-training, which is commonly used to adapt the
PLM independent to the choice of a domain and task.
Also, to compare ours against a more powerful baseline, we modify a recent method (Chen et al., 2020) that alleviates forgetting of PLM during ﬁne-tuning.
Details for each baseline we use are described as follows:
Vanilla Fine-Tuning (FT) :
A baseline that directly ﬁne-tunes the LM on downstream tasks.
Fine-Tuning + more params : A baseline with one more transformer layer at the end of the [Table] [Table]
LM.
We use this baseline to show that the performance gain of our model does not come from the use of additional parameters.
Task-Adaptive Pre-training (TAPT) :
A baseline that further pre-trains the PLM on taskspeciﬁc corpus as in Gururangan et al.
TAPT + RecAdam : A baseline that uses RecAdam (Chen et al., 2020) during further pre-training of PLMs (i [Table]
e., TAPT), to alleviate catastrophic forgetting of the learned general knowledge in PLMs from adaptive pre-training.
Domain-Adaptive Pre-training (DAPT) :
A strong baseline that uses a large-scale domain corpus outside the training set during further pretraining (Gururangan et al., 2020), and requires extra data and large computational overhead.
KALA (pointwise): A variant of KALA that only uses the entity memory and does not use the knowledge graphs.
7. KALA (relational): Our full model that uses KGs to perform relational retrieval from the entity memory.
4. Experimental Setup
We use the uncased BERT-base (Devlin et al., 2019) as the base PLM for all our experiments on QA and NER tasks.
For more details on training and implementation, please see the Appendix B.
4. Experimental Results Performance on QA and NER tasks
On both extractive QA and NER tasks, our KALA outperforms all baselines, including TAPT and
TAPT+RedcAdam (Gururangan et al., 2020; Chen et al., 2020), as shown in Table 1 and 2.
These results show that our KALA is highly effective for the language model adaptation task.
KALA also largely outperforms DAPT (Gururangan et al., 2020) which is trained with extra data and requires a signiﬁcantly higher computational cost compare to KALA (See Figure 1 for the plot of efﬁciency, discussed in Section 5.3).
Effect of Using more Parameters
One may suspect whether the performance of our KALA comes from the increment of parameters.
However, the experimental results in Table 1 and 2 show that increasing the parameters for PLM during ﬁne-tuning (+ more params) yields marginal performance improvements over naive ﬁne-tuning.
This result conﬁrms that the performance improvement of KALA is not due to the increased number of parameters.
Importance of Relational Retrieval
The performance gap between KALA (relational) and KALA (point-wise) shows the effectiveness of relational retrieval for language model adaptation, which allows us to incorporate relational knowledge into the PLM.
The relational retrieval also helps address unseen entities, as discussed in Section 5.4.
Analysis and Discussion 5. Ablation Studies
We perform an ablation study to see how much each component contributes to the performance gain.
KFM Parameters
We ﬁrst analyze the effect of feature modulation parameters (i.e., gamma and beta) in transformers by ablating a subset of them in Table 3, in which we observe that using both [Table] [Table] [Table]
gamma and beta after both layer normalization on a transformer layer obtains the best performance.
Architectural Variants
We now examine the effectiveness of the proposed knowledge conditioning scheme in our KALA framework.
To this end, we use or adapt the knowledge integration methods from previous literature, to compare their effectiveness.
Speciﬁcally, we couple the following ﬁve components with KALA: Entity-as-Experts (Févry et al., 2020),
Adapter (Houlsby et al., 2019), KTNet (Yang et al., 2019), ERNIE (Zhang et al., 2019), and ERICA (Qin et al., 2021).
Note that, most of them were proposed for improving pre-training from scratch, while we adapt them for ﬁne-tuning under our KALA framework (The details are given in Appendix B.4).
As shown in Table 4, our KFM used in KALA outperforms all variants, demonstrating the effectiveness of feature modulation in the middle of transformer layers for ﬁne-tuning. 5. Robustness to Other PLMs
Although we believe our experimental results on
Table 1, 2 with BERT (Devlin et al., 2019) are enough to show the effectiveness of KALA across different pre-trained language models (PLMs), one might be curious that KALA can work on even other PLMs such as RoBERTa (Liu et al., 2020). Thus, to address such concerns, we additionally conduct experiments on RoBERTa.
As shown in
Table 5, we observe that our KALA outperforms all baselines except for one case (Fine-Tuning +
[Figure]
[Table] more params on NewsQA). Thus, we believe that our KALA would be useful to any PLMs, not depending on speciﬁc PLMs.
5. Efﬁciency
Figure 1 illustrates the performance and training
FLOPs of KALA against baselines on the NewsQA dataset.
We observe that the performance of TAPT decreases with the increased number of iterations, which could be due to forgetting of the knowledge from the PLM.
On the other hand, DAPT, while not suffering from performance loss, requires huge computational costs as it trains on 112 times larger data for further pre-training (See Appendix B. for detailed explanations on training data).
On the other hand, our KALA outperforms DAPT without using external data, while requiring 17 times fewer computational costs, which shows that KALA is not only effective but also highly efﬁcient.
To further compare the efﬁciency in various aspects, we report GPU memory, training wall time, and training FLOPs for baselines and ours in Table 6. Through this, we verify that our KALA is more efﬁcient to train for language model adaptation settings than baselines. Note that the resource requirement of KALA could be further reduced by adjusting the size of the entity memory (e.g., removing less frequent entities). Therefore, to show the ﬂexibility of our KALA on the typical resource constraint, we provide the experimental results on two different settings (i.e., tuning the number of entities in the entity memory) – KALA with memory size of 200 and 62.8k (full memory) in Appendix C.6.
Context: A nonsense mutation in exon 17 ( codon 556 ) of the RB gene was found to be present homozygously in both the retinal and the pineal tumours.
Fact: (retinal, instance of, gene) [Figure] Fine-Tuning [Figure]
KALA (Ours)
NewsQA T5-small
EM
F
Fine-tuning 48.
64.24 TAPT 48.
64.30 + RecAdam 48.
KALA (ours) 51.
[Figure]
5. Effectiveness on Unseen Entities
One remarkable advantage of our KALA is its ability to represent an unseen entity by aggregating features of its neighbors from a given KG.
To analyze this, we ﬁrst divide all contexts into one of
Seen and Unseen, where Seen denotes the context with less than 3 unseen entities, and then measure the performance on the two subsets.
As shown in
Figure 4, we observe that the performance gain of KALA over the baselines is much larger on the Unseen subset, which demonstrates the effectiveness of KALA’s relational retrieval scheme to represent unseen entities.
DAPT also largely outperforms ﬁne-tuning and TAPT as it is trained on an extremely large external corpus for adaptive pre-training.
However, KALA even outperforms
DAPT in most cases, verifying that our knowledgeaugmentation method is more effective for tackling domain-speciﬁc tasks.
The visualization of embeddings of seen and unseen entities in Figure 2 shows that KALA embeds the unseen entities more closely to the seen entities 4 , which explains
KALA’s good performance on the Unseen subset.
5. Case Study
To better see how our KFM (§3.2) works, we show the context and its fact, and then visualize representations from the PLM modulated by the KFM.
As shown in Figure 5 right, the token ‘##on’ is not aligned with their corresponding tokens, such as ‘ex’ (for exon ) and ‘cod’ (for codon ), in the baseline.
However, with our feature modulation that transforms multiple tokens associated with the single entity equally, the two tokens (e.g., (‘ex’, ‘##on’)), composing one entity, are closely embedded.
Also, while the baseline cannot handle the unseen entity consisting of three tokens: ‘re’, ‘##tina’, and ‘##l’,
KALA embeds them closely by representing the unseen retinal from the representation of its neighborhood gene derived by the domain knowledge – ( retinal , instance of , gene ).
5. Extension to Generative Model
Our KALA framework is also applicable to encoder-decoder PLMs by applying the KFM to the encoder.
Therefore, we further validate KALA’s effectiveness on the encoder-decoder PLMs on the generative QA task (Lee et al., 2021) with T5small (Raffel et al., 2020).
Table 6 shows that
KALA largely outperforms baselines even with such a generative PLM.
Conclusion
In this paper, we introduced KALA, a novel framework for language model adaptation, which modulates the intermediate representations of a PLM by conditioning it with the entity memory and the relational facts from KGs.
We validated KALA on various domains of QA and NER tasks, on which
KALA signiﬁcantly outperforms relevant baselines while being computationally efﬁcient.
We demonstrate that the success of KALA comes from both
KFM and relational retrieval, allowing the PLM to recognize entities but also handle unseen ones that might frequently appear in domain-speciﬁc tasks.
There are many other avenues for future work, including the application of KALA on pre-training of knowledge-augmented PLMs from scratch.
Ethical Statements
Enhancing the domain converge of pre-traind language models (PLMs) with external knowledge is increasingly important, since the PLMs cannot observe all the data during training and cannot memorize all the necessary knowledge for solving down-stream tasks. Our KALA contributes to this problem by augmenting domain knowledge graphs for PLMs. However, we have to still consider the accurateness of knowledge, i.e., the fact in the knowledge graph may not be correct, which affects the model to generate incorrect answers. Also, the model’s prediction performance is still far from optimal.
Thus, we should be aware of model’s failure from errors in knowledge and prediction, especially on high-risk domains (e.g., biomedicine).
Acknowledgement
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artiﬁcial Intelligence Graduate School Program (KAIST) and No. 2021-0-02068, Artiﬁcial Intelligence Innovation Hub)), AITRICS, Samsung
Electronics (IO201214-08145-01), and the Engineering Research Center Program through the
National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921).
A Details on KG Construction
In this work, we propose to use the Knowledge
Graph (KG) that can deﬁne the relational information among entities that only appear in each dataset.
However, unfortunately, most of the task datasets do not contain such relational facts on its context, thus we need to construct them manually to obtain the knowledge graph.
In this section, we explain the way of constructing the knowledge graph that we used, consisting of facts of entities for each context in the task dataset.
Relation extraction is the way how we obtain the factual knowledge from the text of the target dataset.
To do so, we ﬁrst need to extract entities and their corresponding mentions from the text, and then link it to the existing entities in wikidata (Vrandecic and Krötzsch, 2014).
In order to do this, we use the existing library named as spaCy 5 , and opensourced implementation of Entity Linker 6 .
To sum up, in our work, a set of entities E ( i ) and corresponding mentions M ( i ) for the given input x ( i ) are obtained through this step.
Regarding a concrete example, please see format (a) in Figure 7.
In the example, “Text” indicates the entity mention within the input x , the “start” and “end” indicates its mention position denoted as ( m α , m ω ) , and “id” indicates the wikidata id for the entity identiﬁcation used in the next step.
To extract the relation among entities that we obtained above, we use the scheme of Relation Extraction (RE).
In other words, we use the trained
RE model to build our own knowledge base (KB) instead of using the existing KG directly from the existing general-domain KB 7 .
Speciﬁcally, we ﬁrst ﬁne-tune the BERT-base model (Devlin et al., 2019) for 2 epochs with 600k distantly supervised data used in Qin et al.
(2021), where the Wikipedia document and the Wikidata triplets are aligned.
Then, we use the ﬁne-tuned BERT model to extract the relations between entity pairs in the text.
We use the model with a simple bilinear layer on top of it, which is widely used scheme in the relation extraction literature (Yao et al., 2019).
For an example of the extracted fact, please see format (b) in Figure 7.
In the example, “h” denotes the wikidata id of the head entity, “r” denotes the wikidata id of the extracted relation, and “t” denotes the wikidata id of the tail entity.
In the relation extraction, the model returns the categorical distribution over the top 100 frequent relations.
In general, the relation of top-1 probability is used as the relation for the corresponding entity pair.
However, this approach sometimes results in predicting no_relation on most entity pairs.
Thus, to obtain more relations, we further use the relation of top-2 probability in the case where no_relation has a top-
[Table]
probability
but the top-2 probability is larger than a certain threshold (e.g., > 0 . 1 ).
In Figure 7, we summarize our KG construction pipeline.
In Table 7, we report the hyperparameters related to our
KG construction.
B Experimental Setup
In this section, we introduce the detailed setups for our models and baselines used in Table 1, 2, and 4.
B. Implementation Details
We use the Pytorch (Paszke et al., 2019) for the implementation of all models.
Also, to easily implement the language model, we use the huggingface library (Wolf et al., 2020) containing various transformer-based pre-trained language models (PLMs) and their checkpoints.
Details for KALA
In this paragraph, we describe the implementation details of the components, such as four linear layers in the proposed
KFM, architectural speciﬁcations in the attentionbased GNN, and initialization of both the entity memory and relational embeddings, in the following.
In terms of the functions h 1 , h 2 , h 3 , and h 4 in the KFM of Equation 2, we use two linear layers with the ReLU (Nair and Hinton, 2010) activation function, where the dimension is set to 768 .
For relational retrieval, we implement the novel
GNN model based on GATv2 (Brody et al., 2021) provided by the torch-geometric package (Fey and
Lenssen, 2019).
Speciﬁcally, we stack two GNN layers with the RELU activation function and also use the dropout with a probability of 0.1.
For attention in our GNN, we mask the nodes of the null entity, so that the attention score becomes zero for them.
Moreover, to obtain the context representation of the entity (See Footnote 3 in the main paper) used in the GNN attention, we use the scatter operation 8 for reduced computational cost.
For Entity Memory, we experimentally found that initializing the embeddings of the entity memory with the contextualized features obtained from the pre-trained language model could be helpful.
Therefore, the dimension of the entity embedding is set to the same as the language model d = 768 .
For relation embeddings, we randomly initialize them, where the dimension size is set to 128.
Location of KLM in the PLM
Note that, the number and location of the KFM layers inside the
PLM are hyperparameters.
However, we empirically found that inserting one to three KFM layers at the end of the PLM (i.e., after the 9th - 11th layers of the BERT-base language model) is beneﬁcial to the performance (See Appendix C.4 for experiments on diverse layer locations).
B. Dataset Details
Here we describe the dataset details with its statistics for two different tasks: extractive question answering (QA) and named entity recognition (NER).
Question Answering
We evaluate models on three domain-speciﬁc datasets: NewsQA, Relation, and Medication.
Notably, NewsQA (Trischler et al., 2017) is curated from CNN news articles.
Relation and Medication are originally part of the emrQA (Pampari et al., 2018), which is an automatically constructed question answering dataset based on the electrical medical record from n2c challenges 9 .
However, Yue et al.
(2020) extract two major subsets by dividing the entire dataset into Relation and Medication and suggest the usage of sampled questions from the original emrQA dataset.
Following the suggestion of Yue et al.
(2020), we use only 1% of generated questions of
Relation for training, validation, and testing.
Also, we only use 1% of generated questions of Medication for training and use 5% of generated questions of Medication for validation and testing.
Since the original emrQA is automatically generated based on templates, the quality is poor – it means that the original emrQA dataset was inappropriate to evaluate the ability of the model to reason over the clinical text since the most of questions can be
Hyperparameters
[Table] [Table]
answered by the simple text matching.
To overcome this limitation, Yue et al.
(2020) suggests two ways to make the task more difﬁcult.
First, they divide the question templates into easy and hard versions and then use the hard question only.
Second, they suggest replacing medical terminologies in the question of the test set into synonyms to avoid the trivial question which can be solvable with a simple text matching.
We use both methods to Relation and Medication datasets to report the performance of every model.
For more details on
Relation and Medication datasets, please refer to the original paper (Yue et al., 2020).
The statistics of training, validation, and test sets on all QA datasets are provided in Table 8.
Named Entity Recognition
We use three different domain-speciﬁc datasets for evaluating our
KALA on NER tasks: CoNLL-2003 (Sang and
Meulder, 2003) (News), WNUT-17 (Derczynski et al., 2017) (Social Network Service) and NCBIDisease (Dogan et al., 2014) (Biomedical).
The
CoNLL-2003 is constructed from the manually curated 1,393 English news articles, including 301.4k tokens, which has 9 class labels.
The WNUT17 dataset consists of 65,124 emerging and rare entities from social media (e.g., Twitter, Reddit,
YouTube, to name a few), which has 13 class la-
[Table]
[Figure]
bels.
The NCBI-Disease dataset consists of the 793 PubMed articles from the biomedical domain, which contains 6,892 disease mentions anddisease concepts, and also has 3 class labels.
The statistics of training, validation, and test sets are provided in Table 10.
B. Training details
All experiments are constrained to be done with a single 12GB Geforce RTX 2080 Ti GPU for fairness in terms of memory and the availability on the academic budget, except for the DAPT and generative QA which use a single 48GB Quadro 8000 GPU.
KALA training needs 3 hours in wall time with a single GPU.
For all experiments, we select the best checkpoint on the validation set.
For the summary of training setups, please see Tableand 11.
Fine-tuning Setup
In the following three paragraphs, we explain the setting of ﬁne-tuning for
QA, NER, and generative QA tasks.
For all experiments on extractive QA tasks, we ﬁne-tune the
Pre-trained Language Model (PLM) for 2 epochs with the weight decay of 0.01, learning rate of 3e-5, maximum sequence length of 384, batch size of 12, linear learning rate decay of 0.06 warmup rate, and half precision (Micikevicius et al., 2018).
For all experiments on NER tasks, we ﬁnetune the PLM for 20 epochs, where the learning rate is set to 5e-5, maximum sequence length is set to 128, and batch size is set to 32.
We use
AdamW (Loshchilov and Hutter, 2019) as an optimizer using BERT-base as the PLM.
For the generative QA task in Table 6, we ﬁnetune the T5-small (Raffel et al., 2020) for 4 epochs with the learning rate of 1e-4, maximum sequence length of 512, and batch size of 64.
We also use the Adafactor (Shazeer and Stern, 2018) optimizer.
Instead of training with the same optimizer as in
BERT for QA and NER, we instead use the independent AdamW optimizer with the learning rate of 1e-4 and weight decay of 0.01 to train the KALA module with T5.
Adaptive Pre-training Setup
In this paragraph, we describe the experimental settings of adaptive pre-training baselines, namely TAPT, TAPT (+
RecAdam), and DAPT.
For QA tasks, we further pre-train the PLM for {1,3,5,10} epochs and then report the best performance among them.
Speciﬁcally, reported TAPT result on NewsQA, Relation, and Medication are obtained by 1 epoch of further pre-training.
We use the weight decay of 0.01, learning rate of 5e-5, maximum sequence length of 384, batch size of 12, and linear learning rate decay of 0.06 warmup rate, with a half-precision.
Also, the masking ratio for the pre-training objective is set to 0.15, following the existing strategy introduced in the original BERT paper (Devlin et al., 2019).
For NER tasks, we further pre-train the PLM for 3 epochs across all datasets.
In particular, the learning rate is set to 5e-5, batch size is set to 32, and the maximum sequence length is set to 128.
We also use AdamW (Loshchilov and Hutter, 2019) as the optimizer for all experiments.
In the case of T5-small for generative QA in Table 6, we further pre-train the PLM for 4 epochs with the learning rate of 0.001, batch size of 64, maximum sequence length of 384, and Adafactor (Shazeer and Stern, 2018) optimizer.
Regarding the setting of TAPT (+ RecAdam) on all tasks, we follow the best setting in the original paper (Chen et al., 2020) – sigmoid as an annealing function with annealing parameters: k = 0 .
5 , t 0 = 250 , and the pretraining coefﬁcient of 5000.
For training with DAPT, we need an external corpus having a large amount of data for adaptive pre-training.
Thus, we ﬁrst choose the datasets of two domains – News and Medical.
Speciﬁcally, as the source of corpus for the News domain, we use the sampled set of 10 million News from the
RealNews dataset used in Gururangan et al.
As the source of corpus for the Medical domain, we use the set of approximately 100k passages from the Medical textbook provided in Jin et al.
The size of pre-training data used in DAPT is much larger than TAPT.
In other words, for experiments on NewsQA, TAPT only uses ﬁne-tuning contexts containing 5.8 million words from the NewsQA training dataset, while DAPT uses more than a hundred times larger data – enormous contexts containing about 618 million words from the RealNews database.
For both News and Medical domains, we further pre-train the BERT-base model forepochs with the batch size of 64, to match the similar computational cost used in Gururangan et al.
Other experimental details are the same as
TAPT described above.
B. Architectural Variant Details
In this subsection, we describe the details of architectural variants reported in Section 5.1.
For all variants, we use the same KGs used in KALA.
Entity-as-Experts (Févry et al.
(2020); EaE) utilizes the entity memory similar to our work, but they use the parametric dense retrieval more like the memory neural network (Sukhbaatar et al., 2015).
Similar to Févry et al.
; Verga et al.
(2021), we change the formulation of query and memory retrieval by using the mention representation of the entity from the intermediate hidden states of PLMs, which is formally deﬁned as follows:
[Equation]
where h e represents the average of token representations of the entity mention m = ( m α , m ω ) .
We also give the supervised retrieval loss ( ELLoss in Févry et al.
(2020)), when training the EaE model.
With this retrieval, EaE also can represent the unseen entity e / ∈ E train if we know the mention boundary of the given entity on the context.
We believe it is expected to work well, if the entity memory is pre-trained on the enormous text along with the pre-training of the language model from the scratch.
However, it might underperform for the language model adaptation scenario, since it can fall into the problem of circular reasoning – the PLM does not properly represent the unseen entity, but it should predict which entity it is similar from the representation.
Regarding the integration of the knowledge from the entity memory into the
PLM, the retrieved entity representation v is simply added (Peters et al., 2019) to the hidden representations H after the transformer block as follows:
[Equation]
where h is Multi-Layer Perceptrons (MLPs).
Adapter (Houlsby et al., 2019) is introduced to ﬁne-tune the PLM only with a few trainable parameters, instead of ﬁne-tuning the whole parameters of the PLM.
To adapt this original implementation into our KALA framework, we replace our
Knowledge-conditioned Feature Modulation with it, where the Adapter is used as the knowledge integration module.
We interleave the layer of Adapter after the feed-forward layer ( F F ) and before the residual connection of the transformer block.
Also, instead of only providing the LM hidden states as an input, we concatenate the knowledge representation in Equation 3 to the LM hidden states.
Note that we ﬁne-tune the whole parameters following our KALA setting, unlike ﬁne-tuning the parameters of only Adapter layers in Houlsby et al.
ERNIE (Zhang et al., 2019) is a notable PLM model that utilizes the external KB as an input for the language model.
The key feature of ERNIE can be summarized into two folds.
First, they use the multi-head self-attention scheme (Vaswani et al., 2017) to contextualize the input entities.
Second,
ERNIE fuses the entity representation at the end of the PLM by adding it to the corresponding language representation.
We assume that those two features are important points of ERNIE.
Therefore, instead of using a Graph Neural Network (GNN) layer, we use a multi-head self-attention layer to contextualize the entity embeddings.
Then, we add it to a representation of the entity from the PLM, which is the same as the design in equation 5.
KT-Net (Yang et al., 2019) uses knowledge as an external input in the ﬁne-tuning stage for extractive
QA.
Since they have a typical layer for integrating existing KB (Miller, 1995; Carlson et al., 2010) with the PLM, we only adopt the self-matching layer as the architecture variant of the KFM layer used in our KALA framework.
The computation of the self-matching matrix in KT-Net is costly, i.e., it requires a large computational cost that is approximately 12 times larger than KALA.
ERICA (Qin et al., 2021) uses contrastive learning in LM pre-training to reﬂect the relational knowledge into the language model.
We use the
Entity Discrimination task from ERICA on the primary task of ﬁne-tuning.
We would like to note that, as reported in Section 5 of the original paper (Qin et al., 2021), the use of ERICA on ﬁne-tuning has no effect, since the size and diversity of entities and relations in downstream training data are limited.
Such limited information rather harms the performance, as it can hinder the generalization.
In other words, contrastive learning cannot reﬂect the entity and relation in the test dataset.
B. FLOPs Computation
In this subsection, we give detailed descriptions of how the FLOPs in Figure 1 are measured.
We majorly follow the script from the ELECTRA (Clark et al., 2020) repository to compute the approximated FLOPs for all models including ours.
For
FLOPs computation of our KALA, we additionally include the FLOPs of the entity embedding layer, linear layers for h 1 , h 2 , h 3 , h 4 , and GNN layer.
Since the GNN layer is implemented based on the sparse implementation, we ﬁrst calculate the FLOPs of the message propagation over one edge, and then multiply it to the average number of edges per node.
Also, in terms of the computation on mentions, we consider the maximum sequence length of the context rather than the average number of mentions, to set the upper bound of FLOPs for our KALA.
Note that, in NewsQA training data, the average number of nodes is 57, the average number of edges for each node is 0.64, and the average number of mentions in the context is 92.68.
C Additional Experimental Results
In this section, we provide the analyses on the forgetting of TAPT, entity memory, number of entities and facts, location of the KLM layer, and values of
Gamma and Beta.
[Figure]
T r a i n i n g
M L M
L o ss 2.
T e s t
M L M
L o ss
[Figure]
c
[Figure]
r e
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure]
G P U
M
[Figure]
m o r y 1e
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
C. Analysis on forgetting of TAPT
In Figure 1, we observe that the performance of
TAPT decreases as the number of training steps increases.
To get a concrete intuition on this particular phenomenon, we analysis what happens in the Pre-trained Language Model (PLM), when we further pre-train it on the task-speciﬁc corpus.
Speciﬁcally, in Figure 8, we visualize the Masked
Language Model (MLM) loss of TAPT on both domain-speciﬁc corpus from the Relation dataset and general corpus from the sampled Wikipedia documents during the adaptive pre-traing.
As Figure 8 shows, the test MLM loss increases while the training MLM loss persistently increases as the training step increases.
This result indicates that
TAPT on domain-speciﬁc corpus may yield the catastrophic forgetting of the general knowledge in the PLM.
C. Effects of the Size of Entity Memory
In this subsection, we analyze how the size of entity memory affects the performance of our KALA.
In Figure 9, we plot the performance of KALA on the NewsQA dataset by varying the number of entity elements in the memory.
Note that, we reduce the size of the entity memory by eliminating the entity appearing fewer times.
Thus, the results are obtained by only considering the entities that appear more than [1000 , 100 , 10 , 5 , 0] times, e.g., 0 means the model with full entity memory.
As shown in Figure 9, we observe that the size of the 1 2 3 4 5 6 7 Number of entities I m p r o v e m e n t
CONLL
[Figure]
2003WNUT
[Figure]
17NCBI-Disease
[Figure]
[Figure]
Number of
[Figure]
I m p r
o v e m e n
[Figure]
[Figure]
entity memory is larger, the performance of our
KALA is better in general.
However, interestingly, we also observe that the smallest size of the entity memory shows decent performance, which might be due to the fact that some parameters in the entity memory are stale.
For more discussions on it including visualization, please refer to Appendix D.2.
Finally, we would like to note that, in Figure 1, we report the performance of our KALA in the case of [1000 , 5 , 0] (i.e., considering entities appearing more than [1000 , 5 , 0] times).
C.
Effects of the Number of Entity and Fact
In this subsection, we aim to analyze which numbers of entities and facts per context are appropriate to achieve good performance in NER tasks.
Specifically, we ﬁrst collect the contexts having more than or equal to the k number of entities (or facts), and then calculate the performance difference from our KALA to the ﬁne-tuning baseline.
As shown in Figure 10, while there are no obvious patterns, performance improvements from the baseline are consistent across a varying number of entities and facts.
This result suggests that our KALA is indeed beneﬁcial when entities and facts are given to the model, whereas the appropriate number of entities and facts to obtain the best performance against the baseline is different across datasets.
[Figure] [Figure]
C. Effects of the Location of KFM
In the main paper and Appendix B. [Figure] we describe that the location of the KFM layer inside the PLM architecture is the hyperparameter.
However, someone might wonder which location of KFM yields the best performance [Figure] and what is the reason for this.
Therefore, in this section [Figure] we analyze where we obtain the best performance in various locations of the KFM layer on the NewsQA dataset.
Specifically, in Figure [Figure] we show the performance of our KALA with varying the location of the KFM layer insider the BERT-base model.
The results demonstrate that the model with the KFM on the last layer of the BERT-base outperforms all the other choices.
This might be because, as the ﬁnal layer of the PLM is generally considered as the most task-speciﬁc layer, our KFM interleaved in the latest layer of BERT expressively injects the task-speciﬁc information from the entity memory and KGs, to such a task-speciﬁc layer.
C. Analysis on Values of Gamma and Beta
To see how much amount of value on gamma and beta is used to shift and scale the intermediate hidden representations in transformer layers, we visualize the modulation values, namely gamma and beta, in Figure 12.
We ﬁrst observe that, as shown in Figure 12, the distribution of values of gamma and beta approximately follow the Gaussian distribution, with zero mean for beta and one mean for gamma.
Also, we notice that the scale of values remain nearly around the mean point, which suggests that the small amount of shifting to intermediate hidden representations on transformer layers is enough to contribute to the performance gain, as we can see in the main results of Table 1, 2.
C. Detailed Efﬁciency Comparison
While we provide the efﬁciency on FLOPs in Figure 1, we further provide the efﬁciency on GPU memory, wall time, and FLOPs for training each method in Table 6.
Speciﬁcally, we measure the computational cost on the NewsQA dataset with
BERT-base, where we use the single Geforce RTX 2080 Ti GPU on the same machine.
For our KALA, as we can ﬂexibly manage the cost of GPU memory by reducing the number of entities in entity memory (See Figure 9 with Appendix C.2 for more analysis on the effects of the size of entity memory), we provide the experimental results on two settings – KALA with memory size 0.2k and 62.8k (full memory). As
shown in Table we conﬁrm that the computational cost of our KALA with the full memory is similar to the cost of the more params baseline that uses one additional transformer layer on top of BERT-base.
However, by reducing the number of entities in the memory, we can achieve better efﬁciency than more params in terms of GPU memory and FLOPs.
Also, we observe that the training cost (i.e., Wall Time and FLOPs) of TAPT and DAPT is high, especially on DAPT, thus we verify that our KALA
is more efﬁcient to train for domain adaptation settings.
D Additional Visualization Results
Here we provide the frequency distribution of entities, additional case studies, and more illustrations of textual examples and embedding spaces.
D. Additional Representation Visualization
While we already show the contextualized representations of seen and unseen entities in the latent [Figure] space in Figure 2 right, we further visualize them including the missing baselines of Figure 2, such as Fine-tuning or TAPT, in Figure 13 on the NCBIDisease dataset.
Similar to Figure 2, we observe that all baselines fail to closely embed the unseen entities in the representation space of seen entities.
While this visualization result does not give a strong evidence of why our KALA outperforms other baselines, we clearly observe that KALA is beneﬁcial to represent unseen entities in the feature space of seen entities, which suggests that such an advantage of our KALA helps the PLM to generalize over the test dataset, where the context contains unseen entities.
D. Entity Frequency Distribution
We visualize the frequency of entities in Figureand 15.
The entity frequency denotes the number of mentions of their associated entities within the entire text corpus of the training dataset.
As shown in Figure 14 and 15 of QA and NER datasets, the entity frequency follows the long-tail distribution, where most entities appear a few times.
For instance, in the NewsQA dataset, more than 20k entities among entire 60k entities appear only once in the training dataset, whereas one entity ( CNN 10 ) appears approximately 20k times.
This observation suggests that most of the elements in the entity memory are not utilized frequently.
In other words, only few entities are accurately trained with many training instances, whereas there exists the stale embeddings which are rarely updated.
This observation raises an interesting research question on the efﬁcient usage of the entity memory, as we can see in Figure 9 that the small size of entity memory could result in the better performance (See Appendix C.2).
We leave the more in-depth analysis on the entity memory as the future work.
[Figure]
D. Additional Case Study
In addition to the case study in Figure 5, we further show the case on the question answering task in Figure 16, like in Section 5.5, With this example, we explain how the factual knowledge in KGs could be utilized to solve the task via our KALA.
The question in the example is “who was kidnapped because of her neighbor”.
We observe that DAPT answers this question as Araceli Valencia .
This prediction may come from matching the word ‘her’ in the question to the feminine name ‘Araceli Valencia’ in the context.
In contrast, our KALA predicts the
Jaime Andrade as an answer, which is the ground truth.
We suspect that this might be because of the fact “(Jaime Andrade, spouse, Valencia)” in the knowledge graph, which relates the ‘Valencia’ to the ‘Jaime Andrade’.
Although it is not clear how it directly affects the model’s performance, we can reason that KALA can successfully answer the question by utilizing the existing facts.
D. Additional Data Visualization
In Figure 17 and 18, we visualize the examples of the context with its seen and unseen entities and its relational facts.
We ﬁrst conﬁrm
that the quality of facts is moderate to use.
For instance, in the ﬁrst example of Figure 17, the fact in the context that
Omar_bin_Laden is son of Osama_bin_Laden , is also appeared in the knowledge graph.
In addition, we observe that there are facts that link unseen entities to the seen entities in both Figure 17 and 18.
Thus, while some of the facts in the knowledge graph are not accurate, we can represent the unseen entities with their relation to the seen entities.
We expect that there is a still room to improve in terms of the quality of KGs, allowing our KALA to modulate the entity representation more accurately.
We leave the study on this as the future work.
Context PHOENIX, Arizona (CNN) – Jamie Andrade had just gotten out of the shower when the men came to snatch him.
Jamie Andrade was kept in this closet for three days without food or water, police say.
His wife, Araceli Valencia, was mopping the kitchen in … (ellipse)…
Question who was kidnapped because of her neighbour? Answer Jaime Andrade Facts ( Sampled ) (Valencia, spouse, Jaime Andrade Jr.) (Jamie Andrade Jr., spouse, Valencia) KALA (Ours) DAPT
DAPT prediction Araceli Valencia,
[Figure]
Context MADRID, Spain (CNN) – One of Osama bin Laden’s sons has been denied asylum in Spain, an Interior Ministry spokeswoman told CNN on Wednesday.
Omar bin Laden pictured earlier this year during television interview in Rome, Italy.
Omar bin Laden, who is in his late 20s, stepped off a plane at Madrid’s Barajas International
Airport during a stopover late Monday and informed authorities that he planned to request political asylum, the spokeswoman said.
Bin
Laden has publicly called on his father to abandon terrorism.
He prepared his formal asylum request Tuesday at the airport with the help of a translator, filing it around 1 p.m., the spokeswoman said.
The Interior Ministry, which had 72 hours to reply to the request, was required to seek the opinion of the U.N. High Commissioner for
Refugees on the matter.
The UNHCR recommended … (ellipse) …
Question 1 Where was Omar previously denied?
Answer 1 asylum in Britain.
Facts ( Sampled ) (Bin Laden, significant event, Flight) (International Airport, country, Spain) (International Airport, [UNK], Madrid) (Omar bin Laden, father, Osama Bin
Laden)
(Spain, diplomatic relation, Italy) (Osama Bin Laden, child, Omar Bin
Laden) (Italy, diplomatic relation, Spain)
Question 2 Did Spain give a reason for turning down the asylum?
Answer 2 was given
Question
Who was denied asylum in Britain?
Answer 3 Omar bin Laden
Question 4 What family member of Omar bin Laden was associated with terrorism?
Answer 4 his father
Context (CNN) – unseeded Frenchwoman Aravane Rezai produced one of the shocks of the year on Sunday by defeating favorite Venus
Williams in straight sets to win the final of the Madrid Open.
The 23year-old Rezai – who had only claimed WTA Tour titles at Strasbourg and Bali prior to Madrid – continued her remarkable week with a 67-5 victory, adding Williams’ scalp to her earlier surprise victories over former world number one’s Junstine Henin and Jelena Jankovic.
Williams, who returns to No.2 in the world behind younger sister
Serena on Monday, lost the opening set in just 27 minutes and then failed to take advantage of a 4-1 lead in the.
“I just cannot believe this,” world number 24 Rezai – who must now enter calculations for the French Open – told reporters.
“Venus played very well and I’ve always respected her as a player and a champion.
I just tried my best today and it worked well for me.”
Williams, who was looking to secure her 44 th career title, only converted two of her 13 break points in the batch – a statistic that contributed greatly to her defeat.
Question 1 Which player was the favourite?
Answer 1 Venus Williams
Facts (Venus Williams, sibling, Aravane Rezai)
(Final, part of, Year) (Mutua Madrid Open, located in the administrative territorial entity, Madrid)
(Victories, instance of, Military rank) (Surprise, instance of, Military rank) (Mutua Madrid Open, instance of,
Military rank)
(Final, instance of, Military rank)
Question 2 Which title number was this?
Answer 2 44th
Question
When did the Mardrid Open final take place?
Answer 3 Sunday
Context The adenomatous polyposis coli ( APC ) tumour - suppressor protein controls the Wnt signalling pathway by forming a complex with glycogen synthase kinase 3beta ( GSK - 3beta ), axin / conductin and betacatenin.
Facts ( Sampled ) (complex, subclass of, protein) (GSK, instance of, protein) (glycogen, instance of, protein) (APC, instance of, protein)
Context HLA typing for HLA - B27, HLA - B60, and HLA - DR was performed by polymerase chain reaction with sequence - specific primers, and zygosity was assessed using microsatellite markers.
Facts ( Sampled ) (microsatellite, subclass of, primers) (DR1, instance of, microsatellite) (microsatellite, subclass of, typing)
Context We identified four germline mutations in three breast cancer families and in one breast - ovarian cancer family.
among these were one frameshift mutation, one nonsense mutation, one novel splice site mutation, and one missense mutation.
Facts ( Sampled ) (frameshift mutation, subclass of,
Germline mutations)
(Nonsense mutation, subclass of,
Germline mutations) (splice site mutation, subclass of, Germline mutations) (missense mutations, subclass of,
Germline mutations)
(Nonsense mutation, subclass of, cancers) (frameshift mutation, subclass of, cancers) (missense mutations, subclass of, cancers)
Context A nonsense mutation in exon 17 ( codon 556 ) of the RB gene was found to be present homozygously in both the retinal and the pineal tumours.
Facts ( Sampled ) (retinal, instance of, gene)
(Nonsense mutation, subclass of, gene)
Context Sixteen different p16 germline mutations were found infamilies, while one germline mutation, Arg24His, was detected in the
CDK4 gene.
Facts ( Sampled ) (p16, subclass of, Germline mutations) (Germline mutations, subclass of, gene) (p16, instance of, gene)
Context Aspartylglucosaminuria ( AGU ) is a rare disorder of glycoprotein metabolism caused by the deficiency of the lysosomal enzyme aspartylglucosaminidase ( AGA ).
Facts ( Sampled ) (Aspartylglucosaminuria, subclass of, deficiency)
Context Detection of heterozygous carriers of the ataxiatelangiectasia ( ATM ) gene by G2 phase chromosomal radiosensitivity of peripheral blood lymphocytes.
Facts ( Sampled ) (ATM, instance of, gene) (G2 phase, part of, blood) (G2 phase, instance of, gene)
Context Recently, we reported five Austrian families with generalized atrophic benign epidermolysis bullosa who share the same COL17A mutation.
Facts ( Sampled ) (epidermolysis bullosa, instance of, mutations)