Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoning
Abstract
Commonsense reasoning systems should be able to generalize to diverse reasoning cases.
However, most state-of-the-art approaches depend on expensive data annotations and overﬁt to a speciﬁc benchmark without learning how to perform general semantic reasoning.
To overcome these drawbacks, zero-shot QA systems have shown promise as a robust learning scheme by transforming a commonsense knowledge graph (KG) into synthetic QAform samples for model training.
Considering the increasing type of different commonsense
KGs, this paper aims to extend the zero-shot transfer learning scenario into multiple-source settings, where different KGs can be utilized synergetically.
Towards this goal, we propose to mitigate the loss of knowledge from the interference among the different knowledge sources, by developing a modular variant of the knowledge aggregation as a new zero-shot commonsense reasoning framework.
Results on ﬁve commonsense reasoning benchmarks demonstrate the efﬁcacy of our framework, improving the performance with multiple KGs.
Introduction
The ability to understand natural language through commonsense reasoning is one of the core focuses in the ﬁeld of natural language processing.
To measure and study the different aspects of commonsense reasoning, several datasets are developed, such as SocialIQA (Sap et al., 2019b), CommonsenseQA (Talmor et al., 2018) and PhysicalIQA (Bisk et al., 2020),
each requiring different type of commonsense knowledge ( e.g. , social, taxonomic, causal, declarative, etc ) to select the correct answer.
While large-scale neural systems (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b) have shown human-level accuracy on these benchmarks, recent studies (Mitra et al., 2019) also criticize that these models solve individual datasets, rather than learning how to perform general semantic reasoning.
To this end, Ma et al.
(2021) suggested zero-shot evaluation as a genuine measure for the reasoning capability of the machine.
Inspired by this new metric, in this work, we focus on building unsupervised zero-shot multiplechoice QA systems.
That is, we target an arbitrary commonsense reasoning task where conventional approaches (that rely heavily on task-speciﬁc supervision) are not applicable to such zero-shot learning scenarios.
To learn QA models without expensive annotation efforts, recent works (Ma et al., 2021;
Banerjee and Baral, 2020; Malaviya et al., 2020) propose to generate a synthetic QA dataset using a commonsense KG such as ATOMIC (Sap et al., 2019a) and ConceptNet (Speer et al., 2017).
Such an approach mostly focuses only on one speciﬁc type of reasoning relations ( e.g. , if-then relation, or declarative relation), neglecting the fact that real-world QA systems require simultaneously considering different types of reasoning abilities ( e.g. , declarative and social, or causal and physical reasoning;
Ilievski et al., 2021; Chang et al., 2021).
To consider different types of reasoning, this paper extends ideas from the aforementioned zeroshot learning to the multi-source case such that it beneﬁts from different types of commonsense knowledge on individual KGs.
For example,
ATOMIC (Sap et al., 2019a) focuses on social commonsense while ConceptNet (Speer et al., 2017) contains conceptual knowledge.
A practical approach is multi-task learning (MTL; Caruana, 1997;
Liu et al., 2019a), which learns a shared encoder for different synthetic QA datasets from multiple
KGs.
Despite its effectiveness, MTL scheme suffers from interference among different KGs, which results in forgetting previously learned knowledge when trained on new KG which has different kinds of knowledge (Pilault et al., 2021;
Pfeiffer et al., 2021; Wang et al., 2021a; Wu et al., 2020).
To address these limitations, we propose a novel, modularized framework that aims to learn multiple expert models for KGs, then conduct zero-shot fusion to allow collaboration among KGs.
For this purpose, we leverage AdapterFusion (Pfeiffer et al., 2021) where multiple tiny modules between Transformer blocks called adapters (Houlsby et al., 2019) can be combined after independent training, thus allowing a continual integration of the adapters without retraining the entire framework.
Speciﬁcally, we treat the adapters as different KG-speciﬁc experts, and combine them using an attention-like fusion module.
To improve the fusion of adapters, we suggest a KG-alignment adapter that guides to the apt expert adapters .
Here, we use KGs in three different synthetic supervision training: (1)
KG-speciﬁc QA datasets to train the KG-speciﬁc expert adapters, (2) a KG classiﬁcation datasets to train the KG-alignment adapter, and (3) a balanced mixture of KG-speciﬁc QA datasets to train the fusion module.
Our modularized method alleviates the interference between different KGs, which is the pitfall of MTL from our empirical observation, and thus combines multiple KGs into a synergetic zero-shot framework.
Our contributions are: (1) We suggest a simple, yet effective KG modularization strategy for the use of multiple KGs in commonsense reasoning.
We then explore the use of AdapterFusion (Pfeiffer et al., 2021) for better knowledge aggregation based on the KG modularization in zero-shot setting.
We believe that such modularized transfer learning is critical to using different knowledge sources synergetically against interference between them.
In extensive experiments on various commonsense reasoning benchmarks, our framework achieves signiﬁcant improvements over baselines using a single KG, even using multiple KGs, which implies the robustness in commonsense reasoning.
Related Work & Preliminaries 2. Zero-shot Commonsense Reasoning
Many researchers have recently focused on building unsupervised models without any benchmark supervisions ( i.e. , zero-shot learning).
In such zeroshot setting, KGs are often used as an external resource for improving model prior ( e.g. , continually learned from pre-trained language models) (Banerjee and Baral, 2020; Bosselut and Choi, 2019; Ma et al., 2021), especially for commonsense reasoning, as much existing work couples language models with neural/symbolic commonsense KGs.
However, most of existing work are either assuming the existence of the alignment information between tasks and KGs (Banerjee and Baral, 2020) or an integrated KG (Ma et al., 2021).
For example,
ATOMIC 2020 (Hwang et al., 2021), a commonsense
KG which incorporates tuples from ConceptNet and ATOMIC with new relations and further crowdsourcing, combines multiple KGs into a new integrated KG, but as widely known (Ilievski et al., 2020; Hwang et al., 2021), heterogeneous schema between different KGs may limit triplets that can be integrated.
1 Rather than such symbolic KG integration with the inevitable loss of knowledge, in this work, we explore the neural KG integration leveraging the multiple KGs without additional processing and alignment information between KG and task.
2. Transfer Learning with Modular Approaches
The idea of having specialized parameters, or socalled experts, has been widely studied to integrate multiple sources of knowledge via transfer learning.
The adapter module (Rebufﬁ et al., 2017;
Houlsby et al., 2019) has been explored as one of such approaches, introducing a small number of task-speciﬁc parameters at every layer of pretrained language model (PLM) while sharing the parameters of underlying PLM which is ﬁxed.
To address the limitations of transfer learning due to high re-training cost, many works utilize the multiple adapter modules for individual tasks with different domains (Puigcerver et al., 2020;
Bapna et al., 2019; Rücklé et al., 2020; Madotto et al., 2021) considering each adapter to be an expert of each domain.
Similar to our work, K-Adapter (Wang et al., 2021a)
encodes factual and linguistic knowledge to each adapter, but in this paper, we further explore how to mitigate catastrophic forgetting or interference among multiple adapters for better knowledge transfer in zero-shot setting.
2. Multi-Task Learning
MTL (Liu et al., 2019a; Zhang and Yang, 2017;
Caruana, 1997) learns a shared representation while aggregating knowledge across multiple learning tasks, often leading to better generalization ability of a model.
However, parametric aggregation of knowledge with MTL has following limitations:
(1) retraining the full model when adding new
tasks (Houlsby et al., 2019; Pfeiffer et al., 2021, 2020b) (2) catastrophic forgetting and interference between tasks leading to difﬁculties of solving each task equally well (Pilault et al., 2021; Wu et al., 2020; Yu et al., 2020) and (3) inconsistent effect (Lourie et al., 2021).
To deal with these challenges, Mixture-of-Experts (MoE) is a parameterized generalization of ensembling techniques, which has been adapted for MTL with gating network trained to optimize each task (Ma et al., 2018).
However, simple linear gating networks are too shallow and thus may destruct task knowledge for commonsense reasoning.
To address this problem, AdapterFusion (Pfeiffer et al., 2021) has been proposed to fuse task speciﬁc parameters called adapters for the given target task leveraging attention-like mechanism.
AdapterFusion aggregates adapters, which is trained independently for each task, in a non-destructive manner mitigating aforementioned MTL problems such as forgetting and interference between tasks.
Recently, it has been used for zero-shot cross-lingual transfer framework (Pfeiffer et al., 2020c; Wang et al., 2021b), which motivates our work to transfer multi-source knowledge with less interference for zero-shot commonsense reasoning.
Modularized Zero-shot Framework
In our setup, we repurpose synthetic QA generation (Ma et al., 2021) for the task of knowledgedriven zero-shot learning for commonsense reasoning, i.e. , we transform a KG into multiple ( Q i , A i ) pairs where Q i is a natural language question and
A i =
{ A i, 1 , ...,
A i,m } is the set of options with m answer candidates.
Speciﬁcally, given a triple ( e head , r, e tail ) in a KG, where e head , e tail and r denote head/tail entity and relation respectively, we transform e head and r into a natural language question Q i using templates.
For the option set A i , we use the combination of the correct answer e tail and m − 1 distractors which are tail entities from other triples sampled randomly (Ma et al., 2021). Details are described in Appendix B.
Formally, we denote ( Q i , A i ) as one QA sample.
The goal is to learn a QA model from the synthetic QA sample.
In a downstream task ( e.g. , reasoning benchmarks such as SocialIQA and CommonsenseQA), we need to predict answers given non-synthetic test samples ( Q test , A test ) .
In the training stage, we are given K KG-driven datasets {D kQA } Kk =1 from K different KGs, where D kQA is
[Table]
a dataset with N k samples for KG k as follows:
[Equation]
where label is the index of the correct answer for each sample.
In this work, as shown in Table 1, we generate four synthetic QA datasets from ATOMIC , ConceptNet , WikiData [Table] and
WordNet (More details are in Appendix C).
For effective use of multiple KGs at once with less interference, we present a modularized framework, which is a novel approach to knowledge transfer for the zero-shot setting as shown in Figure 1.
As a modular approach, we train the multiple KG-speciﬁc adapters ( expert adapters ) with each dataset from KG.
Based on these pre-trained adapters, we use a zero-shot fusion method to aggregate knowledge of each adapter leveraging
AdapterFusion (Pfeiffer et al., 2021) as a base architecture with the balanced mixture of each
KG dataset.
Further, for better knowledge fusion, we suggest a KG-alignment aware adapter ( KGClassiﬁer adapter ) as a guide for detecting alignment with given sample in zero-shot reasoning.
Here, we utilize KG classiﬁcation dataset by verifying the synthetic QAs.
Algorithm 1 in Appendix outlines the overall process of our proposed framework. We summarize the notations in Appendix A.
3. KG Modularization
First, we modularize the KGs to preserve their intrinsic knowledge.
Considering the importance of using a suitable and well-aligned KG (Ma et al., 2019, 2021) on a downstream task, the subtle difference between each KG should be learned by the model without any interference from each other.
Accordingly, we adopt the adapter module (Houlsby et al., 2019) which repurposes a pretrained language model (PLM) to incorporate each [Figure]
KG as tiny modules in between Transformer blocks.
Speciﬁcally, as illustrated in Figure 2 (except for green area), the adapter training strategy involves injecting new layers (parameterized by Φ ) into the original PLM (parameterized by θ ).
The weights of the original PLM are untouched, while the new adapter layers are initialized at random.
Formally, we call each adapter trained with D kQA as an expert adapter for KG k , parameterized by Φ kQA .
When a QA sample ( Q i , A i ) is given for dataset
D kQA , we ﬁrst concatenate question Q i and each answer option
A i = { A i, 1 , ...,
A i,m } to generate input sequences T i =
{ T i, 1 , ..., T i,m } .
Then, we compute a score S i,j (Ma et al., 2021) for the answer candidate
A i,j is computed as follows:
[Equation]
where w t is a word token in the sequence T i,j and
P is the conditional probability from Transformer blocks parameterized by θ and Φ .
To train the adapter Φ kQA , we use the marginal ranking loss (Ma et al., 2021) as follows:
[Equation]
where η represents the margin.
[Equation]
where KG-invariant parameters θ are ﬁxed and only
KG-dependent parameters Φ kQA are learned, which enables to store the corresponding knowledge separately without any interference.
Further, we can parallelize the training of adapter for all KGs.
The efﬁciency of adapter training allows our modularization to be more scalable.
3. Zero-shot Fusion
Once the expert adapters are learned, we combine the knowledge from each expert adapter using an attention-like mechanism.
We present a novel fusion strategy as shown in Figure 2, which is referred to as the zero-shot fusion.
In contrast to AdapterFusion (Pfeiffer et al., 2021) where the focus is learning to transfer knowledge to a speciﬁc target task, our zero-shot fusion aims to generalize this transfer to any arbitrary target task.
Speciﬁcally, the zero-shot fusion parameters Ψ learn to combine ﬁxed expert adapters which are parameterized by Φ 1 QA , ..., Φ K QA .
In each Transformer layer l of PLM with the injected fusion layer, the zero-shot fusion parameters Ψ QA consist of query, key, and value matrices, denoted by W Ql , W Kl , and W Vl respectively.
These parameters are used to learn the balancing between the representation of each expert adapters through attention-like mechanism.
While ﬁxing both the parameters θ and all expert adapters Φ 1 QA , ..., Φ KQA , the only trainable weights Ψ QA on the fusion layer learns to combine the knowledge from different K expert adapters by using the subset of {D kQA } Kk =1 by random sampling.
Here, we balance the ratio between the K knowledge-driven datasets as N samples (details are in Appendix D).
Formally,
[Equation]
where Ψ refers to the initialized zero-shot fusion parameters.
More speciﬁcally, in the l -th Transformer layer, let h lPLM and h k,lE be the representations of underlying PLM parameterized by θ and an expert adapter parameterized by Φ kQA , respectively.
Then, using the hidden representation h lPLM of PLM as a query, the fusion layer performs the attention-like function as follows:
[Equation]
[Figure]
where z l is passed to the next Transformer layer.
Given a sample, the zero-shot fusion learns the suitable balancing parameters between the expert adapters for zero-shot reasoning.
Eventually, it learns to identify generalizability across commonsense reasoning tasks.
3. KG-Classiﬁer Adapter
AdapterFusion uses the PLM hidden representation h lPLM as a query which is learned when training on a speciﬁc downstream task.
In our zero-shot setting, however, we use a mixture of synthetic QA for fusion training, which is not exactly a training dataset for a downstream task.
To compensate for this issue, we present KG-Classiﬁer adapter , which is a KG alignment-aware adapter, which is motivated from the fact that the ability to ﬁnd which
KG has an alignment with the given sample can be helpful as a role of providing a guidance for better performance (Ma et al., 2019, 2021).
Speciﬁcally, we propose a novel training task for
KG-Classiﬁer adapter , which requires predicting the KG for the given sample of the task.
For that, given {D kQA } Kk =1 , we ﬁrst transform a QA sample ( Q i , A i ) into a new KG classiﬁcation sample [ Q i ; A i,label ] where [; ] is the concatenation.
Then, we obtain a new label y i ∈ { 0 , 1 } K indicating the corresponding KG source.
The samples are in
Appendix E. Formally, KG classiﬁcation dataset
D KGC is deﬁned as:
[Equation]
where M is the total size of {D kQA } Kk =1 .
Based on D KGC , we learn the KG-Classiﬁer adapter parameterized by θ and Φ KGC .
First, a classiﬁcation sample i is encoded into h CLS ∈
R H then scored as ˆ y i ∈ R K with a linear layer
W KGC ∈ R K × H , i.e. , ˆ y i =
W KGC h CLS .
Once ˆ y i is normalized by a softmax layer, the network is trained to minimize the cross-entropy loss L KGC between the prediction ˆ y i and its ground truth
y i :
[Equation]
We propose to use the representation of KGClassiﬁer adapter as a query in attention-like mechanism, referred to as the zero-shot fusion with KGClassiﬁer adapter .
That is, using the hidden representation h lKGC of a KG-Classiﬁer adapter parameterized by Φ KGC as a query, we substitute Q l in
Eq.
(11) as follows:
[Equation]
The overall zero-shot fusion architecture including
KG-Classiﬁer is illustrated in Figure 2.
Experiments
In this section we evaluate the efﬁcacy of our framework on ﬁve commonsense reasoning tasks.
We denote KG-Classiﬁer adapter by KG-C adapter .
4. Experimental Settings
All our experiments are conducted in a zero-shot setting, in which the models do not have access to the ofﬁcial training data or labels of the benchmark.
For the evaluation, we use the validation set of each benchmark 2 , however, the validation set of each benchmark can be role as an test set since it is not used for hyperparameter tuning or model selection.
We use accuracy as a metric.
4.1. Benchmarks
We evaluate our proposed framework on ﬁve question-answering benchmarks for commonsense reasoning: SocialIQA (SIQA) (Sap et al., 2019b),
CommonsenseQA (CSQA) (Talmor et al., 2018),
Abductive NLI (a-NLI) (Bhagavatula et al., 2020),
PhysicalIQA (PIQA) (Bisk et al., 2020), and WinoGrande (WG) (Sakaguchi et al., 2020).
Each commonsense benchmark evaluates a speciﬁc kind of knowledge: social commonsense for SIQA, concept-level commonsense for CSQA, abductive
[Table]
reasoning for a-NLI, physical commonsense for
PIQA, and pronoun resolution ability for WG.
The details are presented in Appendix G.
4.1. Baselines
We compare our framework with the following baselines.
First, to show the characteristics of each benchmark, we use the random or the most frequent label as Random and Majority baseline, respectively.
RoBERTa-L and GPT2-L is the performance of each PLM without any ﬁnetuning.
Also, as the baseline for the unsupervised learning model using KGs, we report the performance of Self-talk (Shwartz et al., 2020),
COMET-DynaGen (Bosselut and Choi, 2019), SMLM (Banerjee and Baral, 2020) as presented in original papers.
For further analysis in § 4.4 and § 4.5, we set the following models that are pre-trained on the synthetic QA datasets from KGs as baselines:
• Single-Task Learning (STL) :
The model is pre-trained on a synthetic QA dataset generated from a single KG.
Speciﬁcally, we experiment two architectural choices: PLM (STLPLM) and PLM with adapters (STL-Adapter).
For each architecture, there are four STL models for each of synthetic QA datasets derived from ATOMIC , ConceptNet , WikiData , and WordNet .
We note that the trained STLAdapter is an expert adapter from a speciﬁc KG in our framework.
The performance of each STL baseline is shown in Appendix I Table 9 and Table 10.
• Multi-Task Learning (MTL) :
The model is pre-trained on multiple synthetic QA datasets, each of which is generated from a KG.
We experiment with a PLM trained on all four aforementioned synthetic QA datasets.
We note that the difference between STL-PLM and MTL is whether to use one synthetic QA dataset or multiple synthetic QA datasets for its training.
4.1. Implementations
We employ RoBERTa-L (Liu et al., 2019b) from
Hugging Face’s transformers toolkit for all experiments.
We follow the default settings from Ma et al.
Our implementation uses Adapter (Houlsby et al., 2019) and AdapterFusion (Pfeiffer et al., 2021) as a base model architecture from AdpaterHub (Pfeiffer et al., 2020a).
We run our experiments with three different random seeds.
The implementation details are described in Appendix H.
4. Main Results
Table 2 shows the zero-shot evaluation results on ﬁve benchmark datasets.
Generally, zero-shot fusion scores higher than the baselines across all benchmarks, and further, zero-shot fusion shows the best performance in all benchmarks except WG.
We note that although Ma et al.
(2021) uses the synthetic QA dataset after sample ﬁltering, our method achieves comparable performance with the best performance in WG, even with the raw dataset.
Also, the average score of all evaluation benchmarks (the last column of Table 2) shows that zero-shot fusion has generalisability in commonsense reasoning.
In addition, zero-shot fusion achieves consistent improvements over MTL.
These results indicate that our proposed zero-shot fusion method attributes to fusing the knowledge of multiple KGs more synergetically regardless of the task.
Moreover, as an ablation, we compare the zeroshot fusion with and without KG-C adapter to explore the efﬁcacy of the KG-C adapter .
We can observe that zero-shot fusion with KG-C adapter improves the average accuracy by 0.4%, which implies that the use of KG-C adapter improves the overall performance and makes our method generalize better on most of the evaluation benchmarks.
4. Impact of the KG-Classiﬁer Adapter
To assess the effects of the KG-C adapter itself, we visualize and compare the ﬁnal layer [CLS] token representation between PLM and KG-C adapter .
Figure 3 shows t-SNE (Van der Maaten and Hinton, 2008) plots of all representation of ﬁve benchmark datasets.
In this ﬁgure, every sample is mapped into a 1024-dimensional feature space through
RoBERTa-L model and projected back into a twodimensional plane by t-SNE.
We can observe that
KG-C adapter can separate the samples of different benchmarks well despite being unseen data.
It veriﬁes that KG-awareness acquired with the KG classiﬁcation task is beneﬁcial to categorize the given sample.
The KG-C adapter can thus generate a relevant KG-aware query for a given sample and help to fuse representations from suitable expert adapters in our proposed framework.
Further, we explore how the KG-C adapter affects zero-shot fusion which is based on an attention-like mechanism (Pfeiffer et al., 2021) compared to zero-shot fusion without KG-C adapter .
Here, while zero-shot fusion without KGC adapter simply uses the representation of PLM as a query, zero-shot fusion with KG-C adapter leverages the representation of KG-C adapter .
To illustrate this strength, we visualize the attention probability of [CLS] token from each fusion layer as a representative in Figure 4.
The column of the darker cell indicates the adapter that has the bigger inﬂuence on the fused representation.
We can observe that zero-shot fusion with KG-C adapter fuses the knowledge from different experts with a subtle difference rather than focusing on a single expert severely.
This implies that KG-C adapter [Figure] [Figure] enables the delicate balancing between multiple knowledge sources based on the KG-alignment awareness, which leads to performance improvements in commonsense reasoning tasks.
Interestingly, both cases have the ability not to focus on the expert adapter based on WikiData , which can be seen as a redundant expert.
This observation would beneﬁt from the further study that explores the optimal combination of KGs by expert selection or rejection.
4. Mitigating Interference
In this experiment, we compare the amount of interference in the MTL and zero-shot fusion with
KG-C adapter .
We propose a novel evaluation metric, the interference ratio , which is the percentage of the incorrectly predicted samples by the multiKG models among the correctly predicted samples from the STL models in common.
Using the interference ratio, we can precisely compare the negative effects of multi-KG models on knowledge aggregation since the only reason to get the correct samples wrong is the interference caused by learning with additional KGs.
We present the interference ratio of the models on ﬁve benchmark datasets in Figure 5.
This ﬁgure shows that MTL has the higher interference ratio than the competing models across all benchmarks.
Our method achieves a substantially better ratio, especially when KG-C adapter is used.
This demonstrates the efﬁcacy of our framework in mitigating interference between knowledge, which is one of the major problems of MTL.
4. Visualization of Knowledge Aggregation
To verify the ability of our model to aggregate different types of KGs, we compare the relative performance gains of MTL and zero-shot fusion with
KG-C adapter when increasing the number of KGs.
The performance of all KG-combinations for each framework is presented in Table 9 and Table 10.
We visualize the improvement of performance for ﬁve benchmark development sets, leveraging heatmaps in Figure 6.
Here, for the sake of brevity, we denote our framework with KG-C adapter as our method.
For MTL in Figure 6 (a), the color of the cell denotes the relative improvement of MTL with the combination of KGs over the best performance among the STL-PLM of KGs.
Also, for our method in Figure 6 (b), the relative improvement is measured based on the best performance among the
STL-Adapter of KGs, considering the difference of the base architecture for MTL (i.e. PLM) and zeroshot fusion (i.e. PLM with adapter).
The green and red colors denote the increase and decrease of performance, respectively, when using multiple KGs together.
The greener color on the cells indicates that the approach beneﬁts from an increasing number of KGs, which implies aggregating knowledge successfully.
In Figure 6, while the MTL tends to show the decrease of the performance when more KGs are
[Figure]
utilized for training, our method obtains relative performance improvement across most of benchmarks.
In both framework, the slightly degraded performance of the combination of KGs without
ATOMIC could be due to the strong alignment between ATOMIC and SIQA.
Except for the above case, we can observe that as more KGs are leveraged, the color of the cell gets greener, which implies that our method gains more advantages for better performance.
This demonstrates that our method enables knowledge aggregation for multiple KGs synergetically.
Conclusion
Despite the existence of various types of commonsense KGs, utilizing multiple KGs has not been explored enough in the commonsense reasoning ﬁeld.
Motivated by this, this paper proposes a modularized transfer learning framework to fuse the knowledge from multiple KGs efﬁciently for zero-shot commonsense reasoning.
Our framework consists of KG modularization for expert adapter , zero-shot fusion and KG-Classiﬁer adapter .
Extensive experiments show that our framework obtains strong improvements over MTL on ﬁve commonsense reasoning benchmarks.
In the future, our work can be extended to adapt our methods to further various multiple KGs with studies of appropriate scale for KG modularization.
In addition, based on our hypothesis that the existence of an optimal combination, we can explore the study for the optional use of modularized KG experts for the best transfer learning. Acknowledgements
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No. 2020-0-01361,
Artiﬁcial Intelligence Graduate School Program (Yonsei University)) and (No. 2022-0-00077, AI
Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous Data) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021-11-1055).
Jinyoung Yeo is a corresponding author. arXiv preprint arXiv Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan
Hong, and Ed H Chi task relationships in multi-task learning with multi-gate mixtureof-experts.
In ACM SIGKDD .
Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. 2019.
Towards generalizable neuro-symbolic systems for commonsense question answering.
EMNLP .
Kaixin Ma, Filip Ilievski, Jonathan Francis, Yonatan
Bisk, Eric Nyberg, and Alessandro Oltramari. 2021.
Knowledge-driven data construction for zero-shot evaluation in commonsense question answering.
Andrea Madotto, Zhaojiang Lin, Yejin Bang, and Pascale Fung. 2021.
The adapter-bot: All-in-one controllable conversational model.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine
Bosselut, and Yejin Choi. 2020.
Commonsense knowledge base completion with structural and semantic context.
In AAAI .
George A Miller. 1995.
Wordnet: a lexical database for english.
Communications of the ACM .
Arindam Mitra, Pratyay Banerjee, Kuntal Kumar Pal,
Swaroop Mishra, and Chitta Baral. 2019.
Exploring ways to incorporate additional knowledge to improve natural language commonsense question answering.
arXiv preprint arXiv:1909.08855 .
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé,
Kyunghyun Cho, and Iryna Gurevych. 2021.
Adapterfusion: Non-destructive task composition for transfer learning.
EACL .
Aish arXiv
Jonas Pfeiffer
Iryna Gurevych,
Sebastian Ruder. 2020c.
Mad-x: An adapter-based framework for multi-task cross-lingual transfer.
arXiv preprint arXiv:2005.00052 .
Joan
Susano
Daniel
Houlsby arXiv:2009.
Sylvestre-Alvise Rebufﬁ, Hakan Bilen, and Andrea
Vedaldi.
arXiv preprint arXiv:1705.08045 .
Andreas Rücklé, Jonas Pfeiffer, and Iryna Gurevych.
EMNLP .
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi adversarial winograd schema challenge scale.
In AAAI .
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin,
Brendan Roof, Noah A Smith, and Yejin Choi if- reasoning
AAAI .
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019b
Socialiqa sense reasoning about social interactions.
EMNLP .
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
Unsuper vised commonsense question with self talk.
EMNLP .
Robyn Speer, Joshua Chin, and Catherine Havasi.
eral
AAAI .
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and
Jonathan Berant.
commonsense knowledge.
NAACL .
Laurens Van der Maaten and Geoffrey Hinton. 2008.
Visualizing sne
JMLR .
Denny Vrandeˇci´c and Markus Krötzsch. 2014.
Wiki- collaborative knowledgebase.
ACM .
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021a
ACL .
Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig
2021b sembling varieties.
Sen Wu, Hongyang R Zhang, and Christopher Ré.
ICLR .
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.
NIPS .
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey
A List of Notations
We summarize the notations used in this paper in
Table 7.
B Synthetic QA
We generate QA for four KGs ( ATOMIC ,
ConceptNet , WikiData and WordNet )
based on synthetic QA generation (Ma et al., 2021) without sample ﬁltering.
We use the preﬁxes for relation of triplet as shown in Table 3 for generating synthetic QA (refer to Ma et al.
Table 4 shows the statistics of the synthetic QA dataset from KGs.
The samples of synthetic QA with source triplet are shown in Table [Table]
, PersonX
[Table] to [Table]
oReact [Table] a [Table]
[Table]
oWant
[Table]
a result,
[Table]
to
[Table]
Others then
Causes
[Table]
CapableOf
[Table]
CausesDesire
[Table]
IsA
[Table]
SymbolOf
[Table]
[Table]
LocatedNear
[Table]
Desires
[Table]
HasProperty
[Table]
PartOf
[Table]
HasFirstSubevent
[Table]
HasLastSubevent
[Table]
[Table]
A variety of KGs have been proposed to provide large-scale high quality collection of different commonsense knowledge types: ATOMIC (Sap et al., [Table] 2019a) focuses on inferential knowledge organized as typed if-then relations with variables ( e
[Table]
g. , “ if X pays Y a compliment, then Y will likely return the compliment ”).
ConceptNet (Speer et al., 2017) mainly consists of taxonomic and lexical knowledge ( e.g. , RelatedTo, Synonym, and IsA) and physical commonsense knowledge ( e.g. , MadeOf and PartOf).
WikiData (Vrandeˇci´c and Krötzsch, 2014) is a general-domain KG which has a close relation with Wikipedia.
WordNet (Miller, 1995) is a large lexical source of words and taxonomical system.
D Dataset for Zero-shot Fusion
For zero-shot fusion training, we use balanced mixture of synthetic QA from different KGs by random sampling.
The statistics of dataset for zero-shot fusion is shown in Table 6.
For validation dataset, we balance between the ATOMIC , ConceptNet and
WordNet due to the lack of synthetic QA validation dataset from WikiData . [Table]
Notation
Meaning ( e head , r, e tail )
Triple of KG (head entity, relation, tail entity)
Q i
Natural language Question of sample
i
A i =
{ A i, 1 , ...,
A i,m }
A set of answer options of sample i , A i,j denotes j -th answer option of sample i (1 ≤ j ≤ m )
T i =
{ T i, 1 , ..., T i,m }
[Table]
sequences generated by concatenation of Q i and A i w
[Table]
i
[Table]
W Ql
[Table]
W Kl
[Table]
W Vl
[Table]
h lPLM
[Table]
h k,lE
[Table]
h lKGC
[Table]
E KG-Classiﬁcation Dataset
We suggest KG-Classiﬁcation dataset D KGC for
KG-Classiﬁer adapter training.
The example of transformation from synthetic QA dataset D QA is shown in Table 8.
The dataset size is equal to the whole dataset of synthetic QA (refer to Table 4).
[Table]
Dana is seen as risky.
A
[Figure]
QA → KG-Classiﬁcation ConceptNet
Q: pentode is a [MASK]
A1: ascocarp A2: girls footwear A3: vacuum tube ( (cid:88) )
S: pentode is a vacuum tube.
A: ConceptNet
QA → KG-Classiﬁcation WikiData
Q: badminton is a [MASK]
A1: fable A2: juvenile justice A3: type of sport ( (cid:88) )
S: badminton is a type of sport.
A: WikiData
[Figure]
KG
[Figure]
Classiﬁcation WordNet
Q: princewood is part of [MASK]
A
[Figure]
shaddock A2: genus Cordia ( (cid:88)
[Figure]
A
[Figure]
S
[Figure]
princewood
[Figure]
part of genus Cordia
[Figure]
F Zero-shot architecture with parameters
We describe the illustration of the zero-shot fusion architecture with parameters in Figure [Figure]
G Commonsense Reasoning Benchmarks
SocialIQA (SIQA) (Sap et al., 2019b) requires reasoning for emotional and social intelligence in everyday situations.
Each QA consists of a context that comes from ATOMIC , a question which is based on the relations in ATOMIC , and 3 answer candidates.
It contains 38,000 multiple-choice questions, which is generated by crowdsourcing.
CommonsenseQA (CSQA) (Talmor et al., 2018) evaluates a broad range of concept-level commonsense reasoning.
Each multiple-choice question, answer and distractors are designed by crowdsourcing based on the ConceptNet .
Abductive NLI (a-NLI) (Bhagavatula et al., 2020) asks to infer the most plausible explanation based on the given causal situation to test abductive reasoning in narratives.
Each sample consists of the beginning and the end of the story with two possible options to be an explanation for the given situation.
PhysicalIQA (PIQA) (Bisk et al., 2020) requires physical commonsense reasoning to select the most sensible solution for the given goal among the two choices.
Its dataset is comprised of over 16, training samples, 2K validation samples, and 3K test samples.
HellaSWAG (HSWAG) (Zellers et al., 2019) is an evolved version of SWAG (Zellers et al., 2018), which asks to infer the most proper story based on the given situation.
The dataset consists of 70K questions with four answer options.
H Implementation Details
In all our experiments, we use max sequence length 128, batch size 32, weight decay 0.01, adam β0.9, adam β 2 0.99, adam epsilion 1 e − 8 , warm-up proportion 0.05, and margin 1.0.
The experiments are conducted split across NVIDIA GeForceand NVIDIA RTX A5000.
H. Baselines
The baseline models for STL-PLM and MTL are trained with learning rate 1 e − 5 for single epoch.
H. Adapter
For expert adapters , we use learning rate 8 e − 5 after tuning in { 5 e − 6 , 8 e − 6 , 1 e − 5 , 5 e − 5 , 8 e − 5 , 1 e − 4 } .
For KG-Classiﬁer adapter , we use learning rate 1 e − 5 , batch size 64 for ﬁve epochs.
H. Zero-shot fusion
After experiment with learning rates { 1 e − 5 , 8 e − 5 } , we empirically ﬁnd that a learning rate of 1 e −works well on zero-shot fusion without/with KGClassiﬁer adapter , respectively.
Here, we set the attention drop probability 0.1.
As we used extremely smaller subset of the synthetic QA dataset, zeroshot fusions are trained for ﬁve epochs.
I Knowledge aggregation of zero-shot fusion
In order to validate the efﬁcacy on knowledge aggregation of zero-shot fusion over the STL, we present the results of each framework with various combination of KGs in Table 9 and Table 10.
[Table]
and WN represent ATOMIC , ConceptNet , WikiData and WordNet , respectively.
We run our experiment with seed 42.
[Table]
Input: PLM parameters θ , K KGs Output: Reasoning model parameters ( θ, { Φ kQA } Kk =1 , Φ KGC , Ψ QA ) {D kQA } Kk =1 ← Generate synthetic QA samples from multiple KGs (Eq. 1) D
KGC ← Generate KG classiﬁcation samples from multiple KGs (Eq. 9) for each KG k = 1 , ..., K do Φ kQA ← argmin Φ L QA ( D kQA ; θ, Φ) (Eq. 4)
Φ KGC ← argmin Φ (cid:80) Mi
=1 L KGC ( D KGC ; θ, Φ) (Eq. 10)
Ψ QA ← argmin Ψ (cid:80)
Kk =1 L QA ( D kQA ; θ, { Φ kQA } Kk =1 , Φ KGC , Ψ) (Eq. 5 and 11) return ( θ, { Φ kQA } Kk =1 , Φ KGC , Ψ QA )