Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers (2020)

Abstract: Following the major success of neural language models (LMs) such as BERT or GPT-2 on a variety of language understanding tasks, recent work focused on injecting (structured) knowledge from external resources into these models. While on the one hand, joint pre-training (i.e., training from scratch, adding objectives based on external knowledge to the primary LM objective) may be prohibitively computationally expensive, post-hoc fine-tuning on external knowledge, on the other hand, may lead to the catastrophic forgetting of distributional knowledge. In this work, we investigate models for complementing the distributional knowledge of BERT with conceptual knowledge from ConceptNet and its corresponding Open Mind Common Sense (OMCS) corpus, respectively, using adapter training. While overall results on the GLUE benchmark paint an inconclusive picture, a deeper analysis reveals that our adapter-based models substantially outperform BERT (up to 15-20 performance points) on inference tasks that require the type of conceptual knowledge explicitly present in ConceptNet and OMCS. We also open source all our experiments and relevant code under: https://github.com/wluper/retrograph .

## introduction self-supervised neural models like elmo @xcite , bert @xcite , gpt @xcite , or xl-net @xcite have rendered language modeling a very suitable pretraining task for learning language representations that are useful for a wide range of language understanding tasks @xcite . although shown versatile w.r.t. the types of knowledge @xcite they encode, much like their predecessors -static word embedding models @xcite -neural lms still only "consume" the distributional information from large corpora. yet, a number of structured knowledge sources exist -knowledge bases (kbs) @xcite and lexico-semantic networks @xcite @xcite -encoding many types of knowledge that are underrepresented in text corpora. starting from this observation, most recent efforts focused on injecting factual @xcite @xcite and linguistic knowledge @xcite into pretrained lms and demonstrated the usefulness of such knowledge in language understanding tasks @xcite . joint pretraining models, on the one hand, augment distributional lm objectives with additional objectives based on external resources @xcite @xcite and train the extended model from scratch. for models like bert, this implies computationally expensive retraining from scratch of the encoding transformer network. post-hoc fine-tuning models @xcite @xcite , on the other hand, use the objectives based on external resources to fine-tune the encoder's parameters, pretrained via distributional lm objectives. if the amount of fine-tuning data is substantial, however, this approach may lead to catastrophic forgetting of distributional knowledge obtained in pretraining @xcite . in this work, similar to the concurrent work of @xcite , we turn to the recently proposed adapter-based fine-tuning paradigm @xcite , which remedies the shortcomings of both joint pretraining and standard post-hoc fine-tuning. adapterbased training injects additional parameters into the encoder and only tunes their values: original transformer parameters are kept fixed. be-cause of this, adapter training preserves the distributional information obtained in lm pretraining, without the need for any distributional (re-)training. while @xcite inject factual knowledge from wikidata @xcite into bert, in this work, we investigate two resources that are commonly assumed to contain generalpurpose and common sense knowledge: 1 concept-net @xcite and the open mind common sense (omcs) corpus @xcite , from which the conceptnet graph was (semi-)automatically extracted. for our first model, dubbed cn-adapt, we first create a synthetic corpus by randomly traversing the con-ceptnet graph and then learn adapter parameters with masked language modelling (mlm) training @xcite on that synthetic corpus. for our second model, named om-adapt, we learn the adapter parameters via mlm training directly on the omcs corpus. we evaluate both models on the glue benchmark, where we observe limited improvements over bert on a subset of glue tasks. however, a more detailed inspection reveals large improvements over the base bert model (up to 20 matthews correlation points) on language inference (nli) subsets labeled as requiring world knowledge or knowledge about named entities. investigating further, we relate this result to the fact that conceptnet and omcs contain much more of what in downstream is considered to be factual world knowledge than what is judged as common sense knowledge. our findings pinpoint the need for more detailed analyses of compatibility between (1) the types of knowledge contained by external resources; and (2) the types of knowledge that benefit concrete downstream tasks; within the emerging body of work on injecting knowledge into pretrained transformers. ## knowledge injection models in this work, we are primarily set to investigate if injecting specific types of knowledge (given in the external resource) benefits downstream inference that clearly requires those exact types of knowledge. because of this, we use the arguably most straightforward mechanisms for injecting the con-ceptnet and omcs information into bert, and leave the exploration of potentially more effective knowledge injection objectives for future work. we 1 our results in §3.2 scrutinize this assumption. unwrapping conceptnet. following established previous work @xcite , we induce a synthetic corpus from conceptnet by randomly traversing its graph. we convert relation strings into nl phrases (e.g., synonyms to is a synonym of ) and duplicate the object node of a triple, using it as the subject for the next sentence. for example, from the path "alcoholism causes ----→ stigma hascontext ------→ christianity partof ---→ religion" we create the text "alcoholism causes stigma. stigma is used in the context of christianity. christianity is part of religion.". we set the walk lengths to 30 relations and sample the starting and neighboring nodes from uniform distributions. in total, we performed 2,268,485 walks, resulting with the corpus of 34,560,307 synthetic sentences. adapter-based training. we follow @xcite and adopt the adapter-based architecture for which they report solid performance across the board. we inject bottleneck adapters into bert's transformer layers. in each transformer layer, we insert two bottleneck adapters: one after the multi-head attention sub-layer and another after the feed-forward sub-layer. let @xmath0, input to a bottleneck adapter. the bottleneck adapter, consisting of two feed-forward layers and a residual connection, yields the following output: ## evaluation we first briefly describe the downstream tasks and training details, and then proceed with the discussion of results obtained with our adapter models. ## conclusion we presented two simple strategies for injecting external knowledge from conceptnet and omcs corpus, respectively, into bert via bottleneck adapters. additional adapter parameters store the external knowledge and allow for the preservation of the rich distributional knowledge acquired in bert's pretraining in the original transformer parameters. we demonstrated the effectiveness of these models in language understanding settings that require precisely the type of knowledge that one finds in conceptnet and omcs, in which our adapter-based models outperform bert by up to 20 performance points. our findings stress the importance of having detailed analyses that com-pare (a) the types of knowledge found in external resources being injected against (b) the types of knowledge that a concrete downstream reasoning tasks requires. we hope this work motivates further research effort in the direction of fine-grained knowledge typing, both of explicit knowledge in external resources and the implicit knowledge stored in pretrained transformers.