Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense (2022)

Abstract: We study how to enhance text representation via textual commonsense. We point out that commonsense has the nature of domain discrepancy. Namely, commonsense has different data formats and is domain-independent from the downstream task. This nature brings challenges to introducing commonsense in general text understanding tasks. A typical method of introducing textual knowledge is continuing pre-training over the commonsense corpus. However, it will cause catastrophic forgetting to the downstream task due to the domain discrepancy. In addition, previous methods of directly using textual descriptions as extra input information cannot apply to large-scale commonsense. In this paper, we propose to use large-scale out-of-domain commonsense to enhance text representation. In order to effectively incorporate the commonsense, we proposed OK-Transformer (Out-of-domain Knowledge enhanced Transformer). OK-Transformer effectively integrates commonsense descriptions and enhances them to the target text representation. In addition, OK-Transformer can adapt to the Transformer-based language models (e.g. BERT, RoBERTa) for free, without pre-training on large-scale unsupervised corpora. We have verified the effectiveness of OK-Transformer in multiple applications such as commonsense reasoning, general text classification, and low-resource commonsense settings.

## introduction although unsupervised language models have achieved big success on many tasks @xcite , they are incapable of learning low-frequency knowledge. for example, in the masked language model task in fig. 1 , even if we replace "kevin was" (left) with "jim was" (right), bert @xcite still predicts the masked word as sick, crying, dying, etc. this is because similar texts in its training corpus rarely describe the subject of "comforted". to improve the model's ability to generalize and understand low-frequency knowledge, we propose to incorporate commonsense into language models. in fig. 1 , to make correct predictions, we need to enhance the language model with the commonsense c 1 . however, commonsense has the nature of domain discrepancy. the downstream task and the commonsense knowledge have distribution discrepancies. taking the commonsense knowledge base we use (i.e. atomic2020 @xcite ) as an example, the distribution discrepancy is specifically manifested in (1) their data formats. the format of a commonsense description usually belongs to some specific patterns (e.g. "... as a result ...", "... because ..."), while the downstream tasks can have arbitrary patterns. (2) the commonsense belongs to the domain of event causality, while the downstream tasks may belong to arbitrary domains. here we highlight the challenges caused by the domain discrepancy. to introduce external textual knowledge to a pre-trained language model, a common practice is to continue pre-training the language model on the corpus of the external knowledge @xcite . however, the study @xcite also found that continuing pre-training requires external knowledge and downstream tasks to have similar domains. due to its domain discrepancy, introducing commonsense through continuing pre-training will cause catastrophic forgetting to downstream tasks, thereby injuring the effectiveness. we have verified this empirically in sec 6.3. therefore, the domain discrepancy prevents us from introducing commonsense by continuing pre-training. to enhance the representation of the target text with external commonsense, we propose to directly use its candidate commonsense as an extra input. another challenge is the scale of the commonsense. although expbert also allows extra textual commonsense as input, it only captures smallscale commonsense with a fixed size. in addition, when we introduce commonsense from a large-scale knowledge base for general purpose (i.e. atomic2020), unrelated commonsense (e.g. c 2 and c 3 in fig. 1 ) will certainly occur. however, expbert lacks the ability to distinguish related and unrelated commonsense. therefore, the power of large-scale commonsense knowledge was restricted in expbert. we will verify this empirically in sec 6.3. in order to incorporate the large-scale outof-domain commonsense, we propose the ok-transformer (out-of-domain knowledge enhanced transformer) on the basis of transformer @xcite . ok-transformer has two modules. the knowledge enhancement module is used to encode the target text with commonsense, and the knowledge integration module is used to encode and integrate all candidate commonsense. ok-transformer has two advantages. first, it fully represents the contextual information of the textual commonsense. second, it can be adapted to existing pre-trained language models (e.g. bert and roberta) for free. that is, we are able to adapt ok-transformer to the pre-trained language models, without pre-training ok-transformer over large-scale unsupervised corpora from scratch. some other methods are related to our work, such as introducing structured knowledge @xcite @xcite and plain text knowledge @xcite in language models. these methods do not represent the specific inductive bias of commonsense knowledge and therefore are not suitable to introduce commonsense. we will compare these studies with more details in sec 2. ## related work in this section, we compare different ways to introduce knowledge into language models. we divide the knowledge introduction methods into (1) continuing pre-training method @xcite and (2) explicit introduction in the downstream task @xcite . continuing pre-training the language model is effective when the external knowledge is similar to the downstream task @xcite . however, commonsense and downstream tasks have domain discrepancies, so continuing pre-training is unsuitable for introducing commonsense. we have empirically verified this in sec 6.3. introducing explicit knowledge in downstream tasks we classify the knowledge into structured knowledge, plain text, and semi-structured knowledge, depending on its form. the entries of structured knowledge are represented as individual embeddings @xcite @xcite , while commonsense descriptions in this paper can be represented more accurately by the contextual information of their word sequences. ## problem setup: commonsense as the extra input we consider a text classification task where the text x and its label y are provided for training. assuming that the candidate commonsense descriptions for enhancing x come from a large-scale commonsense knowledge base (i.e. atomic2020), we retrieve candidate commonsense for x as the extra input. we denote the commonsense descriptions for x as cs(x) = {c 1 • • • c n }, where each c i is a commonsense description. the retrieval process will be shown in sec 6. the model takes both x and cs(x) as the input. since atomic2020 contains if-then knowledge for general purposes, the problem setup can be expanded to a broad range of text understanding tasks. the goal of training is to find parameter θ that minimizes the loss of training examples given the texts and candidate commonsense descriptions: ## ok-transformer in this section, we propose ok-transformer based on transformer to introduce extra commonsense descriptions. we first show ok-transformer on an abstract level in sec 4.1. then we elaborate two modules within it, i.e. knowledge enhancement and knowledge integration, in sec 4.2 and sec 4.3, respectively. ## adaptation to pre-trained language models in this section, we take bert as an example to illustrate how we adapt ok-transfomer to existing pre-trained language models. we denote the adapted model as ok-bert. an important manifestation of the effectiveness of the transformer structure is its applications in large-scale pre-trained models (e.g. bert, roberta). in order to introduce external knowledge, many other studies conduct training over large-scale unsupervised corpus @xcite . however, ok-transformer is able to directly adapt to the existing pre-trained language models for free. in other words, when adapting ok-transformer to ok-bert, we directly use the parameters of each transformer layer of bert to initialize the ok-transformer layers of ok-bert. this property greatly improves the applicability of ok-bert. in the rest of this section, we will describe how transformer (1) , transformer (2) , and transformer 3) are adapted respectively in sec 5.1, and how to fine-tune ok-bert in sec 5.2. ## experiments we evaluate the effectiveness of our proposed models in three scenarios: cloze-style commonsense reasoning, text classification, and low-resource commonsense settings. all the experiments run over a computer with 4 nvidia tesla v100 gpus. ## conclusion in this paper, we study how to use commonsense to enhance the general text representation. we first analyzed the challenges brought by the domain discrepancy of commonsense. then, we propose ok-transformer to allow commonsense integration and enhancement. in the experiments, we verified the effectiveness of our proposed models in a variety of scenarios, including commonsense reasoning, general text classification, and low-resource commonsense. our models consistently outperform the baselines. we have also empirically analyzed other properties (e.g. interpretability) of the model.