Free Lunch for Efficient Textual Commonsense Integration in Language Models (2023)

Abstract: Recent years have witnessed the emergence of textual commonsense knowledge bases, aimed at providing more nuanced and context-rich knowledge. The integration of external commonsense into language models has been shown to be a key enabler in advancing the state-of-the-art for a wide range of NLP tasks. However, incorporating textual commonsense descriptions is computationally expensive, as compared to encoding conventional symbolic knowledge. In this paper, we propose a method to improve its efficiency without modifying the model. Our idea is to group training samples with similar commonsense descriptions into a single batch, thus reusing the encoded description across multiple samples. We theoretically investigate this problem and demonstrate that its upper bound can be reduced to the classic graph k-cut problem . Consequently, we propose a spectral clustering-based algorithm to solve this problem. Extensive experiments illustrate that the proposed batch partitioning approach effectively reduces the computational cost while preserving performance. The efficiency improvement is more pronounced on larger datasets and on devices with more memory capacity, attesting to its practical utility for large-scale applications.

## introduction while pre-trained language models have made substantial progress in natural language processing, they still lack certain knowledge. thus it is critical to incorporate external knowledge sources @xcite @xcite . previous research has primarily focused on incorporating symbolic knowledge from structured knowledge graphs. recently, realizing the lack of expressiveness and contextualization of symbolic knowledge, many forms of commonsense knowledge bases are constructed, such as if-then knowledge @xcite and discourse knowledge @xcite . the integration of such textual commonsense knowledge into language models has been shown to improve the state of the art for various tasks, such as named entity recognition @xcite and commonsense knowledge base completion @xcite . however, integrating such commonsense knowledge are computationally expensive. commonsense knowledge in text form requires more complex encoders (e.g. transformer @xcite ), as opposed to the simple lookup operation for discrete symbolic knowledge. the feedforward and back-propagation process for the text encoder is significantly more computationally expensive than the standalone symbolic knowledge embeddings. therefore, it is essential to reduce the computational cost for efficient integration of textual commonsense knowledge, particularly for large-scale applications. in this paper, we propose a method to accelerate the process of incorporating textual commonsense knowledge into language models. our approach is based on the observation that if multiple training samples in a mini-batch share the same commonsense description, the encoding for that description can be reused across those samples. in other words, we only need to encode each distinct description in a mini-batch once. for example, consider the training samples x 1•••4 and the associated commonsense t 1•••4 in fig. 1 . in the batch partitioning in fig. 1a , the samples in one batch have no shared descriptions, requiring seven times of commonsense encoding for t i . however, in the batch partitioning shown in fig. 1b , each description will be encoded only once, resulting in only four times of encoding for t i . the cost of encoding the commonsense is significantly reduced by effective partitioning of the training samples. therefore, our goal is to group the training samples in such a way as to minimize the total number of distinct commonsense descriptions per mini-batch. to optimize the batch partitioning, we begin by theoretically analyzing the objective ( §2.1). our the main contributions of this paper are as follows: (1) we propose the use of batch partitioning for improving the efficiency of textual commonsense integration for language models. (2) we theoretically demonstrate that the batch partitioning problem can be reduced to the classic graph kcut problem, and we use the well-studied spectral clustering to optimize it. (3) we empirically show that the efficiency of integrating commonsense descriptions can be significantly improved without sacrificing effectiveness. the acceleration is even more pronounced for large-scale training. ## the batch partitioning problem in this section, we analyze the training efficiency w.r.t. batch partitioning. we first show in § 2.1 that the complexity of the model depends on the number of corresponding knowledge descriptions per sample. then, in § 2.2, we formally define this batch partitioning problem. ## related work integrating knowledge into language models has been one of the focuses of language modeling research in recent years. the main integration methods currently include using pre-trained entity embeddings, and constructing knowledgeaware corpora. ernie @xcite , know-bert @xcite , and kglm @xcite are typical methods using pre-trained entity embeddings. ernie uses wikidata (vrandečić and krötzsch, 2014) as the knowledge base and uses transe @xcite to encode knowledge. knowbert, on the other hand, uses skip-gram like objective @xcite based on wikipedia descriptions as the pre-trained entity embeddings. in addition, knowbert adds a loss on entity linking to the pre-trained objective. kglm @xcite allows modification/updating of knowledge by building a local knowledge graph for the target sentence. wklm @xcite constructs a corpus of incorrect knowledge descriptions by replacing wikipedia's entities with different entities of the same type. it trains the model to identify incorrect and correct knowl-edge descriptions. recently, models that integrate textual knowledge have also been proposed. in this paper, we adopt the model structure in ok-transformer @xcite . textual knowledge bases noting the deficiencies of symbolic knowledge in terms of expressiveness and contextual information representation, some work has started to use text as a form of knowledge. atomic @xcite is a large-scale manually annotated common-sense textual knowledge base that includes social interaction, event-centered, physical entity. atomic contains knowledge like (personx reaches personx's home, before, personx needs to park the car). aser @xcite ) is an eventuality knowledge graph of activities, states, events, and their relations. its knowledge atoms are in natural language form, e.g. (i do not have lunch, succession, i am hungry). comet @xcite is an extension of atomic based on the generative language model. it mainly solves the problem of insufficient coverage of atomic. some primitive research @xcite has started to apply these textual knowledge bases in some specific tasks. ok-transformer @xcite ) is proposed to integrate textual knowledge for general purposes. however, in our experimental tests, it takes too much time in encoding the commonsense. to our knowledge, there is still a lack of research on how to integrate textual knowledge into general text understanding tasks efficiently. comparison with dense textual knowledge retriever when introducing external texts, another style is to use a retriever that returns only top k candidate texts in terms of similarity @xcite @xcite . however, this method requires a heavy pre-training process to learn the retriever. on the other hand, for the textual knowledge base we use in this paper, we can directly use the manually labeled trigger words for each knowledge description to retrieve knowledge. therefore, in this paper, we focus on how to efficiently and effectively integrate knowledge from a textual knowledge base. high-performance language models more general techniques for high-performance language models have also received extensive studies. the main approaches of previous studies include (1) model compression and quantization @xcite , and (2) efficient repre-sentation of long texts @xcite . however, the model compression approaches require heavy pre-training before they can be adapted to language models. moreover, the techniques for optimizing the efficiency for long text do not have significant effects on short texts @xcite . besides, each commonsense description we considered in this paper tends to be short. in addition, these works have not considered the characteristics of the knowledge integration problem in this paper, i.e., a training sample corresponds to multiple candidate textual knowledge from the knowledge base. ## experiments in this section, we conducted extensive experiments to evaluate batch partitioning. we aim to address the following key questions: 1. ( § 5.2) how much is the efficiency improvement of batch partitioning? can it improve efficiency without sacrificing effectiveness? 2. ( § 5.3) what is the scalability of batch partitioning as an acceleration method, and can it be applied to large-scale training? ## conclusion in this paper, we study how to improve the efficiency of incorporating commonsense knowledge in language models. due to the high encoding costs of commonsense descriptions, it is crucial to reduce their encoding complexity. our idea is that by carefully dividing samples with similar descriptions into the same batch, the knowledge encoding utilization can be improved. with such an idea, we theoretically analyze the optimization objective of this batch partitioning. we found that the upper bound of this problem can be reduced to the classical graph k-cut problem. we propose to use the well-studied spectral clustering algorithm to optimize the batch partitioning. by experimenting with a variety of tasks, we show that the proposed batch partitioning approaches its upper bound in terms of both effectiveness and efficiency. and the method is more applicable for larger datasets and on devices with more capabilities.