Language Model Pre-Training with Sparse Latent Typing (2022)

Abstract: Modern large-scale Pre-trained Language Models (PLMs) have achieved tremendous success on a wide range of downstream tasks. However, most of the LM pre-training objectives only focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences. In this paper, we manage to push the language models to obtain a deeper understanding of sentences by proposing a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge. Besides, the language model pre-trained with such an objective also significantly improves Information Extraction related downstream tasks in both supervised and few-shot settings. Our code is publicly available at https://github.com/renll/SparseLT.

## introduction transformer-based pre-trained language models (plms) have achieved significant success on a wide range of nlp tasks. however, typical pre-training objectives for plms only focus on teaching the model to directly reconstruct text-level words or sentences, but have not sought to obtain deeper sentence understanding by learning latent-level interpretable representations. for example, transformerdecoder models like the openai gpt series @xcite @xcite the task of language modeling for pre-training, and transformer-encoder models like bert @xcite and roberta @xcite are trained by predicting the masked tokens within a sentence. both of these training objectives merely train the models to recover the masked tokens or predict the next words or sentences, while ignoring to learn latent-level representations of sentences that could be potentially useful for both better language understanding and downstream tasks. pre-training a language model to learn latent representations is extremely hard: first, there are no ground-truth labels for the latent representations that could be used for reliable supervised learning. during pre-training, the model is only given an unlabeled text corpus over which to identify latent representations such as sentence-level keywords and structures. this means the training process must be strictly self-supervised @xcite . furthermore, to be interpretable, the latent representations for natural language texts are supposed to be discrete, which further complicates the design of a completely differentiable training framework. to push the language models to learn deeper understandings of sentences, in this paper, we propose a novel pre-training framework, sparse latent typing, that enables the language model to sparsely extract sentence-level keywords with meaningful latent types. we have tackled all abovementioned challenges and our framework is fully differentiable and completely self-supervised. as shown in figure 1 , given an input sentence from the pre-training corpus, we introduce a latent typing mechanism to jointly selects and classifies the keywords from the sentence into a category of randomly initialized latent types. we implement such a latent classification model based on gumbel sampling @xcite to make sure the overall pre-training framework is differentiable. since there are no ground-truth labels available for the selected keywords and latent types, we incorporate an one-layer transformer decoder into the training pipeline to map the fused token and latent type representations back to the original sentence, and use the sentence reconstruction loss to control for adequate usefulness of the latent representations. our approach provides the decoder model with a shortcut to directly access the encoded token representations, so that the latent representation for each of the input tokens can be learned as an auxiliary type representation. for pre-training objectives, in addition to minimizing the sentence reconstruction error, we also introduce a novel typing sparsity loss to minimize the number of token representation selected for latent typing. a kl-divergence based diversity loss is also proposed to encourage a diverse selection of the latent types. experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge. besides, the language model pre-trained with such an objective also significantly improves information extraction related downstream tasks in both supervised and few-shot settings. in summary, our contributions are three-fold: latent structure learning there are also several studies @xcite that incorporate latent structure learning into language model pre-training. particularly, @xcite also proposes to use a transformer decoder layer to reconstruct the original sentence to provide training signals. however, instead of learning coarse-grained sentence representations, we focus on learning fine-grained latent type representation that are interpretable and useful at the token level. to meet this end, we propose a series of novel training objectives and architecture designs to facilitate a sparse selection and typing of the token representations in the latent space. ## problem formulation given a text corpus d composed of text sentences s, we use @xmath0, • • • , w n }, s ∼ s to represent a sentence consisting of n tokens. assuming a text encoder f : @xmath1, • • • , x n , our latent type classifier h : @xmath2, x ) then selects a subset of token representations x1 , • • • , xt , @xmath3, • • • , z t . each of the token types z i is selected from a latent embedding space c consisting of v @xmath4. the text decoder g : (z, x ) → s then reconstructs the original sentence s through the pair of latent types and selected token representations (z, x ). the objective of sparse latent typing is to find pairs of latent types and token representations that are as compact as possible but still contain the necessary information for reconstructing the original input sentences. formally, we want to minimize the following joint objective, ## learning sparse latent types to tackle the non-differentiable problem of the size of the selected typing pairs @xmath5, x )|, we first take a closer look at the latent type classifier h which decides the latent type z i of each token representation x i . our insight is that we can regard the action of not selecting a token representation as a frozen zero type vector c 1 = 0 ∈ c. we then do an element-wise multiplication between z i and x i to obtain the representations @xmath6. the advantages of this approach are that (1) the element-wise multiplication naturally prevents the gradient from being propagated to the token representations that are classified as the zero type vector c 1 , (2) the element-wise multiplication directly modulates the gradients of the token representations with the latent type vectors. this can in principle provide better guidance to the text encoder with the information of the latent vectors than can be provided by other vector fusion operators such as element-wise addition or vector concatenation. based on this framework, we developed a novel typing sparsity loss in section 4.2 to approximately minimize the typing pairs size t . while our approach is generally applicable for any text encoder and decoder, specific neural architectures used in this work are discussed in section 5.1. in our framework, the latent type classifier h is simplified as a mapping h ′ : @xmath7. the simplified text decoder g ′ : @xmath8. ⊗ is the vector fusion operator and should be interpreted as element-wise multiplication in this work. the proposed architecture for sparse latent typing is illustrated in figure 2 , which is further explained in the following subsections. ## experiments in our experiments, we first conduct intrinsic evaluation to investigate whether the model can successfully learn word selections with meaningful latent types during pre-training. then, we apply our model on both supervised and few-shot ie tasks to evaluate the effectiveness of our pre-training framework on downstream tasks. ## analysis in this section, we address the following research questions on sparse latent typing. how are the latent types distributed over the encoded token representations? we draw a t-sne (van der @xcite plot of the encoded token representations x of 1,000 sentences (30792 tokens in total) randomly sampled from the pre-training corpus in figure 3 . the token representations are colored with their corresponding latent type indices. from the figure, we can observe that the token representations begin to be distributed as individual islands with the same colors after 300k steps of pre-training from scratch. this implies that our sparse latent typing objective can effectively encourage the clustering of the token representations in a small latent space defined by 64 randomly initialized type embeddings. we can also observe a similar trend of latent clustering for the bert-sparselt model, which is illustrated in appendix b, figure 4 . can typing sparsity loss effectively control the sparsity of token selections? we pretrained three bert-sparselt models with different weighting factors β of the typing sparsity loss to investigate its influence on latent type selections and sentence reconstruction. (additional samples for β = 0.05 are included in appendix b, to what extent are the learned latent types interpretable? in ## conclusion in this paper, we propose a novel language model pre-training framework that encourages the model to sparsely extract sentence-level keywords with meaningful latent types in a completely selfsupervised manner. experimental results and analysis demonstrate that incorporating sparse latent type learning early in the pre-training stage will not only facilitate the model to learn sentence-level keyword selections with interpretable latent types, but also improves downstream information extraction tasks in both supervised and few-shot settings.