O o MM ix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification (2021)

Abstract: Recent studies on neural networks with pre-trained weights (i.e., BERT) have mainly focused on a low-dimensional subspace, where the embedding vectors computed from input words (or their contexts) are located. In this work, we propose a new approach, called OoMMix, to finding and regularizing the remainder of the space, referred to as out-of-manifold, which cannot be accessed through the words. Specifically, we synthesize the out-of-manifold embeddings based on two embeddings obtained from actually-observed words, to utilize them for fine-tuning the network. A discriminator is trained to detect whether an input embedding is located inside the manifold or not, and simultaneously, a generator is optimized to produce new embeddings that can be easily identified as out-of-manifold by the discriminator. These two modules successfully collaborate in a unified and end-to-end manner for regularizing the out-of-manifold. Our extensive evaluation on various text classification benchmarks demonstrates the effectiveness of our approach, as well as its good compatibility with existing data augmentation techniques which aim to enhance the manifold.

## introduction neural networks with a word embedding table have been the most popular approach to a wide range of nlp applications. the great success of transformer-based contextual embeddings as well as masked language models @xcite @xcite makes it possible to exploit the pre-trained weights, fully optimized by using large-scale corpora, and it brought a major breakthrough to many problems. for this reason, most recent work on text classification has achieved state-of-the-art performances by finetuning the network initialized with the pre-trained weight @xcite . however, they suffer from extreme over-parameterization due to the large pre-trained weight, which allows them to be easily overfitted to its relatively small training data. along with outstanding performances of the pretrained weight, researchers have tried to reveal the underlying structure encoded in its embedding space @xcite . one of the important findings is that the contextual embeddings computed from words usually form a low-dimensional manifold @xcite . in particular, a quantitative analysis on the space @xcite , which measured the effective dimension size of bert after applying pca on its contextual embedding vectors, showed that 33% of dimensions covers 80% of the variance. in other words, only the low-dimensional subspace is utilized for finetuning bert, although a high-dimensional space (i.e., model weights with a high capacity) is provided for training. based on this finding on contextual embedding space, we aim to regularize the contextual embedding space for addressing the problem of over-parameterization, while focusing on the outside of the manifold (i.e., out-of-manifold) that cannot be accessed through the words. in this work, we propose a novel approach to discovering and leveraging the out-of-manifold for contextual embedding regularization. the key idea of our out-of-manifold regularization is to produce the embeddings that are located outside the manifold and utilize them to fine-tune the network for a target task. to effectively interact with the contextual embedding of bert, we adopt two additional modules, named as embedding generator and manifold discriminator. specifically, 1) the generator synthesizes the out-of-manifold embeddings by linearly interpolating two input embeddings computed from actually-observed words, and 2) the discriminator identifies whether an input embedding comes from the generator (i.e., the synthesized embed-ding) or the sequence of words (i.e., the actual embedding). the joint optimization encourages the generator to output the out-of-manifold embeddings that can be easily distinguished from the actual embeddings by the discriminator, and the discriminator to learn the decision boundary between the in-manifold and out-of-manifold embeddings. in the end, the fine-tuning on the synthesized out-of-manifold embeddings tightly regularizes the contextual embedding space of bert. the experimental results on several text classification benchmarks validate the effectiveness of our approach. in particular, our approach using a parameterized generator significantly outperforms the state-of-the-art mixup approach whose mixing strategy needs to be manually given by a programmer. furthermore, our approach shows good compatibility with various data augmentation techniques, since the target space we focus on for regularization (i.e., out-of-manifold) does not overlap with the space the data augmentation techniques have paid attention to (i.e., in-manifold). the in-depth analyses on our modules provide an insight into how the out-of-manifold regularization manipulates the contextual embedding space of bert. ## related work in this section, we briefly review two approaches to regularizing over-parameterized network based on auxiliary tasks and auxiliary data. ## method in this section, we propose a novel mixup approach, termed as oommix, to regularize the outof-manifold in contextual embedding space for text classification. we first briefly remind the architecture of bert, then introduce two modules used for out-of-manifold regularization, which are embedding generator and manifold discriminator. ## experiments in this section, we present the experimental results supporting the superiority of oommix among the recent mixup approaches in text classification. also, we investigate its compatibility with other data augmentation techniques. finally, we provide in-depth analyses on our approach to further validate the effect of out-of-manifold regularization. ## conclusion this paper proposes oommix to regularize out-ofmanifold in the contextual embedding space. our main motivation is that the embeddings computed from the words only utilize a low-dimensional manifold while a high-dimensional space is available for the model capacity. therefore, oommix discovers the embeddings that are useful for the target task but cannot be accessed through the words. with the help of the manifold discriminator, the embedding generator successfully produces out-of-manifold embeddings with their labels. we demonstrate the effectiveness of oommix and its compatibility with the existing data augmentation techniques. our approach is a bit counter-intuitive in that the embeddings that cannot be accessed through the actual words are helpful for the target model. as the discrete features from texts (i.e., words), embedded into the high-dimensional continuous space where their contexts are encoded, cannot cover the whole space, the uncovered space also should be carefully considered for any target tasks. in this sense, we need to regularize the out-of-manifold to prevent anomalous behavior in that space, which is especially important for a large pre-trained contextual embedding space.