Re-embedding Difficult Samples via Mutual Information Constrained Semantically Oversampling for Imbalanced Text Classification (2021)

Abstract: Difficult samples of the minority class in imbalanced text classification are usually hard to be classified as they are embedded into an overlapping semantic region with the majority class. In this paper, we propose a Mutual Information constrained Semantically Oversampling framework (MISO) that can generate anchor instances to help the backbone network determine the re-embedding position of a non-overlapping representation for each difficult sample. MISO consists of (1) a semantic fusion module that learns entangled semantics among difficult and majority samples with an adaptive multi-head attention mechanism, (2) a mutual information loss that forces our model to learn new representations of entangled semantics in the non-overlapping region of the minority class, and (3) a coupled adversarial encoder-decoder that fine-tunes disentangled semantic representations to remain their correlations with the minority class, and then using these disentangled semantic representations to generate anchor instances for each difficult sample. Experiments on a variety of imbalanced text classification tasks demonstrate that anchor instances help classifiers achieve significant improvements over strong baselines.

## introduction data imbalance is a long-standing challenge in the text classification tasks such as sentiment analysis @xcite , intent detection @xcite and spam detection @xcite , where the distribution of training data over classes is skewed. for example, the number of minority samples accounts for only 28% of training instances in sms spam dataset @xcite and 14% in opin-rank dataset @xcite . data imbalanced issue is more severe in toutiao dataset @xcite with a minoritymajority ratio of 1:122 (hereafter imbalance ratio). as the class distribution tends to be extremely imbalanced, texts from the minority class(es) may be easily categorized into the majority class(es) @xcite @xcite . recent studies have shown that some minority samples, called difficult samples as they locate in the overlapping semantic region, are more important for imbalanced text classification than those far from the overlapping semantic region @xcite . as illustrated in figure 1 , the difficult samples have similar (entangled) embeddings with some majority samples in this overlapping semantic region as they are similar to these majority samples about surface forms (e.g., n-gram or syntax). for example, in yelp.p dataset, a review "my parents didn't want to go back to beautiful miami" is a difficult sample of the minority class. however, many words of this sample have also occurred in a positive review of the majority class "the beauty of miami made jessie reluctant to go back". the latest research on imbalanced learning separated the learning procedure into representation learning (i.e., the backbone network) and classification (i.e., the classifier), and achieved the stateof-the-art performance by freezing the backbone network and fine-tuning the classifier weights to obtain balanced decision boundaries @xcite . however, they ignore that entangled semantic representations of difficult samples make the decision boundaries hard to be clearly determined. to this end, we propose to generate anchor instances, which have similar surface forms with difficult samples but be embedded in the nonoverlapping region of the minority class, to help the backbone network learn disentangled semantic representations for difficult samples. see figure 1 , consider the aforementioned difficult sample, two anchor-instance-generation steps are taken. first, entangled semantics of "beauty", "miami" and "go back" in the difficult sample, are decoupled from the anchor instance. second, semantics of "to my disappointment" and "help me" in some nondifficult samples of the minority class are injected into the anchor instance. in order to make this generation framework feasible, we should answer the following three questions: (1) given a difficult sample paired with a majority sample, how can we capture their entangled semantics? (2) how can we decouple and inject semantics from and into an anchor instance? (3) merging anchor instances into the original data may change the data distribution and hence have a negative impact on non-difficult sample classification. how can we avoid this? to address these problems, we propose a mutual information constrained semantically oversampling (miso) approach with three essential components: a semantic fusion module (sfm), a mutual information (mi) loss, and a coupled adversarial generator (cag) based on encoderdecoder networks. sfm is leveraged to adaptively find entangled semantics among difficult samples and majority samples in the overlapping region. formally, we assume the majority and minority classes as two random variables a and b, their entangled semantics can be modeled as the mutual information between the two classes: a,b) p(a)p(b) . we introduce mi loss for parallel decoupling and injecting, which is symmetric and smooth. semantic representations outputted from sfm constrained by mi loss are fed into cag for generating anchor instances, with an adversarial strategy to ensure that the original data distribution is not destroyed. in addition to the proposed miso framework for imbalanced text classification, other contributions of our work can be summarized as follows. first, the boundary of the minority class learned by mi loss is theoretically proved as a (near-)optimal boundary. second, we further theoretically show that the new distribution after adding anchor instances is consistent with the original distribution of the minority class (see proof.1 and proof.2 in appendix). third, experiment results demonstrate that text classifiers, trained on rebalanced datasets with anchor instances generated by miso, outperform state-of-the-art methods by an average of 2.7% in nine datasets. the average success rate of moving difficult samples into non-overlapping region is 13.7%, which validates the effectiveness and robustness of miso in handling difficult samples. ## problem statement let x + := {x + 1 , ..., x + n + } ∈ r n + ×l be a training set of positive samples with the minority class distribution n + , x -:= {x - 1 , ..., x - n -} ∈ r n -×l be a training set of negative samples with the majority class distribution n -, where x i is the i-th sentence consisting of up to l tokens, n -and n + are the number of instances in the majority and minority classes, respectively. data imbalance can be roughly divided into the slight imbalance (e.g., the goal of miso is to make z close to n + but far from n -. in doing so, we generate a set of anchor instances y + := {y + 1 , ..., y + t } ∈ r t×l with z as their disentangled representations for difficult samples in x + , where t is the number of anchor instances. we further define a marginal distribution over y + as u ψ + ,σ,ω , where ψ + , σ, ω are the parameters of a continuous and differentiable parametric function e ψ + (i.e., the minority encoder-decoder), sfm, and mi loss. ## miso we introduce the overall architecture and then elaborate on each component of miso in this section. ## training and inference training objective in summary, the goal of miso is to use cag to generate anchor instance via constructing z as disentangled semantic representations by sfm jointly with mi loss, s σ • t ω . the final function is defined as follows: training in order to calculate the mutual information, two mines need to be pre-trained before training the entire miso. furthermore, the warmup of the minority encoder is a necessary condition for pre-training mines, ensuring that the inputs of mines are reliable. therefore, we first freeze sfm and the discriminators to pre-train the minority encoder-decoder. secondly, we follow the method proposed by @xcite to pretrain mines. we have found that the training challenge lies in how to train mines when sfm is frozen. to solve this, we simulate the output of sfm using a trick. notably, we concatenate k and q to obtain a set of 2d-dimensional vectors and feed them into a feedforward neural network to obtain a set of d-dimensional vectors q. we use q as the inputs of the decoders to participate in the pre-training of the encoder-decoder. finally, we can use q, q, k as the inputs to pre-train their mines. the whole training process is shown in algorithm 1 (lines 1-9). inference once miso is trained, we can use it to generate anchor instances for difficult samples of the minority class. lines 10-12 in algorithm 1 demonstrate the inference with miso. this oversampling of anchor instances will not stop until the two classes are balanced. ## end 6 experiments we conducted experiments on several text classification tasks to examine the effectiveness of the proposed miso against the previous state-of-theart imbalanced learning methods. ## conclusion in this paper, we have presented an effective mutual information-constrained oversampling strategy, which re-embed difficult samples via a safe and robust method. our method makes the traditional text classification still feasible when dealing with imbalanced data in the real world. in future work, we will try to design a more effective backbone network for re-embedding difficult samples. by calculation, therefore, ∆ is a concave with respect to the discriminator d. when d dd ∆ = 0, ∆ is the maximization. in other words, then the optimal d * is computed as follows: