S et C onv: A N ew A pproach for L earning from I mbalanced D ata (2020)

Abstract: For many real-world classification problems, e.g., sentiment classification, most existing machine learning methods are biased towards the majority class when the Imbalance Ratio (IR) is high. To address this problem, we propose a set convolution (SetConv) operation and an episodic training strategy to extract a single representative for each class, so that classifiers can later be trained on a balanced class distribution. We prove that our proposed algorithm is permutation-invariant despite the order of inputs, and experiments on multiple large-scale benchmark text datasets show the superiority of our proposed framework when compared to other SOTA methods.

## introduction in many real-world nlp applications, the collected data follow a skewed distribution @xcite @xcite , i.e., data from a few classes appear much more frequently than those of other classes. for example, tweets related to incidents such as shooting or fire are usually rarer compared to those about sports or entertainments. these data instances often represent objects of interest as their rareness may carry important and useful knowledge @xcite @xcite . however, most learning algorithms tend to inefficiently utilize them due to their disadvantage in the population @xcite . hence, learning discriminative models with imbalanced class distribution is an important and challenging task to the machine learning community. solutions proposed in previous literature can be generally divided into three categories @xcite : (1) data-level methods that employ undersampling or over-sampling technique to balance the class distributions @xcite @xcite . despite the success of these approaches on many applications, some of their drawbacks have been observed. resampling-based methods need to either remove lots of samples from the majority class or introduce a large amount of synthetic samples to the minority class, which may respectively lose important information or significantly increase the adverse correlation among samples @xcite . it is difficult to set the actual cost value in costsensitive approaches and they are often not given by expert before hand @xcite . also, how to guarantee and utilize the diversity of classification ensembles is still an open problem in ensemble-based methods @xcite . in this paper, we propose a novel set convolution (setconv) operation and a new training strategy named as episodic training to assist learning from imbalanced class distributions. the proposed solution naturally addresses the drawbacks of existing methods. specifically, setconv explicitly learns the weights of convolution kernels based on the intra-class and inter-class correlations, and uses the learned kernels to extract discriminative features from data of each class. it then compresses these features into a single class representative. these representatives are later applied for classification. thus, setconv helps the model to ignore samplespecific noisy information, and focuses on the latent concept not only common to different samples of the same class but also discriminative to other classes. on the other hand, in episodic training, we assign equal weights to different classes and do not perform resampling on data. moreover, at each iteration during training, the model is fed with an episode formed by a set of samples where the class imbalance ratio is preserved. it encourages the model learning to extract discriminative features even when class distribution is highly unbalanced. building models with setconv and episodic training has several additional benefits: ## experiment we evaluate setconv on two typical tasks, including incident detection on social media and sentiment classification. ## conclusion in this paper, we propose a novel permutationinvariant setconv operation and a new training strategy named as episodic training for learning from imbalanced class distributions. the combined utilization of them enables extracting the most discriminative features from data and automatically balancing the class distribution for the subsequent classifier. experiment results demonstrates the superiority of our approach when compared to sota methods. moreover, the proposed method can be easily migrated and applied to data of other types (e.g., images) with few modifications. although the performance of setconv shows its advantage in classification, it may not be appropriate for high-dimensional sparse data. it is because the large amount of 0s in these data may lead to close-to-zero convolution kernels and limit the model's capacity for classification. combining sparse deep learning techniques with setconv is a potential solution to this issue. we leave it for future work.