Label Smoothing for Text Mining (2022)

Abstract: Current text mining models are trained with 0-1 hard label that indicates whether an instance belongs to a class, ignoring rich information of the relevance degree. Soft label, which involved each label of varying degrees than the hard label, is considered more suitable for describing instances. The process of generating soft labels from hard labels is defined as label smoothing (LS). Classical LS methods focus on universal data mining tasks so that they ignore the valuable text features in text mining tasks. This paper presents a novel keyword-based LS method to automatically generate soft labels from hard labels via exploiting the relevance between labels and text instances. Generated soft labels are then incorporated into existing models as auxiliary targets during the training stage, capable of improving models without adding any extra parameters. Results of extensive experiments on text classification and large-scale text retrieval datasets demonstrate that soft labels generated by our method contain rich knowledge of text features, improving the performance of corresponding models under both balanced and unbalanced settings.

## introduction instances in most text mining datasets are usually assigned by one label. such label, called hard label, reflects the logical relationship between the label and the instance. in most cases, hard labels are utilized for training models that learn to discriminate between classes with training objectives to maximize the log probability of the correct class. however, there are various real-world text mining tasks where there are not just two possibilities whether an instance belongs to a specific class since instances may be involved with multiple labels of varying degrees. in such scenarios, soft label @xcite , which involves s1: it was nice. the cashier was chipper. wish they would crank the a/c a little more though. restroom was a little dirty. hard label (1, 0) soft label (0.6, 0.4) the explicit relative importance of each label, is a more reasonable description of an instance. for example, in the sentence s1 shown in the the key to generate informative soft labels for text instances is to exploit the semantic relevance between instances and labels accurately. we are inspired by the fact that a set of highly representative words can represent semantic information of labels and instances (e.g., s1 in to verify the effectiveness of our method, we conduct extensive experiments on the two most typical application scenarios in text mining: text classification and large-scale text retrieval. experimental results demonstrate that models equipped with kwls gain significant improvements over the original models, especially in the highly unbalanced large-scale text retrieval task @xcite . to further analyze the ability of kwls to deal with unbalanced problems, we construct various unbalanced datasets to simulate multifarious unbalanced problems in text mining. experimental results on different unbalanced settings show that kwls may bring extra supervised signals to facilitate model learning. our main contributions are listed as follows: ## related work label smoothing, a form of output distribution regularization, prevents overfitting of a neural network by softening the ground-truth labels to penalize overconfident outputs @xcite , has made tremendous achievements in many data mining fields. for example: @xcite reveal that label smoothing improves model calibration and summarize several behaviors observed while training deep neural networks with label smoothing. @xcite propose an improved mutual information estimator based on binning and show the correlation between compression of softmax layer representations and generalization, which may explain why networks trained with label smoothing generalize so well. the above works analyze the effectiveness of classical label smoothing methods in various fields. the two most classical label smoothing methods are follows: @xcite first introduced label smoothing that improves accuracy by computing cross-entropy not with the "hard" targets from the dataset, but with a weighted mixture of these targets with the uniform distribution, and many state-ofthe-art image classification models have incorporated label smoothing into training procedures ever since. zhang and sabuncu (2020) regard selfdistillation as a label smoothing method and propose a novel instance-specific label smoothing technique that promotes predictive diversity without the need for a separately trained teacher model. however, these two methods mentioned above are proposed for universal data mining tasks so that they cannot leverage the valuable text features in text mining datasets. this paper proposes a kwls method explicitly designed for text mining, which incorporates semantic relevance between labels and instances into soft labels. experimental results demonstrate that our method is more suitable for text data. it is worth noting that the primary purpose of ls is incorporating the possibility (or uncertainty) into the original hard label to facilitate model performances rather than generating the ground truth soft labels. ## task description and formulation given a training set @xmath0, l i )|1 ≤ @xmath1, x i is a sequence of words x @xmath2, w 2 , ..., w n } where w 1 , w 2 , and w n are words in the sentence and n is the length of the sequence. the hard label vector of x i is denoted by l @xmath3, l 2 i , ..., l p i ), where l j @xmath4, 1} denotes whether label l j describes x i , where p is the total number of labels. our task is to generate the soft label the proposed kwls method is introduced in the following section. ## keyword-based label smoothing to represent labels and instances by a set of highly representative words, we propose to first determine keywords for instances and labels then calculate correlations between them. our method mainly involves the following four components. ## conclusion we have presented our keyword-based label smoothing method (kwls) for text mining tasks and demonstrated it's usage and effect on model training. unlike previous works that focus on universal data mining tasks, our method is explicitly designed for text mining, which incorporates semantic relevance between labels and instances into soft labels. like other widely-used tricks of text mining, the technical design of kwls is simple yet effective, making it extremely easy to be applied in different text mining tasks, as shown in experiments. despite its simplicity, using soft labels generated by kwls as an auxiliary training target shows significant superiority in improving model performance, whether the data is balanced or unbalanced. our codes are released on the github. 4