Weighted Contrastive Learning With False Negative Control to Help Long-tailed Product Classification (2023)

Abstract: Item categorization (IC) aims to classify product descriptions into leaf nodes in a categorical taxonomy, which is a key technology used in a wide range of applications. Along with the fact that most datasets often has a long-tailed distribution, classification performances on tail labels tend to be poor due to scarce supervision, causing many issues in real-life applications. To address IC task’s long-tail issue, K-positive contrastive loss (KCL) is proposed on image classification task and can be applied on the IC task when using text-based contrastive learning, e.g., SimCSE. However, one shortcoming of using KCL has been neglected in previous research: false negative (FN) instances may harm the KCL’s representation learning. To address the FN issue in the KCL, we proposed to re-weight the positive pairs in the KCL loss with a regularization that the sum of weights should be constrained to K+1 as close as possible. After controlling FN instances with the proposed method, IC performance has been further improved and is superior to other LT-addressing methods.

## introduction item categorization (ic) aims to classify a product into a node of a taxonomy hierarchy. the textual descriptions of the products are used as the input and thus the task can be formulated as a text classification problem. ic is a fundamental task in e-commerce and the base for many applications such as personal recommendation and query understanding. one of the major challenges in building a highly effective real-life ic system is the serious long-tailed (lt) problem-a few head classes have the majority of the product items, while each of the remaining (large number of) tail classes contains only a few items. consequently, the scarce supervision available for these tail classes tends to cause unsatisfactory classification performance. in the most recent years, several novel lt-addressing methods, e.g., methods utilizing selfsupervision @xcite and contrastive learning (cl) @xcite , have emerged in computer vision. however, the related research on natural language processing (nlp) tasks is still limited. however, when utilizing unsupervised contrastive learning, e.g., k-positive contrastive loss (kcl) @xcite , the issue of appearing false negative (fn) samples hurts model learning. figure 1 shows an example of the fn issue and the performance impact reported in @xcite . in this paper, to build ic models and use kcl to solve the lt issue, we propose a novel method to control the fn issue, which entails assigning different weights for each positive sample in the kcl loss and tries to keep the sum of these weights equal to a predefined value. the experimental results on the three amazon product category datasets show that the proposed contrastive learning methods help on improving the model performance on tail classes and the fn controlling can further improve cl-based lt-addressing method. our main contributions can be summarized as: ## related work many methods have been proposed to address the lt issue. one category of those methods resamples the data to balance the label distribution, e.g., @xcite . another category of methods assign different weights to samples based on their label frequencies, e.g., focal loss @xcite class-balanced loss @xcite , label-distribution-aware margin loss (ldam) @xcite and so on. some of those loss-balanced methods are also applied to the nlp domain in @xcite . in addition, few shot learning @xcite and transfer learning @xcite methods are also proposed for long tail classification. recently, a two-stage training strategy (exampled in @xcite ), which decouples the learning a feature encoder and the learning of a classifier, has become influential in computer vision and shows its superior performance on addressing the lt issue. contrastive learning (cl) has been found to be effective in providing high-quality encoders in a simple self-learning fashion. the cl-based text representation learning has become a hot research topic in nlp. simcse @xcite uses dropout operations to be an effective text augmentation and can learn effective text representations. in the lt-addressing two-stage method, selflearning which discards the influence of label distribution has been used in its representation learning stage, e.g., @xcite . besides simply using self-supervision, including the supervision signal from existing labels can improve the representation learning @xcite . however, introducing semantics information may suffer from the lt issue and hurt the performance of tail classes. to address this issue, k-positive contrastive loss @xcite is proposed to learn balanced feature representations. an instance is called false negative (fn), if any in-batch negative instance shares the label carried by the anchor sample. fn samples are found to be harmful to cl methods and corresponding mitigation methods are proposed @xcite . ## methodology let x denote the title of a product and y its category label. then ic can be formulated as a text classification problem: given a product title x, we a model to predict the class label y, where h and h + are the representations of the anchor sample x and its corresponding positive sample x + , respectively. h - is the representations of negative sample set x -of the given the anchor sample x and positive sample x + . h -∈ h -is the representation of the negative sample x -in x -where x -is the negative sample set given the anchor sample x and its positive sample x + . τ is a temperature hyper-parameter and sim(•, •) denotes the cosine similarity of the two vectors. ## experiments datasets. the experiments are performed on the public amazon dataset @xcite which is a widely used benchmark. following @xcite , we use three categories of amazon product datasets: automotive, beauty, and electronics. each sample in the automotive datasets has a title and a category label. all three datasets have long-tail issue. 1 overall performance. we compare the proposed method with the following models: bert with cross-entropy loss (bert-ce), crt @xcite , unsupervised simcse (simcse us ), and controlling false negative instances with attraction (kcl-fna) and elimination (kcl-fne). note that except for bert-ce, all other baseline models use the two-stage approach to address the long tail issue, in which the classifier is trained using a balanced data set. the experimental results are shown in ## conclusion in large-scaled item categorization tasks, category labels are naturally distributed in a long tail pattern, which challenges the performance on tail classes due to severe supervision missing. to address this challenge, we adopt a two-stage lt-addressing method that was originally proposed in the image classification task. to make this method work on our text classification task, we use the recently proposed simcse @xcite to do an effective text transformation and kcl loss in the representation learning stage. furthermore, we recognize there are false negative samples caused by using the kcl loss and propose a novel controlling method to reduce the corresponding negative influences. the experimental results prove that the proposed method helps improve the performance on long-tailed data and the false negative controlling can further help boost the performance when using kcl. while we worked on item classification in this paper, we will extend the model to other problems.