Robust Product Classification with Instance-Dependent Noise (2022)

Abstract: Noisy labels in large E-commerce product data (i.e., product items are placed into incorrect categories) is a critical issue for product categorization task because they are unavoidable, non-trivial to remove and degrade prediction performance significantly. Training a product title classification model which is robust to noisy labels in the data is very important to make product classification applications more practical. In this paper, we study the impact of instance-dependent noise to performance of product title classification by comparing our data denoising algorithm and different noise-resistance training algorithms which were designed to prevent a classifier model from over-fitting to noise. We develop a simple yet effective Deep Neural Network for product title classification to use as a base classifier. Along with recent methods of stimulating instance-dependent noise, we propose a novel noise stimulation algorithm based on product title similarity. Our experiments cover multiple datasets, various noise methods and different training solutions. Results uncover the limit of classification task when noise rate is not negligible and data distribution is highly skewed.

## introduction product classification is a quintessential ecommerce machine learning problem in which product items are placed into their respective categories. with recent advancements of deep @xcite ., text only) and multimodal (e.g., text and image) models have been developed to predict larger numbers of items and categories with better accuracy @xcite @xcite . however, one of the fundamental assumptions behind such models is the availability of large and highquality labeled datasets. access to such datasets is usually costly or infeasible in some settings. large product datasets usually suffer from annotation er-rors, i.e., products are assigned to incorrect categories, partially due to complex category structure, confusing categories and similar titles. the problem of noisy labels is even more severe when product category distribution is highly imbalanced with heavy-tail @xcite . therefore, a text classifier which is robust to noisy labels present in training data is critical for highperforming product classification applications. while machine learning in the presence of label noise has been studied for decades, most of prior studies experimented in computer vision domain @xcite , and only a few research was conducted in text classification @xcite . without an annotated dataset with manually-identified label noise, classical approaches for label noise stimulation assume class-conditional noise (ccn) where the probability of an item having label corrupted depends on the original and noisy labels. with this assumption, all products of "men's watches" category have the sample probability to be assigned "women's watches" label. this is not generally correct. for instance, product titles having phrase "men's watches" are less likely mis-labeled. recent research addresses more general label noise, i.e., instance-dependent noise (idn), that an item is mis-labeled with a probability depending on its original label and features. in this paper, we present a comprehensive study on improving product title classification in the presence of idn. we develop a simple yet effective deep neural network for text classification and show that our model performs well on different product title datasets ranging from small to medium sizes, balanced to skewed distributions, and tens to over a hundred categories. to generate noisy labels for experiments, our first contribution is an idn stimulation algorithm which flips an item's label based on its similarity to items of other categories. noisy label data generated by our method is com-pared with prior idn stimulation methods for their impact to model accuracy degradation. to make the model robust to label noise, our second contribution is a data augmentation method that reduces noise rate and thus improves model's accuracy. we compare three state-of-the-art deep neural network training algorithms to train a classifier on data with label noise generated by different methods. from experimental results we discuss lessons learned for product title classification in production. to the best of our knowledge, this work is the first time that noise-resistance model training is studied in e-commerce domain, which is our third contribution. ## related work automatic product categorization has been well studied to address its challenges including large number of items and categories, and hierarchical categories structure @xcite @xcite . the largescale nature of product data leads to a critical issue of noisy labels. for example, an e-commerce website reported that 15% of product listings by sellers have incorrect labels @xcite . @xcite attempted to use a latent topic model to help manually inspect noisy categories and remove incorrect samples. our current study focuses on fully automated methods for data denoising and noise-resistance training to prevent models from over-fitting to noisy samples. training deep neural networks (dnn) with noisy labels is challenging because dnn's large learning capacity make them highly susceptible to over-fitting to noise @xcite . early work stacked dnn with layers to model noise-transition matrix assuming classconditional noise, i.e., noisy label ŷ only depends on true label y but not on the input x @xcite . because noise transition matrix can be difficult to learn or not feasible in real-world settings, other directions targeted to selecting clean samples in each mini-batch and use them to update dnn's parameters @xcite . among those, coteaching @xcite and coteaching + @xcite showed the effectiveness of cross-training two networks simultaneously in that each network sends selective samples for the other to learn. a more realistic assumption of noisy labels is instance-dependent noise (idn) in which probability of noisy label ŷ depends on true label y and input x @xcite . among stateof-the-art work on idn, self-evolution average label -seal @xcite and progressive label correction -plc @xcite are representatives of label refurbishment @xcite that uses softmax output to assign soft labels to training instances. we compare seal, plc and coteaching + on training a product title classifier with label noise. ## datasets in this study, we employ 6 public datasets for product classification. while some datasets have multimodal inputs, e.g., product titles, descriptions, images, we use only product title inputs and leave other fields for a future work. this restriction may prevent us from achieving the best possible performance by incorporating other information-rich inputs @xcite . however, our main motivation is to evaluate noise-resistance training approaches. for each dataset, we filter-out category labels with less than 10 samples, then apply stratified random sampling to split 10% for testing and 90% for training. we leave a study of few-shot learning for product title classification for future work. hyper-parameters of models and training algorithms are fine-tuned within training sets when needed. in experiments with noisy labels, only training samples have label corrupted while testing sets are unchanged. this assures a realistic evaluation that model accuracies are measured against ground-truth disregarding how the model was trained. to measure skewness of data label distribution, we calculate kl-divergence from the actual category distribution to uniform distribution. data statistics are shown in as shown in ## base model for product title categorization we develop a product title classifier based on lstm-cnns architecture proposed in @xcite . the network architecture is depicted in figure 1 . input encoding layer is a concatenation of word-embeddings (looking-up function against glove pre-trained embeddings @xcite ) and character embeddings (output of a character-cnn layer). the sequence of embedding vectors is passed to a bidirectional recurrent neural network of lstm cells @xcite . prediction is carried by a dense layer whose input is last hidden state of bidirectional lstm. the dnn is implemented to evaluate our implementation, we compare model performance with fine-tunning the pretrained bert-base uncased language model @xcite . results on 6 datasets with clean label are reported in ## instance-dependent noise stimulation a common approach for automated idn generation is to train one or a set of classifiers on clean label data, and use such classifiers to generate noisy labels for the whole dataset. related studies can be different on how to maintain a pool of classifiers, e.g., different checkpoints of a single models or different model architectures, and label placement strategies, e.g., whether replacing clean label samples with noisy counterparts or allowing a sample word-emb char-cnn lstm-cell lstm-cell lstm-cell lstm-cell lstm-cell lstm-cell word-emb char-cnn word-emb char-cnn fully-connected layer … … … concatenation concatenation softmax layer 5s materials fastenal tape 108' t a p e padding padding … character embedding convolution max pooling character representation char-cnn to have multiple copies with different labels. we follow @xcite to use replacement strategy which is considered a more difficult setting. we implement four different idn algorithms, and adjust parameters to generate noisy label data with noise rates (i.e., ratio of noisy label samples over data size) in two levels: 0.2 (low) and 0.4 (medium). last-epoch idn: we train a base classifier for 10 epochs to obtain the network corresponding to last epoch checkpoint. the trained network is executed on training data to obtain prediction confidence score (i.e., output of softmax layer) for every sample. following the formula of noise type-i described in @xcite , we corrupt item category from the most confident label to the second confident label. this method uses a noise factor parameter to control noise rate, thus we run different trials to probe the noise factors that give us noise rates of interest. ## conclusion in this paper, we evaluate a denoising algorithm and three training approaches for product title classification with category labels corrupted by instancedependent noise. we introduce a new idn stimulation algorithm and compare with three idn algorithms from prior studies to explore model performance on a wider range of noise type. therefore our study can evaluate model robustness to idn more reliably. overall we find that coteaching + achieves highest average improvement and be our recommendation when applying to new product data without prior knowledge of noise cause or true distribution. seal can be a good method when we have clean validation data to evaluate. however, all methods studied in this paper have difficulties to address noise in large scale data with highly imbalanced class distribution, especially when noise rate is high. for such extreme setting, application of data denoising and noise-resistance training algorithms could not yield to reasonable performance for applying to production. for a future work, we plan to combine multiple techniques including transformer-based classifier as a more advanced model and stacking data denoising with noise-resistance training.