Consistent Text Categorization using Data Augmentation in e-Commerce (2023)

Abstract: The categorization of massive e-Commerce data is a crucial, well-studied task, which is prevalent in industrial settings. In this work, we aim to improve an existing product categorization model that is already in use by a major web company, serving multiple applications. At its core, the product categorization model is a text classification model that takes a product title as an input and outputs the most suitable category out of thousands of available candidates. Upon a closer inspection, we found inconsistencies in the labeling of similar items. For example, minor modifications of the product title pertaining to colors or measurements majorly impacted the model’s output. This phenomenon can negatively affect downstream recommendation or search applications, leading to a sub-optimal user experience. To address this issue, we propose a new framework for consistent text categorization. Our goal is to improve the model’s consistency while maintaining its production-level performance. We use a semi-supervised approach for data augmentation and presents two different methods for utilizing unlabeled samples. One method relies directly on existing catalogs, while the other uses a generative model. We compare the pros and cons of each approach and present our experimental results.

## introduction in the last two decades, widespread use of ecommerce platforms such as amazon and ebay has contributed to a substantial growth in online retail. such platforms rely on both explicit and implicit product features in order to deliver a satisfying user experience. there, the inferred product category is typically a crucial signal for many application such as browsing, search and recommendation. we focus on improving an existing product categorization model, we refer to as 'the categorizer', that is employed by our company for fast categorization of billions of items on a daily basis. it classifies e-commerce items, such as products or deals, based on a predefined hierarchy of categories, namely gpt (google product taxonomy). given a product title, the categorizer assigns the most appropriate label in the taxonomy. the model itself is highly scalable and effective, so it is well-suited for settings with large and rapidly growing item catalogs. in our company, the categorizer is used as a standalone component in various e-commerce related services, such as recommendation, search, and ad ranking. a recent examination of the categorizer's output revealed inconsistencies in the labeling of similar items. it was evident that in some cases small variations in product titles, such as those relating to colors or measurements, significantly affect the categorizer's output. this inconsistency negatively impacts search and recommendation algorithms that rely on the inferred category, leading to a poor user experience. the concept of consistency in nlp tasks has been studied in various research works, including robustness to paraphrasing @xcite and robustness to adversarial attacks @xcite . other works relate consistency issues with the misuse of spurious features during the learning phase @xcite @xcite . when examining the performance of the categorizer in terms of accuracy alone, the inconsistency issue may be overlooked. but, since many recommendation pipelines depend on the output of the product categorizer, an inconsistent model can have severe implications on the user experience. in most cases, the differences include returning the parent category or a sibling category, rather than a completely different category path. to tackle this inconsistency problem, we use different data augmentation techniques and enrich the training data with item versioning, leading to a more consistent model. data augmentation for improving various nlp tasks has been widely studied and surveyed @xcite , and particularly in the context of consistency @xcite . generating such data, both manually @xcite and automatically @xcite @xcite , has shown to contribute to the robustness of learnt models in different settings. we chose to use data augmentation, without changing the current architecture of the already-in-use product categorizer for two main reasons. first, for scalability reasons, any change in the architecture might degrade the model's ability to infer the categories of billions of items per day. second, maintaining the current model architecture expedites the productization process and requires only minimal engineering effort. this work defines a new framework, consistent semi-supervised learning (consistent-ssl), for consistent text categorization in the context of e-commerce (section 2). we use an unlabeled clustered dataset as a source of legit item versioning. the dataset is derived from product catalogs, and includes clusters of different versions of items. we present two different methods to utilize this unlabeled clustered data: a self-training method and a generative approach (section 3). we describe the datasets and the experimental framework we use for the evaluation of the proposed methods (section 4). finally, we detail results, showing an improvement in the consistency rate of 4-10% above the baseline model, and discuss the advantages and weaknesses of each method (section 5). ## consistent semi-supervised learning we now formalize our notion of consistent classification and introduce the settings for consistent semi-supervised learning (consistent-ssl). ## methods we present two methods for consistent-ssl, consistent self training (cst) and consistent generative augmentation (cga). both methods utilize the unlabeled samples from d u for data augmentation. in each method we create an augmented set d aug using d u and train a classifier f on d l ∪d aug . this approach optimizes indirectly the objective of eq. ( 3 ), as we add additional training samples d aug that consists of different versions of the same items. the goal is to expose f to a more diverse set of item versions in training time, making it more robust to minor changes. let us review our approach using an illustrative example. consider a dataset that contains clothing items. assuming that d l , which was sampled from the distribution p, exhibits a spurious correlation between color of an item to its category (e.g. most of the black items are coats and most of the red items are dresses), then a classifier that was trained solely on d l will tend to rely on the color of the item when it predicts its category. when applying the model, v could change the items' colors and therefore the classifier will not be consistent (e.g. if v transforms a black coat to a red one, the classifier might predict different categories). but, assuming the training data includes an item in multiple colors (e.g. black coat, red coat, blue coat, etc.), with the same label (e.g. coats & jackets), then a model that is trained on such data will not relate a specific color to a specific label. such a model will be encouraged to ignore the color of an item when it predicts the label, and therefore will be more robust to changes in color. note that colors here are only an example of one kind of versioning of items. spurious features in the data could be related to colors, measurements, models, materials etc. ## empirical evaluation we now present our experimental results. we note that in all of our experiments, we use a model that is based on fasttext @xcite architecture, and has an hierarchical structure. this specific model is found to perform well on our task, as it takes into account the hierarchical structure nature of the labels. for more details, see appendix c. ## results and discussion we train each examined model 5 times and present the mean score of the achieved results. for each model, we compare the weighted average f1 score for the accuracy test and the consistency rate of the consistency test. the colorssizes-blind model performs similarly to the baseline for both measurements; the slight changes are within the std range, thus making the differences insignificant compared to the baseline model. this is an evidence that the item versioning is more complex than just changing the size or color and includes title rephrasing concepts that are hard to tackle in a trivial way. in addition, the results show that both of the cst versions, complete and sub-sampled, achieve significantly higher consistency rates than the baseline, gaining lifts of 10% and 7% respectively. on the other hand, both of the methods yield lower f1 scores, reducing lift by 1.65% and 0.6% respectively. a possible cause of the degradation in the f1 score is the differences between the data distribution of d l , which we sample the accuracy test from, and the data distribution of d u which we use to augment our training data. the fact that using the sub-sampled version of d u mitigates most of this degradation supports this claim. an additional cause could be the usage of the noisy pseudo-labels in the augmented set instead of the unavailable ground truth labels. note that the amount of added data using d u to tackle consistency is bigger than the original d l , which aims to tackle accuracy. the focus in terms of the training shifts from an accuracy problem to a consistency problem, thus hurting the f1 of the new model. the higher consistency rate of cst-full compared to the cst-sub-sampled can be explained by a difference of more than 1m samples in the size of d aug . similarly, the cga method also improves the consistency rate, gaining lift of 4.5%, and doesn't significantly affect the accuracy score. as mentioned, we use a threshold @xmath0.7, thus including 440k samples in d aug . these additional samples correspond to a similar distribution as d l .the improvement in both the consistency and the accuracy indicates that the generative model is able to correctly learn the real-world item versioning and produce a significant amount of data with high accuracy labels and the same distribution as in the accuracy test. summarizing the above, our experiments highlight three key factors in the consistent-ssl framework: 1) scale -enriching the learning set with more examples of item versioning increases the consistency. 2) quality -augmenting the data with real-world samples is better than using generative ones in term of performance. 3) distribution -preserving the original distribution in the augmented set is important for maintaining good accuracy. ## conclusions this work presents a new framework for consistent text categorization in the context of e-commerce. the aim of this work is to improve a product categorization model that serves various services of a major web company. we address the labeling inconsistency issues found in the categorization of similar items, leading to poor user experience in related recommendation and search applications. our framework utilizes an unlabeled clustered dataset in two ways: a self-training approach and a generativeaugmentation method. we performed a thorough investigation of the two approaches and investigated several factors that majorly influence their performance. our experimental results suggest that both proposed methods improve the consistency rate by 4% to 10%, while maintaining the accuracy of the current production model. finally, our study illustrates the trade off between the quality and the scale of the augmented dataset, and its impact on the performance of both methods.