Augmenting Training Data for Massive Semantic Matching Models in Low-Traffic E -commerce Stores (2022)

Abstract: Extreme multi-label classification (XMC) systems have been successfully applied in e-commerce (Shen et al., 2020; Dahiya et al., 2021) for retrieving products based on customer behavior. Such systems require large amounts of customer behavior data (e.g. queries, clicks, purchases) for training. However, behavioral data is limited in low-traffic e-commerce stores, impacting performance of these systems. In this paper, we present a technique that augments behavioral training data via query reformulation. We use the Aggregated Label eXtreme Multi-label Classification (AL-XMC) system (Shen et al., 2020) as an example semantic matching model and show via crowd-sourced human judgments that, when the training data is augmented through query reformulations, the quality of AL-XMC improves over a baseline that does not use query reformulation. We also show in online A/B tests that our method significantly improves business metrics for the AL-XMC model.

## approach a pipeline diagram of our system is shown in figure 1 . the main goal of this work is to increase good quality query-product pairs with which to train an al-xmc model; we do this by generating alternative queries from known queries using a fine-tuned t5 model. to fine-tune the t5 model, we identify purchase-intent-preserving query pairs from historical customer data. in section 3 we describe our approach to constructing the data used to fine-tune the t5 model. ## constructing query reformulations we regard customer interaction data as tuples of the form ⟨q, t, p⟩, each consisting of a query q, interaction type t (eg: search, purchase, etc.) and product p. it is generally the case that newer stores do not have sufficient interaction data to train robust semantic models. to increase the data available to train semantic models, we can either increase the number of queries associated with the product, the number of products associated with the queries, or both. in a typical e-commerce site the number of products is usually fixed. it is not usually possible or desirable to artificially augment this set. instead we fine-tune a language model to generate query reformulations and associate the reformulated queries to products using the same interaction type t as the original query. the reformulated queries increase the variance of the tokens, including those from rare queries, in accordance with their distribution in the target store. in order for this to happen, the reformulated queries need to encompass the same purchase intent as the original queries. for a language model to create high-fidelity reformulations, we need an adequate corpus of intentpreserving query paraphrase pairs. however, generating such a corpus is not trivial. queries vary widely in specificity, from highly generic (gifts for teens) to highly specific (hp 63xl). also, a figure 2 : top 6 similarities of two example queries, sorted by pmi. while all the top matches to aquarium filter brush appear to refer to the same product as the key, many of the top matches to heart name tag for dog refer to products that are not good matches. ## experiments from a low resource store we collect 2.4m instances of behavioral association between queries and products. we select behaviors over a threshold level of activity (e.g. clicks and purchases), comprising 0.83m pairs, and use that to train a baseline al-xmc model; we call this data local. to apply query reformulation (qr), we first fine-tune a pretrained t5-base model for a sequence-to-sequence prediction task that uses query pairs from the clusters we generated in section 3. 4 we fine-tune for two epochs using batches of 32 intent-preserving query pairs and gradient accumulation with a factor of 16, for an effective batch size of 512. we size accuracy 130k 79.67% 460k 86.67% 4m 90.33% 40m 93.00% in addition to the local baseline and local+qr reformulation approaches, we compare our data augmentation method to integrative knowledge transfer (ikt) @xcite ) from a higher-resource store. to do this, we identify queries in the high-resource store that show strong interaction with a product that is available in both stores. we add those query-product associations to the training set in the target store. such an approach is of course only possible in a mature ecosystem where previously-established high-resource stores exist. ## evaluation to intrinsically evaluate the impact of fine-tuning data size on reformulation intent preservation, we randomly select 300 qr tuples and determine to what degree intent is preserved during reformulation, using a variety of data conditions to fine-tune t5. we count a reformulation as accurate if it has the same manually judged purchase intent as the input query and a search engine returns a similar set of products for both queries. to extrinsically evaluate, we consider the effectiveness of the al-xmc models when built under different data augmentation conditions. we consider both offline and online evaluation paradigms. ## discussion we compare the generated queries to those transferred by ikt48 and find that only 20k of the reformulated queries were not present in the ikt48 set. this indicates that the t5 model is able to generate high fidelity queries that customers are likely to use. in addition, data augmentation with query reformulations is able to achieve the same results as that with ikt48 with only about a third of the data. thus, the reformulated queries are able to capture the relevant information of the target locale. although ikt48 contains nearly the same information as qr, it is also noisier in the sense that it transfers queries that are relevant in the source (high-resource) store but may not be relevant in the target low-resource store. this noise necessitates several times more data to achieve the same performance. model training with the larger ikt data set takes approximately 4 times as long as training with the query reformulations dataset. ## related work data augmentation in the e-commerce space has dealt with reformulating queries in several ways. @xcite expand user queries using semantically similar terms according to a learned embedding space, while @xcite use query annotations to identify words to expand queries. both methods rely on a large amount of natural query reformulation by users to train their models, which we avoid. @xcite and @xcite propose synonym or paraphrase replacement reformulation approaches. the noisiness of these approaches, some of which expect significant context, are a bad fit for the product query use case. other works that leverage data enhancement via model-based generation include @xcite , who augment queries in black-box question answering tasks, @xcite , who generate bilingual parallel data via backtranslation, and @xcite who, similar to us, though in a machine translation context, reformulate training data via language model-based generation, in this case focusing on rare word replacement. ## conclusion in this paper, we have presented a method to augment training data for semantic models in lowresource e-commerce stores. our method is generic enough to be applied as training data augmentation for any model and does not require transfer from high-resource store data. we have shown that augmenting training data using query reformulations improves upon a baseline store-specific al-xmc semantic matching model in both offline evaluations as well as online business metrics. our methods increase the training data of a test low-traffic store from 0.83m to 16.14m, resulting in a quality improvement of 19.57% over the baseline model. we also see a significant boost to business metrics in online a/b tests. next we will explore similar data augmentation techniques for generating multilingual query reformulations, using the mt5 pretrained model @xcite . in addition to low traffic stores, this technique may even be applied to yet-to-be-launched locales where training data is missing completely, by forming pseudo-queries from product descriptions.