Learning Robust Models for e-Commerce Product Search (2020)

Abstract: Showing items that do not match search query intent degrades customer experience in e-commerce. These mismatches result from counterfactual biases of the ranking algorithms toward noisy behavioral signals such as clicks and purchases in the search logs. Mitigating the problem requires a large labeled dataset, which is expensive and time-consuming to obtain. In this paper, we develop a deep, end-to-end model that learns to effectively classify mismatches and to generate hard mismatched examples to improve the classifier. We train the model end-to-end by introducing a latent variable into the cross-entropy loss that alternates between using the real and generated samples. This not only makes the classifier more robust but also boosts the overall ranking performance. Our model achieves a relative gain compared to baselines by over 26% in F-score, and over 17% in Area Under PR curve. On live search traffic, our model gains significant improvement in multiple countries.

## introduction deep learning models have shown excellent performance in the natural language domain, and this success has inspired practitioners to adapt these models to information retrieval tasks @xcite . however, deep learning has not succeeded in these tasks due to the lack of massive labeled datasets @xcite . another reason is that word-based representations @xcite are less useful in representing complex, informal search queries @xcite and hence provide limited understanding of the search intent. in the absence of explicit knowledge of which documents are "matched" with a search query and which are "mismatched", it is hard to learn robust deep learning models that understand the query intent and find high-quality, relevant documents. text-based product search is even more challenging. simple modifications to the input query (or a product title) can completely change the search intent (or the product type, respectively). take, for example, the query gray iphone x by which a user is looking for a specific phone. slightly modified queries such as iphone x charger and case for iphone x refer to different products. therefore, it is hard for distributed representations to capture the nuances. moreover, noisy user-behavioral signals from clicks and purchases (e.g., users purchased a phone while searching for a charger) can lead to biases in the ranking algorithms. as such, even top-ranked items may not match the search intent. in this paper, we consider the problem of identifying query-item mismatches to enhance the ranking performance in product search. this task typically requires a large labeled dataset of matches and mismatches that we will respectively refer to as negative and positive samples. even if we can partly afford the expensive and time-consuming labeling, acquired datasets are unbalanced and lack hard positive samples, preventing the classifier from learning a robust decision boundary. however, the above examples gray iphone x and iphone x charger motivate that meaningful positive samples can be artificially generated by leveraging the labeled data. in fact, we can heuristically construct a large number of negatives by observing which items are commonly purchased in response to the corresponding query. the question is that can we use such negatives to synthesize hard-to-classify positives to robustify the classifier? we illustrate the goal of the generation in figure 1 . to this end, we develop a deep, end-to-end model that learns to identify mismatched queryitem pairs and is also capable of generating mismatched queries given an item. the task of the generator is twofold: it has to be able to generate hard-to-classify samples so that the classifier to the extreme right is a telescope, which is an easy-to-classify example (red box). close to the classification boundary is a hiking shoe (orange dotted box) which is a hard-to-classify positive. we want to train a generator that can learn to generate such hard samples. learns a more robust decision boundary; it also needs to generate realistic queries. using matched query-item pairs allows the generator to synthesize hard-to-classify mismatches based on an efficient encoder-decoder architecture. this has a distinct advantage over generating samples from noise, as in generative adversarial networks @xcite or via dithering the learned representations to make the model more robust @xcite . we include our classifier and generator in an end-to-end model. the classifier only requires continuous representations of the generated query as the second input instead of a discrete text sequence. this key property enables us to use efficient gradient-based optimization techniques and bypass reinforcement learning-based methods @xcite , which are significantly more complex, and also recently developed heuristic approaches to generate adversarial text samples @xcite . to achieve this, we modify the objective function in a way that makes the end-to-end training possible via sampling a binary latent variable, avoiding the min-max optimization for gans @xcite . we perform extensive experiments on a mismatch dataset in an e-commerce company. the proposed model outperforms deep learning baselines by over 26% in f-score and 17% in relative aupr score and performs significantly better than gbdt models, which are widely used in practice. including the query generator helps achieve higher gains than merely dithering the vector representation of the query. we also show that the generative model can indeed generate hard-to-classify mismatches. when integrated with the ranking component of a real-world product search engine, our model outperforms the baseline methods in multiple countries on an online a/b test evaluation. ## proposed model: quarts we present our proposed model, namely quarts (query-based adversarial learning for robust textual search) in figure 2 . quarts is composed of three components: (i) an lstm and attentionbased classifier, (ii) a variational encoder-decoder query generator (ved) and (iii) a state combiner. due to space constraints, we defer the details of (i) and (ii) in the appendix. the lstm classifier (i) is adapted from the entailment model in @xcite , with some changes to fit the product search task (see appendix a.1). the ved generator (ii) takes a matched pair (i, q) as input and outputs a new query q gen so that the pair (i, q gen ) is mismatched while q gen stays lexically similar to q. as an example, if @xmath0, space gray and @xmath1, we can generate q @xmath2. in this case, q gen is similar to q, but (i, q gen ) constitutes a product mismatch. to have an end-to-end model, we combine the query representations computed by the classifier and the generator to form a proper input to the attention layer. we need to make sure that the modifications still allow us to efficiently backpropagate the gradients of the loss function during training. to achieve this, we add a merging layer shown by the orange box in figure 2 . this layer computes sh gen + (1 -s)h, @xmath3, q))z where h, h gen are the corresponding lstm representations of the input q and q gen , and z ∼ bernoulli(p) is a random binary variable that controls whether the input query q or the generated query q gen is used. when @xmath4, quarts essentially computes the probability of mismatch. let us explain how the real label y and the switch z combine to yield the desired outputs. as @xmath5, q) is a real positive, we want to leverage it to train the classifier f θ (•). in this case, @xmath6. when @xmath7, we can either use this sample to train the classifier or use it to generate adversarial representations h gen . this process is controlled by z. when @xmath8, we use h gen , else h. the value of z determines whether we want to use the datapoint as-is for training, or instead use the "fake" query via the ved module. a second consideration is how to enable efficient training on f θ (•) and the generator g ψ (•). let x @xmath9, q gen ) be the datapoint we will use to train f θ (•) using the output from g ψ (•). in this case, since @xmath10, @xmath11, we use z as a proxy "label" to train f θ (•). for samples @xmath12, 2, . . . , n , we sample z i ∼ bernoulli(p) for some @xmath13, 1) to decide which negative samples have labels flipped. we modify the cross entropy loss as below, with l θ being the weighted cross-entropy loss: note that (1) is differentiable in θ, ψ and notably h gen -the generated representations of q gen . since we do not use the actual generated query, we need not resort to heuristics or policy gradient-based optimization methods to minimize (1). before train-ing quarts end-to-end, we pre-train the classifier and the ved on proper data. the pseudocode of the end-to-end training is shown in algorithm 1. we used a human-labeled dataset of query-item pairs, obtained from an e-commerce search platform. there are in total @xmath14.2m pairs of which only a small fraction are mismatches. a separate test set of ∼ 100k labeled pairs was used to evaluate all methods. we further have 3m query-item pairs that are deemed "matched" by considering items that are purchased frequently in response to those queries from the search logs. this acts as the augmentation dataset for the quarts model. ## conclusion and future work we developed an end-to-end model with hard to classify query generation for retrieval in ecommerce product search. we built upon ideas for textual entailment, and used a word by word attention layer to help create item representations conditioned on an input query. we trained a generator that yields representations of queries that are mismatched to a source item, while at the same time being "realistic". this allows us to address the figure 4: word-by-word attention for a mismatched (top) and matched (bottom) query-item pair. rows represent query words, columns represent item words, with lighter shares representing larger weights. protector is attended to more on the left whereas case is attended to more in the right class imbalance of our datasets, while also generating samples that help robustly train the classifier. to train the model end to end, we modified the cross-entropy loss, allowing us to avoid optimizing a minimax objective. experiments on an offline dataset and live product search traffic showed that our method improves significantly over baselines.