BERT Goes Shopping: Comparing Distributional Models for Product Representations (2021)

Abstract: Word embeddings (e.g., word2vec) have been applied successfully to eCommerce products through prod2vec. Inspired by the recent performance improvements on several NLP tasks brought by contextualized embeddings, we propose to transfer BERT-like architectures to eCommerce: our model - Prod2BERT - is trained to generate representations of products through masked session modeling. Through extensive experiments over multiple shops, different tasks, and a range of design choices, we systematically compare the accuracy of Prod2BERT and prod2vec embeddings: while Prod2BERT is found to be superior in several scenarios, we highlight the importance of resources and hyperparameters in the best performing models. Finally, we provide guidelines to practitioners for training embeddings under a variety of computational and data constraints.

## introduction distributional semantics @xcite is built on the assumption that the meaning of a word is given by the contexts in which it appears: word embeddings obtained from co-occurrence patterns through word2vec @xcite , proved to be both accurate by themselves in representing lexical meaning, and very useful as components of larger natural language processing (nlp) architectures @xcite . the empirical success and scalability of word2vec gave rise to many domain-specific models @xcite @xcite : in ecommerce, prod2vec is trained replacing words in a sentence with product interactions in a shopping session @xcite , eventually generating vector representations of the products. the key intuition is the same underlying word2vec -you can tell a lot about a product by the company it keeps (in shopping sessions). the model enjoyed immediate success in the field and is now essential to nlp and information retrieval (ir) use cases in ecommerce @xcite . as a key improvement over word2vec, the nlp community has recently introduced contextualized representations, in which a word like play would have different embeddings depending on the general topic (e.g. a sentence about theater vs soccer), whereas in word2vec the word play is going to have only one vector. transformer-based architectures @xcite in large-scale models -such as bert @xcite -achieved sota results in many tasks @xcite . as transformers are being applied outside of nlp @xcite , it is natural to ask whether we are missing a fruitful analogy with product representations. it is a priori reasonable to think that a pair of sneakers can have different representations depending on the shopping context: is the user interested in buying these shoes because they are running shoes, or because these shoes are made by her favorite brand? in this work, we explore the adaptation of bertlike architectures to ecommerce: through extensive experimentation on downstream tasks and empirical benchmarks on typical digital retailers, we discuss advantages and disadvantages of contextualized embeddings when compared to traditional prod2vec. we summarize our main contributions as follows: ## conclusion and future work inspired by the success of transformer-based models in nlp, this work explores contextualized product representations as trained through a bert-inspired neural network, prod2bert. by thoroughly benchmarking prod2bert against prod2vec in a multi-shop setting, we were able to uncover important insights on the relationship between hyperparameters, adaptation strategies and ecommerce performances on one side, and we could quantify for the first time quality gains across different deployment scenarios, on the other. if we were to sum up our findings for interested practitioners, these are our highlights: while our findings are encouraging, there are still many interesting questions to tackle when pushing prod2bert further. in particular, our results require a more detailed discussion with respect to the success of bert for textual representations, with a focus on the differences between words and products: for example, an important aspect of bert is the tokenizer, that splits words into subwords; this component is absent in our setting because there exists no straightforward concept of "sub-product" -while far from conclusive, it should be noted that our preliminary experiments using categories as "morphemes" that attach to product identifiers did not produce significant improvements. we leave the answer to these questions -as well as the possibility of adapting prod2bert to even more tasks -to the next iteration of this project. as a parting note, we would like to emphasize that prod2bert has been so far the largest and (economically) more significant experiment run by coveo: while we do believe that the methodology and findings here presented have significant practical value for the community, we also recognize that, for example, not all possible ablation studies were performed in the present work. as bianchi and hovy (2021) describe, replicating and comparing some models is rapidly becoming prohibitive in term of costs for both companies and universities. even if the debate on the social impact of largescale models often feels very complex @xcite -and, sometimes, removed from our day-to-day duties -prod2bert gave us a glimpse of what unequal access to resources may mean in more meaningful contexts. while we (as in "humanity we") try to find a solution, we (as in "authors we") may find temporary solace knowing that good ol' prod2vec is still pretty competitive.