“Does it come in black?” CLIP -like models are zero-shot recommenders (2022)

Abstract: Product discovery is a crucial component for online shopping. However, item-to-item recommendations today do not allow users to explore changes along selected dimensions: given a query item, can a model suggest something similar but in a different color? We consider item recommendations of the comparative nature (e.g. “something darker”) and show how CLIP-based models can support this use case in a zero-shot manner. Leveraging a large model built for fashion, we introduce GradREC and its industry potential, and offer a first rounded assessment of its strength and weaknesses.

## introduction recommender systems (rss) are one of the most ubiquitous applications of machine learning (ml) in e-commerce @xcite , recently featuring novel benchmarks and extensive use of deep neural networks in @xcite @xcite . while details differ between neural architectures, they all share the principle that products are represented as points in a latent space, learned from user behavior, item meta-data or a combination of both @xcite . fig. 1 represents item-to-item recommendations as movements in the product space: starting from a query item -the white dress -, rss help shoppers to move either around their current location, or "jump" to a different one. adding to the blooming literature on substitute, complementary, popularity and exploration-based rss @xcite @xcite , this work presents gradrec, a new type of recommendation that introduces explicit directionality into the mix, by allowing exploration in selected directions through natural language: "something darker" will move the user from the white dress to the grey dress. in particular, we summarize our contributions as follows: first, we introduce gradrec as a new type of recommendation experience and a technical contribution -to the best of our knowledge, gradrec is the first zero-shot approach for language-based comparative recommendations, showing that clip-like @xcite models may enable recommendations to be generated on the fly without the need of explicitly defined labels for training or behavioral data. second, we devise both qualitative and quantitative evaluations to offer a first rounded assessment of the strengths and weaknesses of our proposal, and supplement our analysis with extensive visual examples. third, as part of our submission, we release to the community our fine-tuned weights, publish an interactive web-app for exploration, and open source our code to help reproducing our findings and building on them foot_0 . while we present our results as a preliminary investigation into the untapped capabilities of clip for retail, we do believe our methods to be interesting to a broad set of practitioners: those exploring recommendations for conversational and interactive commerce, and those leveraging deep learning for horizontally scalable saas products foot_1 . finally, while motivated by very practical concerns, this work contains new insights on the topology of the information encoded by over-parameterized neural networks, which could help our understanding of the kind of regularities that these models learn about our world. ## an industry perspective the intersection of product recommendation and natural language is a blooming research area: advances in neural nlp have been recently used for content-based recommendations @xcite , cold-start scenarios @xcite , language grounding @xcite , and explainable rss @xcite . a relatively new use case is provided by the growth in the market of interactive technologies, as intelligent virtual assistants (ivas) are expected to handle recommendations that increasingly encompass the expressiveness of natural language @xcite . while interaction is an opportunity, the limited real estate available to display recommendations is a constraint for ivas @xcite : since scrolling is limited, strategies for moving from one product to another (as in fig. 1 ) are crucial for ivas market penetration. in this work, we consider recommendations which are of the comparative form: given an item of focus -in a chat, a product page, etc. -, the shopper makes use of natural language queries to retrieve a second item (e.g. "shorter", "darker", etc.), related to the first but different along the specified attribute. while state-of-the-art ivas can already provide very simple recommendations through language (amazon, 2022), we are the first to suggest the existence of an entire new dimension and depth to mimic the interactions typical of a real-life shopping experience. when thinking about applying this method in a multi-tenant saas context, it is worth noting how small are the assumptions gradrec actually makes about the underlying inventory: while in the case of fashionclip and its dataset it is true that products often contain information about an attribute's intensity (e.g. "knee-length shorts"), the relationship between them is not explicitly encoded, yet it is inferred by gradrec. moreover, when applying these models across new catalogs, there is no guarantee descriptions would be as rich, or even using the same lexicon to describe the same attribute ("bermudas" vs "knee-length"). these considerations further highlight the strength of using a latent space derived from a general and flexible multi-modal model, and the non-trivial nature of extracting comparative recommendations. ## related work our work sits at the intersection of various recent technical advances in latent space manipulation and iterative ir. many recent works explore latent space manipulation of generative adversarial networks (gans) for purposes of fine-grained image editing @xcite ; jahanian et al. (2020) the idea of iterative search refinement using comparative information and attribute ranking is not new @xcite . however, previous work sit in the standard fully supervised "learning-to-rank" tradition. conversely, our approach operates in a zero-shot fashion by using both clip retrieval and clip representations to generate suggestions onthe-fly. finally, our work builds on top of the recent wave of contrastive-based methods for representational learning: while latent product representations have been extensively studied from multiple angles @xcite 4 gradient recs 4.1 overview gradrec, builds upon the multi-modal space induced by fashionclip. gradrec aims to traverse the latent space such that the intensity of an attribute of interest varies monotonically for products along that path, allowing us to make finegrained recommendations that require compara-tive knowledge. there are independently grounded reasons to expect this method to work. first, we have solid evidence that embedding spaces are able to encode recognizable "concepts" (e.g. lexical knowledge in word2vec @xcite , facial expressions in gans @xcite ). second, we perform an extensive evaluation of the fashionclip product space, focusing on attributes such as color and occasion: our qualitative assessment (section 4.2) verified that embeddings are indeed often clustered, further suggesting that movements in the "concept space" can be represented as paths in the latent space. ## experiments to investigate gradrec strengths and weakness, we offer a preliminary assessment of its capabilities over important fashion dimensions, such as product discovery. ## conclusion we introduced gradrec, a zero-shot approach for comparative recommendations, that showed promising results in our initial investigations. while further evaluation -especially, involving relevance judgments by humans -is needed to fully assess gradrec capabilities, we do believe that our work provides preliminary but novel insights into innovative application of large models in important industry use-cases. formality correctly, is however unable to preserve the visual style of the footwear correctly. as we have highlighted in section 5.4, gradrec performance is sensitive to the initial dataset retrieval performance: in this instance, the query "formal shoes" retrieves predominantly black, leather dress-shoes, thereby steering the traversal in that direction.