Paper: pEBR: A Probabilistic Approach to Embedding Based Retrieval (ICLR-2024)

Reviewer comment (Experiments):
Assessment is sparse. The baseline chosen, DSSM, is rather old (from 2013).

Author response:
Since our proposed method is applicable to various types of networks, thus we choose the classical DSSM model as a representative to present the model performance.

Reviewer comment (Experiments):
Comparison on one dataset and its unclear if it is public.

Author response:
It is the real data in our online e-commerce system, we consider to publish it in the future.

Reviewer comment (Experiments):
Can you comment on the dependence on the amount of training data? Would the precision and recall be higher with much smaller click data than 87M clicks?

Author response:
In general, larger dataset will result in better performance. However, our method has advantages with smaller datasets since we introduce probabilistic distribution assumption to improve the generalization ability.

Reviewer comment (Experiments):
Why not compare to SoTa model for retrieval?

Author response:
Since our proposed method is applicable to various types of networks, thus we choose the classical DSSM model as a representative to present the model performance.

Reviewer comment (Experiments):
Have you tried evaluated it on other public datsets?

Author response:
Not yet, but we consider to publish our dataset in the future.

Reviewer comment (Novelty):
It would be great if the authors could enhance the related work with probabilistic embedding approaches, especially a few from the domain of images and metric learning, and also draw some parallels. For example, Probabilistic Embeddings for Cross-Modal Retrieval, CVPR 2021.

Author response:
We think our work quite different from the probabilistic embedding work presented at CVPR 2021. As the probabilistic embedding is mainly focus on the learning of polysemy, our work aims to learn popularity. Therefore, we didn't delve into a detailed discussion of probabilistic embedding. But it is also a kind of probabilistic method, we will consider to discuss more about these methods in the future.

Reviewer comment (Experiments):
It is very difficult to assess the result. Could you please give more details for the dataset? What type of user log are these? How does the model that you train look like? What features do you have? How large is the set of unique items? The dataset as well as the architecture used is described very briefly.

Author response:
The datasets are user click logs collected from a large-scale e-commerce system, which have tens of millions of unique queries and items. The model we employed is based on the DSSM architecture which has a query tower and an item tower. The query tower takes the input of user input query, and the item tower takes the attributes of item like title, brand, model, color, size and so on.

Reviewer comment (Experiments):
Why did you select such a large number for k? Does this artificially inflate the metrics you measure? Usually, we would try to retrieve a small set of elements to feed to a ranker.

Author response:
The choice of $k$ depends on the efficiency requirements of online system. We choose a large $k$ for two reasons: (1) our e-commerce system is large-scale, which have billions of candidate items and we need to retrieval enough items for subsequent ranking stage. (2) We have cascade ranking modules which serves like a funnel, progressively reducing the computation at each stage.

Reviewer comment (Evaluation):
How significant is the result that you achieve? How does this affect the ranking stage?

Author response:
We have achieve significant improvement in online A/B tests. It retrieves more items for popular queries, while less items for specific queries, thus the total number for ranking stage does not change much.

Reviewer comment (Novelty):
The authors claim that the paper is the first to introduce probabilistic modelling into embedding based retrieval, which remains doubtful to me. Probabilistic embedding has a long history in machine learning, as well as probabilistic information retrieval, at least dating back to probabilistic ranking principle (Robertson, 1977), which essentially seeks to model the relevance of items to a query. Such literature was not reviewed in the paper. Furthermore, this formulation for modelling the retrieval probability and learning embeddings based on contrastive losses is not new to the community (see [1] and the references therein).

Author response:
We know that there have been several literatures about probabilistic embeddings. However, we mainly focus on learning the probabilistic distribution of the relevance score of query and item, which is distinct from the learning of probabilistic embeddings. Furthermore, most probabilistic embeddings aim to capture the polysemy of query or item, our work addresses the learning of popularity.

Reviewer comment (Experiments):
In Section 3.3.2, the current model pEBR also relies on the chosen threshold $t$. Could the authors explain how sensitive it is to the model performance and at what value one should set it in practice?

Author response:
We need to consider both the effectiveness and efficiency of an online system. Larger $t$ brings better retrieval performance but require more online computations. In practice, we conduct several A/B tests to choose different $t$ that can balance the effectiveness and efficiency.

Reviewer comment (Experiments):
The authors claim that without the probabilistic assumption, the model can fall short of generalisation ability. Any empirical evidence to support this, in comparison with the frequentist approaches?

Author response:
Retrieval system usually suffers from the long-tail problem. The frequentist approaches are unable to effectively learn the probabilistic distribution of long-tail queries, as these queries are always short of training data.