Adaptive Hyper-parameter Learning for Deep Semantic Retrieval (2023)

Abstract: Deep semantic retrieval has achieved remarkable success in online E-commerce applications. The majority of methods aim to distinguish positive items and negative items for each query by utilizing margin loss or softmax loss. Despite their decent performance, these methods are highly sensitive to hyper-parameters, i.e., margin and temperature \tau , which measure the similarity of negative pairs and affect the distribution of items in metric space. How to design and choose adaptively parameters for different pairs is still an open challenge. Recently several methods have attempted to alleviate the above problem by learning each parameter through trainable/statistical methods in the recommendation. We argue that those are not suitable for retrieval scenarios, due to the agnosticism and diversity of the queries. To fully overcome this limitation, we propose a novel adaptive metric learning method that designs a simple and universal hyper-parameter-free learning method to improve the performance of retrieval. Specifically, we first propose a method that adaptive obtains the hyper-parameters by relying on the batch similarity without fixed or extra-trainable hyper-parameters. Subsequently, we adopt a symmetric metric learning method to mitigate model collapse issues. Furthermore, the proposed method is general and sheds a highlight on other fields. Extensive experiments demonstrate our method significantly outperforms previous methods on a real-world dataset, highlighting the superiority and effectiveness of our method. This method has been successfully deployed on an online E-commerce search platform and brought substantial economic benefits.

## introduction in recent years, pre-trained deep semantic retrieval models have made significant progress and application, particularly in the e-commerce field, with the research and application of deep learning technology. compared with traditional lexical-based methods, the deep semantic retrieval model has the advantages of high accuracy, low mismatch, and strong generalization. the classical deep semantic retrieval methods could be split into two categories, sparse-based retrieval @xcite @xcite , and dense-based retrieval @xcite @xcite @xcite . although there are differences in representation, they all learn the deep model through an end-to-end paradigm using contrastive learning methods. specifically, they first represent the query and item into dense/spare vectors in metric space. then, they adopt softmax loss or margin loss to distinguish the positive item and the negative item. the hyper-parameters of margin or temperature measure the disagreement of similarity of query and candidate items. nevertheless, we argue that this paradigm also has several limitations as it fails to meet query requirements. for instance, different queries have different metrics for candidate items. some queries are general words, and the distance between the query and candidate items may be smaller than precise words. reflected in the metric space, this query should be a smaller margin in margin loss, and vice versa. analogously, the temperature of the general query should be large, which has a greater entropy for the similarity of query and items. however, the classical deep retrieval method fails to address these issues due to the fixed hyper-parameters. as we know, hyper-parameter searches are highly time-cost expensive to find the optimum by grid approach and have a significant impact on performance. designing and choosing adaptive parameters for different pairs remains an open challenge. recently several methods considered to alleviate the above problem by learning each parameter through train-able/statistical methods in recommender systems. typically, @xcite first presents a concept of user/item bias to measure the margin in metric space for margin loss, which could be trained by background optimization. @xcite proposes to learn an adaptive margin for different users via bi-level optimization @xcite , where a proxy function is built to explicitly update the margin generation-related parameters. however, those are not available for retrieval scenarios, due to the agnosticism and diversity of the query. similarly, @xcite also finds that the temperatures play an important role from the perspective of limitation of normalization and develop an adaptive fine-grained strategy for the temperature with satisfying four desirable properties including adaptivity. this is the first work to explicitly talk about the problem and attempt to study how to adaptively set the proper temperature for recommender systems. we argue that this method is complex and unavailable in retrieval systems because the query is unknown in the real world not like the field of recommendation, where users are given. thus, the above limitation motivates us to design a simple and hyper-parameter-free method to enhance the performance of deep retrieval. toward this end, we present a unified solution to address the problem for both softmax loss and margin loss. more precisely, we first analyze the limitation of the original method from the metric space. subsequently, we present a heuristic method, which generates the margins/temperatures by the inner product of batch samples adaptively. to prevent the collapse of the training process i.e., all items' embedding have high similarity, we adopt the symmetric metric learning method to push the positive items far away from the negative items. we conduct extensive experiments in the real-world e-commerce field, and the results demonstrate the effectiveness of the proposed. the contributions of this paper can be summarized as follows: ## preliminaries in this section, we first give the formulation of the retrieval task and then present the heuristic method for two classical paradigms, i.e., margin loss and softmax loss. finally, we will give a detailed description of the proposed hyper-parameters-free methods. ## approach in this section, we will first talk about the limitation of the loss function above-mentioned and then will give a general heuristic method in accordance with the measure assumptions. finally, we describe the complete method, the symmetric metric learning method in detail. ## experiments in this section, we conduct extensive experiments to evaluate the performance of the proposed method and investigate the effect of different components by ablation studies. ## conclusion and future work this paper addresses the challenge of adaptive hyper-parameter selection for the contrastive learning paradigm in deep retrieval fields. toward this end, we first present a straightforward heuristic method, which uses the batch similarity of items to generate a margin or temperature, eliminating the complex trainable variables. however, our theoretical analysis reveals that this method is prone to model collapse. to prevent the above problem, we adopt the symmetric metric learning method to align the query and items in metric space. experiments verify that our assumptions and method are simple, effective, and significantly outperform other models in real-world datasets. moreover, we have successfully deployed this method on online search platforms, leading to significant commercial value. it is worth noting that, this method could give a great inspiration for other fields, such as recommendation and knowledge graph learning. in future work, we aim to explore the benefits of multi-objective learning with adaptive hyperparameters in the full sample space.