Denoising Attention for Query-aware User Modeling (2024)

Abstract: Personalization of search results has gained increasing attention in the past few years, also thanks to the development of Neural Networks-based approaches for Information Retrieval. Recent works have proposed to build user models at query time by leveraging the Attention mechanism, which allows weighing the contribution of the user-related information w.r.t. the current query.This approach allows giving more importance to the user’s interests related to the current search performed by the user.In this paper, we discuss some shortcomings of the Attention mechanism when employed for personalization and introduce a novel Attention variant, the Denoising Attention, to solve them.Denoising Attention adopts a robust normalization scheme and introduces a filtering mechanism to better discern among the user-related data those helpful for personalization.Experimental evaluation shows improvements in MAP, MRR, and NDCG above 15% w.r.t. other Attention variants at the state-of-the-art.

## introduction the past few years have witnessed an increasing interest in neural models for tackling various tasks of information retrieval @xcite , among which personalized search. two of the main challenges of personalized search are how and when personalization should take place. first, not all the data gathered to represent specific users' preferences are equally related to each of the users' queries, as users usually have multiple and diverse interests. second, personalization is not always beneficial to the retrieval process @xcite as it could cause the information need expressed by the user to be misinterpreted by the system, thus decreasing effectiveness. a recent trend in personalized search @xcite is query-aware user modeling, which consists in building a representation of the user preferences, i.e., the user model, at query time, based on various sources of user interests and by giving more importance to those related to the current search performed by the user. since a user is typically interested in different and even unrelated topics, a desirable property for defining reliable personalization models is the ability to discern between beneficial and noisy user-related information on a query basis. previous works in this context relied on the attention mechanism @xcite to weigh the contribution of distinct sources of user-related information on a query basis. despite the increasing use of the attention mechanism in user modeling, there is still a lack of an in-depth analysis of its effects on personalization, as well as a comparison with simpler operators in this context. in this paper, we first describe and analyze the attention mechanism when used for query-aware user modeling, by highlighting some shortcomings related to its use of the softmax function (section 3). to overcome these limitations, in section 4, we propose the denoising attention mechanism, an attention variant designed to filter out noisy user-related information and produce a balanced representation of the user interests w.r.t. the current search. in section 5, we introduce the task of personalized results re-ranking and the framework we employed in our experimental evaluation. then, we present the research questions we addressed and describe the experimental setup (section 6). finally, in section 7, we compare the denoising attention with other attention variants at the stateof-the-art, evaluating both their effectiveness and their robustness. results clearly show the advantages of denoising attention and the importance of the filtering mechanism it implements. we share all the code to reproduce the experimental evaluation, and we make available the implementation of denoising attention for future works. 1 ## related work personalization of search results has received considerable attention from both academia and industry. the definition of user models is the core issue in personalized search. early user modeling approaches relied on click-based features @xcite @xcite , content-based features @xcite , social network analysis @xcite , language models @xcite , topic modeling @xcite @xcite , ontologies @xcite , and other sources of user-related information to build user representations. researchers have also applied representation learning @xcite techniques to build semantic vector representations of queries, documents, and user-related information for personalizing search results @xcite @xcite @xcite . several recent works employed on the attention mechanism @xcite to weigh and aggregate the available user-related information on a query basis. these models take advantage of the diverse interests that a user may have to conduct query-aware personalization. for example, previous works @xcite @xcite relied on attention to build sessionbased user models for personalizing subsequent searches. @xcite employed the attention to enhance a personalization model based on user re-finding behavior. @xcite leveraged the attention to weigh user-related terms for personalized query suggestion. @xcite proposed an attentive personalized item retrieval model that estimates the importance of each item in the user history. despite the increasing application of attention for user modeling, previous publications did not present an in-depth analysis of its behavior and effects on personalization. the sole exception is represented by the zero attention model @xcite . this attention variant was defined to allow the retrieval model to avoid personalization when no source of user information is related to her current search, which is not possible using the standard attention formulation, as we will discuss in section 3.2. despite promising results, successive works @xcite have shown that the zero attention model performs inconsistently. in this paper, we address the lack of in-depth analysis of the attention mechanism when applied for personalization and propose a novel attention variant to overcome some limitations highlighted by our study of such a mechanism. ## preliminaries on query-aware user modeling users usually have diverse interests in multiple domains. not all those preferences are equally relevant to a specific user's information need. for example, if a user is looking for a new book to read, her fashion-related preferences probably do not matter for personalizing the results of her current query. query-aware user modeling consists in building a user model at query time, based on previously gathered sources of user interest, by giving more importance to those related to the current search performed by the user. in the literature, the definition of a user model with the previous characteristics has been provided by relying on the attention mechanism @xcite , which allows weighing the contribution of the user-related data w.r.t. the current search query. in this section, we first describe the attention mechanism as it is usually employed in the context of personalized search. then, we discuss some of its shortcomings when used for personalization. ## denoising attention mechanism as extensively discussed in section 3.2, the principal issues of the attention are related to its normalization step, described in section 3.1, and specifically to the use of the softmax function to produce the attention weights. to counteract these issues, we need a mechanism able to avoid overly narrowing or overly smoothing the attention weights, which can cause the model either to focus only on a single source of user-related information or to reduce the diversity of their estimated importance. moreover, this mechanism should finely filter out noisy contextual information, thus preventing it from flowing into the user model. finally, it should zero out the user's context vector when personalization could harm the retrieval process, i.e., when all the user-related information is noisy or irrelevant with respect to the current search. in this regard, we propose the denoising attention mechanism. the denoising attention mechanism departs from the softmax function by adopting a more straight-forward and robust normalization scheme, and it introduces a filtering mechanism based on relu @xcite and a threshold value. to complement those changes, we rely on a cosine similarity-based alignment model to evaluate the relatedness of the sources of user-related information w.r.t. the current search. scoring for an alignment model to act in a complementary way with the changes introduced in the next paragraph, we need a function a(q, d) bounded between 0 and 1, as an unbounded function would make it difficult to control which information flows into the user model. to compute the alignment scores e q,d , we then rely on a cosine similarity-based function bounded in [0, 1]: filtering we propose to extend the attention mechanism with an information filter. first, we employ a threshold t to negativize the alignment scores of the user data loosely related to the query: by combining these two operations, we can both control the information flow from the user data to the user model and filter out noisy user-related information that could harm the retrieval process. to overcome the well-known dying relu problem @xcite , we sum the user model to the query representation during training. ## personalized results re-ranking in this section, we introduce the task we have considered for evaluating the proposed user modeling approach, i.e. personalized results re-ranking. moreover, we describe the personalized re-ranking framework we employed for comparative evaluation. this framework allowed us to test different user modeling techniques with ease and isolate their impact from the other system components. in personalized results re-ranking, a retrieval system (first stage retriever) computes a ranked list of documents in response to a search query. then, a personalization component computes new relevance scores for the initially retrieved documents ## experimental setup the experiments reported in this section aim to answer the following research questions: rq1 are query-aware attention-based user models more effective than static user models? rq2 is denoising attention more effective at user modeling than other attention variants? rq3 is denoising attention more robust, i.e., less likely to decrease the system's effectiveness due to noisy user-related data, than other attention variants? to answer the research questions rq1 and rq2, we conducted a comparative evaluation of the retrieval effectiveness of the personalized re-ranking pipeline described in section 5 using several different user models. then, to answer the research question rq3, we compared the number of times the considered user models decreased the retrieval effectiveness of our first-stage retriever, bm25. in the following, we present the datasets we employed for conducting our evaluations (section 6.1), we introduce the baselines we have selected (section 6.2), and we outline the training setup and evaluation procedure (section 6.3). ## results and discussion in this section, we present the results of our comparative evaluation. first, we discuss the retrieval effectiveness of the personalized re-ranking pipeline described in section 5 when considering different user modeling techniques. then, we analyze the robustness of the compared user models, evaluating the probability they decrease the system's effectiveness in the presence of noisy user-related data. we remind the reader that the only difference between the compared personalization models is the technique used for defining the user model, while the other system's components are fixed. ## conclusion in this work, we have addressed some issues related to the use of the attention mechanism for query-aware user modeling and proposed a novel user-data aggregation model called denoising attention, designed to solve the shortcomings of the standard attention formulation and, in particular, filter out noisy user-related information. experimental evaluation in two different search scenarios, namely web search and academic search, shows the benefits of our proposed approach over other attention variants and highlights the potential of correctly managing the user-related information.