Improving Relevance Prediction with Transfer Learning in Large-scale Retrieval Systems
Abstract
Machine learned large-scale retrieval systems require a large amount of training data representing query-item relevance.
However, collecting users’ explicit feedback is costly.
In this paper, we propose to leverage user logs and implicit feedback as auxiliary objectives to improve relevance modeling in retrieval systems.
Speciﬁcally, we adopt a two-tower neural net architecture to model queryitem relevance given both collaborative and content information.
By introducing auxiliary tasks trained with much richer implicit user feedback data, we improve the quality and resolution for the learned representations of queries and items.
Applying these learned representations to an industrial retrieval system has delivered signiﬁcant improvements.
1. Introduction
In this paper, we propose a novel transfer learning model architecture for large-scale retrieval systems.
The retrieval problem is deﬁned as follows: given a query and a large set of candidate items, retrieve the top- k most relevant candidates.
Retrieval systems are useful in many real-world applications such as search (Shen et al., 2014) and recommendation (Covington et al., 2016;
Yi et al., 2018; He et al., 2014).
The recent efforts on building large-scale retrieval systems mostly focus on the following two aspects:
• Better representation learning.
Many machine learning models have been developed to learn the mapping of queries and candidate items to an embedding space (Koren et al., 2009; Krichene et al., 2019).
These models leverage various features such as collaborative and content information (Wang et al., 2012).
• Efﬁcient retrieval algorithms.
Given learned representations, efﬁcient algorithms are proposed to retrieve the top- k relevant items given the similarity (distance) metric associated with the embedding space (Broder, 1997; Guo et al., 2016).
However, it is challenging to design and develop real-world large-scale retrieval systems for many reasons:
• Sparse relevance data.
It is costly to collect users’ true opinions regarding item relevance.
Often, researchers and engineers design human-eval templates with Likert scale questions for relevance (Chang et al., 2015), and solicit feedback via crowd-sourcing platforms ( e g. , Amazon Mechnical Turk).
• Noisy feedback.
In addition, user feedback is often highly subjective and biased, due to human bias in designing the human-eval templates, as well as the subjectivity in providing feedback.
• Multi-modality feature space.
We need to learn relevance in a feature space generated from multiple modalities, e.g. , query content features, candidate content features, context features, and graph features from connections between query and candidate (Wang et al., 2012;
Page et al., 1999; Cui et al., 2010).
In this paper, we propose to learn relevance by leveraging both users’ explicit answers on relevance and users’ implicit feedback such as clicks and other types of user engagement.
Speciﬁcally, we develop a transfer-learning framework which ﬁrst learns the effective query and candidate item representations using a large quantity of users’ implicit feedback, and then reﬁnes these representations using users’ explicit feedback collected from survey responses.
The proposed model architecture is depicted in Figure 2.
Our proposed model is based on a two-tower deep neural network (DNN) commonly deployed in large-scale retrieval systems (Krichene et al., 2019).
This model architecture, as depicted in Figure 1, is capable of learning effective representations from multiple modalities of features.
These representations can be subsequently served using highly efﬁcient nearest neighbor search systems (Guo et al., 2016).
To transfer the knowledge learned from implicit feedback to explicit feedback, we extend the two-tower model by adopting a shared-bottom architecture which has been widely used in the context of multi-task learning (Caruana, 1997).
Speciﬁcally, the ﬁnal loss includes training objectives for both the implicit and explicit feedback tasks.
These two tasks share some hidden layers, and each task has its own independent sub-tower.
At serving time, only the representations learned for explicit feedback are used and evaluated.
Our experiments on an industrial large-scale retrieval system have shown that by transferring knowledge from rich implicit feedback, we can signiﬁcantly improve the prediction accuracy of sparse relevance feedback.
In summary, our contributions are as follows:
• We propose a transfer learning framework which leverages rich implicit feedback in order to learn better representations for sparse explicit feedback.
• We design a novel model architecture which optimizes two training objectives sequentially.
• We evaluate our model on a real-world large-scale retrieval system and demonstrate signiﬁcant improvements.
The rest of this paper is organized as follows:
Sectiondiscusses related work in building large-scale retrieval systems.
Section 3 introduces our problem and training objectives.
Section 4 describes our proposed approach.
Section 5 reports the experimental results on a large-scale retrieval system.
Finally, in Section 6, we conclude with our ﬁndings.
2. Related Work
In this section, we ﬁrst introduce some state-of-the-art industrial retrieval systems, and then discuss the application of multi-task learning and transfer learning techniques in retrieval and recommendation tasks.
2.1. Industry-scale Retrieval Systems
Retrieval systems are widely used in large-scale applications such as search (Shen et al., 2014) and recommendation (Covington et al., 2016;
Yi et al., 2018; He et al., 2014).
In recent years, the industry has moved from reverse index based solutions (Brin & Page, 1998), to machine learned retrieval systems.
Collaborative-ﬁltering based systems (Hu et al., 2008; Beutel et al., 2017) have been very popular and successful until very recently, when they were surpassed by various neural network based retrieval models (Liu et al., 2017;
Yi et al., 2018; Beutel et al., 2018).
A retrieval system involves two key components: representation learning and efﬁcient indexing algorithms (Manning et al., 2010).
Many large-scale industrial retrieval systems have seen success of using two-tower DNN models to learn separate representations for query and candidate items (He et al., 2017; Yang et al., 2018; Krichene et al., 2019).
2.2. Multi-task Learning and Transfer Learning for Retrieval and Recommendation Systems
There has also been work on multi-task retrieval systems for context-aware retrieval applications based on tensor factorization (Zhao et al., 2015).
Unfortunatelly, due to limitations on model capacity and serving time constraints, the model cannot be easily adapted to learn complex feature representations from multiple feature sources.
Many multi-task DNN based recommendation systems (Covington et al., 2016; Ma et al., 2018) are designed for ranking problems where only a small subset of high quality candidates are scored.
These full-blown ranking solutions cannot be easily applied to retrieval problems, where we try to identify thousands of candidates from a large corpus with millions to hundreds of millions of candidate items.
Inspired by these works, we propose a novel framework to combine the beneﬁts of both worlds: (1) the computation efﬁciency of a two-tower model architecture; and (2) the improved model capability of a multi-task DNN architecture (Caruana, 1997).
This enables us to transfer the learning from rich implicit feedback to help sparse explicit feedback tasks.
Our work is closely related to transfer learning (Pan & Yang, 2009; Raina et al., 2007; Pan et al., 2011; 2010) and weakly supervised learning (Oquab et al., 2015; Han et al., 2014;
Papandreou et al., 2015; Zhou, 2017).
3. Problem Description
In this section, we formalize the retrieval problem, and introduce our training data and training objectives.
3.1. The Retrieval Problem
The retrieval problem is deﬁned as follows.
Given a query and a corpus of candidate items, return the top- k relevant items.
Let { x i } Ni =1 ⊂ X
and { y j } Mj =1 ⊂ Y , respectively, be the feature vectors of queries and candidates in feature space
X and Y , where N and M , respectively, denote the number of queries and candidates.
We model the retrieval system as a parameterized scoring function s ( · , · ; θ ) :
X × Y (cid:55)→ R , where θ denotes the model parameters.
Items with top- k scores s ( x , y ; θ ) are selected for a given query at inference time.
We assume the training data is a set of query and item pairs { ( x t , y t ) }
Tt =1 , where y t is the candidate associated with x t which has either explicit or implicit users’ feedback, and T (cid:28) MN in practice.
Our goal is to ﬁt the scoring function based on these T examples.
3.2. Training with User Feedback
When training a machine learning based retrieval system, the ideal way is to use users’ explicit feedback which reﬂects the relevance of an item to a query.
However, asking for users’ explicit feedback is costly; hence, many existing systems use implicit feedback from user logs, such as clicks.
In this paper, we study retrieval systems with both explicit and implicit feedback, where implicit feedback is abundant and explicit feedback is relatively sparse.
3.3. Joint Optimization with Auxiliary Objectives
The goal of our retrieval problem is to learn better representations of queries and candidates such that the similarity between a query candidate pair closely approximates relevance.
Therefore, our main training objective is to minimize the differences between the predicted relevance and the ground truth.
To facilitate representation learning, we introduce an auxiliary objective which captures user engagement on items, such as clicks of an item, purchase of a product for shopping retrieval, or views of a movie for movie recommendation.
Formally, we aim to jointly learn two objectives s exp ( · , · ; θ ) and s imp ( · , · ; θ (cid:48) ) while sharing part of the parameters between θ and θ (cid:48) .
We assume some of the examples ( x t , y t ) are in set E with explicit feedback, and others are in set I with implicit feedback.
In addition, each example ( x t , y t ) ∈
E is associated with label l t ∈ R representing user’ explicit feedback, e.g. , response to the relevance survey.
Note that E and I are not mutually exclusive as some examples can have both implicit and explicit feedback.
We use regression loss to ﬁt users’ explicit feedback on example set in E .
One example loss is the mean squared error (MSE):
[Equation]
where | · | represents the cardinality.
On the other hand, we treat the modeling of implicit feedback as a multi-class classiﬁcation task over the full corpus of items, and use the softmax formulation to model the probability of choosing item y , namely
[Equation]
The maximum likelihood estimation (MLE) can be formulated as
[Equation]
With loss multipliers w and w (cid:48) , we jointly optimize the losses in (1) and (2) by optimizing
[Equation]
4. Model Architecture
In this section, we describe our proposed framework to learn relevance for large-scale retrieval problems.
We extend the two-tower model architecture by introducing a sharedbottom model architecture on both towers.
4.1. Two-tower Model Architecture
Figure 1 provides a high-level illustration of the two-tower
DNN model architecture.
Given a pair of query and item represented by feature vectors x ∈ X , y ∈ Y , respectively, the left and right tower provides two DNN based parameterized embedding functions u : X × R d
(cid:55)→ R k , v : Y × R d (cid:55)→ R k which encode features of query and item to a k -dimensional embedding space.
The scoring function is then computed as the dot product between the query and item embeddings at the top layer, i.e. ,
[Equation]
4.2. Shared-bottom Architecture for Transfer Learning
To enable multi-task learning, we extend the two-tower model by adopting the shared-bottom architecture.
Specifically, we introduce two sub-towers on top of the bottom hidden layers, one for the explicit-feedback task and the other for the implicit-feedback task.
The outputs of bottom hidden layers are fed in parallel to the two sub-towers.
The bottom hidden layers are shared between the two sub-towers (Caruana, 1997), and are referred to as shared-bottom layers.
The ﬁnal model architecture is depicted in Figure 2.
4.3. Training and Serving Schema
During training, we ﬁrst train the model for the auxiliary user engagement objective, which uses the cross entropy loss.
Having learned the shared representations, we ﬁnetune the model for the main relevance objective, which uses the squared loss.
To prevent potential over-ﬁtting caused by the sparse relevance data, we apply stop gradients for the relevance objective on the shared-bottom layers.
For serving, we only need to store and serve the top layer of the two relevance sub-towers to predict the relevance.
5. Evaluation
In this section, we describe the experiments of our proposed framework on one of Google’s large-scale retrieval systems for relevant item recommendations, e.g. , apps.
5.1. Experiment Setup
Our system contains several millions of candidates.
Our training data contains hundreds of thousands of explicit feedback from relevance survey, and billions of implicit feedback from user logs.
We randomly split the data into 90% for training and 10% for evaluation.
Model performance was measured on the eval set by the Root Mean Square Error (RMSE) for relevance prediction.
The model was implemented in Tensorﬂow, of which the output relevance embeddings for queries and candidates were served for retrieval.
The hyper-parameters including model size, learning rate, and training steps were carefully tuned for the best model performance.
5.2. Experiment Results
We study the effects of applying transfer learning to relevance prediction.
The following experiment results suggest that transfer learning signiﬁcantly improves the prediction quality of sparse relevance task and helps avoid over-ﬁtting.
Table 1 reports relevance RMSE (the lower the better) for different combinations of training objectives and feature types.
We see that using implicit feedback leads to a signiﬁcant improvement as compared to using explicit feedback only.
Also, using collaborative information together with content information performs better than the model which uses collaborative information alone.
Table 2 reports relevance RMSE for various model sizes on two sets of training objectives.
As a close approximation to the model size, we report the number of multiplications.
For models trained with sparse explicit feedback only, increasing the model sizes causes over-ﬁtting and consequently degrades the model performance.
In contrast, for models trained with implicit feedback, increasing the model size improves the model performance.
This suggests that implicit feedback regularizes the model and prevents over-ﬁtting.
[Table] [Table]
5.3. Discussions and Future Work
The success of transfer learning hinges on a proper parameterization of both the auxiliary and main tasks.
On one hand, we need sufﬁcient capacity to learn a high-quality representation from a large amount of auxiliary data.
On the other hand, we want to limit the capacity for the main task to avoid over-ﬁtting to its sparse labels.
As a result, our proposed model architecture is slightly different from the traditional pre-trained and ﬁne-tuning model (Hinton &
Salakhutdinov, 2006).
Besides shared layers, each task has its own hidden layers with different capacities.
In addition, we apply a two-stage training with stop gradients to avoid potential issues caused by the extreme data skew between the main task and auxiliary task.
Our experiences have motivated us to continue our work in the following directions:
• We will consider multiple types of user implicit feedback using different multi-task learning frameworks, such as Multi-gate Mixture-of-Expert (Ma et al., 2018) and Sub-Network Routing (Ma et al., 2019).
We will continue to explore new model architectures to combine transfer learning with multi-task learning.
• The auxiliary task requires hyper-parameter tuning to learn the optimal representation for the main task.
We will explore AutoML (Pham et al., 2018) techniques to automate the learning of proper parameterizations across tasks for both the query and the candidate towers.
6. Conclusion
In this paper, we propose a novel model architecture to learn better query and candidate representations via transfer learning.
We extend the two-tower neural network approach to enhance sparse task learning by leveraging auxiliary tasks with rich implicit feedback.
By introducing auxiliary objectives and jointly learning this model using implicit feedback, we observe a signiﬁcant improvement for relevance prediction on one of Google’s large-scale retrieval systems.