LBD: Decouple Relevance and Observation for Individual-Level Unbiased Learning to Rank
AffiliationAddress
Abstract
Using Unbiased Learning to Rank (ULTR) to train the ranking model with biasedclick logs has attracted increased research interest.
The key idea is to explicitlymodel the user’s observation behavior when building the ranker with a large number of click logs.
Considering the simplicity, recent efforts are mainly based on theposition bias hypothesis, in which the observation only depends on the position.
However, this hypothesis does not hold in many scenarios due to the neglect of thedistinct characteristics of individuals in the same position.
On the other hand, directly modeling observation bias for each individual is quite challenging, since theeffects of each individual’s features on relevance and observation are entangled.
It is difficult to ravel out this coupled effect and thus obtain a correct relevancemodel from click data.
To address this issue, we first present the concept of coupling effect for individual-level ULTR.
Then, we develop the novel Lipschitz and
Bernoulli Decoupling (LBD) model to decouple the effects on relevance and observation at the individual level.
We prove theoretically that our proposed methodcould recover the correct relevance order for the ranking objective.
Empirical results on two LTR benchmark datasets show that the proposed model outperformsthe state-of-the-art baselines and verify its effectiveness in debiasing data.
Introduction
Learning to Rank (LTR) has been widely used in modern information retrieval systems, which aimsto learn a ranking model that sorts documents with their relevance.
Recently, using implicit feedback(eg. clicks) instead of relevance labels (typically obtained by human annotation) to train the rankingmodel becomes popular since implicit feedback is an attractive proxy of relevance, which is cheapand relatively easy to obtain on a large scale [36].
However, they inherently contain a lot of bias fromuser behavior [26].
Using Unbiased Learning to Rank (ULTR) to remove these biases has attractedincreasing research interest [1, 27].
The key is to factorize the clicks into relevance probabilities andobservation probabilities, and model a user’s observation probability on an item in a ranking list.
Byreweighing the click signals based on the reciprocal of observation probabilities, ULTR provides anunbiased estimate of the ranking objective.
Most of the existing ULTR methods are based on the position bias hypothesis [27, 4, 41, 16, 3],which assumes that the observation only depends on the position.
We refer to this method as grouplevel ULTR method, in which various heterogeneous features (typically encoded with the information of query, document, and user) share the same observation probability according to some property (like position).
However, the observation bias inherently varies across individuals.
For example,the exact match in result titles and abstracts (which is usually shown in a different presentation style)affects users’ attention and judgment, known as attractive bias [43, 7].
Except for attractive bias,there are lots of factors that can influence a user’s observation [10, 44, 35, 7, 22, 29, 30].
These factors are encoded in (or can be inferred from) the features, which causes different documents tohave a different observation.
To be in line with these real phenomena, we argue there is an urgent need to develop an individuallevel ULTR, which can capture the effect of heterogeneous features for observation behavior.
Thatis to say, the observation model should be a function related to documents.
[Figure]
Unfortunately, this problem is very challenging.
As shown in Figure 1, there exists a coupling effect from features to observation andrelevance.
There are infinite solutions to factorize the click rate into relevance function and observation function (all of them are functions offeatures), but few of them imply a correct ranking model.
In the extreme case, a naive rankingmodel which outputs identical relevance probability still has a chance to perfectly fit the clickdata with a properly designed association between features and observation [39].
A straightforward solution to avoid coupling is to divide thefeatures manually into two separated parts: one related to relevance and the other related to observation.
But it’s intractable in practice because some of these features affect both.
For example, theexact match in result titles and abstracts affects observation (attractive bias), but on the other hand,it is naturally an important feature encoding how relevant the document is.
Besides, result domain[7] and users’ search intent [35] are also common factors affecting both.
To address the coupling effect problem, we find that it is unnecessary to correctly estimate the truerelevance probability (named hard decoupling), yet only need to recover the correct relevance orderfrom click data (named soft decoupling) because of the ranking objective.
Given that we don’t knowthe true relevance, this goal is difficult to achieve directly on the ranking model.
We instead modifythe observation model, which helps to achieve soft decoupling when the click probability estimationis unbiased.
We propose two special observation models, the Lipschitz Observation Model (LOM)which has a Lipschitz constraint β on the observation function, and the Bernoulli Observation Model(BOM) which randomly cancels debiasing operation with a probability of t on the observation function output.
The two models act on different mechanisms, thus they are complementary and can becombined.
We provide a theoretical guarantee that when β and t are bounded, these two observationmodels can achieve soft decoupling whenever estimating unbiased click probability with a rankingmodel.
Given this, we propose a principled novel individual-based ULTR framework, called Lipschitz and Bernoulli Decoupling (LBD) model.
We conducted comprehensive experiments on two
LTR benchmark datasets, which shows that the proposed LBD outperforms the baseline methods andverifies its effectiveness in debiasing data.
We further studied the different application areas of thetwo decoupling techniques and found that the performance can be better when they are combined.
To the best of our knowledge, we are the first to study the coupling effect for ULTR and proposea feasible solution to decouple the effects on relevance and observation.
The main contributions ofthis work are:
We present the concept of coupling effect for individual-level ULTR, as well as two targets to achieve decoupling.
We propose a novel Lipschitz and Bernoulli Decoupling model, which could help to decouple the effects on relevance and observation.
We provide theoretical proof to guarantee the decoupling ability of our proposed techniques.
We conducted comprehensive experiments on two LTR benchmark datasets in the coupling effect scene, which shows that the two decoupling techniques have different application areas, and combining them can outperform the baseline methods.
Related Work
Debiasing Click Data for LTR .
Debiasing click data is an important research direction in the information retrieval field.
We review this line of related work in § A.
Individual-Level Observation .
There is much work finding that the observation bias can vary frominstance to instance.
The user’s observation can be influenced by result type [30], result quality [10],search intent [35], result domain [7, 22], exact match [43], and even the relevance itself [29].
Thesefactors can be inferred from document features.
Particularly, [42] found that in mobile search, theuser may be directly satisfied with the result without clicking any links on certain kinds of documents.
[35] argued that users with different search intent may have different examination behavior.
All these discoveries suggest the necessity to estimate the bias at the individual level.
Recently, researchers attempted to find solutions in ULTR fields to deal with individual-level observation bias.
[21] proposed to manually impose a stronger regularization on the position biasestimated by Unbiased LambdaMART, to alleviate the influence of user’s different click behaviors,while it’s heuristic and has no theoretical proof.
[28] proposed wLambdaMART that estimates theconfidence of click data with a few labeled data to correct bias, without the need to design a specificbias type.
But this method requires extra labeled data.
[37] employed heterogeneous treatment effect estimation techniques to estimate the bias between the position k and the position 1 .
However,this method is based on an assumption that the click rates at the first position reveal their accuraterelevance.
This is only true when the observation probability at the first position is constant.
Preliminaries
In this work, we use bold-faced letters to denote vectors, upper-case letters to denote random variables, and the corresponding lower-case letters to denote values.
P ( · ) denotes the distribution of arandom variable.
Generally, the core of LTR is to learn a ranking model f , which assigns a relevance score to adocument with its query-document features X ∈ R l .
Then, the documents for a query can be sortedin descending order of their scores.
In traditional LTR, the ranking model is directly learned withthe true relevance score R to make the ranking order of the query list close to the optimal order [27].
In the click setting, user’s click logs are used as substitutes of relevance due to their abundance andcheapness.
While click signals are naturally biased from the true relevance [25], much prior workfollow the examination hypothesis to address it, which can be formulated as:
[Equation]
where r ( x ) =
Pr ( R = 1 | X = x ) , o p ( x ) =
Pr ( O = 1 | X = x , P = p ) and c p ( x )
=
Pr ( C =1 | X = x , P = p ) denote the probabilities of relevance, observation and click, and P denotes theposition.
By explicitly modeling the bias effect via observation probability, it is able to attain anunbiased estimate of the ranking objective.
Our method
In this section, we first introduce the coupling effect in § 4.1, and propose two decoupling methodsin § 4.2 and § 4.3.
4.1 Coupling Effect
Let ˆ r ( x ; θ 1 ) denote the estimation of ranking model, and ˆ o ( x ; θ 2 ) denote the estimation of observation model.
We use clicks to train the ranking model and the observation model using Eq.(1), tolet their production close to the click.
What we care about is whether the relevance model is correctwhen the model converges.
To simplify the later analysis, we suppose that the models are powerfulenough to predict unbiased click probability when converges and make the following assumption.
Assumption 1 (Unbiased Click Prediction)
There exists a parameter space Θ ∗ which consists of pairs of ranking model parameter and observation model parameter, such that for all ( θ ∗ 1 , θ ∗ 2 )
∈
Θ ∗ , the predictive click probability is unbiased:
[Equation]
From Assumption 1 we can find that the factorization of relevance probability ˆ r ( x ; θ ∗ 1 ) and observation probability ˆ o p ( x ; θ ∗ 2 ) is unidentifiable given the click probability c p ( x ) , which makes it challenging to uncover the correct ranking model.
This is because the features X have an effect onboth relevance R and observation O .
It’s difficult to distinguish which parts of the features affectrelevance and observation, respectively.
In the extreme case, the ranking model will degenerate intoa trivial case that always predicts the same scores for all documents [39].
We refer to this problemas the coupling effect .
Suppose the correct factorization is c p ( x )
=
r ( x ) · o p ( x ) , where r ( x ) is the true relevance ando p ( x ) is the true observation.
To achieve a good decouple of the features’ effect on relevance andobservation, we aim to find a ranking model that can make ˆ r ( x ; θ 1 ) close to r ( x ) whenever predicting an accurate click probability.
Based on this, we introduce the definition of hard decoupling asfollows.
Definition 1 (Hard Decoupling)
For any features x with the true relevance r ( x ) , a parameterspace Θ ′ ⊆
Θ ∗ can do hard decoupling, if for all ( θ ∗ 1 , θ ∗ 2 )
∈ Θ ′ , we have r ( x ) =
ˆ r ( x ; θ ∗ 1 ) .
Considering the pairwise comparison of the ranking objective, we further relax the hard decouplingconstraint and refer to this as soft decoupling as follows.
Definition 2 (Soft Decoupling)
For any two features x 1 and x 2 , without loss of generality we suppose r ( x 1 ) ≥ r ( x 2 ) .
A parameter space Θ ′ ⊆
Θ ∗ can do soft decoupling, if for all ( θ ∗ 1 , θ ∗ 2 )
∈ Θ ′ ,we have ˆ r ( x 1 ; θ ∗ 1 ) ≥ ˆ r ( x 2 ; θ ∗ 1 ) .
Note that soft decoupling requires identifying the relevance, which is challenging when relevance isunobserved.
In this subsection, we propose two mechanisms (Lipschitz Decoupling and Bernoulli
Decoupling) to do soft decoupling with theoretical guarantees.
The proofs of the theorems in thissection are provided in Appendix § B.
From this point forward, we assume that all of the parametersθ take the value from Θ ∗ , and omit θ of ˆ r ( · ; θ ) and ˆ o p ( · ; θ ) for ease of notation.
4.2 Lipschitz Decoupling
Usually, the rate of change between the observation and features will not be very large in real scenarios.
Based on it, we propose to modify the smoothness of the observation model to make it match thetrue observation function.
It helps to achieve soft decoupling when the click probability estimationis unbiased.
We start with defining the Lipschitz continuity for a function.
Let ||·|| be a norm on R l .
In this work,we employ the Euclidean norm (L-2) when not mentioned otherwise:
|| x
|| =
|| x
|| 2 = (cid:113)(cid:80) li
=1 x 2 i .
Definition 3 (Lipschitz Continuity)
A function f ( x ) : R l → R is α -Lipschitz to the input x if
[Equation]
, where α is referred to as the Lipschitz constant.
Particularly, if an α -Lipschitz function f is differentiable, we have ||∇ x f ( x ) || ≤ α .
We supposer ( x ) , o p ( x ) , ˆ r ( x ) and ˆ o p ( x ) are all differentiable.
Assumption 2 (True Observation Lipschitz)
For each p ∈ [ n ] , the true observation o p ( x ) is αLipschitz.
It assumes that there is an upper bound on the rate at which the true observation changes with thefeatures.
This is a reasonable extension for well-known position-based model [27], which assumesthat α = 0 .
Similarly, we assume the observation estimation is also Lipschitz.
We refer to this kindof observation model as Lipschitz Observation Model (LOM).
Definition 4 (Lipschitz Observation Model)
An observation model is β − Lipschitz observationmodel ( β -LOM), if its estimation ˆ o p ( x ) is β -Lipschitz for all p ∈ [ n ] .
We claim that soft decoupling can be achieved by selecting a lower β , as the following theorem.
Theorem 1 (Lipschitz Decoupling)
For any two features x 1 and x 2 , without loss of generality wesuppose r ( x 1 ) ≥ r ( x 2 ) .
For a β -LOM, if the following condition is satisfied,
[Equation]
(cid:26) inf γ ∈ Γ( x 1 , x 2 ) (cid:90) γ βo p ( x ) + α ˆ o p ( x ) o 2 p ( x ) ||
d x || (cid:27) , where Γ( x 1 , x 2 ) denotes the collection of all curves from point x 1 to point x 2 .
Then we have:ˆ r ( x 1 ) ≥ ˆ r ( x 2 ) .
Remark
If the gap between r ( x 1 ) and r ( x 2 ) is large (i.e., r ( x 1 ) r ( x 2 ) is large) and α is small enough,there exists a β such that the condition can be satisfied.
Namely, when the true observation doesn’tchange too fast with the features, we can select a lower β to achieve soft decoupling.
At a high level, β -LOM controls the capacity of the function space Θ ∗ by restricting the Lipschitzconstant β .
While a low β benefits the decoupling, it also suffers from the risk of underfitting.
For example, if we adopt a 0 -LOM (known as the Position-based Model) in the setting that thetrue observation has a large Lipschitz constant, we may not find a ˆ r ( x ) and a ˆ o p ( x ) such that theirproduct is equal to a given click rate c p ( x ) .
In other words, β has a lower bound.
This can be shownformally in the following theorem.
Theorem 2 (Lower Bound of β ) Let c ( x ) denote a vector containing click rates at each position:
[Equation]
For any two different features x 1 and x 2 , we have
[Equation]
Remark
We consider a special situation where the true Observation is 0-Lipschitz ( α = 0 ).
Inthis case, we can find t 1 , t 2 such that t 1 c x 1 − t 2 c x 2 is a zero vector, and the lower bound of β canbe zero.
When the true observation has a large Lipschitz number, t 1 c x 1 − t 2 c x 2 can no longer be azero vector and β should be larger.
It means the models should change quickly with the input featureto fit clicks.
Theorem 1 and Theorem 2 give the upper bound and lower bound of β , respectively.
In practice,we can tune the value of β to achieve the best effect for soft decoupling.
We refer to this method as Lipschitz Decoupling .
4.3 Bernoulli Decoupling
Unfortunately, when α is large in a given dataset, it’s difficult to select a proper β such that theupper bound and the lower bound are satisfied at the same time.
Under this circumstance, it mayperform even worse than that without any debiasing operation (see § 6.2.2 for discussion).
As acomplementary, we propose another decoupling techiques, named Bernoulli Decoupling.
Unlike
Lipschitz Decoupling which makes constraints on observation model’s estimation ˆ o p ( x ; θ 2 ) , weexplicitly add an additional sampling step to transform its output, as follows:
[Equation]
That is to say, we randomly cancel the debiasing operation with a certain probability t (i.e., forcethe observation model to predict 1 as observation score and let the ranking model learn from clicksdirectly), to prevent it from getting worse when the true observation has a large Lipschitz constant.
We define the observation model based on this new sampling process as t - Bernoulli Observation Model ( t -BOM).
For convenience, we denote the expectation E (ˆ o ′ p
x ; θ 2 , t )) as the estimation of the observationmodel after transformation, and Assumption 1 holds based on this expectation.
It’s worth mentioningthat BOM can also achieve soft decoupling, as the following theorem shows:
Theorem 3 (Bernoulli Decoupling)
For any two different features x 1 and x 2 , without loss of generality we suppose r ( x 2 ) ≥ r ( x 1 ) .
For a t − BOM, if the following condition is satisfied,
[Equation]
then we have: ˆ r ( x 2 ) ≥ ˆ r ( x 1 ) .
Remark
If the true observation o p ( x ) is 0-Lipschitz (i.e., α = 0 ), there must exist a t ≤ 1 makingthe above error condition satisfied to achieve soft decoupling.
For a larger Lipschitz number, we canincrease the value of t appropriately.
Similar to LOM, BOM is another method to control the capacity of Θ ∗ , and it also suffers from therisk of underfitting.
In order to make it fit the click rates, t has an upper bound.
Theorem 4 (Upper Bound of t )
For any feature x and a t − BOM, we have
[Equation]
Theorem 3 and Theorem 4 give the lower bound and the upper bound of t respectively.
Like LOM,we take t as a tuning parameter in practice.
We refer to this method as Bernoulli Decoupling .
Bernoulli Decoupling is a complement to Lipschitz Decoupling, and they can be used simultaneously to improve the efficiency of decoupling.
This is because the range in which the two hyperparameters take effect is an ”or” relationship: Even if β is outside the scope of Theorem 1 and
Theorem 2, as long as t is within the scope of Theorem 3 and Theorem 4, the combined model canstill achieve soft decoupling.
This shows that it is beneficial to combine the two models together.
Finally, we propose our ( β, t ) -Lipschitz and Bernoulli Decoupling (LBD) model, which combinesthe β -LOM and t -BOM.
We treat LBD as our complete solution for individual-based ULTR.
Model Implementation
Up to this point, we have shown that a ( β, t ) -LBD can effectively decouple clicks on a relatively lowLipschitz observation dataset.
In this section, we first introduce the implementation of the proposed
LBD, then present the optimization objective of the proposed model.
We first use a neural network f as the ranking model (take features as input) and a neural networkg as the observation model (take features and position as inputs), and output the relevance and theobservation:
ˆ r = f
( x ) , ˆ o p =
g p ( x ) .
Lipschitz Decoupling Based on Theorem 1 and 2, we need to enforce a proper Lipschitz constraint on the observation model to control the capacity and employ Lipschitz Decoupling.
Sincewe use a neural network to implement the observation model, it’s difficult to directly control the
Lipschitz constant without adjusting the network structure too much.
As an easy alternative, [18]proposed to add a penalty on the gradient norm.
Similar to them, we employ a gradient penalty lossfunction to control the Lipschitz: L gp ( x ) = λ (cid:80) ni
=1 ||∇ x
(ˆ o i )
||
This is based on the fact that adifferentiable function is k -Lipschtiz if and only if it has gradients with the norm at most k everywhere.
The gradient penalty loss encourages the model to have a low Lipschitz constant, and we canadjust λ to control the penalty level and obtain models with different Lipschitz constants.
Thus, λcan be a substitute for β .
Bernoulli Decoupling Based on Theorems 3 and 4, in order to implement Bernoulli Decouplingtechnique and avoid degradation of the debiasing process, we sample a γ from Bernoulli distributionparameterized by 1 − t .
Our final logarithmic observation probability is generated by log
ˆ o ′ p
=γ log ˆ o p , where γ ∼ Bernoulli (1 − t ) .
The final predictive click score is: log ˆ c p = log ˆ r + log ˆ o ′ p .
Objective Function and Training Process Similar to [4], we adopt a list-wise loss based onsoftmax cross entropy.
Given a query q with the document ranking list π q = { x 1 , x 2 , · · · , x n } , wegenerate click ˆ c p for feature x p at each position p .
Suppose the real click signals for each documentin π q are c 1 , c 2 , · · · , c n , we use the following supervise loss to train our models:
[Equation]
Finally, the objective function based on a query q can be written as:
[Equation]
We jointly update the ranking model and the observation model with the derivatives of L ( q ) andrepeat the process until the algorithm converges.
Experiments6.1 Experimental Setup
Dataset We follow the standard setup in CLTR [5, 4, 14, 37] and conduct semi-synthetic experiments on two widely used benchmark datasets: Yahoo!
LETOR 1 [12] and Istella-S 2 [31].
Wefollowed the given data split of training, validation and testing.
To the best of our knowledge, we arethe first to study the coupling effects, thus there is no precedent for the click simulation.
Our clicksimulation details can be found in § C.1.
The simulation of the observation is worth mentioning.
We first select some crux features as thefactors that are related to observation, such as result type, result quality or search intent, etc.
Becausethe exact meaning of the features given by the datasets is unknown, we follow the method in previouswork [37, 14] to select some features as substitutes: (1) use normalized features and relevances inthe total dataset to train ExtRa Trees [17]; (2) based on this model, select and combine the top-10important features as crux features.
We adopted the methodology in [37] to model the feature’sinfluence on observation.
The observation probability is set to be:
[Equation]
where v p is the position-based examination probability at position p by eye-tracking studies [25].
wis a 10-dimensional vector uniformly drawn from [ − η, η ] , where η is a hyper parameter to controlthe dependency between the observation and crux features.
x ′ denotes the crux features of x .
Notethat η can be explained as the coupling level between relevance and observation.
Baselines We combined the state-of-the-art ULTR methods and two LTR models for comparison.
Debiasing methods includes RegressionEM [41], DLA [4], PairDebias [21], HTE [37], Labeled
Data (uses human-annotated relevance labels to train the ranker directly) and Click Data (uses theraw click data to train the ranker directly).
Ranking models include DNN and Linear .
We adoptedthe codes in ULTRA framework [5, 6] to implement RegressionEM, DLA, PairDebias and the ranking models, and kept the same hyperparameters.
Note that Regression-EM, DLA and PairDebiasare all group-level ULTR methods.
HTE is individual-level ULTR, but it’s not a jointly learningalgorithm, so we compared it in another scene.
Our method is referred to as LBD .
We also use three degenerate versions for ablation study: (1)set t = 0 to remove the Bernoulli sampling step (named as LBD Lips ), (2) set λ = 0 to removethe gradient penalty loss function (named as LBD Ber ), and (3) remove both (named as Unlimited ).
Training details can be found in § C.2.
[Table]
6.2 Experimental Results
6.2.1 Performance on different rankers and datasets
We first study the performance of jointly learning-based ULTR models.
Table 1 summarizes the
NDCG@10 results about the performance of different rankers on different datasets.
We see that ourproposed LBD significantly outperforms almost all the other baseline methods since we properlyhandle the coupling effect.
The results of other baselines are compatible with those reported in [6].
Particularly, we have the following findings:
LBD works better than LBD Lips and LBD Ber , which indicates that using Lipschitz Decoupling and Bernoulli Decoupling at the same time is better than using them alone.
The reason seems to be that the two decoupling methods have different conditions (see Theorem 1 and Theorem 3).
When using them together, the scope of decoupling can be expanded.
Unlimited works worst than almost all the best baselines, indicating the existence of the coupling effect.
If we train the observation model with the ranking model unlimitedly, the performance would be quite bad.
Additionally, the coupling effect is more serious when using the Linear
ranker.
Lipschitz Decoupling performs better than Bernoulli Decoupling on Yahoo!
dataset, while the opposite is true on Istella-S dataset.
It shows that the application of two decoupling methods is tightly related to the data distribution
[Figure]
.2.2 Performance on different coupling levels
In this experiment, we investigate the performance under different degrees of coupling level on
Yahoo! dataset, where the ranker is DNN.
We vary η from 0 to 0.6 when generating clicks.1 https://webscope.sandbox.yahoo.com/http://quickrank.isti.cnr.it/istella-dataset/
Figure 2(a) shows the influence of coupling levels on different offline ULTR methods.
We canobserve that our model achieves the best performance under all degrees of coupling level.
When theobservation is 0-Lipschitz (i.e., η = 0 ), our model and DLA perform similarly, since both of thememploy a similar listwise loss function.
The debiasing performance of other baselines diminisheswith the increased coupling level.
For a large coupling level ( η ≥ 0 . 3 ), all baselines perform worsethan the raw click data.
In contrast, our proposed method always performs better than click data,which shows the strong decoupling ability of LBD.
Besides, our method exhibits a downward trendwith the increased coupling level, which demonstrates that it’s more difficult to decouple the effectswhen the true observation has a large Lipschitz constant.
Figure 2(b) shows the performance of different decoupling methods we propose in this work.
Onecan see that for a low degree of coupling level ( η < 0 . 2 ), LBD Lips performs better than LBD Ber ,which implies that LOM can decouple the effects under a low Lipschitz observation dataset.
For alarge coupling level, LOM is worse than BOM in turn.
One explanation is that BOM can avoid thedeterioration of ULTR when the relevance and observation effects are difficult to decouple since itrandomly drops out the debiasing operation.
Besides, LBD always performs better than LBD Lipsand LBD Ber , once again showing that using them together could expand the scope of decoupling.
The previous study focuses most on joint-learning ULTR methods.
To compare the Heterogeneous
Treatment Effect (HTE), we conduct another experiment based on intervention data.
Following[37], we simulate 2,560,000 queries for the intervention experiment before training the rankingmodel (this number is equal to the number of total queries we used to train models).
For eachquery, we randomly chose a k ∈ [ n ] and swapped the document at the first position with the oneat the k -th position.
We collected clicks generated on these ranking lists and calculated the HTEestimator ˆ τ k ( x ) .
Based on ˆ τ k ( x ) , we debiased the click data and use it to train a ranking modeldirectly.
We also used an accurate HTE estimator τ k ( x ) from the click model we used, which canbe regarded as the upper bound of the HTE method.
Based on the clicks corrected with τ k ( x ) , wetrain a ranking model directly (named HTE-oracle).
We also use these clicks to train our model(named LBD-HTE-oracle).
We present the results in Figure 2(c).
We can observe that our LBD (trained without any interventiondata) performs even better than HTE and HTE-oracle in a low degree of coupling level because HTEis based on an assumption that the click rates at the first position can reveal the correct relevance,which is biased when the observation probability is not constant.
For a high degree of couplinglevel, HTE performs better than our method.
One possible reason is that intervention can augmentthe dataset (since each document has an opportunity to display at the top position), which is helpfulwhen the effects are difficult to decouple.
Besides, when our model is trained on the clicks correctedwith the oracle HTE, it can always outperform the vanilla HTE method.
This demonstrates onceagain that HTE is still biased and verifies our decoupling effectiveness.
6.2.3 Hyper Parameter Analysis
We investigate the performance with different hyper-parameters in § C.3.
Conclusion
In this work, we take the first step to study the coupling effect in individual-level ULTR.
We proposetwo goals: hard decoupling, where the true relevance probability should be estimated accurately,and soft decoupling, where the correct relevance order can be recovered.
We modify the observationmodel to achieve soft decoupling when the click probability estimation is unbiased.
We propose twospecial observation models, Lipschitz Observation Model (LOM) and Bernoulli Observation Model(BOM).
We provide a theoretical guarantee that when the hyper-parameters of them are bounded,soft decoupling can be achieved by these methods.
We conduct comprehensive experiments on two
LTR benchmark datasets, which shows that the proposed LBD outperforms the baseline methodsand verifies its effectiveness in debiasing data.
Limitations.
For simplicity, our theoretical results are based on Assumption 1, which requiresthe models to fit clicks perfectly.
It would be interesting to investigate how the decoupling effectwill be affected if click estimation is not so accurate.
Besides, how to decouple relevance andobservation when the coupling effects are extremely severe (i.e., observation function has a large
Lipschitz constant) is an open question.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work? [No] The neg- ative societal impacts are limited.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
is not resource using assets (e g
, code, data, ) or curating/releasing new assets (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]