Paper: Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank (ICLR-2024)

Reviewer comment (Novelty):
In the first practical scenario (many data, small bias factor space), which is common in practice, people are fine without worrying about identifiability, and the analysis and methods in this paper mostly do not apply since the graph is small and likely connected.

Author response:
Below are the responses for your other reviews:

> So there are two scenarios in practice: 1) there are a lot of data, and the bias factor space is small. The analysis and methods in this paper mostly do not apply in such case ...

We acknowledge that node intervention/merging are indeed unnecessary in the identifiability issue. However, **we would like to kindly highlight that Theorem 1 (identifiability checking) is also our central contribution and important analysis**, which builds a theoretical base for current ULTR algorithms and is applicable to all cases. As R ggp4 said, this graph view is useful for nicer properties, more intuition-based understandings and more explainability in ULTR. This might provide a new perspective for future ULTR research.

Reviewer comment (Theory):
Theorem 1 issue: While the trend of using more bias factors is debatable (especially given existing work on coupling/confounding effects), the trend to enrich x is clear. Many real-world applications have personalized feature vectors, making the graph unlikely to be connected even with few bias factors. The paper does not consider this perspective, and people are fine working with ULTR on such datasets. This questions the value of the proposed framework. Additionally, the condition presented is only a sufficient condition.

Author response:
> Theorem 1: Many real-world applications have personalized feature vectors - then the graph is not likely to be connected even with a small number of bias factors ... people are fine working with ULTR with such datasets ... one should also note that the condition is only a sufficient condition.

- We appreciate your insightful suggestion about the feature count. We completely agree that the graph is not likely to be connected when enriching $X$. To demonstrate it, we conducted a thorough experiment for tuning the feature numbers, confirming that increasing X can indeed lead to potential unidentifiability issues. The results are in Appendix E.6 in the revised version. **The current work is okay in public ULTR datasets because the number of features hasn't reached the unidentifiability threshold, as we showed in Section 5.2.**

- We would like to kindly highlight that Theorem 1 is not just a sufficient condition. It's a **necessary and sufficient** condition for identifiability and relevance recovery.

Reviewer comment (Theory):
Theorem 2 issue: The assumptions are too strong to provide meaningful value in this highly practical field. Although it is presented as a simplified "estimate," its value remains very limited.

Author response:
> Theorem 2: The assumptions are too strong to make meaningful value from this theorem. The reviewer understands this is to show a simplified “estimate”, but still the value is quite limited for this highly practical field.

Thanks for pointing it out. Note that this theorem is not critical for our proposed method and the experiments we conducted. Its purpose is to provide an intuition of how "increasing bias factors or decreasing data can potentially lead to unidentifiability" in a simple scenario. We've verified this in a more complex and practical setting in Section 5.2. **To reduce its significance, we've renamed Theorem 2 to Corollary 1.**

Reviewer comment (Theory):
Theorem 3 issue: There is self-contradiction with the paper's motivation. The paper focuses on large bias factor space, but this theorem assumes that each (input feature, bias factor) pair needs N observations from a Bernoulli distribution, with the theorem based on “N is sufficiently large.” This seems meaningless for the target scenario.

Author response:
> Theorem 3: There’s self-contradiction with the motivation of the paper ... the theorem is based on “N is sufficiently large”.

Thanks for pointing it out. After careful inspection, we've discovered that the assumption "N is sufficiently large" is not necessary for our proposed method, as it's only used for applying the central limit theorem. **We have removed this unnecessary assumption and restructured the theorem to present only the variance term, which is all our method requires.**

Reviewer comment (Theory):
Theorem 4 issue: The error bound is only shown for merging two subgraphs. Given the concern with large bias factor space where the number of subgraphs could be high, there is no error bound provided for the entire merging process or consideration of possible error propagation to meaningless values. Showing error bound only for two subgraphs is very limited.

Author response:
> Theorem 4: The error bound is only shown to merge two subgraphs.

Apologies for the confusion about the theorem name "error bound of merging". Its purpose is not to establish the error bound for node merging but to illustrate the error when combining two components, which forms the foundation for the below merging cost (Eq.7). **To address the error bound for node merging, we have created another corollary in Appendix B.5.**

Reviewer comment (Experiments):
On the experimental side, the evaluation is weak because the proposed methods are only compared against a very basic baseline. The authors argue the methods are agnostic to the algorithm and “aptly represent the current research,” but the reviewer strongly disagrees, asserting that these methods should also be tested on more sensible baselines, such as state-of-the-art ULTR methods. Without such evidence, the practicality and relevance of the methods are questionable, especially given their clear caveats (node intervention method is not very practical, node merging method likely introduces errors).

Author response:
> Experiment: will they help state-of-the-art ULTR methods? 

We appreciate your feedback about the evaluation. **We've made changes to the paper, and now it includes results for DLA, Regression-EM, and Two-Tower**. Please check our general response for more details.

Reviewer comment (Theory):
Theorem 3 is straightforward and does not seem useful; the conditions required by Theorems 2 and 4 are stringent (see Q2 below for more details).

Author response:
Thanks for pointing out the problem. This theorem is not critical for our proposed method and the experiments we conducted. Its purpose is to provide an intuition of how "increasing bias factors or decreasing data can potentially lead to unidentifiability" in a simple scenario. We've verified this in a more complex and practical setting in Section 5.2. **We appreciate your suggestion of renaming and we've renamed Theorem 2 to Corollary 1.**

Reviewer comment (Experiments):
In the experiment, there are only two weak baselines. In addition, there are lack of details about the two baselines. However, this article mentions a lot of related works but does not include them as part of the baselines for comparison empirically. This is inconsistent with the requirements in this area of ULTR. Could you add some cutting-edge ULTR methods as baselines?

Author response:
Dear Reviewer tHmn,

Thank you for reviewing our work and aiding in its improvement. In this revision, we've **compared the existing ULTR algorithms**, **clarified the motivation behind Theorem 3**, and **introduced an additional corollary to fortify Theorem 4 for increased reliability**. Please let us know if these efforts address your concerns. We remain open for further discussion if needed. Thank you for your time and consideration.

Best regards,

Authors

Reviewer comment (Theory):
Theorem 2 is a very simple case (… $x$ and $t$ are selected independently and uniformly …). Thus, it is not sufficient to write it as a Theorem; it might be more appropriate to write it as a Proposition.

Author response:
Thanks for pointing out the problem. This theorem is not critical for our proposed method and the experiments we conducted. Its purpose is to provide an intuition of how "increasing bias factors or decreasing data can potentially lead to unidentifiability" in a simple scenario. We've verified this in a more complex and practical setting in Section 5.2. **We appreciate your suggestion of renaming and we've renamed Theorem 2 to Corollary 1.**

Reviewer comment (Theory):
Theorem 3 is simple and straightforward (just by the central limit theorem) and does not seem useful. Could you clarify the purpose and use of Theorem 3? Also, it is not sufficient to write it as a Theorem; it might be more appropriate to write it as a Lemma.

Author response:
Thanks for pointing out this point. We kindly remind that this theorem is necessary in the proposed node intervention method: The purpose is to used to compute the specific form of variance, which serves as the cost function (Eq.3). The basic idea is that there are too many choices that the intervention targets can be selected, and in Theorem 3 we find that minimizing the variance helps facilitate the identifiability. This is why we've chosen variance as our cost function for choosing intervention target, and **we've highlighted this point in Theorem 3, which is now renamed Proposition 1.**

Reviewer comment (Theory):
The condition required in Theorem 4, "A disconnected IG consists of two connected components G1 and G2" is strong and difficult to fulfill in practice.

Author response:
Apologies for the confusion about the theorem name "error bound of merging". Its purpose is not to establish the error bound for node merging but to illustrate the error when combining two components, which forms the foundation for the below merging cost (Eq.7). **To address the error bound for node merging, we have created another corollary in Appendix B.5.**

Reviewer comment (Theory):
The soundness of Theorems 3 and 4 is very critical to this paper for the following reasons: (a) The graph is always disconnected in practice, and will suffer from identifiability problems; (b) To recover the identifiability, we always need node intervention and node merging to recover the connected graph for empirical applications. Thus, the rationality of these two proposed algorithms (node intervention and node merging) becomes critical. Regrettably, Theorems 3 and 4 are slightly weak, which seriously undermines the contribution of this paper.

Author response:
We hope the above responses and revisions could strengthen Theorem 3 and 4. Please confirm whether our explanations address your concerns. If you have any more questions or additional concerns, please feel free to ask, and we will make every effort to address them.

Reviewer comment (Theory):
Since the theory of this paper relies heavily on the examination hypothesis, the graph-equivalence idea is not generalizeable to other general hypotheses on the dataset.

Author response:
We agree that this paper is based on the examination hypothesis. We opted to focus on it because it's the most widely used generation process (see Appendix A). However, we kindly remind that it does not mean the graph idea is not applicable to other hypothesis. **Actually, to generalize this idea to other hypothesis, only small adjustments in Definition 1 and Theorem 1 are required while maintaining the core structure of IG**. We leave it as future work since it is more complicated and falls beyond the scope of this work.

Reviewer comment (Experiments):
Since the methods are still propensity-based, there are a plethora of such ranking models. It would be fair game if the paper uses such methods as baselines to strengthen the validity of the proposed methods.

Author response:
We appreciate your feedback about the evaluation. **We've made changes to the paper, and now it includes results for DLA, Regression-EM, and Two-Tower**. Please check our general response for more details.

Reviewer comment (Experiments):
When choosing the bias factors, we can choose either fewer factors, which makes the graph more likely to be connected, or more factors, which accounts for more bias but poses a more disconnected graph. It would be great if there is any discussion on the tradeoff and the corresponding performance of the two proposed methods.

Author response:
We report the MCC performance of (1) only considering one bias factor; (2) considering all bias factors; and (3)(4) our proposed methods in the K=2 simulation dataset:

|ULTR method|DLA|Regression-EM|Two-Tower|
|-|-|-|-|
|(1) one bias factor|0.641|0.641|0.641|
|(2) all bias factors|0.738|0.634|0.892|
|(3) node intervention|1.000|0.982|1.000|
|(4) node merging|0.987|0.975|0.987|

The trade-off depends on the specific ULTR algorithms: Regression-EM faces more trouble with unidentifiability, whereas DLA and Two-Tower have more issues with bias. This happens because when there's no assurance of identifiability, different ULTR algorithms might end up with various poor model parameters. But when our methods ensure identifiability, they all converge to a shared set of good settings that recover the relevance accurately.

Reviewer comment (Experiments):
In the real world, the dataset is mostly sparse and thus there might be a large number of connected components in IG. How much does the performance of the two methods deteriote with the increasing sparsity? Is there a systematic way to deal with that issue?

Author response:
We presents the `nDCG@10 / MCC` results with the increasing numbers of connected components (K) in the fully-simulation datasets as follows:

||K=1|K=2|K=3|K=4|
|-|-|-|-|-|
|No debias|0.863 / 0.657|0.858 / 0.641|0.859 / 0.638|0.870 / 0.655|
|DLA|1.000 / 1.000|0.900 / 0.738|0.889 / 0.700|0.876 / 0.660|
|+ Node intervention |-|1.000 / 1.000 |1.000 / 1.000|1.000 / 1.000|
|+ Node merging |-|1.000 / 0.987|1.000 / 0.978|0.947 / 0.835|

We can observe that the performance of DLA and Node merging drops as the $K$ increases, while Node intervention always recovers the relevance accurately. Overall, our proposed methods significantly improve the initial results.

> Is there a systematic way to deal with that issue?

If doing intervention experiments is allowed, node intervention is always perfered. Otherwise, node merging is also acceptable since it can siginificantly reduce the unidentifiability issue.

Reviewer comment (Experiments):
In the node merging method, the costs between nodes are computed based on their deterministic features X_t. However, how is it guaranteed that the features reflect their true similarity? For example, we may use document rank as a bias factor when only considering the position bias. But it turns out that the user may notice the documents in the order of: the first several documents (since they’re most noticeable), the last several documents on this page (since users may scroll down), and then documents in the middle. In the more complex settings of several factors, it’s even less obvious which nodes are similar to each other. Is it possible to make the features not deterministic but rather learned throughout multi-task learning?

Author response:
This is an interesting question. This depends on prior knowledge, often needing practitioners to manually define click models based on their experience to represent this bias similarity. While it's beyond this paper's scope, we speculate that automatically learning this similarity might not be viable. This is because unidentifiability arises from incomplete datasets and limited information, which challenges learning the actual similarity.

Reviewer comment (Reproducibility):
In the application of the method, the dataset is mostly online and continuously taking in new data points. How does the proposed method handle the updates efficiently? For example, if two nodes (bias factors) with similar features are already merged but the new datapoints from the user creates an edge between them, is there a way to efficiently deal with this?

Author response:
We can maintain a set recording all the merging pairs. When new data arrives, we can check if it's already in the merging set. If it is, we can separate the two nodes and handle them individually in the upcoming training.

Reviewer comment (Writing):
For an unbiased learning-to-rank algorithm, there is always a ranking algorithm base. It is not clear how the proposed model incorporates the existing ranking models.

Author response:
**Our proposed algorithms work with the dataset**: First, we use algorithm 1 to assess if the dataset is identifiable. If it's not, we use algorithm 2 and 3 to adjust the dataset and ensure that its IG is connected. Once that's done, **we apply existing ULTR algorithms and train ranking models using this processed dataset**. This means our method is compatible with various models. We've explained this in Section 5.1 (in initial submission) and explicitly mentioned it in the Introduction (in the revised paper).

Reviewer comment (Experiments):
It is not clear how this approach can be applied to existing ranking models in the context of unbiased learning-to-rank, where there should be a ranking function (often biased) and the goal is to build a debiasing method that can be incorporated into the biased ranking models.

Author response:
**Our proposed algorithms work with the dataset**: First, we use algorithm 1 to assess if the dataset is identifiable. If it's not, we use algorithm 2 and 3 to adjust the dataset and ensure that its IG is connected. Once that's done, **we apply existing ULTR algorithms and train ranking models using this processed dataset**. This means our method is compatible with various models. We've explained this in Section 5.1 (in initial submission) and explicitly mentioned it in the Introduction (in the revised paper).

Reviewer comment (Experiments):
In the experiment part, the authors only compare no debias and a simple examine hypothesis method, without comparison to more existing unbiased learning-to-rank algorithms.

Author response:
We appreciate your feedback about the evaluation. **We've made changes to the paper, and now it includes results for DLA, Regression-EM, and Two-Tower**. Please check our general response for more details. Our initial submission is linked with DLA, the paper you first shared, as mentioned in Appendix E.3. As for the second paper you shared (known as Pairwise-Debiasing), we didn't compare with it for specific reasons:

- It is not a theoretically sound method, which is proved in [1] (Section 7.1) and [2] (Section 4.1.4).

- It's not based on examination hypothesis, which is our framework developed on.

[1] Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank.

[2] Unbiased Learning to Rank: Online or Offline?

Reviewer comment (Novelty):
The authors should compare and discuss the connections to existing unbiased learning-to-rank algorithms such as “Unbiased Learning to Rank with Unbiased Propensity Estimation” and “An Unbiased Pairwise Learning-to-Rank Algorithm”.

Author response:
We appreciate your feedback about the evaluation. **We've made changes to the paper, and now it includes results for DLA, Regression-EM, and Two-Tower**. Please check our general response for more details. Our initial submission is linked with DLA, the paper you first shared, as mentioned in Appendix E.3. As for the second paper you shared (known as Pairwise-Debiasing), we didn't compare with it for specific reasons:

- It is not a theoretically sound method, which is proved in [1] (Section 7.1) and [2] (Section 4.1.4).

- It's not based on examination hypothesis, which is our framework developed on.

[1] Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank.

[2] Unbiased Learning to Rank: Online or Offline?