Paper: Fair Off-Policy Learning from Observational Data (ICLR-2024)

Reviewer comment (Evaluation):
**Results of the real-world experiments do not seem promising.** The proposed approach improves "action fairness", however, it does not improve or even sometimes worsens the "value fairness" compared to the naive, unconstrained policy. Justifications are needed for this point.

Author response:
Results on real-world data: In Table 2, we first report the results for our framework with only action fairness and without value fairness (second row, “FairPol only with action fairness”). Here, value fairness is indeed worse compared to the unrestricted policy (first row). However, this is to be expected as FairPol enforces action fairness, which can worsen value fairness (we refer to our toy example in Appendix C. for a detailed discussion on this issue). In contrast, the (estimated) action fairness is improved by a large margin. Furthermore, comparing our results for value fairness (third and fourth row of Table 2) directly with the unrestricted policy (first row) is misleading because the unrestricted policy does not ensure action fairness. Instead, FairPol with max-min fairness (fourth row of Table 2) is effective in improving value fairness compared to FairPol with only action fairness (second row of Table 2). In summary, **the results using real-world data confirm the effectiveness of our framework.**

    **Action:** We followed your suggestion and expanded our discussion of our results using real-world data in Sec. 6.

Reviewer comment (Writing):
**The clarity of writing has some room for improvement.** Aside from the above points, this paper has some additional ambiguous parts. Please refer to Questions 2 and 3.

Author response:
Clarity of writing: Thank you for pointing this out. We improved our writing along your suggestions. Please refer to our “Response to Questions”.

Reviewer comment (Experiments):
**Experimental details missing for synthetic experiments.** The paper states that results are provided for three policy scores ($m \in$ {DM, IPS, DR}), but Table 1 reports results for only one, without specifying which. No additional results were found in the Appendix. Clarification is requested, and it is suggested to report the number of random seeds used.

Author response:
Experimental details: **In Table 1, we reported the results for all methods** $\mathrm{m} \in \{\mathrm{DM}, \mathrm{IPW}, \mathrm{DR}\}$. Here, for each method, we reported three rows corresponding to $m$. In particular, we do not observe large differences in results between the three methods. Thus, we fixed $m = \mathrm{DR}$ (corresponding to the doubly robust method) in our other experiments (e.g., Table 2; Fig. 3. and 4.).

    **Action:** We added a clarification to Sec. 6.

Reviewer comment (Writing):
**Clarification on notation of regression functions.** On page 4, the paper denotes "$\mu_j(X, S) = \mathbb{E}[Y | X, S, A = j], j = \{ 0, 1 \}$, are the outcome regression functions" (of DM and DR). The reviewer notes that $\mathbb{E}[Y | X, S, A = j]$ by definition refers to the conditional expected reward rather than the outcome of regression. It would be more appropriate to write $\mu_j(X, S) \approx \mathbb{E}[Y | X, S, A = j]$ (using "$\approx$" instead of "$=$").

Author response:
“Outcome regression”: Thank you for pointing out this ambiguity. **Action:** We replaced the naming “outcome regression” with “response surface”, which is consistent with previous work on off-policy learning and treatment effect estimation.

Reviewer comment (Writing):
**Clarity on focus of fairness type in the Introduction.** Reading the Introduction, it was unclear whether the paper focuses on user-side or action-side fairness. Since there are discussions on item fairness in recommender systems (e.g., Singh & Joachims, 18), it might help to explicitly state whether sensitive features belong to users' covariates or actions' features. The reviewer notes that Section 2 makes it clear that the focus is on user-side fairness.

Author response:
User vs action covariates: Thank you. You are correct in that we defined our covariates $X$ to be user-specific attributes, not attributes of the action/treatment. **Action:** We added a clarification to Sec. 3.

Reviewer comment (Novelty):
While the proposed method can learn optimal policies under both action and policy value fairness, the underlying technique seems to be very similar to the existing methods. The technical contribution of this paper is unclear to me. Specifically, the proposed solution includes two steps: the first step aims to learn fair representation to ensure action fairness, while the second step incorporates the policy value fairness constraints to the objective function. For the first step, the idea of leveraging adversarial objectives to achieve independence between sensitive attribute and representation is similar to the concept of GAN, a common approach for fair representation learning. For the second step, the learning objective with fairness constraint is also straightforward.

Author response:
* Thank you for giving us the opportunity to clarify our technical contributions. You are correct in that we build our framework on existing literature streams, e.g., fair representation learning. However, we believe that our paper makes several non-trivial technical contributions beyond that. In particular: (i) We adapt fairness notions from related literature and **formalize these for off-policy learning from observational data**. (ii) We provide a **theoretical understanding of how these fairness notions interact in the context of off-policy learning** (see Lemma 1 and 2, as well as our toy examples in Appendix C and D). (iii) We propose a **practical framework to learn fair optimal policies. **While we build upon ideas from fair representation learning to achieve action fairness, we combine these with the off-policy learning methodology** and provide novel theoretical guarantees** in the form of generalization bounds (Theorem 1). Hence, we believe that our paper contains several technical contributions that add to the state-of-the-art in algorithmic fairness and off-policy learning in a non-trivial manner.

    **Action:** We clarified our technical contributions in Sec.1. We also expanded our discussion (see end of Sec. 6), in which we also reiterated our contributions

Reviewer comment (Experiments):
The paper doesn’t compare with any algorithms in experiments. While the authors claim that existing methods are proposed in a different setting and cannot serve as the baseline, I wonder whether the methods can be adjusted and applied. For example, when learning fair representation in the first step, many approaches in fair representation learning may be applied and serve as baselines.

Author response:
* **No baselines available:** We agree that, generally, a thorough comparison with baselines is beneficial to provide a rigorous empirical evaluation. Unfortunately, there are **no** baselines directly available as our setting of fair policy learning has not yet been studied before.  
Nevertheless, we took great strides to adapt existing methods as baselines. In the context of our work, we see two potential literature streams that could provide methods baselines for our experiments: (i) Methods for standard off-policy learning from observational data (not considering fairness); and (ii) methods that incorporate fairness notions into off-policy learning. In the following, we lay out how we build upon methods from (i) and (ii) for benchmarking: 
    * For (i), the following three methods are considered standard in the literature [1]: the direct method, the IPW method, and the DR method. **Our proposed framework works for all of these, and we provided experiments for all three baselines **(see Table 1). We did not observe large differences in performance.
    * For (ii), **we are not aware of any other work that aims at achieving the same fairness notions (i.e., combining action and value fairness) in the context of off-policy learning**. Any potential benchmarking experiment with existing methods would thus involve comparing methods that satisfy different fairness notions. Hence, **we follow established fairness literature [e.g., 2, 3] and do not compare methods targeted at different fairness notions**. Instead, we investigated in our experiments how well our proposed method achieves our proposed fairness notions, and how this affects the performance (policy value). For this purpose, **we provided several ablations studies:** for example, we remove parts of our method (value fairness), and **we compare with an unrestricted policy** that does not satisfy any fairness notions.

    We agree that our approach for fair representation learning (adversarial domain confusion loss) could in principle be replaced by any other approach that aims at enforcing independence from the sensitive attribute.  Hence, we provide an **additional ablation study (see new Appendix K)**, where we compare the first step of our framework with two other approaches: (i) Adversarial learning using gradient reversal; and (ii) regularization using a Wasserstein distance. Our chosen approach (adversarial domain confusion loss) performs best, which is also consistent with prior literature [1].


    **Action:** We **added new experiments** based on fair representation learning (see our **new Appendix K**).

Reviewer comment (Theory):
Many approaches have been proposed to learn ML models under fairness constraints such as demographic parity, envy-free fairness, max-min fairness. Except for the objective function, what are the technical challenges of off-policy learning compared to tractional supervised learning? In other words, why cannot methods proposed in fair supervised learning be modified and directly adapted to off-policy learning?

Author response:
* **Differences to standard supervised learning:** Thank you for the question. As you mentioned correctly, one important difference between off-policy learning and (standard) supervised learning is the **difference in objective function**: In off-policy learning, we are interested in maximizing the policy value, which is estimated using nuisance parameters that depend on the sensitive attribute. Note that we can not simply remove dependence from the sensitive attribute, as this could lead to unobserved confounding and thus biased policy value estimates. Hence, combining off-policy learning with fairness results in challenges for both (i) fairness considerations; and (ii) finite-sample performance when introducing fairness constraints during policy learning. 

    In our paper, we address these challenges as follows: For (i), we provide a **theoretical understanding** of different fairness notions in off-policy learning (Lemma 1 and 2). We also show explicitly how “action-fair” policies may still be considered unfair due to discrepancies in policy values for different sensitive groups (see our toy examples in Appendix C and D). For (ii), we investigate the off-policy learning problem with fairness constraints from a statistical perspective and provide theoretical learning guarantees in the form of generalization bounds (Theorem 1).

Reviewer comment (Experiments):
Can we adapt existing methods to your setting and treat them as baselines?

Author response:
* **Additional baselines:** Thank you for your question. We agree that, generally, a thorough comparison with baselines is beneficial to provide a rigorous empirical evaluation. Unfortunately, there are **no** baselines directly available. Nevertheless, we took great strides to adapt existing methods as baselines where possible (see our “Response to Weaknesses”, point 2). We also **added new experiments** using benchmarks from fair representation learning (see our **new Appendix K**).

Reviewer comment (Reproducibility):
One concern is that the paper does not discuss how they deal with observational data. In particular, the proposed method depends on several estimators like DM, IPW, DR. But some quantities are unknown in the observational setting, like \mu_1, \mu_0, \pi_b. I would be great if the authors can discuss how these quantities are obtained, and how they affect the theoretical and empirical results.

Author response:
1. **Observational data/ nuisance parameters:** Thank you for your question. You are correct that our method (or rather any method for off-policy learning) depends on the policy score $\psi^\mathrm{m}(\pi, \mathcal{D})$ and, therefore, on the unknown nuisance parameters $\mu_1$, $\mu_0$, and $\mu_0$ (which correspond to the choice of $\mathrm{m} \in \{\mathrm{DM}, \mathrm{IPW}, \mathrm{DR}\}$, respectively). Under Assumption 1, these nuisance parameters can be estimated from the observational data using arbitrary regression/machine learning algorithms (see Sec. 3). **In our experiments on synthetic data, we do not focus on the question of estimation errors in the nuisance parameters but rather aim to explore the impact of our proposed fairness notions**. For this purpose, we refrain from estimating the nuisance parameters but instead use the ground-truth nuisance parameters, which are known in our simulations (we refer to Appendix G for details). Note that we do this for all methods to ensure a fair comparison of the results.  

    Motivated by your comment, we added **new experimental results** (see our **new Appendix L**). Therein, we re-ran our experiment from Table 1 but now use TARNet as a state-of-the-art method [1] to estimate nuisance parameters (instead of using the ground-truth). This allows us to analyze the robustness of our results on synthetic data with respect to nuisance errors. Our new results show that our results **remain robust** when using estimated nuisance parameters. 

    **Action:** We provided a more elaborate explanation for our evaluation. Furthermore, we **added new experimental results** by using TARNet for estimating nuisance parameters instead of using the ground truth (see our **new Appendix L)**. We added details regarding the implementation of TARNet to Appendix M.

Reviewer comment (Novelty):
The contribution seems limited. The fairness notions proposed in this paper appear in many prior works. The approach to achieving "action fairness" seems to be borrowed from prior works. The approach to achieving "value" fairness is a direct application of counterfactual estimator.

Author response:
3. **Contribution/novelty of fairness notions:** Thanks. You are correct in that our proposed fairness notions are adapted from existing literature streams. For example, “action fairness” resembles “demographic parity” in standard fair machine learning, and both “max-min” and “envy-free” fairness have also been proposed in works from optimization (e.g., from resource allocation). The reason is that the fairness notions are deeply grounded in distributive justice. Please note that **we do not claim to be the first to propose the mentioned fairness notions**. Instead, our contributions are as follows: 
    1. (i) We propose to adapt existing fairness notions from related literature (e.g., resource allocation), but **formalize these for off-policy learning from observational data**; 
    2. (ii) We provide a **theoretical understanding of how these fairness notions interact in the context of off-policy learning** (see Lemma 1 and 2, as well as our toy examples in Appendix C and D); 
    3. (iii) We propose a **practical framework to learn fair optimal policies and provide theoretical guarantees** in the form of generalization bounds.

    **Action:** We clarified our contributions in Sec.1. We also expanded our discussion (see end of Sec. 6), in which we also reiterated our contributions

Reviewer comment (Theory):
In assumption 1, it should be P(A=1 | X=x, S=s), right? Otherwise, equation (3) and (4) might be ill-defined since \pi_b(X,S) can be zero.

Author response:
1. Thank you. You are correct: it should be $\mathbb{P}(A=1 \mid X = x, S = s)$. We apologize for the typo and corrected it in our revised paper.

Reviewer comment (Theory):
Theorem 1 is hard to understand. I think it must depend on the logging/behavior policy. Maybe it is captured in K? It would be great if the authors can have a clear definition of these notations. And it does not seem to take into account the fact that we are in the observational setting as I mentioned in 1.

Author response:
3. Thank you. You are correct: our bounds in Theorem 1 do indeed depend on the nuisance parameters (in particular on the logging policy/propensity score) through the constant $K_m$, where $\mathrm{m} \in \{\mathrm{DM}, \mathrm{IPW}, \mathrm{DR}\}$ indicates the policy score (the thus the choice of nuisance parameters). In our proof (see Appendix B), we show that $K_{\mathrm{DM}} = 1$, $K_{\mathrm{IPW}} = \frac{1}{2 \eta}$, and $K_{\mathrm{DR}} = \frac{\eta + 1}{\eta}$, where $\eta$ is a constant that satisfies $\mathbb{P}(\eta \leq \pi_b(X, S) \leq 1 - \eta) = 1$. Note that, in Theorem 1, we consider the ground-truth nuisance parameters because it has been shown that, under mild assumptions (e.g., uniform consistency of nuisance estimators), estimation errors in the nuisance parameters do not affect the leading term in the convergence rate (see Sec. 3.1, Sec.3.2, Lemma 4 in [1]). This is also consistent with previous work that provides generalization bounds for off-policy learning [2, 3].

    **Action:** We moved the definition of $K_{\mathrm{DM}}$ to the main paper to clarify the dependence of the bounds on the policy score.