Paper: Long-Term Impacts of Model Retraining with Strategic Feedback (ICLR-2024)

Reviewer comment (Theory):
While the paper investigates long-term interactions, it makes the simplifying assumption that the joint distribution \(P_{XY}\) is fixed over time. This is clearly not the case, especially when who stays in the system or who comes into the system is determined by the previous interaction of the ML system with different demographic groups. User retention and the effect of retraining on the representation disparity seem to be neglected in the current work.

Author response:
In this paper, we mainly focus on retraining with different data sources and strategic agents, which is a novel setting that was never considered by previous literature. Adding participation dynamics may indeed model reality better, but it will add more complexity to the model, which may be a future direction.

Reviewer comment (Theory):
How Assumption 3.1 is justified? How does having a fixed hypothesis class allow us to ignore algorithmic bias? It clearly varies with \(D^t\). Is it assumed that it does not vary much with \(D^t\)? Or is it assumed that the algorithmic bias is very small compared to other biases for all possible \(D^t\) values? If so, can you justify this, perhaps by going over an example scenario and by providing experimental evidence?

Author response:
When we consider a specific class of models, the algorithmic bias may not be negligible in practice, but the focus of our paper lies in the effects brought by the retraining process. Instead, algorithmic bias has already been extensively studied in the statistical learning theory and it is inevitable to some degree. With *model-annotated* samples, the sample complexity becomes very large, which already makes algorithmic bias as small as possible (the remaining bias is caused by the distribution itself). Our empirical results contain algorithmic bias, and it still demonstrates the validity of the theoretical results.

Reviewer comment (Theory):
Theorem 3.5 does not shed light on the lower limit of the qualification rate. Can it get worse than the qualification rate under \(P_{XY}\)?

Author response:
Due to the monotonic likelihood assumption, the best response will not make agents' qualifications lower than $q_0$, so the lower limit will be $q_0$.

Reviewer comment (Theory):
If the decision-maker knows that there are long-term impacts, then why should it use a one-shot trained classifier like logistic regression? Isn’t it better to incorporate the knowledge of the long-term impacts of strategic agents and classifier-annotated samples within the loss function to be optimized?

Author response:
There are frameworks considering using repeated risk minimization to update a single classifier in long term [Perdomo et al., 2020]. However, the training strategy needs the loss function to satisfy rigorous assumptions and is not likely to converge to an optimal point. Thus, as a more spontaneous solution, the decision-maker tends to gather data each time to update the classifier, which is the common practice nowadays. Our paper aims to investigate whether this practice will bring unexpected outcomes.

Reviewer comment (Theory):
About implementation of the early stopping mechanism. Does the decision maker know that it has systematic bias? How does it know the identities of the advantaged and disadvantaged groups? If these are not known, then how can this early stopping mechanism be implemented?

Author response:
In practice, although the decision-maker may not know the exact magnitude of systematic bias, but it usually has sufficient prior knowledge of the group identities and the direction of the systematic bias. For example, in a college admission scenario, the admission may already have good knowledge on which ethnic group (race) is historically underrepresented, thereby assuming a negative systematic bias exists.

Reviewer comment (Evaluation):
The appendix includes results where all examples are human-annotated. Investigating the figure there, can’t we conclude that it is better to forget about model-annotated samples and train only with human-annotated samples (although they are much fewer in number)?

Author response:
In our current experiment, we assume the decision-maker can obtain 2000 *human-annotated* samples each round, and the underlying distribution $P_{XY}$ is relatively easy to learn. In practice, the availability of *human-annotated* samples may be more limited, making the algorithmic bias significantly larger due to the low sample complexity.

Reviewer comment (Theory):
The main weakness of the theoretical results in this work is that the authors focus on changes in “marginal” quantities, such as the acceptance rate \(E_{S_{t}}[A(f_{t}, P^{t})]\) instead of changes in conditional quantities, such as the acceptance rate conditional on a particular choice of covariates \(x\), which is given by \(E_{S^{t}}[f_{t}(X) \mid X=x].\) As a result, it is difficult to understand to what extent their results derive from the data provided from the strategic agents (covariate shift), the systemic bias in the human-annotated samples (one form of conditional shift), or the fact that the conditional distribution of the classifier \(f_{t}(X) \mid X=x\) does not (second form of conditional shift). To what extent do the results derive from changes in the covariate distribution and changes in the classifier? In the absence of systemic bias, Theorem 3.3 seems to derive from the fact that after agents respond strategically to the model, they are more qualified to get a positive label (agents sampled from \(P_{X}^{t}\) are more likely–based on \(P_{Y|X}\) – to get a positive label than agents sampled from \(P_{X}\)). Thus the acceptance rate of agents would increase over time, simply because there are now more qualified agents. It would be helpful if the authors could clarify their presentation and discuss how much these different forces yield changes in the distribution of \(f_{t}(X) \mid X=x\) – the conditional distribution of the classifier.

Author response:
Firstly, our model mainly consists of 3 parts as shown in Eqn.(3) of the main paper: *human-annotated* samples, *model-annotated* samples, and agents' strategic best responses. We study them together since each of them is likely to play a role under the strategic retraining setting. Moreover, our model admits different proportions of *human-annotated* samples and *model-annotated* samples, which means the results are suitable for a process updated with only *model-annotated* samples or *human-annotated* samples. In App. G.3 (Lemma G.2), the proof details reveal that both $\mathbb{E}[f_t(X) | X = x]$ and $\overline{q}_t$ are increasing. However, note that the increasing $a_t$ is not due to the increasing $q_t$. As Prop. 3.4 shows, $q_t$ can decrease over time. It is just because the model "misannotate" some agents to be qualified.

Reviewer comment (Theory):
The assumption that agents’ covariate modification translates to genuine changes in the label is a somewhat strong assumption. This means that the classic “gaming” type behavior (where agents may manipulate their covariates while their labels remain fixed) is not permitted. This assumption greatly reduces the complexity of the problem because when “gaming” is not permitted, the strategic behavior problem becomes equivalent to covariate shift (no gaming essentially implies that \(P_{Y|X}\) is fixed over time). Since this assumption is a departure from previous works in strategic classification, where previous works permit an agent to “game” and take meaningful actions that change their label [Kleinberg and Raghavan, 2020], the authors should flag it and make the contrast more clear in their paper. Although the authors have a related work section in their appendix, it is urged that they include some of these references in the main text of the paper.

Author response:
Firstly, our work takes interest in how the qualification rate changes as the retraining goes on, so we assume the covariate shift because feature shifts will not produce a qualification rate change. Although the earliest works on strategic classification consider only the feature shifts, it is quite common for recent works to consider covariate shifts ([Rosenfeld, et al., 2020; Raab \& Liu, 2021; Guldogan et al., 2022]. Moreover, the works in causal strategic learning ([Miller et al., 2020]) suggest that the decision-maker can only consider causal features to ensure improvement.

Reviewer comment (Experiments):
It is somewhat confusing that the covariate distributions of the model-annotated samples and the human-annotated samples are different. The motivation for permitting the covariates of human-annotated samples to be sampled from the prior-best-response distribution is unclear. Given the dynamic nature of the setup, it seems more natural for both sets of samples to have the same covariate distribution (the post-best-response covariate distribution).

Author response:
We assume the *human-annotated* samples are drawn from *prior-best-response* distribution $P_{XY}$ because the human annotation process is independent of the decision-making process.  At each round $t$, each human agent is classified by the model $f_t$ and best responds to $f_{t-1}$, the decision-maker never confuses the agents by simultaneously using human experts to label agents. Instead, human experts never participate in the interaction and human annotation is another process for the decision-maker to obtain information about the whole population by acquiring data from public datasets or some third parties, and then label them to recover the population distribution. This distribution is free of best response because the samples are not subject to any decision. To better illustrate this idea, consider an example as follows. In a loan application scenario, the bank obtains personal finance data from credit agency and other demographic data from the government to produce *human-annotated* data, while they keep using an automated decision model to reject or approve loan application and update this model with the aid of *human-annotated* samples. We also agree with the interesting nature of considering the *human-annotation* for *post-best-response* distribution and our model can incorporate this setting. Details are discussed in the general response.

Reviewer comment (Writing):
It is not clear from the introduction how we should think about human-annotated samples. It would be helpful if the authors could clarify early on whether we view human-annotated samples to be reliable (the conditional distribution of the human-annotated samples is the same as \(P_{Y|X}\) from the true distribution) or we believe the human-annotated samples to be inaccurate (the conditional distribution of the human-annotated samples does not have the same \(P_{Y|X}\) as the true distribution). Later in the paper, it becomes clear that it is the latter. Furthermore, should we think of the human-annotated samples as being strategically supplied? Later in the paper, it seems like the human-annotated samples are not strategically supplied, because the covariates are drawn from the prior-best-response distribution and the labels are given by a systematically biased (but not necessarily strategic) decision maker. The confusion arises because the authors write that one of their motivating questions is – “how is the agent population reshaped over time when the model is retrained with strategic feedback?”

Author response:
The model can either let *human-annotated* samples be reliable or not, but it is likely that they are not accurate and have a systematic bias due to several reasons (App. B gives examples). However, all results in Sec. 3.1 - 3.2 hold no matter whether systematic bias exists. This is why we do not state it until Sec. 3.3. The systematic bias exists not because of strategic supply. It exists mainly because of the complexity of labeling a sample under human-related decision-making scenarios. Sec. 4 and App. B illustrates this in detail.

Reviewer comment (Theory):
There is interest in the setting where agents are strategic and the model is updated with only model-annotated samples. It is unclear whether a fixed point will arise [Frankel et al, 2022] and to what extent this setting is related to the repeated risk minimization in performative prediction [Brown et al, 2022, Perdomo et al, 2020].

Author response:
Our model can demonstrate this setting because the ratio between *model-annotated* and *human-annotated* samples is not constant. Specifically, Thm. 3.3, Prop. 3.4 will always hold when there are only *model-annotated* samples. Other results also need no modification.

Reviewer comment (Experiments):
It would be helpful to see a discussion of the different retraining strategies, e.g., (1) using the strategic agents’ post-best-response covariate distribution labeled with model to retrain, (2) using the strategic agents’ post-best-response covariate distribution with ground-truth (accurate) labels to retrain, (3) using the prior-best-response covariate distribution, (4) using i.i.d. data with ground-truth (accurate) labels to retrain.

Author response:
thanks for this helpful suggestion. Just as the answer of the previous question, the ratio between *model-annotated* and *human-annotated* samples should not be a problem. For the *prior-best-response* and *post-best-response* strategies, we discuss it in the general response.

Reviewer comment (Theory):
Assuming groups have equal base rates is entirely unrealistic.

Author response:
The assumption that groups have equal base rates is only used in the discussion of section 4 on fairness. In particular, we are concerned about how retraining with systematic bias under the strategic setting will influence fairness given the two groups are equally qualified and should be treated equally, so we make the equal base rate assumption. Furthermore, Thm 4.1 holds when group $i$ has a base rate larger than or equal to the one of group $j$ (not necessarily equal). For other situations under unequal base rates, we discuss it in detail in the general response to reviewers.

Reviewer comment (Novelty):
If the purpose is to study "long-term impacts of model retraining", the proposed solution for fairness can't be "stop retraining the model".

Author response:
The key of the section on fairness is to reveal the long-term fairness dynamics, where we discover that even when the systematic bias of human annotation always exists, simply retraining both groups for several rounds will lead to a fair classifier without any other explicit corrections. Instead of implementing the early stopping mechanism, we can consider a more common setting where a "hard fairness intervention" is performed at each round to only let the decision-maker choose from fair models. This is also discussed in detail in the general response.

Reviewer comment (Theory):
Assuming the decision-maker has some acceptance bias, and then relying on the same decision-maker to self-identify it and correct it, is not ideal.

Author response:
The key of the section on fairness is to reveal the long-term fairness dynamics, where we discover that even when the systematic bias of human annotation always exists, simply retraining both groups for several rounds will lead to a fair classifier without any other explicit corrections. Instead of implementing the early stopping mechanism, we can consider a more common setting where a "hard fairness intervention" is performed at each round to only let the decision-maker choose from fair models. This is also discussed in detail in the general response.

Reviewer comment (Experiments):
It would be extremely interesting to have explored how strategic feedback (either individually or in the form of collective action) could've helped with improving fairness (and ideally using different base rates as a source of unfairness, which is still simplistic but already a great deal more realistic than a bias in acceptance rates alone).

Author response:
The assumption that groups have equal base rates is only used in the discussion of section 4 on fairness. In particular, we are concerned about how retraining with systematic bias under the strategic setting will influence fairness given the two groups are equally qualified and should be treated equally, so we make the equal base rate assumption. Furthermore, Thm 4.1 holds when group $i$ has a base rate larger than or equal to the one of group $j$ (not necessarily equal). For other situations under unequal base rates, we discuss it in detail in the general response to reviewers.

Reviewer comment (Evaluation):
The reason why early stopping can improve DP is because the retraining process (without "refinement") simply monotonically increases acceptance rate, which just showcases the modelling flaws.

Author response:
What we hope to show in Sec. 4 is that although the acceptance rate increases as the retraining process goes on, it is possible to have positive impact on fairness if we apply the early stopping mechanism.

Reviewer comment (Experiments):
It's unclear to which degree the conclusions hold on any real-world scenario.

Author response:
The theoretical model needs us to know $P_{XY}, P_X$. This is why we need to estimate the distribution through the KDE estimator. The common datasets for strategic learning settings are limited in samples, so many previous works have a protocol to generate samples. Moreover, except for the uniform synthetic dataset, all other datasets violate some of the assumptions in the main paper, and the results still verify the validity of our theoretical results.

Reviewer comment (Experiments):
Of the two non-synthetic datasets ("German" and "credit approval"): (1) the experiments for "German" seem to have been ran on a synthetic version of the dataset as a result of fitting a KDE on the original data, and (2) the experiments on "credit approval" use only a small subset of features (two continuous features).

Author response:
The theoretical model needs us to know $P_{XY}, P_X$. This is why we need to estimate the distribution through the KDE estimator. The common datasets for strategic learning settings are limited in samples, so many previous works have a protocol to generate samples. Moreover, except for the uniform synthetic dataset, all other datasets violate some of the assumptions in the main paper, and the results still verify the validity of our theoretical results.

Reviewer comment (Theory):
As there are a series of simplifying assumptions that are unlikely to hold in real-world settings, it seems important to verify to which degree violating these assumptions changes the conclusions.

Author response:
The theoretical model needs us to know $P_{XY}, P_X$. This is why we need to estimate the distribution through the KDE estimator. The common datasets for strategic learning settings are limited in samples, so many previous works have a protocol to generate samples. Moreover, except for the uniform synthetic dataset, all other datasets violate some of the assumptions in the main paper, and the results still verify the validity of our theoretical results.

Reviewer comment (Evaluation):
Why does the qualification rate only change for the model-annotated samples under strategic feedback, and not for human-annotated samples?

Author response:
We assume the *human-annotated* samples are drawn from *prior-best-response* distribution $P_{XY}$ because the human annotation process is independent of the decision-making process.  At each round $t$, each human agent is classified by the model $f_t$ and best respond to $f_{t-1}$, the *decision-maker* never confuses the agents by simultaneously using human experts to label agents. Instead, human experts never participate in the interaction and human annotation is another process for the decision-maker to obtain information about the whole population by acquiring data from public dataset or some third-parties, and then label them to recover the population distribution. This distribution is free of best response because the samples are not subject to any decision. To better illustrate this idea, consider an example as follows. In a loan application scenario, the bank obtains personal finance data from credit agency and other demographic data from government to produce *human-annotated* data, while they keep using an automated decision model to reject or approve loan application and update this model with the aid of *human-annotated* samples. We also agree with the interesting nature of considering the *human-annotation* for *post-best-response* distribution and our model can incorporate this setting. Details are discussed in the general response.

Reviewer comment (Theory):
As the marginal feature distribution $P_X$ changes under strategic feedback, it would be expected that the qualification rate $Q(\mathcal{S}_{o,t-1})$ would also change in response, no? Or are the human-annotated samples at each time step not drawn from the same distribution? And, if so, why not, and how could the decision-maker distinguish in practice whether to use a model or a human annotator?

Author response:
We assume the *human-annotated* samples are drawn from *prior-best-response* distribution $P_{XY}$ because the human annotation process is independent of the decision-making process.  At each round $t$, each human agent is classified by the model $f_t$ and best respond to $f_{t-1}$, the *decision-maker* never confuses the agents by simultaneously using human experts to label agents. Instead, human experts never participate in the interaction and human annotation is another process for the decision-maker to obtain information about the whole population by acquiring data from public dataset or some third-parties, and then label them to recover the population distribution. This distribution is free of best response because the samples are not subject to any decision. To better illustrate this idea, consider an example as follows. In a loan application scenario, the bank obtains personal finance data from credit agency and other demographic data from government to produce *human-annotated* data, while they keep using an automated decision model to reject or approve loan application and update this model with the aid of *human-annotated* samples. We also agree with the interesting nature of considering the *human-annotation* for *post-best-response* distribution and our model can incorporate this setting. Details are discussed in the general response.

Reviewer comment (Theory):
It's assumed that human annotated samples are drawn from the prior-best-response distribution, but how would (1.) the decision-maker distinguish these distributions, and (2.) the agents know whether they were assigned to a model-annotator in order to know whether to employ strategic behavior.

Author response:
We assume the *human-annotated* samples are drawn from *prior-best-response* distribution $P_{XY}$ because the human annotation process is independent of the decision-making process.  At each round $t$, each human agent is classified by the model $f_t$ and best respond to $f_{t-1}$, the *decision-maker* never confuses the agents by simultaneously using human experts to label agents. Instead, human experts never participate in the interaction and human annotation is another process for the decision-maker to obtain information about the whole population by acquiring data from public dataset or some third-parties, and then label them to recover the population distribution. This distribution is free of best response because the samples are not subject to any decision. To better illustrate this idea, consider an example as follows. In a loan application scenario, the bank obtains personal finance data from credit agency and other demographic data from government to produce *human-annotated* data, while they keep using an automated decision model to reject or approve loan application and update this model with the aid of *human-annotated* samples. We also agree with the interesting nature of considering the *human-annotation* for *post-best-response* distribution and our model can incorporate this setting. Details are discussed in the general response.