Paper: The Human-AI Substitution game: active learning from a strategic labeler (ICLR-2024)

Reviewer comment (Theory):
The difference between an adversarial labeler and an abstaining labeler is not discussed. The difference between the concerning problem from active learning within an adversarial environment is not clear. Meanwhile, though it is mentioned that the labeler's strategy is identifiable, it is still unclear if it is the worst case.

Author response:
To clarify, in the learning game, we can *deduce* that the labeler will not be adversarial. Note that this is not by assumption, but rather by observation on which labeling strategies can be minimax optimal in the learning game.

To see this, if the labeler is adversarial, then this will lead to a payoff of zero. This is because adversarially generated data will not realize the learning outcome, and thus the learner will not pay the labeler (L67-70). Hence, while adversarial labeling is a valid strategy, it will not be minimax optimal in our learning game and will not be adopted by a labeler as an optimal labeling strategy. 

Finally, to your last point, the labeler’s minimax strategy will always be identifiable. Only these strategies ensure the learner’s learning outcome is met, and thus lead to a positive payoff in the learning game.

Reviewer comment (Experiments):
The motivation of the problem is too detailed, such that it can hardly be extended to other applications. It is recommended that real-world experiments should be conducted to validate the problem or it seems too artificial.

Author response:
We believe that our setup is fairly general, involving only a simple twist of the standard active learning setting (L44). It serves to address an inherent tension in any supervised learning setup: if the labeler allows the learner to learn fast (e.g. through active learning), this in turn reduces the final profit of the labeler.

Moreover, we believe our work is timely and well-motivated. Just in the recent months, we have seen (in the real-world) Hollywood screenwriters go on strike to negotiate a deal that prevents companies from training on their data and replacing them with AI [1]. Thus, we believe it is important to build our understanding of the replacement aspect of learning, which we set out to do in our paper.

[1] WGA Negotiations — Status as of May 1, 2023

Reviewer comment (Novelty):
The theoretical results are does not belong too much surprising information and has limited contribution to the algorithm design, since the algorithm design is mainly based on the labeler's strategy that is already pre-defined.

Author response:
We believe our learning algorithm is novel, since we prove that it is near-minimax optimal in the learning game. Contrary to your point, we do not pre-define the labeler’s strategy. Our learning algorithm is proven to be near-optimal with respect to *any* labeling strategy in the learning game.

Reviewer comment (Theory):
The first strong assumption is that **the learner is not allowed to query an example multiple times**. If the learner can query an example multiple times, then the labeler can first abstain from labeling for a while and then eventually give a correct label. It is very natural in practice for a learner to query an example multiple times until they get a label, but this is not allowed in the authors' model.

Author response:
```
(1) The learner is not allowed to query an example multiple times. If the learner can query an example multiple times, then the labeler can first abstaining from labeling for a while and then eventually give a correct label. It is very natural in practice for a learner to query an example multiple times until he gets a label.
```

**Deadlock:** We have considered the setting you propose, as a possible formulation. However, one issue is that some AL algorithms would deadlock. To see this, at time $t$, an algorithm would query the optimal point $x_t$. If the labeler declines to label *and* $x_t$ is not removed from the data pool, the algorithm would still query $x_t$ at $t+1$. This is because the state has not changed, $x_t$ would still be the best point to query, and eligible to be queried since it’s in the data pool. Thus, for the setting you suggest, there would have to be an apt way to modify AL algorithms and ensure the process does not hang. By contrast, in our setting, a data point is removed from the pool at each time step, thus ensuring progress.

**Labeler controlled queries:** If there is a way around the deadlock issue, we also thought that this setup may grant the labeler a lot of power. If the labeler can hold off on labeling a point indefinitely, the labeler can *dictate* the order the learner receives the labels. This is achieved by refusing to label any point but one (say $x’_t$ at time $t$). Thus, the learner’s choice in queries would not matter as labels can only be received in the order $x’_1$, …, $x’_n$, as decided by the labeler. We thought that this setup may differ substantially from the standard AL setting, and our goal is to study the problem closest to the canonical AL setup (L43-44).

Reviewer comment (Theory):
The second strong assumption, given that the learner can query each example only once, is the **"Guaranteeing Learning Outcome"** assumption (Section 1.1): the labeler must label in a way to guarantee that the learner can identify the hypothesis \(h^*\) in the end. However, this requires the labeler to know the hypothesis space \(\mathcal H\) of the learner very well so that they will not abstain from labeling critical examples (e.g., the \(x^*\) above). This is a strong assumption — how can an average human worker know the hypothesis space of a sophisticated machine learner?

Author response:
```
Guaranteeing Learning Outcome" (Section 1.1): the labeler must label in a way to guarantee that the learner can identify the hypothesis in the end. However, this requires the labeler to know the hypothesis space of the learner very well so that he will not abstain from labeling critical examples (e.g., the above). This is a strong assumption -- how can an average human worker knows the hypothesis space of the sophisticated machine learner?...I might change my opinions if the authors can provide some strong motivations or real-world scenarios to support these two assumptions.
```

**Analyzing minimax strategies:** For most of the paper, in the learning game, we think of the labeler as a “[resourceful] data labeling company” (L199). This is because in game theory, one needs to consider players who play optimally in order to study the Nash Equilibria/minimax query-complexity.

**Dealing with myopic labelers:** With that said, in subsection 4.1 and 4.3, we offer some remedies when facing a player that behaves sub-optimally, for instance due to lack of knowledge about $\mathcal{H}$ as you suggest (assume also the learner does not communicate $\mathcal{H}$ for some reason):

1. If the learner knows that the labeler may behave myopically, one idea is to loosen the “learning outcome” to approximate identifiability. Thus, the labeler abstaining on *one* critical example is not fatal ($x^*$ in your example). In subsection 4.1, we show how to extend our learning algorithms to the PAC learning setting.
2. In 4.3, we observe that the E-VS can be used to detect when an abstained point leads to non-identifiability. Thus, the learner can use this to send a *certified* “warning” to the labeler: if this critical point (e.g. $x^*$) is abstained upon, it will provably lead to non-identifiability. In this way, the learner can use the E-VS representation to prevent an unaware labeler from prematurely halting the learning process. Indeed, non-identifiability is something *neither* party wants. The learner wants to learn. The labeler needs to realize the learning outcome, in order to be paid. 

**Worker Unions:** In the specific case of AI automation, we have seen in recent news that Hollywood screenwriters have collectively gone on strike, engaging in negotiations with companies through their union (“Writers Guild of America'') [1]. Thus, in the AI automation setting, we are inclined to think that the labeler may in fact correspond to a union of workers, which may be more resourceful than a single worker.

[1] WGA Negotiations — Status as of May 1, 2023

Finally, as we write in the introduction (paragraph at L29), this paper studies a *general* conflict in incentives that can arise in *any* learner-labeler relationship. Active learning enables the learner to learn fast, but in doing so, leads to lower labeler profit. This calls into question a fundamental assumption in active learning: what is to say that the labeler wants the learner to learn fast? We note that the setting of the labeler as an “average human worker” is only one instantiation of this setup (L34).

Reviewer comment (Theory):
I might change my opinions if the authors can provide some strong motivations or real-world scenarios to support these two assumptions.

Author response:
```
Guaranteeing Learning Outcome" (Section 1.1): the labeler must label in a way to guarantee that the learner can identify the hypothesis in the end. However, this requires the labeler to know the hypothesis space of the learner very well so that he will not abstain from labeling critical examples (e.g., the above). This is a strong assumption -- how can an average human worker knows the hypothesis space of the sophisticated machine learner?...I might change my opinions if the authors can provide some strong motivations or real-world scenarios to support these two assumptions.
```

**Analyzing minimax strategies:** For most of the paper, in the learning game, we think of the labeler as a “[resourceful] data labeling company” (L199). This is because in game theory, one needs to consider players who play optimally in order to study the Nash Equilibria/minimax query-complexity.

**Dealing with myopic labelers:** With that said, in subsection 4.1 and 4.3, we offer some remedies when facing a player that behaves sub-optimally, for instance due to lack of knowledge about $\mathcal{H}$ as you suggest (assume also the learner does not communicate $\mathcal{H}$ for some reason):

1. If the learner knows that the labeler may behave myopically, one idea is to loosen the “learning outcome” to approximate identifiability. Thus, the labeler abstaining on *one* critical example is not fatal ($x^*$ in your example). In subsection 4.1, we show how to extend our learning algorithms to the PAC learning setting.
2. In 4.3, we observe that the E-VS can be used to detect when an abstained point leads to non-identifiability. Thus, the learner can use this to send a *certified* “warning” to the labeler: if this critical point (e.g. $x^*$) is abstained upon, it will provably lead to non-identifiability. In this way, the learner can use the E-VS representation to prevent an unaware labeler from prematurely halting the learning process. Indeed, non-identifiability is something *neither* party wants. The learner wants to learn. The labeler needs to realize the learning outcome, in order to be paid. 

**Worker Unions:** In the specific case of AI automation, we have seen in recent news that Hollywood screenwriters have collectively gone on strike, engaging in negotiations with companies through their union (“Writers Guild of America'') [1]. Thus, in the AI automation setting, we are inclined to think that the labeler may in fact correspond to a union of workers, which may be more resourceful than a single worker.

[1] WGA Negotiations — Status as of May 1, 2023

Finally, as we write in the introduction (paragraph at L29), this paper studies a *general* conflict in incentives that can arise in *any* learner-labeler relationship. Active learning enables the learner to learn fast, but in doing so, leads to lower labeler profit. This calls into question a fundamental assumption in active learning: what is to say that the labeler wants the learner to learn fast? We note that the setting of the labeler as an “average human worker” is only one instantiation of this setup (L34).

Reviewer comment (Theory):
Question — What is the purpose of Section 3.1 (in particular Algorithm 3 and Proposition 3.7)? My rough understanding is that Section 3.1 wants to show that finding the maximal E-VS bisection point \(x\) (line 3 in Algorithm 2) can be done efficiently by Algorithm 3 (correct me if I am wrong). If this is the case, then what is the time-complexity of Algorithm 3? How does Proposition 3.7 show that finding the maximal E-VS bisection point can be done efficiently?

Author response:
```
What is the purpose of Section 3.1 (in particular Algorithm 3 and Proposition 3.7) ? … If this is the case, then what is the time-complexity of Algorithm 3? How does Proposition 3.7 shows that finding the maximal E-VS bisection point can be done efficiently?
```

1. The purpose of section 3.1 is to develop a way to access the E-VS. We start by noting that the canonical approach of sampling to access the VS (L153-4) will not work for E-VS (L158-9). We prove that in the linear case, it would be akin to sampling from an exponential number of polytopes (Prop 3.5). This motivates finding a new way to access the E-VS, and led us to develop Algorithm 3. 
2. As you may know, the (E-)VS can contain an exponential number of hypotheses. Thus, prior AL works typically make use of oracles to access the VS. Just as in the VS-bisection algorithm, we assume access to a sampling oracle $O$. Now, to overcome the issue in point 1, we observe that the E-VS is a subset of the VS. Hence, it suffices to implement a membership oracle for whether $h \in V$ belongs to E-VS. We prove in Prop 3.7 that this membership check can be implemented by one carefully setup call to the C-ERM oracle.
3. Lastly, indeed less is known about computational efficiency in AL algorithms, even for the original VS bisection algorithm. This is because this differs depending on the instance. Instead, the main focus in AL has been on statistical efficiency, which is also our focus in the paper. We agree with you that exploring the computational efficiency of AL algorithms in general is an important future direction.

Reviewer comment (Theory):
Question — What is the purpose of Section 3.9? The authors say this section is to compare with EPI-CAL, but only give a formal result for EPI-CAL (\(\Omega(\sqrt{m})\) samples in Proposition 3.9) and do not give any formal result for the algorithm they proposed for comparison.

Author response:
```
(Q2) What is the purpose of Section 3.9? The authors say that this section is to compare with EPI-CAL, but only give a formal result for EPI-CAL ( samples in Proposition 3.9) and didn't give any formal result for the algorithm they proposed for comparison.
```

Thanks for this note, we will be sure to clarify! As we write, our setup and that of EPI-CAL (section 3.3) are different. Thus, in this subsection, we were mainly interested in understanding whether strategic abstention can also significantly enlarge query complexity for EPI-CAL. We answer this in the affirmative. We will be sure to make this clear in the revision, perhaps moving this subsection to the appendix.

Reviewer comment (Presentation):
The presentation of Section 5.1 and 5.2 is really confusing. Section 5.1 is about the upper bound on the multi-task query complexity. Theorem 5.3 gives an upper bound under a regularity assumption and for the \(c_{all}\) label cost. Proposition 5.1 is a counterexample where the upper bound doesn’t hold without the regularity assumption, but the authors don't say which label cost this Proposition is for. Proposition 5.2 is an example where the upper bound in Theorem 5.3 doesn't hold for the \(c_{one}\) label function. Section 5.2 is about the lower bound (for the \(c_{one}\) cost). Theorem 5.6 gives a lower bound on the multi-task query complexity under two conditions for the \(c_{one}\) cost, while Propositions 5.4 and 5.5 are counterexamples where the lower bound does not hold if the two conditions are not met.

Author response:
```
…Proposition 5.1 is a counterexample where the upper bound doesn't hold without the regularity assumption, but the authors didn't say which label cost this Proposition is for…
```

Thank you for your helpful suggestion on the reorganization! We will modify the section accordingly. Just to answer your question, Prop 5.1 in fact holds under both costs.

Reviewer comment (Writing):
Clarity: the notation is convoluted; $V$ is used for both VS and a hypothesis space variable. For instance, in table 1, what is $V$? it seems introducing a specific notation for VS (e.g., $\mathbf{V}$) or for the hypothesis space variable (e.g., $\mathcal{V}$) could help.

Author response:
Thank you for this formatting suggestion! We will be sure to clarify the difference and adapt the presentation accordingly. To clarify, we meant to write $H[S]$ for the version space in Table 1.

We hope this helped clarify most of the questions. We found your feedback very useful in revising the manuscript, thank you! Please let us know if there are any further questions.

Reviewer comment (Experiments):
There is no experimental reported. It could help clarify many abstract concepts introduced in the paper using even synthetic data. Examples like Table 1 could help practitioners employ the algorithms introduced in the paper in their applications.

Author response:
Hi Reviewer ciAE, with the discussion period nearing the end, we hope you could please consider getting back to us soon regarding our response above and update. In particular, we have added some toy experiments in Appendix A on randomly sampled toy instances similar to (bigger versions) of the tabular setting as in Table 1 (following your suggestion in point 2).

Please let us know what you think. We would very much appreciate a chance to compose an update to address concerns if there are any, before the end of the discussion period. Thank you again for taking the time!

Reviewer comment (Writing):
Please consider separating the notations of $V$ (see 1. in the weaknesses section).

Author response:
Thank you for this formatting suggestion! We will be sure to clarify the difference and adapt the presentation accordingly. To clarify, we meant to write $H[S]$ for the version space in Table 1.

We hope this helped clarify most of the questions. We found your feedback very useful in revising the manuscript, thank you! Please let us know if there are any further questions.

Reviewer comment (Experiments):
The biggest concern that might weaken this paper is its lack of empirical evidence. Despite the theoretical guarantees, some of the settings are probabilistic (PAC or noisy observations for instance), making empirical evaluation of the proposed methods valuable. Including some simple simulation studies could be beneficial, even considering the page limit.

Author response:
Thank you for getting back to us and your continued interest in our paper, we appreciate it! As you know, our paper is intended to be a theoretical paper, and we believe our paper is well-motivated especially in light of recent events. Just in the past few months, we have seen Hollywood screenwriters go on strike to negotiate a deal preventing companies from training on their data and replacing their expertise with AI [1]. Thus, we believe it is high time to build our understanding of the replacement aspect of learning, which is exactly what we set out to do in our paper.

The experiments, which we started from scratch, finished today. Please see Appendix A for experiments on randomly sampled toy instances similar to (bigger versions) of the tabular setting of Table 1 (as suggested by reviewer ciAE). We have colored our edits to the original submission in red.

[1] In Hollywood writers' battle against AI, humans win (for now) (AP News)

Reviewer comment (Theory):
This paper assumes "the learner’s strategy corresponds to some deterministic, querying algorithm". Nevertheless, non-deterministic policies such as Thompson sampling are quite common in some active learning tasks such as Bayesian optimization. How do these non-deterministic policies fit into the framework of this paper?

Author response:
Thank you for your review, Reviewer rYHN! Please see our responses to your questions below:

```
I only have a minor question. This paper assumes "the learner’s strategy corresponds to some deterministic, querying algorithm". Nevertheless, non-deterministic policies such as Thompson sampling is quite common in some active learning tasks such as Bayesian optimization. How does these non-deterministic policies fit into the framework of this paper?
```

To clarify, Bayesian optimization/Thompson sampling is used for finding a maxima $x \in X$ of some unknown $h$. There, the goal is to adaptively query in order to shrink the uncertainty over $h$, to find one of its maxima with low (simple) regret. 

Our setting is quite different in that we want to learn the unknown hypothesis $h$, instead of optimizing $h$. Moreover, our paper studies minimax learning strategies in the learning *game*, in face of a strategic labeler who may be playing an optimal labeling strategy. This game theoretic setup is another major difference that complicates the analysis and differs from the setup of Bayesian Optimization.

We hope this helped clarify most of the questions. We found your feedback very useful in revising the manuscript, thank you! Please let us know if there are any further questions.