Paper: Estimating Fréchet bounds for validating programmatic weak supervision (ICLR-2024)

Reviewer comment (Evaluation):
I’m a bit confused as to what is the underlying message of Section 4.3. It seems to me that the proposed bounds perform poorly in the case with a few LLM generated weak labelers. I currently am understanding the “label set selection” approach to be picking weak labelers based on “accuracy”, except where accuracy is measured by disagreement with the estimated weak labels (i.e., label model outputs). As such, this would simply be pruning away weak labelers that disagree with the labelmodel, which seems somewhat undesirable.

Author response:
**Comment about label set selection:** In Figure 3, accuracy is **not** related to the alignment of weak labels and the label model. In fact, accuracy here is related to the performance of the weakly supervised classifier. For each one of the three plots, we have different sets of weak labels, and for each one of these sets, we estimate a different label model and use that label model to estimate the performance bounds. It is clear from the second and third plots that the third set of weak labels is better than the second because the red curve in the third plot is above the blue line in the second. In other words, the worst-case scenario using the third set of weak labels is better than the best-case scenario using the second set of weak labels. In this example, we are basically doing model selection. Please check Section A.4 for more details about model selection using our performance bounds.

Reviewer comment (Novelty):
There seems to be a lack of discussion about existing work on adversarial methods in PWS (namely [1] and subsequent followup work). I realize this existing work focuses on the classifier error rate, although the relaxed convex program approach seems similar.

Author response:
**Related work:** Thank you for your suggestions for new references. Adversarial learning is indeed related to our work in some ways (but not a substitute). We will update the "Related work" section and incorporate your suggested references in the final version of this paper.

Reviewer comment (Theory):
Primarily my first point in the weaknesses section above: is this assumption too strong? Given imperfect weak labelers, we cannot correctly estimate the true label (with a labelmodel), and, thus, wouldn’t these bounds would be violated in many cases? Any justification about the strength of these assumptions (i.e., allowing to swap in label model estimates with true labels) would be much appreciated.

Author response:
**On the quality of weak labels:** The quality of weak labels is usually **not** a problem for the label model estimation. According to [2], "a sufficient condition (for the identification of the label model) is the standard one of assuming non-adversarial sources, i.e., that all sources have greater (accuracy) than random accuracy". Consequently, our bounds should be valid asymptotically if all weak labelers have non-adversarial accuracy (weak assumption) and the label model is correctly specified. However, the quality of the weak labels will influence how good the weakly supervised classifier is and the size of the gap between the bounds $L$ and $U$ (see "When are the Fréchet bounds informative?" in the general comments above).

Reviewer comment (Theory):
The estimator in equation (2.4) heavily relies on the distribution P_{Y\mid Z} which is not observed. The paper suggested using a label model to approximate such distribution. This requires the label model to be “good”. However, in this case, we can just use it to evaluate classifiers directly without deriving the Frechet bound.

Author response:
**On using the label model solely to evaluate classifiers:**  As we discuss in the second paragraph of our "Introduction" section (specifically, in the last eight lines of page 1), the classifier cannot be evaluated even when the label model $P_{Y \mid Z}$ is perfectly known. This is because the joint distribution $P_{X, Y}$ is required to evaluate a classifier, and $P_{X, Y}$ remains unspecified even if $P_{X, Z}$ and $P_{Y \mid Z}$ are known. Due to such an underspecification, we are required to derive the Frechet bounds to provide upper and lower bounds for the (underspecified) classifier performance.

Reviewer comment (Experiments):
The experiments should test on multiple types of end models (beyond logistic regression) and label models (beyond just Snorkel).

Author response:
**Inclusion of a new label model, different classifiers, and multinomial classification datasets:** Thank you for your suggestions! We have taken care of this point. Please see the full response to this issue above in the general comments.

Reviewer comment (Experiments):
The experiments should test on datasets beyond binary classification.

Author response:
**Inclusion of a new label model, different classifiers, and multinomial classification datasets:** Thank you for your suggestions! We have taken care of this point. Please see the full response to this issue above in the general comments.

Reviewer comment (Evaluation):
The experiments should measure precision/recall on the unbalanced datasets such as *Tennis* and *Census*, where F1 (and not accuracy) is the default metric in WRENCH.

Author response:
**Use of F1 score:** Thanks for pointing out that accuracy might not be suited for some datasets. We have updated the paper using the F1 score when needed.

Reviewer comment (Experiments):
The results would benefit from more directly showcasing the usefulness of the bounds for hyperparameter tuning/model selection, which may be a more interesting use case of the bounds. This should include testing whether the bounds can be used to *relatively* rank different models more extensively, defining explicit selection rules, and comparing their effectiveness for selecting all hyperparameters (and for more complex weak supervision methods).

Author response:
**Model selection procedure/experiment:** Thank you for suggesting this new idea. We have taken care of this point. Please see the full response to this issue above in the general comments.

Reviewer comment (Theory):
Since in practice PWS users are likely to have access to some (albeit limited) labeled data, it would be interesting to see if and how the bounds can be sharpened when incorporating such labeled data.

Author response:
**Sharpening bounds using labeled data:** We could possibly include new constraints on the optimization problems we solve to compute $L$ and $U$ based on the observed labeled data. However, that would drastically change our method.

Reviewer comment (Writing):
While I think that the paper is overall well-written, and it is easy to understand its contribution, I think that the presentation is lacking in some parts. In my opinion, the paper provides no intuition behind the Frechet bounds and what they represent, and what information they are providing in the weak supervision setting. In my interpretation, we assume to have access to (or an estimate of)  $P(Y|Z)$, $P(Y)$, and $P(X,Z)$. Let's say that as a function $g$, we use the accuracy $g(X,Y,Z)$ = indicator$( h(X) = Y)$. From the definition, Frechet bounds look at the "worst-case" and "best-case" joint distribution $X \times Y \times Z$ with respect to the value of the expectation of the function g. In this case, the information on the label $Y$ is given by $Z$. From my intuition (possibly wrong), the worst-case (lower-bound) is given by trying to get as much as possible $g(X) = 0$ (we make a mistake) whenever $h(X)$ is different from the most likely value of $Y$ given by  $P(Y|Z)$. Analogous intuition for the upper bound. In particular, it looks like the gap between the lower bound and upper bound depend on "how much" the output of the classifier $h(X)$ differs from $P(Y|Z)$ given a sample $(X,Z)$. If this interpretation is correct, then it looks like that the lower bound and upper bounds are informative (i.e., close to one another) only if $h(X)$ follows the output of the labels $P(Y|Z)$, in which case it is not clear what is the advantage of training a classifier h.

Author response:
**Interpretation of the Fréchet bounds:** A more direct intuition is seeing Fréchet bounds using ideas from optimal transport (more details in Section A.1). In summary, the Fréchet bounds find the worst and best cases for the distributions $(X, Y) \mid Z = z$ with respect to the expected value of $g$ for each value of $z$. More explicitly speaking, for each $z$, the objective for the Fréchet bound finds the maximum and minimum values for $E_{P_{X, Y\mid Z = z}}[g(X, Y, z)]$ when $P_{X \mid Z = z}$ and $P_{Y \mid Z = z}$ are specified. That is, for each $z$, we solve an optimal transport problem. Consequently, if $g(x,y,z)=\mathbf{1}[h(x)=y]$ and for a fixed $z$, we transport the mass from $P_{X \mid Z = z}$ to $P_{Y \mid Z = z}$ maximizing (resp. minimizing) classification errors in order to compute the lower (resp. upper) bound for accuracy.

Furthermore, we have added a section (Section A.2 in our updated draft) that quantifies the gap between the upper and lower Frechet bounds, where we establish
$ (U - L)^2 = O(\min \\{ H(X \mid Z), H(Y \mid Z) \\})$, where $H(\cdot \mid Z)$ is the conditional entropy given $Z$. Please see the discussion above in the general comments.  

We will include some of this discussion in the main text once the paper is fully reorganized.

Reviewer comment (Theory):
I think that the assumption that we can estimate $P_{X,Z}$ is too strong and not necessary (and not really used). If $X$ is highly dimensional (i.e. images), it is impossible to estimate this distribution within any reasonable finite sample size. I do not think that this assumption is actually necessary for the scope of this paper, in fact all the applications of the results could be formulated substituting $X$ with $h(X)$, in which case $h(X)$ has the same support than the label space Y. In fact, for precision recall etc, we only need to consider a function $g(Y',Y,Z)$ where $Y'$ is the output of the classifier that is being evaluated. I find it surprising that the results do not add any assumption on estimating the distribution $(X,Z)$ as this is a very hard problem in general. I think this is due to the fact that we never actually estimate the distribution itself, but we simply evaluate the function g that is bounded. This is a bit misleading with the introduction, where it is written that we use an estimate of $P(X,Z)$.

Author response:
**On the estimation of $P\_{X,Z}$:** We appreciate your insight that in practice we care only about $h(X)$ and not $X$ in most cases. However, our method does not require estimating $P_{X,Z}$ in the sense of density estimation. In our Eq. (2.5), we only use samples $\{(X\_i,Z\_i)\}$ to approximate $P\_{X,Z}$ using the empirical distribution $\frac{1}{n}\sum\_{i=1}^n \delta\_{\{(X_i,Z_i)\}}$ ($\delta$ is the Dirac measure) and, therefore, approximating expectations using empirical means.

Reviewer comment (Theory):
I think that the Appendix section with the proof of the results is not well written nor easy to follow. I believe that the paper would greatly benefit from having a more organized proof structure, so that an interested reader can learn about the techniques used. As an example, results from other papers are often used (e.g., Beiglbock & Schachermayer for proof of Theorem 2.1). It takes a lot of effort by the reader to understand the applications of those results, as these cited theorem use some assumptions, have different notation, and it is not immediately straightforward how they are translated to this paper setting. For readability, it would be easier to say something along the lines "since [... these assumptions hold ...] then we can apply this result from [citation] that says: ". Another example is the Proof of Theorem 2.4 (e.g., sequence of equations of page 16, or the sequence of Theorem and Lemmas). It would be easier if the proof is organized in a way such that it is easy to follow and understand the different parts of the proof.

Author response:
**Reorganizing the proofs:** Thank you very much for your observation. We will reorganize the proofs in the next version of the paper to make them easier to understand.

Reviewer comment (Novelty):
There are other papers that used somehow similar ideas to consider a "worst-case" (or adversarial) distribution over all possible distributions in a weak supervision setting that I think are very relevant (e.g., [A,B,C,D,E,F]). In particular, this problem was studied mostly for the case of trying to solve the map $Z \to Y$ from weak supervision sources outputs to label without adding any assumption (e.g., independence or parametric assumption) [A,B,E], although some other work consider the more general case of training a classifier h [C,D,F]. The use of the "worst-case" allows to obtain an upper bound (analogous to the U of the Frechet Bound). As an example, the underlying idea is to consider possible joint distribution $Z,Y$ that satisfy some information on the marginal $P(Y|z_i)$ or $P(Z)$ (this information may vary among the work). Some of this work may require to have some labeled data to get this marginal information. Some information about the label is required in general. In some work, they assume to have access to some labeled data to estimate some statistical constraints. On the other hand, the result of this paper removes this assumption, but assumes that you can get information of the labels from the label model $P(Y|Z)$ (however this model could be misspecified).

Author response:
**Related work:** Thank you for your suggestions for new references. Adversarial learning is indeed related to our work in some ways (but not a substitute). We will update the "Related work" section and incorporate your suggested references in the final version of this paper.

Dear reviewer,

In our last paper update, we included, in the main text, the adversarial learning references (Related work) and a discussion about the new theoretical results (Section 2.4).

Thank you!