Paper: RACH-Space: Reconstructing Adaptive Convex Hull Space with applications in weak supervision (ICLR-2024)

Reviewer comment (Presentation):
Overall, the presentation of the paper needs essential improvement in the problem-setting explanation, including the introduction section.

Author response:
We appreciate the feedback from the reviewer. Based on the feedback, we have now re-written the introduction. In the revised version, we state the problem in a self-contained manner. Further, the link between $\mathbf{W}$ and $\mathbf{y}$ is now shown in equations 1.1 and 1.2. We also now discuss the main idea of *RACH-Space* in the introduction and highlight the contributions.

Reviewer comment (Writing):
The definition of "weakly supervised learning" in this paper is not clear, which makes it challenging to understand the motivation of the paper and which applications it might have. The authors referred to some papers in the first and second paragraphs in the introduction section (Shin et al., 2015), (Mintz et al., 2009), (Chen & Batmanghelich, 2020), and (Karamanolakis et al., 2021). Also, the authors mentioned (Ratner et al., 2016). Although those papers use the phrase "weakly supervised learning," the specific problem settings that they are tackling are different from the authors' formulation in Section 1.1. Hence, the authors are more responsible than usual to state the problem setting in a self-contained and rigorous manner, so that readers can understand which symbols correspond to which data in real applications and what types of assumptions are imposed.

Author response:
We appreciate the feedback from the reviewer. Based on the feedback, we have now re-written the introduction. In the revised version, we state the problem in a self-contained manner. Further, the link between $\mathbf{W}$ and $\mathbf{y}$ is now shown in equations 1.1 and 1.2. We also now discuss the main idea of *RACH-Space* in the introduction and highlight the contributions.

Reviewer comment (Theory):
The authors do not mention the link between $\mathbf{W}$ and $\mathbf{y}$. If there is no clear link between $\mathbf{W}$ and $\mathbf{y}$, we do not need to call the problem setting "weakly supervised learning" since $\mathbf{W}$ is no longer a supervision.

Author response:
We appreciate the feedback from the reviewer. Based on the feedback, we have now re-written the introduction. In the revised version, we state the problem in a self-contained manner. Further, the link between $\mathbf{W}$ and $\mathbf{y}$ is now shown in equations 1.1 and 1.2. We also now discuss the main idea of *RACH-Space* in the introduction and highlight the contributions.

Reviewer comment (Writing):
As above, since the problem setting itself is unclear, readers cannot understand the importance of the area and this paper or discuss whether the proposed method is reasonable or not. I encourage the authors to make the assumptions and motivations clearer, ideally in the introduction section.

Author response:
We appreciate the feedback from the reviewer. Based on the feedback, we have now re-written the introduction. In the revised version, we state the problem in a self-contained manner. Further, the link between $\mathbf{W}$ and $\mathbf{y}$ is now shown in equations 1.1 and 1.2. We also now discuss the main idea of *RACH-Space* in the introduction and highlight the contributions.

Reviewer comment (Writing):
The RACH-space's main idea is not written in the Introduction section, which significantly decreases the readability. I strongly encourage the authors to explain in the introduction section what is the main idea of the proposed method and why the idea solves the issues of existing methods.

Author response:
We appreciate the feedback from the reviewer. Based on the feedback, we have now re-written the introduction. In the revised version, we state the problem in a self-contained manner. Further, the link between $\mathbf{W}$ and $\mathbf{y}$ is now shown in equations 1.1 and 1.2. We also now discuss the main idea of *RACH-Space* in the introduction and highlight the contributions.

Reviewer comment (Writing):
The definition of the *Majority Voting* is unclear.

Author response:
In the revised version, we have included a more detailed definition of *Majority Voting*. We did use *Majority Voting* in our numerical experiments. The first method in Table 3 is *Majority Voting*.

Reviewer comment (Writing):
In which application cases, your problem setting in Section 1.1 is applicable? Could you provide specific examples? Specifically, in which case do we have multiple weak supervising signals?

Author response:
>Q1. In which application cases, your problem setting in Section 1.1 is applicable? Could you provide specific examples? Specifically, in which case do we have multiple weak supervising signals?

There are numerous real world scenarios where the problem setting in Section 1.1 is applicable, and all of the $14$ real world datasets are from the public weak supervision benchmark WRENCH. For example, each weak signal can come from different sources of information, or/and be obtained at different timings and environments. Denoted by *#LFs* in Table 2 of our paper, real world scenarios involve different numbers of weak signals.

Reviewer comment (Theory):
The expected error is defined as $\mathbf{E} [\mathbf{w}_\mathbf{i} - \mathbf{y}]$. I assume $\mathbf{E}$ indicates the expectation operator, but what is the random variable of which we consider the expectation here? Also, you are interested in the difference between $\mathbf{w}_\mathbf{i}$ and $\mathbf{y}$. Does it mean that you assume the value of $\mathbf{w}_\mathbf{i}$ is close to $\mathbf{y}$? I do not see such assumptions in the paper.

Author response:
>Q2. The expected error is defined as $\mathbf{E[w_i-y]}$. I assume $\mathbf{E}$ indicates the expectation operator, but what is the random variable of which we consider the expectation here? Also, you are interested in the difference between $\mathbf{w_i}$ and $\mathbf{y}$. Does it mean that you assume the value of $\mathbf{w_i}$ is close to $\mathbf{y}$? I do not see such assumptions in the paper.

We appreciate the comment. This was part of our previous version that caused some confusion about the expected empirical error rate, so we modified the draft to clarify the definition of the expected empirical error rate. Each row of $\mathbf{W}$ represents a weak signal that gives probabilities for data points being in a certain class. So yes, the rows of $\mathbf{W}$, $\mathbf{w_i}$, are weak signals, which by definition are incomplete and noisy representations of the label vector $\mathbf{y}$.

Reviewer comment (Theory):
Related to that, you mentioned "Note, Majority Voting and RACH-Space assume that, on average, the weak labels are better than random." Here, what does *better* mean? Could you mathematically define it?

Author response:
>Q4. Related to that, you mentioned "Note, Majority Voting and RACH-Space assume that, on average, the weak labels are better than random." Here, what does better mean? Could you mathematically define it?

If one uses random labeling i.e., predict the label of a data point using a uniform probability of $\frac{1}{k}$, the expected empirical error rate is $\frac{2}{k}-\frac{2}{k^2}$. Better in this context means that the average expected empirical error rate of a weak signal is lower than $\frac{2}{k}-\frac{2}{k^2}$. In the revised version, we clarify that we do not assume prior knowledge nor a set bound for the average empirical error rates of weak signals. Instead, we update our parameter, now called pseudo-error rate, starting from $\frac{2}{k}-\frac{2}{k^2}$, which is the error rate of the weak signal that simply does random classification of the data points.

Reviewer comment (Experiments):
You mentioned many times *Majority Voting* but you do not define *Majority Voting*. What is that? Also, why do you not use *Majority Voting*? Why don't you compare your proposed method with the *Majority Voting*?

Author response:
>Q6. You mentioned many times *Majority Voting* but you do not define *Majority Voting*. What is that? Also, why do you not use *Majority Voting*? Why don't you compare your proposed method with the *Majority Voting*?

In the revised version, we have included a more detailed definition of *Majority Voting*. We did use *Majority Voting* in our numerical experiments. The first method in Table 3 is *Majority Voting*.

Reviewer comment (Writing):
The background is not very clear, especially for the weak signal matrix $\mathbf{W}$. For example, where does the matrix $\mathbf{W}$ come from?

Author response:
The matrix $\mathbf{W}$ is composed from the $m$ weak signals. Specifically, the rows of $\mathbf{W}$ are the weak signals. The $m$ weak signals are provided as an input to the problem. Based on the feedback, we have now modified the introduction so that it is clear where $\mathbf{W}$ comes from.

Reviewer comment (Writing):
Some preliminaries are missing. Why the expected error rate $\epsilon_i$ is equal to $\mathbb{E}[\mathbf{w}_i - \mathbf{y}]$? Does each row of $\mathbf{W}$ kind of represent the label vector $\mathbf{y}$?

Author response:
We appreciate the comment. This was a typo and has been corrected in the revised version. Each row of $\mathbf{W}$ represents a weak signal that gives probabilities for data points being in a certain class. So yes, each row of $\mathbf{W}$ are weak signals, which are incomplete and noisy representations of the label vector $\mathbf{y}$.

Reviewer comment (Theory):
As stated in this paper, the computation of $\mathcal{H}_1$ takes time $O((nk)^{\lfloor \frac{m}{2} \rfloor})$ which is prohibitive in practice. The author divided $\mathbf{W}$ into several parts and took their average to reduce the value of $m$. Although the author claimed that this does not make a negative impact on the performance, it cannot convince me since no persuasive proof is provided.

Author response:
We agree that the performance of the method depends on the aggregation method. The computation of $\mathcal{H}_1$ is feasible when $m \leq 7$. We ran our experiments to have $m = 5$, and the performance is similar when $m$ is between $2$ to $7$. In the current work, we have not undertaken a theoretical analysis on the performance guarantees regarding the reduction of $m$. This is within the scope of future work for *RACH-Space*.

Reviewer comment (Writing):
It’s very difficult to follow the paragraphs around equation 3.4 and 3.5. In the current form, some choices seem to be arbitrary. One example is the definition A=2W — why not A=3W? There may be principled reasons for this choice, but they are unclear from the current writing. As mentioned in the abstract, there is a geometrical interpretation. It might be helpful to introduce this geometrical interpretation first to give more intuition about the method before diving into the math.

Author response:
In the revised version, we clarified the writing around the mentioned equations. The matrix $A$ is defined as $A=2W$. This is obtained by rearranging equation (1.2) which leads to the linear system $(2W) y=b$. Hence, the choice of $A$ being $2W$ is not arbitrary, and directly follows from the linear system. We have now added a figure and additional contexts to communicate clearly the geometrical interpretation of *RACH-Space*.

Reviewer comment (Theory):
It seems the method heavily depends on “average expected error rate”. How can one know average expected error rate without assuming the distribution of the error rate? How is average expected error rate defined in the paper?

Author response:
The average expected error rate is the average of the empirical expected error rates of the weak signals. Formally, $\epsilon = \frac{1}{m}\sum_{i=1}^{m} \epsilon_i$. In the revised version, we clarify that our formulation does not depend on knowing the average expected error rate. There was some confusion about the assumptions of *RACH-Space* in regards to the expected empirical error rates. In the revised version, we clarify that we do not assume prior knowledge nor a set bound for the average empirical error rates of weak signals. Instead, we update our parameter, now called pseudo-error rate, starting from $\frac{2}{k}-\frac{2}{k^2}$, which is the error rate of the weak signal that simply does random classification of the data points. *RACH-Space* does not require the average empirical error rate to take a certain value, and works independently of the average empirical error rate, which is unknown.

Reviewer comment (Experiments):
The WRENCH dataset has 14 classification datasets; why were four of the datasets dropped in experiments? It is best to include all 14 datasets in order to avoid any possibility of cherry-picking.

Author response:
We included all 14 datasets in our updated version, and *RACH-Space* is still the best performing model compared to all existing label models. See the updated Table 3.

Reviewer comment (Experiments):
It is mentioned that “experiments are done in the original setup on WRENCH”, but if that’s the case, how come the results are different from those reported in the WRENCH paper? In the WRENCH paper, MeTaL had better results than MV. Is the difference caused by dropping the 4 datasets?

Author response:
We note that the experiments are done in the original setup on WRENCH based on the official WRENCH code from the GitHub repository. For some of the label models, our experiments yielded different results than the results on the WRENCH paper. The results we present are obtained from running the experiments on our machine. MeTaL and MV’s overall performance was not changed when the experiment was done on 10 datasets as opposed to 14 datasets.

Reviewer comment (Theory):
The paper claims it makes minimum assumptions as MV, i.e., only assuming weak labels are better than random. However, the proposed method actually implicitly makes more assumptions. For example, it selects the average expected error rate so that b/n is in a safe region. This is already a strong assumption that one gets to select “average expected error rate”. As the average expected error rate is expected to be a single point value. Another assumption roots in the least squared problem formulation — why not formulate it as a least absolute deviation problem? What implicit assumptions are being made to formulate the problem as a least squared problem instead of least absolute deviation?

Author response:
In the revised version, we clarify that our approach is to modify $\widetilde{b}$ without requiring explicit knowledge of the average expected error rate. To make this clear, we introduce a new parameter, the pseudo-error rate. The vector $\widetilde{b}$ is initially defined using the pseudo-error rate and is subsequently updated so that it lies in the safe region. Our updates to $\widetilde{b}$ are completely algorithmic and do not rely on any concrete values of expected empirical error rates of weak signals. We made the choice of the least squares problem, since we can optimize it using standard gradient descent. We have not explored the least absolute deviation and would be happy to compare the performance of the two algorithms in terms of classification accuracy.

Reviewer comment (Theory):
The contributions could be explicitly stated. It is unclear what is proved in the theoretical section.

Author response:
In the revised version, the introduction now discusses the 5 contributions of the paper including the theoretical contributions.

Reviewer comment (Novelty):
Although I am not an expert in the field, I doubt the RACH-Space is particularly ground-breaking. More serious evidence on real world datasets should be added. For instance, you should test the RACH-Space algorithm on a histopathology dataset where it is common to face weakly-supervised learning problems.

Author response:
All the datasets in our numerical experiments are on real world datasets. In particular, these 14 datasets are part of the WRENCH framework used as a public benchmark for weakly supervised learning on real world data.

Reviewer comment (Writing):
The paper's writing quality can be significantly improved. I have identified numerous grammatical errors and vague phrases that should be addressed to enhance the paper's overall readability. Additionally, the literature review section and the section explaining the motivations behind this work lack informativeness for similar reasons.

Author response:
In the revised version of the paper, we have corrected grammatical errors and clarified phrases to improve readability. Additionally, we have rewritten the introduction to more clearly present the paper's motivation and contributions.