Paper: Set Learning for Accurate and Calibrated Models (ICLR-2024)

Reviewer comment (Experiments):
The authors set the maximum number of randomly sampled sets to the total number of training data points, such that the gradients updates remain the same. However, does this effectively make the proposed method sees k multiple more data points compared to the standard setting? It would be interesting to see how the experiment results look like when two algorithms are of the sample complexity.

Author response:
1.) We agree in principle that this would be interesting, and when performing the experiments in the paper we did indeed account for this: To determine the maximal number of update steps to use in our experiments, we performed an extensive grid search using the *vanilla* baseline. We then used that number for all experiments in our paper. We additionally used an *early stopping* training criterion, meaning we stopped training when the training process converged (no more improvements on a held-out validation set). Thus, all methods converged before hitting the maximal number of iterations. This means that increasing the number of update steps would not change the outcome (at least not substantially).

Reviewer comment (Evaluation):
Is there any intuitions behind why the hard loss always outperform the soft loss?

Author response:
We ran additional ablation experiments to compare the hard and the soft loss optimizations against each other. We added plots showing the results from these experiments to a new section F.5 in the Appendix. We compare test set accuracy and calibration performance across all training settings (class-balanced and heavy-tailed) and for all datasets (our findings are outlined in the comment above). In addition, we added a subsection as part of this new section on "Why soft loss optimization produces less accurate classifiers?" to provide a well-informed intuition about why the soft loss produces worse classifiers than the hard loss. This subsection can be found in F.5.1. See the **general response** for more details.

Reviewer comment (Experiments):
The case considered in the experiments seems to be a very special case, and it is difficult to see whether the conclusions drawn can be generalized.

Author response:
The idea behind OKO is to identify the majority class of a set of inputs. Here we focus on the special case where the sets consist of 2 examples from the majority class and 1 example from the minority ($k=1$). We demonstrated empirically that this setting generalizes to other settings. For space reasons we had to move these experiments to Appendix F.4. That section shows that the conclusions generalize (the particular value of $k$ rarely matters but setting $k>0$ is crucial.). We thus decided to focus on $k=1$ in our main paper because we think that this is the setting that is most relevant to practitioners.

Reviewer comment (Writing):
The "set learning" problem you mention in the introduction is not formally defined, nor is the odd-one-out scheme in Section 2.

Author response:
In these sections, we avoided a more formal introduction specifically because we also want to talk about related work, which does not necessarily share the same formalisms as we do. Rather, we want to talk broadly about other methods that use set learning. A very formal introduction (needed for the proofs) can be found in the Appendix.

Reviewer comment (Novelty):
As far as I understand, OKO consists in transforming the classification problem into a binary classification problem, as made in the "error-correcting output coding" decomposition strategy. Could you explain the difference between OKO and ECOC ?

Author response:
This is a very interesting question. We had not seen the connection to ECOC, thank you for pointing it out! We will include it into our related works discussion. The idea behind ECOC is to look at multiclass classification by training several different binary classifiers. In contrast, with OKO one trains just a single final classification model (not several), and that model still makes a multiclass prediction (in ECOC, every model makes a binary decision). OKO is also focused on delivering well-calibrated models, which AFAIK is not a main goal of ECOC.

Reviewer comment (Reproducibility):
In Section 3, you explain how to construct a training example for the OKO setting. I assume this operation is repeated. Could you elucidate how instance generation can be repeated in the global OKO setting ?

Author response:
Algorithm 1 shows how to construct a single training example for OKO. By running this algorithm repeatedly (for multiple steps), one can construct a whole dataset of sets --- which we refer to as $\mathcal{D}^{\prime}$ --- for using OKO.

Reviewer comment (Experiments):
Could you provide information on the results obtained using the soft loss, and/or insights regarding why the hard loss would give better results than the soft loss ?

Author response:
Please see our **general response** where we provide an answer to this. We will add a new section in a revised version of the manuscript.

Reviewer comment (Theory):
Section 4 is very short. Theorem 1 seems to be limited to the example discussed. The section seems to call for a discussion on the different natures of uncertainty (i.e., aleatory vs epistemic uncertainty) and on the different treatments to be applied accordingly.

Author response:
Theorem 1 is just meant to give some intuition about how OKO behaves in a particular setting and help explain why it may exhibit good calibration. It indicates that, for the setting in Theorem 1, input regions with fewer samples, which may be interpreted as a higher degree of epistemic uncertainty, manifest as aleatoric uncertainty in the OKO test time outputs. While it is conceivable that such regions indeed have low aleatoric uncertainty, and that the few samples do indeed characterize the entire region, in safety-critical applications it is often desirable for the network to err towards uncertainty. We will include this point with an explanation of *aleatoric* and *epistemic* uncertainty in the next draft.

Reviewer comment (Experiments):
You set $k=1$ for all experiments: this seems to correspond to a very special case of OKO. Could you explain the impact of this choice ? This somehow mitigates the conclusions that can be drawn from the experimental part, as it seems difficult to generalize them. Can you elaborate ?

Author response:
We did not have space to include detailed results about this in the main text, but Appendix Section F.4 answers this question. The section shows through empirical evidence that this special case generalizes well to more general cases because the particular choice of $k$ does not matter but $k>0$ is crucial.

Reviewer comment (Writing):
Page 1 stated that "By construction, this paradigm ignores information that may be found in correlations between sets of data." I do not think "correlation" is the correct word here. I do not see how correlation is explicitly exploited in the newly defined loss function. Moreover, shouldn't it be the relationship among points in the same set instead of the relationship among different sets?

Author response:
1.) Maybe “shared latent information” might indeed be a better word for what we’re trying to express. We think OKO is better able to tease out such information because it allows the model to compare data points (both from the same set as well as among different sets), so that every gradient descent step is influenced by more than just a single data point. Judging by our empirical results, we think it’s fair to say that our model, indeed, appears to learn more sensible decision boundaries wrt. Calibration.

Reviewer comment (Theory):
Since k does not really matter, can it be zero? In fact, can we have more than 2 instances from the dominating class in a set. In this case, (2) would be very similar to a multi-instance learning method.

Author response:
4.) The connection to Multi-Instance Learning (MIL) for $k=0$ is reasonable. Thank you for pointing it out! We agree that one can have more than $2$ instances from the majority class. Appendix Section F.4 contains ablation for various settings of $k$, including $k=0$. As stated in the main text, the particular choice of $k$ does not matter much but $k > 0$ is substantially better than $k=0$ across all training settings (see Section F.4 in the Appendix). In preliminary experiments, we tried larger numbers of images for the majority class when $k=0$ and did not observe substantial differences between 2, 3, and 4 images in the majority class when $k=0$. Hence, we conclude that $k > 0$ is important but not its particular value. Of course, one could try different combinations of the number of images in the majority class and $k$ but this is an exponentially large combinatorial problem and we leave this to future analyses.

Reviewer comment (Theory):
Page 5 stated that "The key observation from Theorem 1 is that, although x = 1 or x = 2 have zero entropy and are thus low-entropy regions, OKO is still uncertain about these points because they occur infrequently". However, this is not necessarily a good thing (or not good enough). For example, the result of Theorem 1 does not say that the calibration is correct. In fact, I am not even sure how to define the calibration in this example because the limit portability distribution seems weird to me: it is distributed as 1/0 when epsilon is greater than 0 and 0/0 when epsilon = 0. Specifically, the limit of epsilon = 0 is a singular case. Can this be generalized to more general settings? In summary, Theorem 1 is about a fairly special case, and even in that case it does not explicitly show that the calibration is correct.

Author response:
5.) The desirability of the phenomenon highlighted in Theorem 1 is dependent on the application. It is most desirable when one would like to err towards uncertainty for regions of the input space that are sparsely sampled. This might be beneficial for a safety-critical setting where it would be better for the network to indicate less certainty in sample regions with little data rather than having the network assign high certainty from just a sample or two. We can include this point in the next draft, if you want.

Reviewer comment (Presentation):
I am very puzzled by the subsection titled "Relative Cross-Entropy for Calibration" in Section 4. This RC has nothing to do with OKO and should not be part of a section titled "Properties of OKO". Moreover, this subsection reads very out of context. It is unfortunate that relevant discussion is in the appendix, making it very difficult for the reader to see its relevancy to the topic being discussed in the main text.

Author response:
6.) We will connect RC better with OKO and how RC helps understand the empirical fact that OKO produces greater entropy predictions by showing that the cross entropy vs entropy diff is more centered at zero (see our general response).

Reviewer comment (Writing):
Page 6 stated that "RC is no longer proper due to this zero mean." What does "proper" mean?

Author response:
7.) The definition of proper was, unfortunately, moved to the Appendix and we will also move that sentence as well. A proper score is essentially a score that is maximized in expectation when labels are from the same distribution as the calibrating distribution. However, since RC has an expectation of 0 on the same distribution, and it can be positive, or negative, it is in fact not a proper scoring rule.

Reviewer comment (Reproducibility):
Page 6 stated that "we report results for a version of OKO with k = 1 where in addition to the pair class prediction (see Eq. 2) a model is trained to classify the odd class with a second classification head that is discarded at inference time." How is the overall objective including the second classification head defined? A mathematic formula and/or a graph of the network structure would be helpful.

Author response:
8.) The objective of the 2nd head is cross entropy between the odd class label $(y_{i})$ and the logits of the odd class representation $(x_{i})$. In other words, the full loss function in this case was $L_{\text{oko}}^{\text{hard}}(S_y, f_\theta (S_x)) + y_i \log \theta(x_i) $. We will clarify this in the next version of the manuscript.

Reviewer comment (Experiments):
Page 6 stated that "For simplicity and fairness of comparing against single example methods, we set the maximum number of randomly sampled sets to the total number of training data points ntrain in every setting." In my opinion, it is still not a fair comparison since for the OKO method there are 3n instances out of the n sets.

Author response:
9.) We can see your point and where you are coming from, but we would still like to disagree here: It is indeed true that on average, OKO converges sooner, because it processes more information per update step. However, we made sure that every method could run until convergence, so we are very confident that more examples would not have helped the baselines improve their performance. Also, recall that $n$ does not change. So, if you train each method until convergence, it does not matter how often a model has seen the same image.

Reviewer comment (Presentation):
It is unclear to me what I should look for in Figure 4? Why is one method better than another and how is such an advantage shown in the figure? More specific pointers should be helpful.

Author response:
10.) In Figure 4, we show the distribution of entropies of the predicted probability distributions for individual test data points (i.e., at inference time) partitioned into correct and incorrect predictions respectively. The $x$-axis represents correct vs. incorrect predictions (hence, its label is binary), and the $y$-axis represents the entropy, or uncertainty if you will, of the model’s softmax outputs, and therefore its label is continuous. The range of the entropies goes from $\log(1) = 0$ to $\log(C)$ where $C$ is the number of classes in the data.

Reviewer comment (Theory):
Page 9 stated that "OKO is a theoretically grounded learning algorithm that modifies the training objective into a classification problem for sets of data. We show various consistency proofs and theoretical analyses proving that OKO yields smoother logits than standard cross-entropy, corroborated by empirical results." I strongly disagree with this statement. In the main text there is only one theorem, which, as I mentioned above in question 5, is not strong enough. There is no consistency results in the main text.

Author response:
11.) Thank you for this feedback. After seeing your and the other reviewer’s reaction, we agree that our claims were too strong in this regard. Please see the **general response** above for more details about how we intend to fix this.

Reviewer comment (Presentation):
In C.1 it stated "Before proving Proposition 2 from the main text ...." Proposition 2 is in the appendix, not in the main text.

Author response:
12.) Thank you for pointing this out. We will fix this accordingly.

Reviewer comment (Presentation):
It is clear to me that this paper was written as a long, comprehensive paper. However, it was trimmed down to 9 pages in order to submit to ICLR, in a rush. For this reason, the presentation of the topics is very poor. There are too many topics in this 9-page paper, and the author did not make a convincing case advocating for the proposed method. Moreover, because of the careless reduction of topics, there are topics that read out of context (e.g. the introduction of RC), and there are topics that should have been explained more (e.g. theoretical justification of the proposed loss). I think it may serve the readers better by submitting a comprehensive version of the paper to a journal such as JMLR, instead of rushing through a conference.

Author response:
13.) Thank you for the blunt feedback, we appreciate the honesty. As we addressed in the general response to the reviewers, we agree that our claims about the theoretical contributions were too strong. We toned them down, and improved the presentation of our results, and the general flow of our text.  We would kindly ask you to take a look at the revised version of the PDF that we will upload early next week: You seem to agree that our approach is interesting and our empirical results look good. So, we would kindly ask you to consider whether the results we present here are truly something the ICLR community is not interested in.