Paper: Consistent algorithms for multi-label classification with macro-at-$k$ metrics (ICLR-2024)

Reviewer comment (Theory):
The theoretical results could be accompanied by an intuition to ease understanding of the results.

Author response:
We thank the reviewer for the suggestion; we will try to incorporate more intuitions in the next revision. If the reviewer has specific suggestions regarding which results would benefit the most from them, we would be happy to focus on these parts.

Reviewer comment (Experiments):
A comparison to something like binary relevance learning treating each label independently for demonstrating also empirically that these measures cannot be sufficiently tackled by such an approach would be desirable.

Author response:
For metrics for which macro averages can be written in a linear form (see page 5), the optimal solution can also be obtained by binary relevance-like approaches. In the experiments, we explicitly include methods (e.g., Top-K, Macro-R$_{\text{PRIOR}}$) of this type. In the general case, binary relevance methods do not lead to the optimal solution, as discussed in Appendix E.

Reviewer comment (Presentation):
Minor: p. 4 "define in Table 1".

Author response:
We thank the reviewer for noticing and letting us know.

Reviewer comment (Writing):
Minor: p. 7 "tensor measure measure".

Author response:
We thank the reviewer for noticing and letting us know.

Reviewer comment (Novelty):
In https://doi.org/10.1007/s10994-021-06107-2 an @k metric interpolating between Hamming and subset 0/1 loss is presented. To what extent would this relate to the considered @k-metrics in this paper?

Author response:
While the Hamming loss is a part of the framework we consider in our work, the 0/1 loss is not, as it cannot be represented as a function of the confusion tensor we use. That also applies to the family of losses introduced by [Hüllermeier et al. 2022](https://doi.org/10.1007/s10994-021-06107-2), which are a generalization of these two losses. Therefore, we cannot consider them in our framework.  

The key difference between the approach of Hüllermeier et al. and ours is that 
the former measures performance *jointly on each instance*, while we focus on macro-averages. 

In our work, $k$ is a strict requirement on the number of predicted positive labels per instance. In case of the other work, $k$ is a parameter of a loss function that indicates the minimal size of correctly predicted subsets of labels for an instance that will be rewarded by the loss function. It does not limit in any way the number of predicted positive labels.

To see the difference, consider $k = 1$. In our case, we will predict only one label per instance. The Bayes classifier will be a label with the highest value of the linear transformation of the label marginal (with parameters $\mathbf{a}$ and $\mathbf{b}$ depending on the base loss function and data distribution). In case of Hüllermeier et al., $k=1$ leads to the standard Hamming loss without any restriction on the number of predicted labels. The optimal prediction is then a set of labels with marginal probabilities being greater than (or equal) 0.5.

Reviewer comment (Evaluation):
From the results of Table 2, the proposed method seems to greatly sacrifice instance-wise metrics to trade for gains in macro-average metrics. Is it possible for the proposed method to optimize an interpolated version of the objective that flexibly controls the performance tradeoff between instance-wise metrics and macro-averaged metrics?

Author response:
Yes, it is possible to optimize a measure that interpolates between instance-wise metric and macro-metric as long as the resulting objective is a function of confusion tensor $\mathbf{C}$ and ideally meets all assumptions required by the Frank Wolfe procedure.
Such objective is, for example, a linear interpolation between instance-precision@k and macro-f1@k:
$$
\Psi(\mathbf{C}) = (1 - \alpha) \text{Precision}@k(\mathbf{C}) + \alpha \text{Macro-F1}(\mathbf{C}).
$$

Below, we link to the image presenting plots for two datasets, Mediamill and Flickr. The results show that, in this case, the instance-vs-macro curve has a nice concave shape that dominates simple baselines. In particular, we can initially improve macro-measures significantly with only a small drop in instance-measures. We will include this discussion and more results in the appendix.


Plots: https://i.imgur.com/LJTTxyq.png

Reviewer comment (Experiments):
Some baseline methods that claim to also perform good on tail-labels are not discussed in related work, or compared in the experiment section. For example [1] and [2], to name just a few.

Author response:
In the paper, we included a comparison with some popular techniques that are domain-agnostic. However, we will be happy to include a comparison with more methods of that type if the reviewer has some particular in mind. Regarding the ones already mentioned by the reviewer:

- [1] considers multi-class ($y \in [m]$) setting, while in this work we consider multi-label ($\mathbf{y} \in \\{0,1\\}^m$) setting. Post-hoc logit adjustment presented in Section 4 of [1] is the simple weighting by the inverse of priors, which we also use in experiments (denoted as $\textrm{Macro-R}_{\text{PRIOR}}$). 

- [2] heavily relies on the problem domain (textual data) and cannot be applied to all datasets we consider. Please also note that the approach proposed in our submission can be used on top of any method aiming at improving the probability estimates of tail labels. We will try to highlight it more in the revised version of the paper.

Reviewer comment (Theory):
To improve the clarity of the proposed methods, the author may consider a toy synthetic dataset where data distributions are known, and show derivations of the proposed method, and verify the consistency property through simulations.

Author response:
We thank the reviewer for the suggestion! We will add an experiment on synthetic data to improve the presentation of our paper.

Reviewer comment (Novelty):
This submission also seems highly related to [3]. The author should discuss what's the difference, and compare it empirically.

Author response:
Our submission and paper [3] look at the same problem setup, but using two different optimization frameworks. The latter follows  the _Expected Test Utility_ (ETU) framework where one assumes a _given fixed_ test set $\boldsymbol{X}$ and then tries to predict the corresponding labels $\boldsymbol{Y}$. This has important consequences, both theoretical and practical.

From a theoretical perspective, by having a fixed test set, the optimal classifier is always deterministic, and one essentially solves a discrete optimization problem over the space of all at-$k$ predictions. As this becomes computationally intractable except for trivial cases, [3] needs to make an approximation, actually optimizing $\Psi(\mathbb{E}[\mathbf{C}(\boldsymbol{Y}, \hat{\boldsymbol{Y}}) \mid \boldsymbol{X}])$ instead of the actual ETU objective, $\mathbb{E}[\Psi(\mathbf{C}(\boldsymbol{Y}, \hat{\boldsymbol{Y}})) \mid \boldsymbol{X}]$. This objective is then minimized using a block-coordinate-ascent algorithm.

In contrast, the _Population Utility_ (PU) framework adopted in this work means we are looking for (potentially stochastic) classifiers $\boldsymbol{h}$ that work on the level of _a single instance_. Thus, we want to optimize $\Psi(\mathbb{E}[\mathbf{C}(\boldsymbol{y}, \boldsymbol{h}(\boldsymbol{x}))])$. While this resembles the approximation above, there are important differences. Firstly, the expectation is no longer conditioned on the test set $\boldsymbol{X}$. Secondly, instead of matrices, $\boldsymbol{Y}$ and $\boldsymbol{X}$, we now only need vectors $\boldsymbol{y}$ and $\boldsymbol{x}$, as the expectation is taken over single instances.

From a practical point of view, this means that the PU-classifier introduced in this paper can be employed in an online inference setting, where queries are presented one-by-one, instead of being in one large batch. In contrast, if you apply the ETU algorithm [3] in online inference, i.e., independently apply to test sets of size 1, it degenerates back to the top-k prediction.
However, when evaluating on a given test set, the ETU solution can play its strengths, since it is possible to exploit the actual instances present in the test set, which PU cannot do. Thus, when using the same label probability estimator $\boldsymbol{\eta}$, we generally expect ETU to perform better than PU.

To illustrate our expectation empirically, we present below the results obtained for the Flickr data set (the method introduced in [3] is denoted in the table below with $_{BCA}$ subscript).

**Flickr:**
| method                         | iP@5       | iR@5       | mP@5       | mR@5       | mF@5       |
|:-------------------------------|-----------:|-----------:|-----------:|-----------:|-----------:|
| Top-K                          | **16.99**  | **65.90**  | 17.11      | 46.88      | 23.47      |
| Top-K + $w^{\text{POW}}$       | 16.10      | 62.69      | 13.76      | 52.22      | 20.67      |
| Top-K + $w^{\text{LOG}}$       | 16.76      | 65.10      | 15.05      | 49.60      | 21.99      |
| Top-K + $\ell_{\text{FOCAL}}$  | *16.89*    | *65.50*    | 18.54      | 45.53      | 24.14      |
| Top-K + $\ell_{\text{ASYM}}$   | 16.73      | 64.91      | 17.39      | 45.47      | 23.58      |
| Macro-P$_{FW}$                 | 5.66       | 22.71      | *41.74*    | 9.67       | 10.55      |
| Macro-R$_{BCA}$                | 0.39       | 1.63       | **65.80**  | 3.74       | 3.29       |
| Macro-R$_{PRIOR}$              | 12.17      | 47.40      | 13.98      | *53.64*    | 19.71      |
| Macro-R$_{FW}$                 | 12.17      | 47.40      | 13.98      | *53.64*    | 19.71      |
| Macro-R$_{BCA}$                | 12.63      | 49.12      | 12.93      | **55.44**  | 18.86      |
| Macro-F1$_{FW}$                | 11.78      | 45.83      | 34.68      | 30.77      | *29.40*    |
| Macro-F1$_{BCA}$               | 11.23      | 43.53      | 32.36      | 33.21      | **30.70**  |

Reviewer comment (Evaluation):
The discussion on which of the proposed metrics and macro-at-k approach should be chosen for different problems from practitioners' perspectives is missing, and its inclusion would strengthen the presentation.

Author response:
The basic heuristics might be more robust than the FW method in the case of datasets with very long tails and a very small number of positive instances. In this case, simplicity might be preferred over the sophisticated optimization procedure (which is the general rule of thumb in statistics and machine learning). Nevertheless, we believe that by improving the probability estimation for long tail labels (what we do not discuss in this paper), we should also get a more stable and better performance of the FW method. 

Regarding the metrics, macro-averaging treats all the labels equally important. This prevents ignoring tail labels (when used with an imbalance-sensitive binary loss). The budget of $k$ predictions requires the prediction algorithm to choose the labels “wisely.” The choice of the base metric (e.g., precision, recall, F-measure, etc.) depends on a given application, exactly in the same way as in the case of binary classification applications. For example, suppose a user wants to cover as many as possible relevant labels. In that case, they should use recall, but in our setting, the budget per each instance is precisely defined and recall for each label has the same importance.

Reviewer comment (Evaluation):
The manuscript is missing the analysis of computational complexity and computational times (of optimising the classifiers) and how these relate to the size of the budget and other chosen parameters, as well as how these compare between different heuristics.

Author response:
During the inference, the probabilistic classifier returned by the Frank-Wolfe procedure selects top $k$ labels according to selected $\boldsymbol{a}$ and $\boldsymbol{b}$ parameters, what has time complexity the same as other baselines (that is $O(m)$ per instance).

A single iteration of Frank-Wolfe (FW) for macro-at-k metrics has both time and space complexity of $O(n_2m + m)$, where $m$ is the number of labels, $n_2$ is the size of a dataset used to estimate the confusion matrix in the FW algorithm ($\mathcal{S}_2$ in Algorithm 1).
In case of time complexity, we need $m$ operations for calculating the gradient of $\mathbf{C}$ and values of $\boldsymbol{a}$ and $\boldsymbol{b}$, and $n_2m$ operations for selecting top $k$ predictions for each instance in $\mathcal{S}_2$ according to $\boldsymbol{a} \odot \boldsymbol{\eta}(\boldsymbol{x}) + \boldsymbol{b}$ and estimation of next $\mathbf{C}$. 
In case of space complexity, we need $n_2m$ units to store $m$ probability estimates and labels for each instance in $\mathcal{S}_2$, and additional $m$ units are required for $\mathbf{C}$, its gradients, $\boldsymbol{a}$ and $\boldsymbol{b}$, etc.

The Frank-Wolfe algorithm usually requires a few iterations (in our implementation, we stop it if the number of iterations reaches a maximum, which was 20 in our experiments, or $\alpha \le \epsilon$, which was 0.0001 in our experiments). We include running times (in seconds) and a number of iterations in the table below (for a single run of the Frank-Wolfe procedure).

From the presented data, it is hard to observe a simple relation of $k$ and running time. The required number of iterations depends on the difficulty of the optimization problem that depends not only on $k$ but also on label distribution and the quality of achieved probability estimates. 

We implemented the Frank-Wolfe algorithm in Python using Numpy and Pytorch; we believe that further speed-up can be achieved using lower-level programming language and programming methods.
For datasets with a very large number of labels, we can truncate probability estimates very close to 0 and use sparse data structures to speed up the algorithm further.

We will include the above analysis and results in the appendix.

**Mediamill:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 67.59| 11| 70.33| 10| 56.60| 7 |
| Macro-R$_{FW}$  | 27.69| 4 | 27.70| 4 | 28.41| 4 |
| Macro-F1$_{FW}$ | 45.60| 11| 42.44| 10| 29.33| 5 |

**Flickr:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 85.52| 9 | 87.73| 9 | 110.90  | 12|
| Macro-R$_{FW}$  | 50.80| 4 | 51.55| 4 | 50.84| 4 |
| Macro-F1$_{FW}$ | 55.23| 5 | 50.10| 4 | 61.97| 6 |

**RCV1x:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 753.32  | 6 | 799.90  | 6 | 754.67  | 5 |
| Macro-R$_{FW}$  | 645.35  | 4 | 656.32  | 4 | 605.59  | 4 |
| Macro-F1$_{FW}$ | 541.44  | 5 | 537.67  | 5 | 606.54  | 6 |

**AmazonCat:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 1181.99 | 4 | 840.41  | 2 | 1134.49 | 4 |
| Macro-R$_{FW}$  | 1112.24 | 4 | 1126.27 | 4 | 1081.46 | 4 |
| Macro-F1$_{FW}$ | 891.25  | 4 | 1454.04 | 9 | 885.40  | 4 |

Reviewer comment (Evaluation):
What would be the conclusions or "rule of thumb" for selecting particular heuristics from the practitioners' point of view for certain applications or multi-label classification problems?

Author response:
The basic heuristics might be more robust than the FW method in the case of datasets with very long tails and a very small number of positive instances. In this case, simplicity might be preferred over the sophisticated optimization procedure (which is the general rule of thumb in statistics and machine learning). Nevertheless, we believe that by improving the probability estimation for long tail labels (what we do not discuss in this paper), we should also get a more stable and better performance of the FW method. 

Regarding the metrics, macro-averaging treats all the labels equally important. This prevents ignoring tail labels (when used with an imbalance-sensitive binary loss). The budget of $k$ predictions requires the prediction algorithm to choose the labels “wisely.” The choice of the base metric (e.g., precision, recall, F-measure, etc.) depends on a given application, exactly in the same way as in the case of binary classification applications. For example, suppose a user wants to cover as many as possible relevant labels. In that case, they should use recall, but in our setting, the budget per each instance is precisely defined and recall for each label has the same importance.

Reviewer comment (Evaluation):
What are the computational costs of the proposed approach and how do these relate to the size of k?

Author response:
During the inference, the probabilistic classifier returned by the Frank-Wolfe procedure selects top $k$ labels according to selected $\boldsymbol{a}$ and $\boldsymbol{b}$ parameters, what has time complexity the same as other baselines (that is $O(m)$ per instance).

A single iteration of Frank-Wolfe (FW) for macro-at-k metrics has both time and space complexity of $O(n_2m + m)$, where $m$ is the number of labels, $n_2$ is the size of a dataset used to estimate the confusion matrix in the FW algorithm ($\mathcal{S}_2$ in Algorithm 1).
In case of time complexity, we need $m$ operations for calculating the gradient of $\mathbf{C}$ and values of $\boldsymbol{a}$ and $\boldsymbol{b}$, and $n_2m$ operations for selecting top $k$ predictions for each instance in $\mathcal{S}_2$ according to $\boldsymbol{a} \odot \boldsymbol{\eta}(\boldsymbol{x}) + \boldsymbol{b}$ and estimation of next $\mathbf{C}$. 
In case of space complexity, we need $n_2m$ units to store $m$ probability estimates and labels for each instance in $\mathcal{S}_2$, and additional $m$ units are required for $\mathbf{C}$, its gradients, $\boldsymbol{a}$ and $\boldsymbol{b}$, etc.

The Frank-Wolfe algorithm usually requires a few iterations (in our implementation, we stop it if the number of iterations reaches a maximum, which was 20 in our experiments, or $\alpha \le \epsilon$, which was 0.0001 in our experiments). We include running times (in seconds) and a number of iterations in the table below (for a single run of the Frank-Wolfe procedure).

From the presented data, it is hard to observe a simple relation of $k$ and running time. The required number of iterations depends on the difficulty of the optimization problem that depends not only on $k$ but also on label distribution and the quality of achieved probability estimates. 

We implemented the Frank-Wolfe algorithm in Python using Numpy and Pytorch; we believe that further speed-up can be achieved using lower-level programming language and programming methods.
For datasets with a very large number of labels, we can truncate probability estimates very close to 0 and use sparse data structures to speed up the algorithm further.

We will include the above analysis and results in the appendix.

**Mediamill:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 67.59| 11| 70.33| 10| 56.60| 7 |
| Macro-R$_{FW}$  | 27.69| 4 | 27.70| 4 | 28.41| 4 |
| Macro-F1$_{FW}$ | 45.60| 11| 42.44| 10| 29.33| 5 |

**Flickr:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 85.52| 9 | 87.73| 9 | 110.90  | 12|
| Macro-R$_{FW}$  | 50.80| 4 | 51.55| 4 | 50.84| 4 |
| Macro-F1$_{FW}$ | 55.23| 5 | 50.10| 4 | 61.97| 6 |

**RCV1x:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 753.32  | 6 | 799.90  | 6 | 754.67  | 5 |
| Macro-R$_{FW}$  | 645.35  | 4 | 656.32  | 4 | 605.59  | 4 |
| Macro-F1$_{FW}$ | 541.44  | 5 | 537.67  | 5 | 606.54  | 6 |

**AmazonCat:**
| method | time@3/s| iter.@3 | time@5/s | iter.@5 | time@10/s  | iter.@10|
|:----|--:|--:|--:|--:|--:|--:|
| Macro-P$_{FW}$  | 1181.99 | 4 | 840.41  | 2 | 1134.49 | 4 |
| Macro-R$_{FW}$  | 1112.24 | 4 | 1126.27 | 4 | 1081.46 | 4 |
| Macro-F1$_{FW}$ | 891.25  | 4 | 1454.04 | 9 | 885.40  | 4 |