Paper: A Consistent Lebesgue Measure for Multi-label Learning (ICLR-2024)

Reviewer comment (Theory):
I'm confused about one point: why did the author specifically focus on the learning model f? Does f have any impact on the proposed method? It seems to be a standard neural network. If there's anything I missed, please let me know.

Author response:
$\textbf{W1}$: Thank you for your question. We would like to make clarifications to help your understanding. The main focus of the paper is not the learning model $f$. The main focus of the paper is the consistent Lebesgue measure, hence, CLML is a model agnostic optimisation approach that could be applied to different classifiers. Although we use a standard neural network, CLML can be applied to other classifiers. We have also proven that the Lebesgue measure in CLML is consistent irrespective of the underlying classifier.

Reviewer comment (Reproducibility):
The introduction of the proposed method Section 2.3 is unclear. The paper did not provide a detailed explanation of how to optimize the proposed objective function. If an existing method was used, the technical contribution of the proposed method is limited.

Author response:
$\textbf{W2}$: Thank you for your suggestion. This was brought up by previous Reviewers XYJ3 and Fh3C, and we have added additional technical exposition in the appendix of the updated manuscript. To be specific, have included a more detailed exposition of the underlying optimiser, and covariance matrix adaptation, in addition to an exposition on the Lebesgue measure and pseudocode that describes the overall optimisation procedure of CLML.

Reviewer comment (Experiments):
The dataset is relatively small, with the largest dataset containing only 5000 examples.

Author response:
$\textbf{W3.1}$: Thank you for your comment. In our experiments, we have chosen several datasets with over a thousand features, and several datasets with over 50 possible labels. In tabulated multi-label learning, the complexity of the problem can scale with the high possible number of interactions between features and labels, which we discuss in-depth in the appendix. Increasing the number of examples may not necessarily increase the complexity of the problem  (but might increase the experimentation time). In fact, comparing the smaller number of samples to the high degree of feature-to-label interaction dispersion across the dataset, it becomes quite challenging to train a classifier that can accurately predict a set of class labels.

Reviewer comment (Experiments):
The proposed method should be compared with some commonly used loss functions, such as binary cross entropy loss, ranking loss, etc., based on the same base model.

Author response:
$\textbf{W3.3}$: Thank you for your suggestion. We have already compared CLML against the widely-used cross-entropy loss (widely used as a surrogate loss function in most multi-label learning papers). 

Section 5 of the paper titled Effectiveness of the Lebesgue measure over surrogate loss is dedicated to this point. We also have several figures throughout the paper supporting this point. We have also deliberately chosen three of the most common multi-label loss functions: Hamming-loss, one minus Micro-$F_1$, and one minus label ranking average precision.

Reviewer comment (Presentation):
The figures were not plotted carefully; the font size of legends and axis is too small.

Author response:
$\textbf{W3.4}$: Thanks for your comment. We have increased the font sizes of several figures to improve readability.

Reviewer comment (Reproducibility):
How to optimize the objective function proposed in the paper?

Author response:
Thank you for your question. The optimisation is achieved using covariance matrix adaptation, wherein the quality of a solution is determined by its Lebesgue contribution, which is clearly described in Section 2.3. Additional technical exposition has been included in the Appendix of the updated manuscript to further elaborate on this, including a description of covariance matrix adaptation, estimation of the Lebesgue measure, and pseudocode of the proposed CLML.

Reviewer comment (Writing):
There appears to be a typographical error in Equation 2. Specifically, in the loss function L\left(f\left(x\right),y\right), the y should likely be denoted as y^\prime. This inconsistency needs to be rectified to ensure clarity and coherence in the formulation.

Author response:
$\textbf{W1}$: Thank you for pointing this out. Reviewers PXNK and XYJ3 have also previously pointed this out. We have corrected this minor typographical error in the updated manuscript.

Reviewer comment (Reproducibility):
The optimization method in the last paragraph of section 2.3 is not elaborated in detail. To enhance clarity and provide readers with a holistic understanding, the authors are recommended to furnish a more detailed exposition on the employed optimization techniques.

Author response:
$\textbf{W2}$:  Thank you for the valid point. Reviewer XYJ3 also shared the same concern. We have included some additional formulas and explanations in the appendix to clarify the optimisation method, specifically regarding Section 2.3. To be specific, have included a more detailed exposition of the underlying optimiser, covariance matrix adaptation, in addition to an exposition on the Lebesgue measure and pseudocode that describes the overall optimisation procedure of CLML.

Reviewer comment (Experiments):
The methodology proposed in this paper appears to be tailored specifically for tabular data. This inherent design might limit its applicability to large-scale image datasets, which inherently possess a different data structure and complexity. Thus, the generalizability of the approach to diverse data types, especially image datasets, remains questionable.

Author response:
$\textbf{W3}$: Thank you for your comment. As you have shared the same concern as reviewer XYJ3, we will elaborate on a similar point. Although many of the state-of-the-art multi-label papers focus on multi-label object recognition, tabulated data remains an important research area in multi-label learning. 

Furthermore, the purpose of this paper was to present concrete theoretical and empirical evidence that consistency can be achieved using the Lebesgue measure. Therefore, CLML and the contributions of this paper are not intended to be limited to tabulated data but instead are supposed to focus on communicating an effective alternative to traditional optimisation methods for multi-label learning. 

We are hoping that our paper will pioneer future research in multi-label learning by enticing researchers to reconsider the optimisation procedure in multi-label learning. Understandably, applying CLML to contemporary problems efficiently, such as large-scale image datasets, requires additional work due to the additional complexity of the data and learners, which we felt would detract the main focus of this paper from demonstrating the effectiveness of CLML in the aforementioned theoretical and empirical senses. Incorporating computer vision would have significantly increased the length (or necessary content) of the paper and would have potentially fallen out of scope. Again, we would like to stress that the intention of this paper is to provide grounds for an alternative optimisation approach that future work can build upon for more complex tasks.

We would like to continue elaborating on this point as we address your questions in the following discussions.

Reviewer comment (Reproducibility):
Regarding the optimization method presented in the latter part of section 2.3, could the authors delve deeper into the specifics of this method?

Author response:
$\textbf{Q2}$: Thank you for your question. We have included some additional technical exposition, formulas, and explanations in the appendix to clarify the optimisation in Section 2.3. We have included a more detailed exposition of the underlying optimiser, covariance matrix adaptation, in addition to an exposition on the Lebesgue measure and pseudocode that describes the overall optimisation procedure of CLML.

Reviewer comment (Experiments):
Given that the paper's methodology seems predominantly designed for tabular data, how do the authors envision adapting or evolving this method for more complex datasets, such as large-scale image datasets?

Author response:
$\textbf{Q3}$: This is indeed an interesting problem and the future direction will require a lot of work. Contemporary machine-learning tasks related to large-scale image datasets utilise models with much higher complexity than the standard neural network used in this paper. 

We would kindly ask the reviewer to also see our response to Reviewer XYJ3 who shared a similar question. Currently, CLML can be applied directly to image classification problems with only minor amendments to the implementation. However, due to our computational constraints, we chose to focus this work on developing and proving a consistent optimisation method to work on tabulated data. Adapting CLML to contemporary problems $\textit{efficiently}$ like computer vision requires additional research (which we thought would fall out of scope from the main contributions of this paper), and may require finding alternate ways to differentiate or design a differentiable approximate of the Lebesgue measure to enable migrating back to gradient-based search. Naturally, this would also require a thorough investigation on whether such an approximate would still be consistent with the desired loss functions.

Reviewer comment (Experiments):
Have the authors considered expanding the variety of multi-label loss functions to further enhance the model's performance? Additionally, would assigning different weights to distinct loss functions potentially lead to further improvements in the model's efficacy? It would be interesting to understand the impact of such variations on the overall performance.

Author response:
$\textbf{Q4}$: Thank you for your questions. Regarding the first point, we have used three of the most commonly used multi-label loss functions. Increasing the variety of loss functions will not necessarily enhance model performance, $\textit{i.e.}$, loss functions that are not relevant to the specific multi-label learning problem may not necessarily guide the learner toward promising solutions. There may also be some overlap between the information quantified between loss functions, therefore increasing the sheer number of loss functions optimised after a certain point may reflect diminishing returns. 

We chose Hamming-Loss, Micro-$F_1$, and label ranking average precision deliberately knowing that: (1) we wanted to improve the performance of each loss function specifically; and (2) there exists some potential conflict between them, so it is necessary to optimise them jointly if we are interested in finding solutions to the trade-offs. 

Regarding your second point, weighting the loss functions is not required under the Lebesgue measure, and can be potentially detrimental to the performance. Weighing the distinct loss functions to determine the relative importance may not guarantee consistency. Weight vectors may direct the optimiser away from the Bayes risk of a subset of the loss
functions if those loss functions are weighted low. For a fair comparison, weighting methods require careful design of weight vector distribution which over-complicates the matter and
falls far outside the scope of this paper. Furthermore, the Lebesgue measure naturally accounts for the relations between distinct loss functions, as it is an analytical method that integrates over the area covered by a loss vector in a multi-dimensional loss landscape. We kindly refer the reader to see evidence in the training loss curves in Fig. 7 (d) and (e) in the appendix. The training curves demonstrate that earlier during the optimisation process, $\mathcal{L}_1$ and $\mathcal{L}_3$ are minimised more effectively than $\mathcal{L}_2$. Eventually, CLML is able to correct itself to further improve $\mathcal{L}_2$. These patterns indicate a complex loss landscape with varying difficulty in each loss dimension that CLML can automatically account for. Knowing this behaviour beforehand may suggest that $\mathcal{L}_2$ needs to be weighted differently than $\mathcal{L}_1$ and $\mathcal{L}_3$, although eventually CLML can cope with these differences without such methods.

Reviewer comment (Experiments):
A potential weakness of the algorithm is its limitation to tabular data, which may render it ineffective for handling large-scale image datasets prevalent in contemporary research and applications.

Author response:
$\textbf{W1}$: Thank you for your comment. Although many of the state-of-the-art multi-label papers focus on multi-label object/image recognition, tabulated data remains an important research area in multi-label learning. 

The purpose of this paper was to present concrete theoretical and empirical evidence that consistency can be achieved using the Lebesgue measure. Therefore, CLML and the contributions of this paper are not intended to be limited to tabulated data (in fact, CLML can technically be applied to image classification problems, and in-fact most domains including regression and combinatorial optimisation, with only minor amendments to the code), but instead are supposed to focus on communicating an effective alternative to traditional optimisation methods for multi-label learning. We would kindly urge the reviewer to read our response to Reviewer Fh3C's W3. and Q3. (both of which relate to the topic of computer vision), where we provide further elaboration in the context of future work.

From here, we are hoping that our paper will pioneer future research in multi-label learning by enticing researchers to reconsider the optimisation procedure in multi-label learning. Although CLML can be applied to contemporary problems such as large-scale image datasets, much additional work is required to address efficiency due to the additional complexity of the data and learners, which we felt would detract the main focus of this paper from demonstrating the effectiveness of CLML in the aforementioned theoretical and empirical senses. Incorporating computer vision efficiently and effectively would have significantly increased the length (or necessary content) of the paper and would have potentially fallen out of scope. Again, we would like to stress that the intention of this paper is to provide grounds for an alternative optimisation approach that future work can build upon for more complex tasks.

Reviewer comment (Theory):
In Section 2.3, the optimization algorithms are merely summarized in brief statements rather than being detailed through comprehensive formulas. It raises the question of whether the complexity of these optimization methods contributes to the model's inability to handle large-scale datasets.

Author response:
$\textbf{W2}$: Thank you for the valid point. We have included some additional formulas and explanations in the appendix to clarify the optimisation, specifically regarding Section 2.3. We have included a more detailed exposition of the underlying optimiser, covariance matrix adaptation, in addition to exposition on the Lebesgue measure and pseudocode that describes the overall optimisation procedure of CLML.

Reviewer comment (Writing):
In Equation (2), the loss function and surrogate loss function incorrectly use $ y $ instead of $ y' $. This appears to be a typographical error.

Author response:
$\textbf{W3}$: Thank you for pointing this out. Reviewer PXNK and Fh3C also commented on this. We have corrected this minor typographical error in the updated manuscript.

Reviewer comment (Theory):
Could the authors provide more detailed formulations or explanations for the optimization methods mentioned in Section 2.3? I'm curious to understand their specifics. Additionally, could the authors clarify whether these methods are scalable to handle large-scale datasets?

Author response:
Thank you for the question! We have included a more detailed exposition of the underlying optimiser, covariance matrix adaptation, in addition to an exposition on the Lebesgue measure and pseudocode that describes the overall optimisation procedure of CLML. We hope that our revision will clarify the optimisation methods in Section 2.3 to a satisfactory degree. 

In tabulated multi-label learning, the complexity of the learning problem can scale with the high number of interactions between features and labels, which can be a challenge even for datasets with a relatively small number of examples. We discuss this point in-depth in the appendix. The computational complexity of CLML during inference scales linearly with the number of features and labels. Therefore, a majority of the current complexity arises from the covariance update step, which is quadratic with the number of parameters in the network. Due to our computational constraints, we have chosen to focus this work on proving that consistent optimisation in multi-label learning can be achieved, while presenting empirical evidence to support our theory, on tabulated learning problems. However, this does not limit our work to tabulated problems, in fact, we have proven the consistency of CLML irrespective of the underlying classifier/task. CLML can be adapted to contemporary problems such as computer vision with minor amendments to the code. The challenge is that doing so efficiently would require some additional research (which we thought would fall out of scope from the main contributions of this paper). For this reason, we chose to focus on demonstrating the empirical and theoretical advantages of CLML against state-of-the-art multi-label learners on tabulated problems to entice further research in multi-label learning to reconsider the optimisation process.

Reviewer comment (Experiments):
This paper is an application of the Lebesgue method, and there are numerous other methods for multi-objective optimization. The paper does not compare its method with other multi-objective optimization techniques.

Author response:
$\textbf{W2}$: Thanks again for your suggestion. First, we would like to point out that although there is a lot of research on multi-objective optimisation, there is scarce work on multi-objective classification for multi-label data (even more so for tabulated data). To the best of our knowledge, this area has not been explored sufficiently in our specific problem domain. Furthermore, most existing multi-objective optimisation methods cannot directly be applied to multi-label learning problems, and instead are suited to single-label (binary or multi-class) classification, regression, or even combinatory optimisation problems. The major contributions of our work, and what ultimately sets it apart from other multi-objective papers, is that we aim to: (1) consistently optimise a set of non-differentiable or non-convex multi-label loss functions, where consistency is defined under the Bayes risk framework, and (2) propose to use the Lebesgue measure to achieving such an optimisation, which can be expressed analytically. Expressing the Lebesgue measure analytically is paramount as it opens up future research to approximate certain parts of the Lebesgue measure to suit automatic differentiation. This may enable consistent gradient-based optimisation.

Regarding the specific use of the Lebesgue measure as a quantifier, there are certainly other methods for quantifying performance in a multi-dimensional loss landscape. However, the key contribution was to achieve consistent optimisation over the desired loss functions using the Lebesgue measure. This is not always the case for other multi-objective measures.

For example, decomposition-based methods can use the Tchebycheff aggregation of a set of weighted loss functions. The consistency of decomposition-based methods can not be guaranteed since the weight vectors may direct the optimiser away from the Bayes risk of a subset of the loss functions. For a fair comparison, decomposition-based methods require careful design of weight vector distribution, which over-complicates the matter and falls far outside the scope of this paper.

Reviewer comment (Novelty):
The method proposed in this paper does not exhibit a significant empirical difference from the latest method, CLIF. The experimental results are rather marginal improvement.

Author response:
$\textbf{W4}$: Thanks for your comment. We would like to emphasise that CLML achieved a $13.4$ percent overall improvement in performance against CLIF. This represents a clear improvement over a state-of-the-art method that previously outperformed many well-known and state-of-the-art multi-label learners. 

The improvement was achieved with a simpler neural network that did not utilise $\textit{any}$ of the state-of-the-art label graphs or semantic embeddings that CLIF uses. Furthermore, the results support our theory signifying the importance of consistency in achieving an overall better learning performance.

Reviewer comment (Theory):
What is the specific rationale behind choosing Lebesgue's multi-objective optimization method?

Author response:
$\textbf{Q1}$: Thank you, that is a good question. We would like to refer the reviewer to our answer to W.2 for some additional insight, which we will also further elaborate on here to answer your question.

One of our main contributions was to consistently optimise a set of non-differentiable or non-convex multi-label loss functions, where consistency is defined under the Bayes risk framework. Without loss of generality, the set can consist of any number of multi-label loss functions that may or may not be practical to the target domain. The Lebesgue measure integrates over an area between a reference point (which we define as the origin vector in multi-dimensional space) and a multi-dimensional point that corresponds to a loss vector generated from some arbitrary model $f$. 

The Lebesgue measure has some desirable properties which make it a natural choice for optimising multiple loss functions simultaneously. First, the area covered by a loss vector is determined simply by its position in the multi-dimensional loss landscape, where the position in each dimension corresponds to a value obtained by a desired loss function. This property ensures that an improvement of the Lebesgue measure directly, and analytically, corresponds to an improvement of the underlying desired loss functions. Second, in comparison to other multi-objective methods which were covered in our response to W.2, the Lebesgue measure does not require specifying weight vectors for each loss function, $\textit{i.e.}$, the relative importance between loss functions do not need to be known prior to the optimisation process. This makes it quite effective to apply the Lebesgue measure on novel problems that have no prior domain knowledge. To support this, we kindly refer the reviewer to the training curves in Fig. 7 (d) and (e) in the appendix. The training curves demonstrate that earlier during the optimisation process, $\mathcal{L}_1$ and $\mathcal{L}_3$ are minimised more effectively than $\mathcal{L}_2$. Eventually, CLML is able to correct the trajectory to further improve $\mathcal{L}_2$. These patterns indicate a complex loss landscape with varying difficulty in each loss dimension that CLML can automatically account for.

Reviewer comment (Novelty):
In cases where classifiers resulting from different loss functions may conflict, why is the universal method proposed in this paper still necessary?

Author response:
$\textbf{Q2}$: Thanks for your question, this is a good point. We have partially addressed this in W1., and we will elaborate on this topic further here.

It is still essential to propose a universal method. There are two scenarios to consider, both of which can be addressed by CLML. First, CLML can find a classifier that obtains an overall good performance with trade-offs on several loss functions. This can be desirable in problems where there is a set of applicable or practical loss functions that need to be considered in tandem. In many real-world problems, it is essential to find a solution that can achieve acceptable performance on several metrics, and not a single metric at the expense of others. Second, according to our proof, we also demonstrate that CLML can eventually find a Bayes risk predictor for each of the underlying desired loss functions as it is consistent with each loss function (assuming that one abstracts away the classifier). During optimisation, these Bayes risk predictors can be archived. Hence, CLML can potentially find optimal classifiers for each function $\textit{and}$ find good classifiers with desirable trade-offs between all loss functions.

Reviewer comment (Theory):
Is there an issue with defining p(x) as equivalent to p(x|y) (above Eq 1), where the former depends solely on x, while the latter depends on both x and y simultaneously?

Author response:
$\textbf{Q3}$: Thank you for bringing this to our attention. We originally denoted $p(\textbf{x})$ as the conditional probability $p(\textbf{y}|\textbf{x})$ for simplicity and stated so in the paragraph above Eq. 1. However, in retrospect this can cause unnecessary confusion so we have rectified this by clearly stating the full conditional probability $p(\textbf{y}|\textbf{x})$ in the updated manuscript.

Reviewer comment (Writing):
Is there a problem with Eq 2, where the latter y should be y'?

Author response:
$\textbf{Q4}$: Thank you for pointing this out. Reviewers XYJ3 and Fh3C also commented on this. We have corrected this minor typographical error in the updated manuscript.

Reviewer comment (Theory):
In the absence of a specified R, how does optimizing lambda(P(f)) achieve multi-objective optimization? If R consists of only one element, it is possible that H(F, R) could be empty.

Author response:
$\textbf{Q5}$: Thank you for your question. This is a good point. Initially, $R$ consists of the unit vector in the multi-dimensional loss landscape. We have included this information in the updated manuscript. The unit vector denotes the worst possible performance on all loss functions. From there, CLML will find a function that can achieve better performance than the unit loss vector. During the early stages of the optimisation it is likely that the set of mutually-non dominating vectors will be small, which runs the risk of $H(F,R)$ being empty. However, we use the Monte Carlo sampling method to sample loss vectors in $Z$ to approximate the actual hypervolume efficiently. This mitigates the risk of $H(F,R)$ being empty. This point may become more clear with the additional exposition that we have provided in the abstract relating to the optimisation procedure in Section 2.3.