From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper6127/Area_Chair_cfv6
metareview: This work proposes a new re-weighting method based on OT: it views the training set as an imbalanced distribution over its samples and transported by OT to a balanced distribution obtained from the meta set. The paper is well written, and the proposed method is verified on multiple image and text datasets.

Most of the comments were addressed by the authors in the discussion period. Three reviewers raised their scores. 

The major remaining concern is regarding the relationship/novelty w.r.t. [1]. As clarified by the author, the problems, algorithms, tasks, and datasets are different. The novelty of this work is also supported by most reviewers. 

-- 4F4H: "Re-weighting from optimal transport (OT) perspective is novel."

-- DJ7Q: "The idea of sample-reweighting is not new. But the method proposed by the author that learning the sample weights by minimizing the OT distance between the imbalanced training set and a balanced meta set seems reasonably novel."

-- 8X8s: "It is a very interesting idea to reweight the training samples based on OT in the training phase, this approach is therefore more flexible than the previous OT-based approaches to long-tailed problems. Whereas previous approaches have involved adjustments to the model's results in the post-processing stage, the authors go further and apply OT to the training stage and propose a corresponding solution. "

LeMk also raised the score to 4 due to "the contribution of this paper includes using OT loss in class imbalance and empirical contributions on many datasets". 

Based on the above, the work meets the bar of a NeurIPS publication from the novelty point of view, and the AC recommends acceptance.

In the final version, please clearly discuss the position and contribution of this work w.r.t [1] based on LeMk's final comments.

-- "the two papers are the same in high-level thinking but different in the tasks. [1] considers a more general setting of distribution shift and this paper focuses on class imbalance."

-- “The authors said the key difference is that this paper disengages the dependence of the weight learning on the concerned classifier while [1] does not.... in [1] the weight learning is independent of the classifier when using hidden-layer-output transformation as the non-linear transformation of data. I would suggest the authors carefully check this point to avoid any potential misunderstanding about the position of this paper with related work.”

[1] Influence-Balanced Loss for Imbalanced Visual Classification, ICCV, 2021.
recommendation: Accept
confidence: Certain
award: No

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer LeMk 
comment: Thanks again for your time. Previously, you gave a relatively low rating to our work and raised some doubts. We have provided clarification and additional experimental results in our response and revision, which we hope can address your concerns. Please let us know if you have any further concerns.  We notice that some of the reviewers have increased their ratings after seeing our response. We would appreciate it if you could kindly take our response and other reviewers' comments into consideration.

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer 8X8s
comment: Thanks for your time and comments.

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer 4F4H 
comment: Thank you very much!

From: NeurIPS.cc/2022/Conference/Paper6127/Reviewer_4F4H
title: Re) Response to Reviewer 4F4H 
comment: Thank you for the response.
After reading the response and the revised version of the manuscript, most of my concerns are clarified.
Thus, I increased the initial rating to 6.

From: NeurIPS.cc/2022/Conference/Paper6127/Reviewer_8X8s
title: Response to authors
comment: Thank you for your response and clarification.

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer 8X8s Part 2
comment: Q3:  Thanks for your suggestion and these are interesting directions. To the best of our knowledge, our proposed OT loss is a plug-and-play loss and may be available for handling the imbalance segmentation and detection tasks without significant changes. Besides, we are not sure whether our proposed method is applied to the imbalanced regression task due to the continuous output in the regression task.

Taking the image detection as an example, recall that the proposal feature maps are extracted (Roi Pooling operation) in the original Faster-RCNN and would be passed to the following full-connected layer for final classification. We could adapt our method here to re-weight the extracted feature maps by introducing our OT loss function. Therefore, in the final classification, the small-size objects and tailed class objects could be assigned large weights to achieve balanced classification. What is different from the image classification task is that, here, we need to design a suitable meta dataset. We believe that, some prior information in the detection task would be helpful for building this meta dataset. Similarly, our method could be adopted in image segmentation by learning and assigning a suitable weight at pixel level before final pixel-wise classification. 

Q4:  Thanks for your suggestion. Considering some baselines did not report the variance of the experiments, we have added the std about our proposed method in our revision. 

Q5-7: Thanks for your reminding. We have moved the Related work right behind the introduction.  We have fixed some typos in our revision. We will improve the presentation within the whole manuscript in the camera ready.

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer 8X8s Part 1
comment: Thank you for your time. The typos, relevant missing references and baselines have been cited, discussed and compared in the revised version and the major changes are marked in blue. 
 
Q1: Thanks for your references, which have been cited and discussed in the Related work of our revision. We agree with you that the technique of re-weighting with OT is related to methods in many domains. However, we find that ours are different from these methods, which will be described below in detail.


Reference [1] is developed to address the problem of unsupervised domain adaptation under the setting of generalized target shift (joint class-conditional and label shifts). Although [1] uses the OT distance and re-weighting strategy, there are some fundamental differences between [1] and ours. First, we develop a novel automatic re-weighting method for imbalanced classification while [1] focuses on unsupervised domain adaptation. Second, [1] is proposed to re-weight each class within the source domain by estimating target label proportion, which aims to correctly match class-conditional probabilities in source and target domain by optimal assignment; while ours aims to re-weight each sample in the training set and learn it by directly minimizing the OT distance between a to-be-learned distribution and another balanced distribution. Let alone, we design different cost functions and additionally propose an amortized approach for the learning of weights. 
 
 

The Multi-Source Domain Adaptation via Weighted Joint Distribution Optimal Transport (MSDA-WJDOT) in Reference [2], addresses the problem of domain adaptation on an unlabeled target dataset using knowledge from multiple labeled source datasets. Specifically, it introduces a domain-level weight vector to measure the importance of each source domain on the target domain and aims at finding simultaneously an Optimal Transport-based alignment between the source and target distributions and a re-weighting of the sources distributions. Our method is proposed to solve the imbalanced classification problem by representing the training set as a weighted distribution, where the sample-level weight vector is learned by enforcing the to-be-learned distribution to stay close to another balanced distribution. Different from MSDA-WJDOT more concerned about the learned transport plan between target and source distribution, ours aims to re-weight the training loss by the learned weights to optimize a shared classifier. 

The Reference [3], which is published in the Electronic Journal of Statistics on May 16, 2022,  studies an optimal re-weighting that minimizes the Wasserstein distance between the empirical measures of the two samples, leading to an expression of the weights in terms of Nearest Neighbors. Reference [3] considers the Nearest Neighbor regression under covariate shift while we focus on the imbalanced classification problem, where our motivation is disengaging the dependence of the weight learning on the concerned classifier at each training stage by introducing the OT distance. Besides, we also propose to optimize the weights using different cost functions and develop an amortizing way by constructing a weight net. 

Although our proposed method is related to these references for using OT distance and re-weighting strategy, they are different methods in terms of task and technical details.

Q2:  Thanks for your nice suggestions. Please note that our proposed method falls into the group of re-weighting. Thus, our focus is mainly placed on the comparison to the related re-weighing methods. Further, for standardized image benchmarks, our baselines are mainly those works published in 2018 to 2021. For text datasets, we also introduce the baseline published in ICLR 2022. Notably, we have added the comparison between ours and Influence-Balanced Loss (2021) in our revision. If the reviewer can kindly help point out any specific paper that we need to compare with, we would appreciate it very much and be very happy to do so in the next version.

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer DJ7Q
comment: Thank you for your time. The typos, relevant missing references and baselines have been cited, discussed and compared in the revised version and the major changes are marked in blue. 


Q1: Thanks for your comments. We had explained the reason in submitted version (see Line 36-40 in Introduction.)

Although these existing re-weighting methods have received widespread attention, most may be limited to optimizing the weights by the classification loss on the meta set. Specifically, at each training stage, there are three steps: (a) Firstly, they usually optimize the classifier (and also feature extractor) using the classification loss on the training set and the corresponding weights, where the updated classifier depends on the to-be-learned weights; (b) Secondly, they optimize the weights using the classification loss on the meta-set, where the updated weights consequently rely on the to-be-learned classifier; (c) Thirdly, the updated weights in step (b) are further used to optimize the classifier. Therefore, we can find that the gradient of weights is coupled with the to-be-learned classifier at each training stage. However, a point needing to pay attention to is that the classifier is updated at each training step as the primary concern in the imbalanced issue. Hence, the dependence of weights on the classifier at the training stage may lead to inaccurate learning of the weights.

Q2: Thanks for your comments. Most existing re-weighting methods optimize the weights using the classification loss on the meta-set, indicating the undesired dependence of weights on the to-be-learned classifier at the training stage. Our proposed method bypasses this problem by viewing the learning of weights as the distribution approximation. Specifically, we minimize the statistical distance between the to-be-learned distribution over samples from the imbalanced training set and another balanced distribution over samples from the meta set. Optimizing the weights based on the OT loss instead of the classification loss on meta set, our proposed method disengages the dependence of the weight learning on the concerned classifier at each iteration, producing desired performance.


Q3:Thanks for your suggestion. We had reported the running time of our proposed method on different datasets in Appendix. Compared with the cross-entropy that  optimizes the classification loss on the meta-set, our proposed method has an acceptable time complexity with a better performance. Following your suggestion, we also discuss the computational complexity of optimizing the OT distance the in Appendix of our revision, described here for clarity. The optimization of the OT distance between discrete distributions of size n$, is a linear program and can be solved with combinatorial algorithms such as the simplex methods, which dampens the utility of the method when handling large datasets. Recently, the regularization scheme proposed by Cuturi in Ref [1] allows a very fast computation of a transportation plan, denoted as Sinkhorn algorithm. The Sinkhorn algorithm requires the computational cost of n^{2} \log (n)/ \varepsilon^{2} to reach \varepsilon-accuracy, which is adopted by us to solve the OT problem.

Reference [1]: M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transportation,” in Proc. Advances Neural Inf. Process. Syst., 2013, pp. 2292–2300.

Q4: Thanks for bringing these references to us. We have added the comparisons of ours with IB, IB+CB, and IB+ Focal loss on CIFAR-LT-10 and CIFAR-LT-100, and the comparison of ours with DisAlign on Places-LT in our revision. We can find that ours outperforms IB-based methods and DisAlign, proving its effectiveness.

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer LeMk
comment: Thank you for your time. The typos, relevant missing references and baselines have been cited, discussed and compared in the revised version and the major changes are marked in blue. 
Q1 (a): Thanks for your comments. Although using w = p_{meta}(y)/p_{train}(y) is simple, conceptually, it assigns each sample in a class with the same weight, ignoring the differences among samples within this class. Besides, please refer to Table 3 in our revision paper, where we report the performance of computed weight on the Places-LT dataset as suggested. We can see that this weight has a poor performance and only outperforms the original cross-entropy (i.e., without any way to solve the imbalance issue). Therefore, developing an effective method to learn different sample weights is quite necessary. 
Q1 (b):  Thanks for the reference. We believe that in terms of high-level thinking, the paper is related. But there are fundamental differences between [1]] and ours. Specifically, 

1. Reference [1] and ours focus on different tasks. Our proposed method deals with the imbalanced classification tasks, such as text, image, and 3-dimensional point cloud while Fang et al in [1] performed the label-noise and class-prior-shift experiments on image datasets. [1] mostly focus on distribution shift in image classification, thus, whether the method in [1] can be applied in the imbalanced problems and datasets (text, image, and point cloud) studied in our paper is untested, while our effort on making OT work effectively in these problems is nontrivial.

2. Besides, we view the training set as an imbalanced distribution over its samples, which is transported by OT to a balanced distribution obtained from the meta set. The weights of the training samples are the probability mass of the imbalanced distribution and are learned by minimizing the OT distance between the two distributions. Designing in this way, our proposed method provides a more flexible way of learning the weights of samples. Firstly, the MMD used by [1] uses the kernel when computing the distance between features from two distributions; while our proposed method avoids the kernel and provides two flexible definitions of the cost functions due to the introduction of OT distance. Secondly, if we understand [1] in the right way, the classifier combined with the feature extractor is used to specify the non-linear transformation function of data after a careful discussion; while ours does not introduce the to-be-learned classifier into the cost function in OT loss, which is our key point for disengaging the dependence of the weight learning on the concerned classifier at each iteration. Besides, our proposed method still work effectively even with the label-aware cost, where the OT loss is influenced by neither feature extractor nor classifier. Therefore, these above-mentioned points distinguish our proposed method from the method in [1]. Please refer to Section F in the Appendix of our revision paper to check the label-aware experiment results, where we only use label information in meta-set to optimize the sample weights.

Q1 (c):  Thanks for your comments. The L2RW + RANDOM result is summarized in Tab 3. of our revision. Actually, the L2RW + RANDOM method only performs well in the noisy label setting in L2RW, where L2RW + RANDOM in imbalance classification setting is only a little better than the baseline method. Notably, the classification performance  of L2RW + RANDOM and ours is 32.23 and 39.68 respectively. This result also demonstrates the success in the noisy label task does not necessarily mean success in the imbalanced classification task.

Q2: Thanks for your reference, which has been cited in our revision. Actually, after a careful reading of the reference, we find that MentorNet and ours are totally different in tasks, motivations, and the proposed approaches. More specifically,  MentorNet is proposed to overcome the overfitting of deep networks on corrupted labels, which is used to supervise the training of the base deep networks, namely, StudentNet. During training, MentorNet provides a curriculum (sample weighting scheme) for StudentNet to focus on the sample the label of which is probably correct. However, our proposed method aims to deal with the imbalanced classification task by learning the weights of samples in the training set. Although we additionally provide an amortized method to learn the weights by constructing an explicit weight net to output the weights of samples, optimizing weights directly is also available in our submitted version. In summary, we aim to learn the weights for the imbalanced classification tasks and solve this problem by viewing the learning of weights as the distribution approximation problem, which is hence naturally optimized by OT distance.

From: NeurIPS.cc/2022/Conference/Paper6127/Authors
title: Response to Reviewer 4F4H
comment: Thank you for your time. The typos, relevant missing references and baselines have been cited, discussed and compared in the revised version and the major changes are marked in blue. 
Q1: Thanks for your comments. Although re-weighting methods (L2RW and its followers) have received widespread attention, most of them may be limited to optimizing the weights by the classification loss on the meta set. As we analyzed in the submission, the gradient of weights is usually coupled with the to-be-learned classifier at each training iteration. Considering classifier is the major concern in imbalanced issue, the dependence of weights on classifier at the training stage may lead to inaccurate learning or low convergence of the weights. Thus, our motivation is to disengage the dependence of weights on the classifier at the training stage. In other words, we aim to seek another loss on the meta set to optimize weights, where this loss function is expected to be classifier-independent. Following most of re-weighting methods using a balanced meta set, we view the learning of weights as the task of distribution approximation, where we formulate the training set as the to-be-learned distribution and enforce it to approximate the balanced meta set for learning the weights of samples. Designed in this way, our proposed method can learn the effective weights, whose gradient no longer relies on the to-be-learned classifier, which can produce a good performance and accelerate convergence. 
 
Q2: Thanks for your comments.  Please note that our proposed method falls in the group of re-weighting. Therefore, we mainly focus on considering the work of re-weighting the samples as our baselines. Among them, most methods adopt the classification loss on the meta set to learn the weight vector during the training stage, including L2RW, Meta-Weight-Net, Meta-class-weight, and MetaSAug. Following them, We build the meta-set in the same way (i.e., using a balanced validation set) and learn the weight vector by optimizing our proposed  OT loss instead of the classification loss on the meta set during the training stage. Thus, we believe the comparison between ours and these related methods is fair in terms of the use of the meta set.

To further explore the effect of meta-set on the performance in the long-tailed classification task, we additionally build a meta-set directly from the given imbalanced training set, which is available for all methods. The experimental results of this meta-set are summarized in Section E in the Appendix of our revision. We report the results in SST-2, a text classification dataset, due to the limited rebuttal time. We find that our proposed framework is very robust to the meta-set. Even with the meta-set built on the imbalanced training set, our proposed method  is still effective. That is to say, our method's effectiveness is quite robust to the selection of the meta set. We will provide more results of the meta-set from a given imbalanced training set instead of the validation set on other datasets.

Q3: Thanks for your references, which have been cited in our revision. Since both our proposed method and Influence-Balanced Loss (IB) in [1] aim to re-weighting samples, we further compare the performance between IB (also IB + Focal loss and IB + Class Balanced loss) and ours on CIFAR-LT-10 and CIFAR-LT-100 , and the performance of Balanced Meta-Softmax and ours on Places-LT in our revision. We can find ours outperforms IB-based methods and Balanced Meta-Softmax, showing the effectiveness of our proposed method.

From: NeurIPS.cc/2022/Conference/Paper6127/Reviewer_4F4H
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: Imbalanced classification is a significant problem that many real-world datasets are imbalanced and the model trained on these datasets can yield poor performance. To tackle this problem, this paper proposes a novel re-weighting method using optimal transport (OT). Using a balanced meta set, the weight vectors are learned by minimizing the distance between an imbalanced training set and balanced meta set. This paper demonstrates the effectiveness on multiple benchmark datasets including image and text classification.

strengths_and_weaknesses: Strengths
- The idea of minimizing the distance between an imbalanced training set and a balanced meta set is intuitive, and easy to understand. 
- Re-weighting from optimal transport (OT) perspective is novel.
- The proposed method shows good performance on multiple benchmark datasets, including image and text classification. 
- The paper is easy to follow.

Weaknesses
- The motivation of using OT for re-weighting is not fully convincing. What other automatic re-weighting methods have not been able to solve? And so, how OT can benefit ‘re-weighting’? 
- Using a balanced meta-set seems unfair compared with other methods which do not use a balanced meta-set. For example, ten training samples per class are selected from the CIFAR dataset. In the case of IF=100, the least frequent class has only 5 samples on CIFAR-100-LT. It means, the meta-set has more samples than the training set, which can give a significant information for the minority classes. For fair comparison, in my opinion, other methods should also additionally use the meta-set in their training. 
- This paper missed some important recent baselines. For example, [1] automatically re-weights samples using the gradients and does not require a meta-set. [2] proposed Balanced Softmax to fix the label distribution shift between training and testing. 

[1] Influence-Balanced Loss for Imbalanced Visual Classification, ICCV, 2021.

[2] Balanced meta-softmax for long-tailed visual recognition, NeurIPS, 2020.

questions: Please refer to the weaknesses.
limitations: The authors addressed the limitations and potential negative societal impact.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper6127/Reviewer_LeMk
rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: In this paper, the authors proposed a reweighting method for class imbalance classification based on optimal transport. Experiments on both image and text data demonstrate the effectiveness of the proposed method. 
strengths_and_weaknesses: Strengths:
1. The paper is clear and well-written.
2. The paper provides lots of empirical evidence to demonstrate the effectiveness of the proposed method.

Weaknesses:
1. This paper looks quite similar to an existing work [1]. 
- In [1], class imbalanced problem (i.e., called class-prior-shift in [1]) is studied as a case study under the same setting (i.e., also using a small balanced meta set), and according to importance weighting, the optimal weights can be computed directly (i.e., w = p_{meta}(y)/p_{train}(y)). I'm wondering why not use this simple method to compute weights if such a balanced meta set is available? 
- Also, compared with [1], the core difference may be that this paper learns weights by minimizing an OT distance while [1] is by minimizing a maximum mean discrepancy (MMD). What is the critical contribution of this paper?
- In L2RW, their baseline RANDOM (i.e, using random weights from a rectified Gaussian distribution) is quite strong. It would be interesting to compare the proposed method with the RANDOM baseline. 

2. There is another crucial work worth to be referred and discussing in this area [2].

References:

[1] Fang et al. "Rethinking importance weighting for deep learning under distribution shift." In NeurIPS 2020.  

[2] Jiang et al. "Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels." In ICML 2018.

questions: 1. I'm wondering if the method proposed in this paper can handle other settings, e.g., label noise as in the baseline L2RW?
2. In Stage1 of the proposed method, if t1 is large, would it cause an overfitting issue by pre-training on an unweighted imbalanced dataset? How to decide the value of this parameter in practice? 

--------------Post-rebuttal comments--------------

Thank the authors very much for the rebuttal and the additional results!

After carefully reading the rebuttal, some of my concerns are resolved by the additional experimental results of baselines. But my major concern still remains, i.e., the novelty of this paper may not be so strong given [1] as prior work. I agree with the authors that the two papers are the same in high-level thinking but different in the tasks. [1] considers a more general setting of distribution shift and this paper focuses on class imbalance. Also, the contribution of this paper includes using OT loss in class imbalance and empirical contributions on many datasets. I will slightly raise my score for this and the additional experimental results. 

However, the authors said the key difference is that this paper disengages the dependence of the weight learning on the concerned classifier while [1] does not. This may not be true. If I understand correctly, in [1] the weight learning is independent of the classifier when using hidden-layer-output transformation as the non-linear transformation of data. I would suggest the authors carefully check this point to avoid any potential misunderstanding about the position of this paper with related work.



limitations: It seems that the authors haven't adequately addressed the limitations and potential negative societal impact of the work.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper6127/Reviewer_DJ7Q
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper argues that the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset and the balanced test set. ​Besides, the general optimizing the weights by the classification loss may lead to inaccurate learning of the weights. Therefore, the authors formulate the learning of weight vector as the distribution approximation problem by minimizing the OT distance between to-be-learned distribution of samples in imbalanced training set and another balanced meta set. To show the effectiveness of the proposed method, they conduct extensive experiments on imbalanced classification tasks including image, text, and point cloud. 
strengths_and_weaknesses: Pros:
1. The idea of sample-reweighting is not new. But the method proposed by the author that learning the sample weights by minimizing the OT distance between the imbalanced training set and a balanced meta set seems reasonably novel.
2. The paper is well organized and easy to follow.

Cons:
1. ​Why does the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights? A further explanation is required to highlight the reasonability of the proposed method.
2. The reasonability of the statement that "the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset and the balanced test set" also needs further consideration or explanation.
3. An analysis or discussion about the computation complexity of optimizing the OT distance is necessary.
questions: 1. ​Why does the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights?
2. Why are the weights obtained when the imbalanced training set is approximated to a balanced meta data better for training a classifier?
3.  An analysis or discussion about the computation complexity of optimizing the OT distance is necessary.
4. Missing the comparison with the highly related IB (Influence-balanced loss for imbalanced visual classification, ICCV 2021) and DisAlign (Distribution alignment: A unified framework for long-tail visual recognition, CVPR 2021.).
5. Typos:
1) $\theta$ in Eq.(2) should be bolded.
2) In Eq.(10), should $\eta$-->$\lambda$?
limitations: A hold-out balanced meta-data is required.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper6127/Reviewer_8X8s
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: The authors propose a new sampling method to cope with imbalance learning, which is based on optimal transport. The advantages of the method are illustrated on several different types of datasets. Unlike the traditional reweight approach, the authors look at the problem from the perspective of distribution matching, which leverages the OT distance between the distributions to guide the learning of weight vector or weight net.
strengths_and_weaknesses: It is a very interesting idea to reweight the training samples based on OT in the training phase, this approach is therefore more flexible than the previous OT-based approaches to long-tailed problems. Whereas previous approaches have involved adjustments to the model's results in the post-processing stage, the authors go further and apply OT to the training stage and propose a corresponding solution. The authors' experiments are well developed and they validate the superiority of their approach on multiple data sets.

In addition to the above mentioned points, I have some concerns as follows:
1) The technique presented by the authors for reweighting with OT actually proposes a similar approach in many domains. For example, the usage of OT to address domain adaptation under conditional domain matching and label shift is proposed in [1]. Similarly, reweighting of samples using OT is described in [1]. A similar approach is suggested in [2] and [3]. I hope the author can explain the difference between them.
2) The authors do not seem to be comparing their experiments with the most advanced methods, and as far as I can tell, it seems that many recently published methods are better than the results in the authors' experiments. I wish the authors had compared their experiments with more methods than the ones in the paper.
3) Also, it seems that the author's method can only be applied to classification. I wonder if the author's method can cope with imbalance in object detection, or object segmentation, and also if the method can be extended to imbalance regression? The authors should explain more about the limitations of this work.

From the results on CIFAR and ImageNet, some of the authors' results seem to have very weak boosts, for example, almost less than 1%. The authors' method seems to be limited in terms of experimental effectiveness, and I would like to see the authors take multiple experiments and average them, and report the variance of the experiments. This is what the NeurIPS conference promotes the need to report. Of course, I note that the authors state that these values are reported for text classification, but I think the authors should report these values for all experiments.

In Table 2, the author has conducted comparative experiments on CIFAR-100 to compare which cost matrix is better, but some of the results do not appear to be very different from each other, so it is difficult to derive any significant differences between these different methods in terms of the results.

The writing of the paper also needs further improvement, some issues are as follows:
1) The description of OT is very inadequate, which is very difficult to understand for many people from outside this field. I also hope that the authors will explain how Sinkhorn can be used to solve OT problems.
2) The author should present Related Work after Introduction to make it easier for others to learn about related work in this field.
3) There is some confusion in the author's notation format, e.g. for scalars, vectors, sets, parameters, etc. The author should have followed some notational conventions. It will make the text look more rigorous.

In line 313, "... indicates the effectiveness" should be "indicate the effectiveness".



[1] Optimal Transport for Conditional Domain Matching and Label Shift.

[2] Multi-source Domain Adaptation via Weighted Joint Distributions Optimal Transport.

[3] Reweighting samples under covariate shift using a Wasserstein distance criterion.


questions: See above.
limitations: See above.
ethics_flag: No
soundness: 2 fair
presentation: 3 good
contribution: 2 fair