Original: L ONG - TAIL LEARNING VIA LOGIT ADJUSTMENT
Revised: L ONG - TAIL LEARNING VIA LOGIT ADJUSTMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Real-world classiﬁcation problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few  samples.
Revised: Real-world classiﬁcation problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels have only a few associated samples.

Original: This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels.
Revised: This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels.

Original: In this paper, we present two simple modiﬁcations of standard softmax cross-entropy training to cope with these challenges.
Revised: In this paper, we present a statistical framework that uniﬁes and generalises several recent proposals to cope with these challenges.

Original: Our techniques revisit the classic idea of logit adjustment based on the label frequencies,  either applied post-hoc to a trained model, or enforced in the loss during training.Such adjustment encourages a large relative margin between logits of rare positive labels versus dominant negative classes.
Revised: Our framework revisits the classic idea of logit adjustment based on the label frequencies, which encourages a large relative margin between logits of rare positive versus dominant negative labels. This yields two techniques for long-tail learning, where such adjustment is either applied post-hoc to a trained model, or enforced in the loss during training.

Original: These techniques unify and generalise several recent proposals in the literature, while possessing stronger statistical grounding and empirical performance on several real-world datasets.
Revised: These techniques are statistically grounded, and practically effective on four real-world datasets with long-tailed label distributions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Real-world classiﬁcation problems typically exhibit a long-tailed label distribution, wherein most labels are associated with only a few samples (Van Horn & Perona, 2017; Buda et al., 2017; Liu et al., 2019).
Revised: Real-world classiﬁcation problems typically exhibit a long-tailed label distribution, wherein most labels are associated with only a few samples (Van Horn & Perona, 2017; Buda et al., 2017; Liu et al., 2019).

Original: Owing to this paucity of samples, generalisation on such labels is challenging; moreover, naïve learning on such data is susceptible to an undesirable bias towards dominant labels.
Revised: Owing to this paucity of samples, generalisation on such labels is challenging; moreover, naïve learning on such data is susceptible to an undesirable bias towards dominant labels.

Original: This problem has been widely studied in the literature on learning under class imbalance (Kubat et al., 1997; Chawla et al., 2002; He & Garcia, 2009) and the related problem of cost-sensitive learning (Elkan, 2001).
Revised: This problem has been widely studied in the literature on learning under class imbalance (Kubat et al., 1997; Chawla et al., 2002; He & Garcia, 2009), and the related problem of cost-sensitive learning (Elkan, 2001).

Original: Recently, long-tail learning has received renewed interest in the context of neural networks.
Revised: Recently, long-tail learning has received renewed interest in the context of neural networks.

Original: Two active strands of work involve post-hoc normalisation of the classiﬁcation weights (Zhang et al., 2019;
Revised: Two active strands of work involve post-hoc normalisation of the classiﬁcation weights (Zhang et al., 2019;

Original: Kim & Kim, 2019; Kang et al., 2020; Ye et al., 2020), and modiﬁcation of the underlying loss to account for varying class penalties (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al., 2020).
Revised: Kim & Kim, 2019; Kang et al., 2020; Ye et al., 2020), and modiﬁcation of the underlying loss to account for varying class penalties (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al., 2020).

Original: Each of these strands is intuitive, and has proven empirically successful.
Revised: Each of these strands are intuitive, and have proven empirically successful.

Original: However, they are not without limitation: e.g., weight normalisation crucially relies on the weight norms being smaller for rare classes; however, this assumption is sensitive to the choice of optimiser (see §2).
Revised: However, they are not without limitation: e.g., weight normalisation crucially relies on the weight norms being smaller for rare classes; however, this assumption is sensitive to the choice of optimiser (see §2).

Original: On the other hand, loss modiﬁcation sacriﬁces the consistency that underpins the canonical softmax cross-entropy (see §5.1).
Revised: On the other hand, loss modiﬁcation sacriﬁces the consistency that underpins the canonical softmax cross-entropy (see §5.1).

Original: Consequently, such techniques may prove suboptimal even in simple settings (see §6.1).
Revised: Consequently, such techniques may prove suboptimal even in simple settings (see §6.1).

Original: In this paper, we present two simple modiﬁcations of softmax cross-entropy training that unify several of the above proposals, and overcome their limitations.
Revised: In this paper, we establish a statistical framework for long-tail learning that offers a uniﬁed view of post-hoc normalisation and loss modiﬁcation techniques, while overcoming their limitations.

Original: Our techniques revisit the classic idea of logit adjustment based on label frequencies (Provost, 2000; Zhou & Liu, 2006; Collell et al., 2016), applied either post-hoc on a trained model, or as a modiﬁcation of the training loss. Conceptually, logit adjustment encourages a large relative margin between a pair of rare positivelabel and dominant negative class.
Revised: Our framework revisits the classic idea of logit adjustment based on label frequencies (Provost, 2000; Zhou & Liu, 2006; Collell et al., 2016), which encourages a large relative margin between a pair of rare positive and dominant negative labels. Such adjustment can be achieved by shifting the learned logits post-hoc, or augmenting the softmax cross-entropy with a pairwise label margin (cf.

Original: This has a ﬁrm statistical grounding: unlike recent techniques, it is consistent for minimising the balanced error (cf.
Revised: While similar in nature to recent techniques, our logit adjustment approaches additionally have a ﬁrm statistical grounding: they are Fisher consistent for minimising the balanced error (cf.

Original: (2)), a common metric in long-tail settings which averages the per-class errors.
Revised: (2)), a common metric in long-tail settings which averages the per-class errors.

Original: This  grounding translates into strong empirical performance on  real-world datasets.
Revised: This statistical grounding translates into strong empirical performance on four real-world datasets with long-tailed label distributions.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: (i) we  present two realisations of logit adjustment for long-tail learning, applied either post-hoc (§4.1 or during training (§5.1) (ii) we establish that logit adjustment overcomes limitations in recent proposals (see Table 1), and in particular is Fisher consistent for minimising the balanced error(cf. (2)); (iii) we conﬁrm the efﬁcacy of the proposed  techniques  on  real-world datasets(§6). In the course of our analysis, we present an extension of the softmax cross-entropy with a pairwise label margin (cf. (11)), which offers control of the relative contribution of labels to the overall loss.
Revised: (i) we establish a statistical framework for long-tail learning (§3) based on logit adjustment that provides a uniﬁed view of post-hoc correction and loss modiﬁcation (ii) we present two realisations of logit adjustment, applied either post-hoc (§4.1) or during training (§5.1); unlike recent proposals (Table 1), these are consistent for minimising the balanced error (iii) we conﬁrm the efﬁcacy of the proposed logit adjustment techniques compared to several baselines on four real-world datasets with long-tailed label distributions (§6).

Original: [Table]
Revised: [Table]

Original: Reference
Revised: Reference

Original: [Table]
Revised: [Table]

Original: × (Kang et al., 2020)
Revised: × (Kang et al., 2020)

Original: [Table]
Revised: [Table]

Original: Softmax with rare +
Revised: Softmax with rare +

Original: [Table]
Revised: [Table]

Original: upweighting × (Cao et al., 2019)
Revised: upweighting × (Cao et al., 2019)

Original: [Table]
Revised: [Table]

Original: Softmaxwith
Revised: Softmax

Original: [Table]
Revised: [Table]

Original: -ve downweighting × (Tan et al., 2020)
Revised: downweighting × (Tan et al., 2020)

Original: [Table]
Revised: [Table]

Original: Post
Revised: Post

Original: [Table]
Revised: [Table]

Original: hoc logit
Revised: hoc logit

Original: [Table]
Revised: [Table]

Original: X
Revised: X

Original: This paper (cf.
Revised: This paper (cf.

Original: [Table]
Revised: [Table]

Original: Softmax with logit
Revised: Softmax with logit

Original: [Table]
Revised: [Table]

Original: X
Revised: X

Original: This paper (cf.
Revised: This paper (cf.

Original:  P ROBLEM SETUP AND RELATED WORK
Revised:  P ROBLEM SETUP AND RELATED WORK

Original: Consider a multiclass classiﬁcation problem with instances X and labels Y =
Revised: Consider a multiclass classiﬁcation problem with instances X and labels Y =

Original: [ L ] .
Revised: [ L ] .

Original: = { 1 , 2 , . . .
Revised: = { 1 , 2 , . . .

Original: , L } .
Revised: , L } .

Original: Given a sample S
Revised: Given a sample S

Original: =
Revised: =

Original: { ( x n , y n ) } Nn =1 ∼ P N for unknown distribution P over X × Y , our goal is to learn a scorer f : X → R L that minimises the misclassiﬁcation error P x,y
Revised: { ( x n , y n ) } Nn =1 ∼ P N for unknown distribution P over X × Y , our goal is to learn a scorer f : X → R L that minimises the misclassiﬁcation error P x,y

Original: [Equation]
Revised: [Equation]

Original: Typically, one minimises a surrogate loss ‘ : Y × R L → R, such as the softmax cross-entropy, ‘ ( y, f ( x )) = log[Equation] i
Revised: Typically, one minimises a surrogate loss ‘ : Y × R L → R such as the softmax cross-entropy, ‘ ( y, f ( x )) = log

Original: [Equation]
Revised: [Equation]

Original: For p y ( x ) ∝ e f y ( x ), we may view p ( x ) . = [ p 1 ( x ) , . . . , p L ( x )] ∈ ∆ L as an estimate of P ( y | x ) .
Revised: We may view the resulting softmax probabilities p y ( x ) ∝ e f y ( x ) as estimates of P ( y | x ) .

Original: The setting of learning under class imbalance or long-tail learning is where the distribution P ( y ) is highly skewed, so that many rare (or “tail”) labels have a low probability of occurrence.
Revised: The setting of learning under class imbalance or long-tail learning is where the distribution P ( y ) is highly skewed, so that many rare (or “tail”) labels have a low probability of occurrence.

Original: Here, the misclassiﬁcation error is not a suitable measure of performance: a trivial predictor which classiﬁes every instance to the majority label will attain a low misclassiﬁcation error.
Revised: Here, the misclassiﬁcation error is not a suitable measure of performance: a trivial predictor which classiﬁes every instance to the majority label will attain a low misclassiﬁcation error.

Original: To cope with this, a natural alternative is the balanced error (Chan & Stolfo, 1998; Brodersen et al., 2010; Menon et al., 2013), which averages each of the per-class error rates: [Equation]
Revised: To cope with this, a natural alternative is the balanced error (Chan & Stolfo, 1998; Brodersen et al., 2010; Menon et al., 2013), which averages each of the per-class error rates: under a uniform label distribution

Original: This can be seen as implicitly using a balanced class-probability function P bal ( y | x ) ∝

Original: [Equation]
Revised: [Equation]

Original: L · P

Original: [Equation] x | [Equation] ) , as opposed to the native P ( y | x ) ∝ P ( y ) · P ( x | y ) that is employed in the misclassiﬁcation error.
Revised: This can be seen as implicitly using a balanced class-probability function P bal ( y | x ) ∝ 1 L · P ( x | y ) , as opposed to the native P ( y | x ) ∝ P ( y ) · P ( x | y ) that is employed in the misclassiﬁcation error.

Original: Broadly, extant approaches to coping with class imbalance modify:
Revised: Broadly, extant approaches to coping with class imbalance modify:

Original: (i) the inputs to a model, for example by over- or under-sampling (Kubat & Matwin, 1997; Chawla et al., 2002; Wallace et al., 2011; Mikolov et al., 2013; Mahajan et al.,2018;
Revised: (i) the inputs to a model, for example by over- or under-sampling (Kubat & Matwin, 1997; Chawla et al., 2002; Wallace et al., ; Mikolov et al., ; Mahajan et al.,

Original:  Yin et al., 2018) (ii) the outputs of a model, for example by post-hoc correction of the decision threshold (Fawcett & Provost, 1996; Collell et al., 2016) or weights (Kim & Kim, 2019; Kang et al., 2020) (iii) the internals of a model, for example by modifying the loss function (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al.,  2020).
Revised: ; Yin et al., 2018); (ii) the outputs of a model, for example by post-hoc correction of the decision threshold (Fawcett & Provost, 1996; Collell et al., 2016) or weights (Kim & Kim, 2019; Kang et al., 2020); or (iii) the training procedure of a model, for example by modifying the loss function (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al., 2020; Jamal et al., 2020).

Original: One may easily combine approaches from the ﬁrst stream with those from the latter two.
Revised: One may easily combine approaches from the ﬁrst stream with those from the latter two.

Original: Consequently, we focus on the latter two in this work, and describe some representative recent examples from each.
Revised: Consequently, we focus on the latter two in this work, and describe some representative recent examples from each.

Original: Post-hoc weight normalisation .
Revised: Post-hoc weight normalisation .

Original: Suppose f y ( x ) = w > y Φ( x ) for classiﬁcation weights w y ∈ R D and representations Φ: X → R D , as learned by a neural network.
Revised: Suppose f y ( x ) = w > y Φ( x ) for classiﬁcation weights w y ∈ R D and representations Φ: X → R D , as learned by a neural network.

Original: (We may add per-label bias terms to f y by adding a constant feature to Φ .)
Revised: (We may add per-label bias terms to f y by adding a constant feature to Φ .)

Original: A fruitful avenue of exploration involves decoupling of representation and classiﬁer learning (Zhang et al., 2019).
Revised: A fruitful avenue of exploration involves decoupling of representation and classiﬁer learning (Zhang et al., 2019).

Original: Concretely, we ﬁrst learn { w y , Φ } via standard training on the long-tailed training sample S , and then for x ∈ X predict the label
Revised: Concretely, we ﬁrst learn { w y , Φ } via standard training on the long-tailed training sample S , and then for x ∈ X predict the label

Original: [Equation]
Revised: [Equation]

Original: for τ > 0 , where ν y =
Revised: for τ > 0 , where ν y =

Original: P ( y ) in Kim & Kim (2019); Ye et al.
Revised: P ( y ) in Kim & Kim (2019); Ye et al.

Original: (2020) and ν y =
Revised: (2020) and ν y =

Original: k w y k 2 in Kang et al.
Revised: k w y k 2 in Kang et al.

Original: Further to the above, one may also enforce k w y k 2 = 1 during training (Kim & Kim, 2019).
Revised: Intuitively, either choice of ν y upweights the contribution of rare labels through weight

Original: [Figure]
Revised: [Figure]

Revised: ||

Original: [Figure]
Revised: [Figure]

Original: strongly correlated frequency, while
Revised: y ||

Original: Adam, norms are anti-correlated or of the
Revised: [Figure]

Original: Consequently, weight normalisation under Adam is ineffective for combatting class imbalance.
Revised: [Figure] || [Figure]

Original: Intuitively, either choice of ν y upweights the contribution of rare labels through weight normalisation .
Revised: y || [Figure] normalisation .

Original: The choice ν y = k w y k 2 is motivated by the observations that k w y k 2 tends to correlate with P ( y ) .
Revised: The choice ν y = k w y k 2 is motivated by the observations that k w y k 2 tends to correlate with P ( y ) .

Revised: Further to the above, one may enforce k w y k 2 = 1 during training (Kim & Kim, 2019).

Original: Loss modiﬁcation .
Revised: Loss modiﬁcation .

Original: A classic means of coping with class imbalance is to balance the loss, wherein ‘ ( y, f ( x )) is weighted by P ( y ) − 1 (Xie & Manski, 1989; Morik et al., 1999): for example,
Revised: A classic means of coping with class imbalance is to balance the loss, wherein ‘ ( y, f ( x )) is weighted by P ( y ) − 1 (Xie & Manski, 1989; Morik et al., 1999): e.g., applied to (1),

Original: [Equation]
Revised: [Equation]

Original: While intuitive, balancing has minimal effect in separable settings: solutions that achieve zero training loss will necessarily remain optimal even under weighting (Byrd & Lipton, 2019).
Revised: While intuitive, balancing has minimal effect in separable settings: solutions that achieve zero training loss will necessarily remain optimal even under weighting (Byrd & Lipton, 2019).

Original: Intuitively, one wouldlike instead  to shift the separator closer to a dominant class.
Revised: Intuitively, one would instead like to shift the separator closer to a dominant class.

Original: Li et al.
Revised: Li et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: Masnadi-Shirazi & Vasconcelos (2010) thus proposed to add per-class margins into the hinge loss.
Revised: Masnadi-Shirazi & Vasconcelos (2010) thus proposed to add per-class margins into the hinge loss.

Original: (Cao et al., 2019) similarly proposed to add a per-class margin into the softmax cross-entropy:
Revised: Cao et al. (2019) similarly proposed to add a per-class margin into the softmax cross-entropy:

Original: [Equation]
Revised: [Equation]

Original: where δ y ∝ P ( y ) − 1 / 4 .
Revised: where δ y ∝ P ( y ) − 1 / 4 .

Original: This upweights rare “positive” y to encourage a larger value of f y ( x ) − f y 0 ( x ) , i.e., the margin between y and any “negative” y 0 

Revised: This upweights rare “positive” y to encourage a larger gap f y ( x ) − f y 0 ( x ) , i.e., the margin between y and any “negative” y 0

Original: = y .
Revised: = y .

Original: Separately, Tan et al.
Revised: Separately, Tan et al.

Original: (2020) proposed
Revised: (2020) proposed

Original: [Equation]
Revised: [Equation]

Original: where δ y 0 ≤ 0 is an non-decreasing transform of P ( y 0 ) .
Revised: where δ y 0 ≤ 0 is an non-decreasing transform of P ( y 0 ) .

Original: Note that in the original softmax cross-entropy with δ y 

Revised: Note that in the original softmax cross-entropy with δ y

Original: = 0 , a rare label often receives a strong inhibitory gradient signal as it disproportionately appear as a negative for dominant labels.
Revised: = 0 , a rare label often receives a strong inhibitory gradient signal as it disproportionately appear as a negative for dominant labels.

Revised: This can be modulated by letting δ y 0 (cid:28) 0 .

Original: Limitations of existing approaches .
Revised: Limitations of existing approaches .

Original: Each of the above methods are intuitive, and have shown strong empirical performance.
Revised: Each of the above methods are intuitive, and have shown strong empirical performance.

Original: However, a closer analysis identiﬁes some subtle limitations.
Revised: However, a closer analysis identiﬁes some subtle limitations.

Original: Limitations of weight normalisation .
Revised: Limitations of weight normalisation .

Original: Post-hoc weight normalisation with ν y =
Revised: Post-hoc weight normalisation with ν y =

Original: k w y k 2 per Kang et al.
Revised: k w y k 2 per Kang et al.

Original: (2020) is motivated by the observation that the weight norm k w y k 2 tends to correlate with P ( y ) .
Revised: (2020) is motivated by the observation that the weight norm k w y k 2 tends to correlate with P ( y ) .

Original: However, this assumption is highly dependent on the choice of optimiser, as Figure 1 illustrates: for ResNet-32 models trained on long-tailed versions of CIFAR-10 and CIFAR-100, when using the Adam optimiser, the norms are either anti -correlated or independent of the class priors.
Revised: However, this assumption is highly dependent on the choice of optimiser, as Figure 1 illustrates: for ResNet-32 models trained on long-tailed versions of CIFAR-10 and CIFAR-100, when using the Adam optimiser, the norms are either anti -correlated or independent of P ( y ) .

Original: Weight normalisation thus cannot achieve the desired effect on rare labels.
Revised: Weight normalisation thus cannot achieve the desired effect of boosting rare labels’ scores.

Original: One may hope to side-step this by simply using ν y =
Revised: One may hope to side-step this by simply using ν y =

Original: P ( y ) ; unfortunately,even this choice has  limitations (see §4.2).
Revised: P ( y ) ; unfortunately, this choice has more subtle limitations (see §4.2).

Original: Limitations of loss modiﬁcation .
Revised: Limitations of loss modiﬁcation .

Original: Enforcing a per-label margin per (5) and (6) is intuitive, as it allows for shifting the decision boundary away from rare classes.
Revised: Enforcing a per-label margin per (5) and (6) is intuitive, as it allows for shifting the decision boundary away from rare classes.

Original: However, when doing so, it is important to ensure Fisher consistency (Lin, 2004) (or classiﬁcation calibration (Bartlett et al., 2006)) of the resulting loss for the balanced error.
Revised: However, when doing so, it is important to ensure Fisher consistency (Lin, 2004) (or classiﬁcation calibration (Bartlett et al., 2006)) of the resulting loss for the balanced error.

Original: That is, the minimiser of the expected loss (equally, the empirical risk in the inﬁnite sample limit) should result in a minimal balanced error.
Revised: That is, the minimiser of the expected loss (equally, the empirical risk in the inﬁnite sample limit) should result in a minimal balanced error.

Original: Unfortunately, both (5) and (6) are not consistent in this sense, even for binary problems; see §5.1, §6.1 for details.
Revised: Unfortunately, both (5) and (6) are not consistent in this sense, even for binary problems; see §5.1, §6.1 for details.

Original:  L OGIT ADJUSTMENT FOR LONG - TAIL LEARNING : A STATISTICAL VIEW
Revised:  L OGIT ADJUSTMENT FOR LONG - TAIL LEARNING : A STATISTICAL VIEW

Original: The above suggests that there is scope for improving performance on long-tail problems, both in terms of post-hoc correction and loss modiﬁcation.
Revised: The above suggests that there is scope for improving performance on long-tail problems, both in terms of post-hoc correction and loss modiﬁcation.

Original: We now show how a statistical perspective on the problem suggests simple procedures of each type, which overcome the limitations discussed above.
Revised: We now show how a statistical perspective on the problem suggests simple procedures of each type, which overcome the limitations discussed above.

Original: Recall that our goal is to minimise the balanced error (2).
Revised: Recall that our goal is to minimise the balanced error (2).

Original: A classical result is that the best possible or Bayes-optimal scorer for this problem, i.e., f ∗ ∈ argmin f : X → R L BER( f ) , satisﬁes the following (Menon et al., 2013),  (Collell et al., 2016, Theorem 1).
Revised: A classical result is that the best possible or Bayes-optimal scorer for this problem, i.e., f ∗ ∈ argmin f : X → R L BER( f ) , satisﬁes the following (Menon et al., 2013), (Koyejo et al., 2014, Corollary 4), (Collell et al., 2016, Theorem 1):

Original: [Equation]
Revised: [Equation]

Original: where P bal is the balanced class-probability as per §2.
Revised: where P bal is the balanced class-probability as per §2.

Original: In words, the Bayes-optimal prediction is the label under which the given instance x ∈ X is most likely.
Revised: In words, the Bayes-optimal prediction is the label under which the given instance x ∈ X is most likely.

Original: Consequently, for ﬁxed class-conditionals
Revised: Consequently, for ﬁxed class-conditionals

Original: P ( x | y ) , varying the class priors P ( y ) arbitrarily will not affect the optimal scorers.
Revised: P ( x | y ) , varying the class priors P ( y ) arbitrarily will not affect the optimal scorers.

Original: This is intuitively desirable: the balanced error is agnostic to the level of imbalance in the label distribution.
Revised: This is intuitively desirable: the balanced error is agnostic to the level of imbalance in the label distribution.

Original: To further probe (7), suppose the underlying class-probabilities P ( y | x ) ∝ exp( s ∗ y ( x )) , for(unknown) scorer s ∗ : X → R L .
Revised: To further probe (7), suppose the underlying class-probabilities P ( y | x ) ∝ exp( s ∗ y ( x )) , for scorer s ∗ : X → R L .

Original: Since by deﬁnition P bal ( y | x ) ∝ P ( y | x ) / P ( y ) , (7) becomes
Revised: Since by deﬁnition P bal ( y | x ) ∝ P ( y | x ) / P ( y ) , (7) becomes

Original: [Equation]
Revised: [Equation]

Original: i.e., we translate the (unknown) distributional scores or logits based on the class priors.
Revised: i.e., we translate the optimal logits based on the class priors.

Original: This simple fact immediately suggests two means of optimising for the balanced error:
Revised: While s ∗ is of course unknown in practice, (8) immediately suggests two means of optimising for the balanced error:

Original: (i) train a model to estimate the standard P ( y | x ) (e.g., by minimising the standard softmax-cross entropy on the long-tailed data), and then explicitly modify its logits post-hoc as per (8) (ii) train a model to estimate the balanced P bal ( y | x ) , whose logits are implicitly modiﬁed as per (8)
Revised: (i) train a model to estimate the standard P ( y | x ) (e.g., by minimising the standard softmax-cross entropy on the long-tailed data), and then explicitly modify its logits post-hoc as per (8) (ii) train a model to estimate the balanced P bal ( y | x ) , whose logits are implicitly modiﬁed as per (8).

Original: Such logit adjustment techniques — which have been a classic approach to class-imbalance (Provost, 2000) — neatly align with the post-hoc and loss modiﬁcation streams discussed in §2.
Revised: Such logit adjustment techniques — which have been a classic approach to class-imbalance (Provost, 2000) — neatly align with the post-hoc and loss modiﬁcation streams discussed in §2.

Original: However, unlike most previous techniques from these streams, logit adjustment is endowed with a clear statistical grounding: by construction, the optimal solution under such adjustment coincides with the Bayes-optimal solution (7) for the balanced error, i.e., it is Fisher consistent for minimising the balanced error.
Revised: However, unlike most previous techniques from these streams, logit adjustment is endowed with a clear statistical grounding: by construction, the optimal solution under such adjustment coincides with the Bayes-optimal solution (7) for the balanced error, i.e., it is Fisher consistent for minimising the balanced error.

Original: We now study each of the techniques (i) and (ii) in turn.
Revised: We now study each of the techniques (i) and (ii) in turn.

Original:  P OST - HOC LOGIT ADJUSTMENT
Revised:  P OST - HOC LOGIT ADJUSTMENT

Original: We now propose a post-hoc logit adjustment scheme for a classiﬁer trained on long-tailed data.
Revised: We now propose a post-hoc logit adjustment scheme for a classiﬁer trained on long-tailed data.

Original: We further show this bears similarity to recent weight normalisation schemes, but has a subtle advantage.
Revised: We further show this has subtle advantages over recent weight normalisation schemes.

Original: 4. T HE POST - HOC LOGIT ADJUSTMENT PROCEDURE
Revised: 4. T HE POST - HOC LOGIT ADJUSTMENT PROCEDURE

Original: While employing  softmax cross-entropy to train a neural network, we aim to approximate the underlying P ( y | x ) with p y ( x ) ∝ exp( f y ( x )) for logits
Revised: When employing the softmax cross-entropy to train a neural network, we aim to approximate the underlying P ( y | x ) with p y ( x ) ∝ exp( f y ( x )) for logits

Original: f y ( x ) = w > y Φ( x ) .
Revised: f y ( x ) = w > y Φ( x ) .

Original: Given learned { w, Φ } , one typically predicts the label argmax
Revised: Given learned { w, Φ } , one typically predicts the label argmax

Original: y ∈
Revised: y ∈

Original: [ L ] f y ( x ) , i.e., the most likely label under the model’s
Revised: [ L ] f y ( x ) , i.e., the most likely label under the model’s

Original: P ( y | x ) .
Revised: P ( y | x ) .

Original: In post-hoc logit adjustment , we propose to instead predict, for suitable τ > 0 :
Revised: In post-hoc logit adjustment , we propose to instead predict, for suitable τ > 0 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where π ∈ ∆ L  are estimates of the class priors P ( y ) , e.g., the empiricalclass frequencies on the training sample S .
Revised: where π ∈ ∆ L (for simplex ∆ ) are estimates of the class priors P ( y ) , e.g., the empirical frequencies on the training sample S .

Original: Effectively, (9) adds a label-dependent offset to each of the logits.
Revised: Effectively, (9) adds a label-dependent offset to each of the logits.

Original: When τ = 1 , this can be seen as applying (8) with a plugin estimate of P ( y | x ) , i.e., p y ( x ) ∝ exp( w > y Φ( x )) .
Revised: When τ = 1 , this can be seen as applying (8) with a plugin estimate of P ( y | x ) , i.e., p y ( x ) ∝ exp( w > y Φ( x )) .

Original: When τ 6 = 1 , this can be seen as applying (8) to temperature scaled estimates ¯ p y ( x ) ∝ exp( τ − 1 · w > y Φ( x )) .
Revised: When τ 6 = 1 , this can be seen as applying (8) to temperature scaled estimates ¯ p y ( x ) ∝ exp( τ − 1 · w > y Φ( x )) .

Original: To unpack this, recall that (8) justiﬁes post-hoc logit thresholding given access to the true probabilities P ( y | x ) .
Revised: To unpack this, recall that (8) justiﬁes post-hoc logit thresholding given access to the true probabilities P ( y | x ) .

Original: In practice,the outputs of a sufﬁciently high-capacity neural networks often produce uncalibrated estimates of these probabilities (Guo et al., 2017).
Revised: In practice, high-capacity neural networks often produce uncalibrated estimates of these probabilities (Guo et al., 2017).

Original: Temperature scaling serves as a means to calibrate the estimates, which is routinely employed for distillation (Hinton et al., 2015).
Revised: Temperature scaling is a means to calibrate the estimates, and is routinely employed for distillation (Hinton et al., 2015).

Original: One may treat τ as a tuning parameter to be chosen based on holdout calibration, e.g., the expected calibration error (Murphy & Winkler, 1987; Guo et al., 2017), probabilistic sharpness (Gneiting et al.,
Revised: One may treat τ as a tuning parameter to be chosen based on holdout calibration, e.g., the expected calibration error (Murphy & Winkler, 1987; Guo et al., 2017), probabilistic sharpness (Gneiting et al.,

Original: ; Kuleshov et al., 2018), or a proper scoring rule such as the log-loss or squared error (Gneiting & Raftery, 2007).
Revised: ; Kuleshov et al., 2018), or a proper scoring rule such as the log-loss or squared error (Gneiting & Raftery, 2007).

Original: One may alternately ﬁx τ = 1 and aim to learn inherently calibrated probabilities, e.g., via label smoothing (Szegedy et al., 2016; Müller et al., 2019).
Revised: One may alternately ﬁx τ = 1 and aim to learn inherently calibrated probabilities, e.g., via label smoothing (Szegedy et al., 2016; Müller et al., 2019).

Original: Post-hoc logit adjustment with τ = 1 is not a new idea in the class imbalance literature. Indeed, this is a standard technique when creating stratiﬁed samples (King & Zeng, 2001), and when training binary classiﬁers (Fawcett & Provost, 1996; Provost, 2000; Maloof, 2003).
Revised: Post-hoc logit adjustment with τ = 1 is not a new idea in the classical label imbalance literature (Fawcett & Provost, 1996; Provost, 2000; Maloof, 2003; Zhou & Liu, 2006;

Original: In multiclass settings, this has been explored in Zhou & Liu (2006); Collell et al.
Revised: Collell et al., 2016); however, it has had limited exploration in the recent long-tail learning literature.

Original: However, τ 6 = 1 is important in practical usage of neural networks, owing to their  lack of calibration.
Revised: Further, the case τ 6 = 1 is important in practical usage of neural networks, owing to their typical lack of probabilistic calibration (Guo et al., 2017).

Original: Further, we now explicate that post-hoc logit adjustment  has an important advantage over post-hoc weight normalisation.
Revised: Interestingly, post-hoc logit adjustment also has an important advantage over recently proposed weight normalisation techniques, as we now discuss.

Original: 4. C OMPARISON TO POST - HOC WEIGHT NORMALISAITON
Revised: 4. C OMPARISON TO POST - HOC WEIGHT NORMALISAITON

Original: Recall that weight normalisation involves learning logits
Revised: Recall that weight normalisation involves learning logits

Original: f y ( x ) = w > y Φ( x ) , and then post-hoc normalising the weights via w y /ν τy for τ > 0 .
Revised: f y ( x ) = w > y Φ( x ) , and then post-hoc normalising the weights via w y /ν τy for τ > 0 .

Original: We demonstrated in §2 that using ν y =
Revised: We demonstrated in §2 that using ν y =

Original: k w y k 2 may be ineffective when using adaptive optimisers.
Revised: k w y k 2 may be ineffective when using adaptive optimisers.

Original: However, even with ν y = π y , there is a subtle contrast to post-hoc logit adjustment: while the former performs a multiplicative update to the logits, the latter performs an additive update.
Revised: However, even with ν y = π y , there is a subtle contrast to post-hoc logit adjustment: while the former performs a multiplicative update to the logits, the latter performs an additive update.

Original: The two techniques may thus yield different orderings over labels, since
Revised: The two techniques may thus yield different orderings over labels, since

Original: [Equation]
Revised: [Equation]

Original: Weight normalisation is thus not consistent for minimising the balanced error, unlike logit adjustment.

Original: Indeed, if a rare label y has negative score w > y Φ( x ) < 0 , and there is another label with positive score, then it is impossible for the weight normalisation to give y the highest score.
Revised: Observe that if a rare label y has negative score w > y Φ( x ) < 0 , and there is another label with positive score, then it is impossible for the weight normalisation to give y the highest score.

Original: By contrast, under logit adjustment, w > y Φ( x ) − ln π y will be lower for dominant classes, regardless of the original sign.
Revised: By contrast, under logit adjustment, w > y Φ( x ) − ln π y will be lower for dominant classes, regardless of the original sign.

Revised: Weight normalisation is thus not consistent for the balanced error, unlike logit adjustment.

Original:  T HE LOGIT ADJUSTED SOFTMAX CROSS - ENTROPY
Revised:  T HE LOGIT ADJUSTED SOFTMAX CROSS - ENTROPY

Original: We now show how to directly bake logit adjustment into the softmax cross-entropy.
Revised: We now show how to directly encode logit adjustment into the softmax cross-entropy.

Original: The resulting approach has an intuitive relation to existing loss modiﬁcation techniques.
Revised: The resulting approach has an intuitive relation to existing loss modiﬁcation techniques.

Original: 5. T HE LOGIT ADJUSTED LOSS
Revised: 5. T HE LOGIT ADJUSTED LOSS

Original: From §3, the second approach to optimising for the balanced error is to directly model P bal ( y | x ) ∝
Revised: From §3, the second approach to optimising for the balanced error is to directly model P bal ( y | x ) ∝

Original: P ( y | x ) / P ( y ) .
Revised: P ( y | x ) / P ( y ) .

Original: To do so, consider the following logit adjusted softmax cross-entropy loss for τ > 0 :
Revised: To do so, consider the following logit adjusted softmax cross-entropy loss for τ > 0 :

Original: [Equation]
Revised: [Equation]

Original: Given a scorer that minimises the above, we now predict argmax y ∈
Revised: Given a scorer that minimises the above, we now predict argmax y ∈

Original: [ L ] f y ( x ) as usual.
Revised: [ L ] f y ( x ) as usual.

Original: Compared to the standard softmax cross-entropy (1), the above applies a label-dependent offset to each logit.
Revised: Compared to the standard softmax cross-entropy (1), the above applies a label-dependent offset to each logit.

Original: Compared to (9), we directly enforce the class prior offset while learning the logits, rather than doing this post-hoc.
Revised: Compared to (9), we directly enforce the class prior offset while learning the logits, rather than doing this post-hoc.

Original: The two approaches have a deeper connection: observe that (10) is equivalent to using a scorer of the form g y ( x ) =
Revised: The two approaches have a deeper connection: observe that (10) is equivalent to using a scorer of the form g y ( x ) =

Original: f y ( x ) + τ · log π y . We thus have argmax y ∈
Revised: f y ( x ) + τ · log π y , with argmax y ∈

Original: [ L ] f y ( x ) = argmax
Revised: [ L ] f y ( x ) = argmax

Original: y ∈
Revised: y ∈

Original: [ L ] g y ( x ) − τ · log π y .
Revised: [ L ] g y ( x ) − τ · log π y .

Original: Consequently, one can equivalently view learning with this loss as learning a standard scorer g ( x ) , and post-hoc adjusting its logits to make a prediction.
Revised: Consequently, one can equivalently view learning with this loss as learning a standard scorer g ( x ) , and post-hoc adjusting its logits.

Original: For convex objectives, we thus do not expect any difference between the solutions of the two approaches.

Original: For non-convex objectives, as encountered in neural networks, the bias endowed by adding τ · log π y to the logits ishowever likely to result in a different local minima.
Revised: For non-convex objectives, as encountered in neural networks, the bias endowed by adding τ · log π y to the logits is likely to result in a different local minima, typically with improved performance.

Original: For more insight into the logit-adjusted loss, consider the following pairwise margin loss
Revised: For more insight into the logit adjusted loss, consider the following pairwise margin loss

Original: [Equation]
Revised: [Equation]

Original: for label weights α y > 0 , and pairwise label margins ∆ yy 0 representing the desired gap between scores for y and y 0 .
Revised: for label weights α y > 0 , and pairwise label margins ∆ yy 0 representing the desired gap between scores for y and y 0 .

Original: For τ = 1 , our logit adjusted loss (10) corresponds to (11) with α y = 1 and ∆ yy 

Revised: For τ = 1 , our logit adjusted loss (10) corresponds to (11) with α y = 1 and ∆ yy

Original: = log (cid:16) π y 
π
Revised: = log (cid:16) π y

Original: [Equation] (cid:17) .
Revised: π y (cid:17) .

Original: This demands a larger margin between rare positive ( π y ∼ 0 ) and dominant negative ( π y 0 ∼ 1 ) labels, so that scores for dominant classes do not overwhelm those for rare ones.
Revised: This demands a larger margin between rare positive ( π y ∼ 0 ) and dominant negative ( π y 0 ∼ 1 ) labels, so that scores for dominant classes do not overwhelm those for rare ones.

Original: Furthermore, existing loss modiﬁcation techniques can be viewed as special cases of (11).
Revised: Existing loss modiﬁcation techniques can be viewed as special cases of (11).

Original: For example, α y = 1 π y and ∆ yy 0 = 0 yields the balanced loss (4).
Revised: For example, α y = 1 /π y and ∆ yy 0 = 0 yields the balanced loss (4).

Original: When α y = 1 , the choice ∆ yy 

Revised: When α y = 1 , the choice ∆ yy

Original: = π − 1 / 4 y yields (5).
Revised: = π − 1 / 4 y yields (5).

Original: Finally, ∆ yy 

Revised: Finally, ∆ yy

Original: = log F ( π y 0 ) yields (6), where F : [0 , 1] → (0 , 1] is some nondecreasing function, e.g., F ( z ) = z τ for τ > 0 .
Revised: = log F ( π y 0 ) yields (6), where F : [0 , 1] → (0 , 1] is some non-decreasing function.

Original: These losses thus either consider the frequency of the positive y or negative y 0 , but not both simultaneously.
Revised: These losses thus either consider the frequency of the positive y or negative y 0 , but not both .

Original: Remarkably, thefollowing result shows that the speciﬁc choice that leads to our loss in (10) has a ﬁrm statistical grounding: it ensures Fisher consistency  for the balanced error.
Revised: Remarkably, the speciﬁc choice that leads to our loss in (10) has a ﬁrm statistical grounding: it ensures Fisher consistency (in the sense of, e.g., Bartlett et al. for the balanced error.

Original: Theorem 1.
Revised: Theorem 1.

Original: For any δ ∈ R L + , the pairwise loss in (11) is Fisher consistent with weights and margins
Revised: For any δ ∈ R L + , the pairwise loss in (11) is Fisher consistent with weights and margins

Original: [Equation]
Revised: [Equation]

Original: By invoking Theorem 1 with δ y =
Revised: Letting δ y =

Original: π y , we immediately deduce that the logit-adjusted loss of (10) is consistent.
Revised: π y , we immediately deduce that the logit-adjusted loss of (10) is consistent, provided our π y is a consistent estimate of P ( y ) .

Original: Similarly, δ y = 1 recovers the classic result that the balanced loss is consistent.
Revised: Similarly, δ y = 1 recovers the classic result that the balanced loss is consistent.

Original: While Theorem 1 only provides a sufﬁcient condition in multi-class setting, one can provide a necessary and sufﬁcient conditionfor consistency that rules out other choices  in the binary case; see Appendix B.1.
Revised: While Theorem 1 only provides a sufﬁcient condition in multi-class setting, one can provide a necessary and sufﬁcient condition that rules out other choices of ∆ in the binary case.

Original: Lemma 2.
Revised: Theorem 2.

Original:  Let  σ ( z ) =
Revised: Suppose Y = {± 1 } . Let δ y . = ∆ y, − y , and σ ( z ) =

Original: (1 + exp
Revised: (1 + exp

Original: z ))
Revised: z ))

Original: − 1 .
Revised: − 1 .

Original: The losses in (13) are consistent for the balanced error iff
Revised: Then, the pairwise margin loss in (11) is Fisher consistent for the balanced error iff

Original: [Equation]
Revised: [Equation]

Original: 5. D ISCUSSION AND EXTENSIONS
Revised: 5. D ISCUSSION AND EXTENSIONS

Original: Logit adjustment techniques (10) are complementary to the existing  approaches in the literature.
Revised: Our pairwise margin loss in (11) subsumes several existing loss-correction approaches in the literature.

Original:  For example, Theorem 1 implies that it is sensible to combine logit adjustment with loss weighting; e.g., one may pick ∆ [-yy 
 = τ · log ( π y 0 /π y ) , and α y-]
Revised: Further, it suggests the exploration of new choices of ∆ . For example, we shall see the efﬁcacy of combining the ∆ [-yy 
 = τ · log ( π y 0 /π y ) , and α y-]implicit in the adaptive loss of Cao et al.

Original: = π τ − 1 y .
Revised: with our proposed ∆ .

Original: One may also generalise the formulation in Theorem 1, and employ ∆ yy 

Revised: One may also generalise the formulation in Theorem 1, and employ ∆ yy

Original: = τ 1 · log π y − τ 2 · log π y 0 , where τ 1 , τ 2 are constants.
Revised: = τ 1 · log π y − τ 2 · log π y 0 , where τ 1 , τ 2 > 0 .

Original: This interpolates between the logit adjusted loss ( τ 1 = τ 2 ) and a version of the equalisedmargin loss ( τ 1 = 0 ).
Revised: This interpolates between our loss ( τ 1 = τ 2 ) and a version of the equalised loss ( τ 1 = 0 ).

Revised: For τ = − 1 , a similar loss to (10) has been considered in the context of negative sampling for scalability (Yi et al., 2019): here, one samples a subset of negatives based on π , and corrects the logits to obtain an unbiased estimate of the loss based on all negatives (Bengio & Senecal, 2008).

Revised: Losses of the general form (11) have also been explored for structured prediction (Pletscher et al., 2010).

Original: Cao et al.
Revised: Cao et al.

Original: (2019, Theorem 2) provides a rigorous generalisation bound for the adaptive margin loss under the assumption of separabletraining data with binary labels.
Revised: (2019, Theorem 2) provides a rigorous generalisation bound for the adaptive margin loss under the assumption of separable data with binary labels.

Original: The inconsistency of the loss with respect to the balanced error concerns the more general scenario of non-separable multiclassdata data, which may occur, e.g., owing to label noise or limitation in model capacity.
Revised: The inconsistency of the loss with respect to the balanced error concerns the more general scenario of non-separable multiclass data, which may occur, e.g., owing to label noise or limited model capacity.

Original: We shall subsequently demonstrate that encouraging consistency  can lead to gains in practical settings.
Revised: We shall subsequently demonstrate that encouraging consistency is not merely of theoretical interest, and can lead to gains in practice.

Original: For τ = − 1 , a similar loss to (10) has been considered in the context of negative sampling for scalability (Yi et al., 2019): here, one samples a subset of negatives based on π , and corrects the logits to obtain an unbiased estimate of the loss based on all negatives (Bengio & Senecal, 2008).

Original: Losses of the general form (11) have also been explored for structured prediction (Pletscher et al., 2010).

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: We now present experiments conﬁrming our main claims: (i) on simple binary problems, existing weight normalisation and loss modiﬁcation techniques may not converge to the optimal solution (§6.1); (ii) on real-world datasets, our post-hoc logit adjustment  outperforms weight normalisation, and one can obtain further gains via our logit adjusted softmax cross-entropy (§6.2).
Revised: We now present experiments conﬁrming our main claims: (i) on simple binary problems, existing weight normalisation and loss modiﬁcation techniques may not converge to the optimal solution (§6.1); (ii) on real-world datasets, our post-hoc logit adjustment generally outperforms weight normalisation, and one can obtain further gains via our logit adjusted softmax cross-entropy (§6.2).

Original: 6. R ESULTS ON SYNTHETIC DATASET
Revised: 6. R ESULTS ON SYNTHETIC DATASET

Original: We consider a binary classiﬁcation task, wherein samples from class y ∈ {± 1 } are drawn from a 2D Gaussian with isotropic covariance and means µ y
Revised: We begin with a binary classiﬁcation task, wherein samples from class y ∈ {± 1 } are drawn from a 2D Gaussian with isotropic covariance and means µ y

Original: =
Revised: =

Original: y · (+1 , +1) .
Revised: y · (+1 , +1) .

Original: We introduce class imbalance by setting P ( y = +1)
Revised: We introduce class imbalance by setting P ( y = +1)

Original: = 5% .
Revised: = 5% .

Original: The Bayes-optimal classiﬁer for the balanced error is (see Appendix F)
Revised: The Bayes-optimal classiﬁer for the balanced error is (see Appendix F)

Original: [Equation]
Revised: [Equation]

Original: i.e., it is a linear separator passing through the origin.
Revised: i.e., it is a linear separator passing through the origin.

Original: We compare this separator against those found by several margin losses based on (11): standard ERM ( ∆ yy 0 = 0 ), the adaptive loss (Cao et al., 2019) ( ∆ yy 

Revised: We compare this separator against those found by several margin losses based on (11): standard ERM ( ∆ yy 0 = 0 ), the adaptive loss (Cao et al., 2019) ( ∆ yy

Original: = π − 1 / 4 y ), an instantiation of the equalised loss (Tan et al., 2020) ( ∆ yy 

Revised: = π − 1 / 4 y ), an instantiation of the equalised loss (Tan et al., 2020) ( ∆ yy

Original: = log π y 0 ), and our logit adjusted loss ( ∆ yy 

Revised: = log π y 0 ), and our logit adjusted loss ( ∆ yy

Original: = log π y 

Revised: = log π y

Original: π y ).
Revised: π y ).

Original: For each loss, we train an afﬁne classiﬁer on a sample of 10 ,instances, and evaluate the balanced error on a test set of 10 , 000 samples over 100 independent trials.
Revised: For each loss, we train an afﬁne classiﬁer on a sample of 10 ,instances, and evaluate the balanced error on a test set of 10 , 000 samples over 100 independent trials.

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Figure 2 conﬁrms that the logit adjusted margin loss attains a balanced error close to that of the
Revised: Figure 2 conﬁrms that the logit adjusted margin loss attains a balanced error close to that of the

Original: Bayes-optimal, which is visually reﬂected by its learned separator closely matching that in (12).
Revised: Bayes-optimal, which is visually reﬂected by its learned separator closely matching that in (12).

Original: This is in line with our claim of the logit adjusted margin loss being consistent for the balanced error, unlike other approaches.
Revised: This is in line with our claim of the logit adjusted margin loss being consistent for the balanced error, unlike other approaches.

Original: Figure 2 also compares post-hoc weight normalisation and logit adjustment for varying scaling parameter τ (c.f. (3), (9)).
Revised: Figure 2 also compares post-hoc weight normalisation and logit adjustment for varying scaling parameter τ (c.f. (3), (9)).

Original: Logit adjustment is seen to approach the performance of the Bayes predictor; any weight normalisation is however seen to hamper performance.
Revised: Logit adjustment is seen to approach the performance of the Bayes predictor; any weight normalisation is however seen to hamper performance.

Original: This veriﬁes the consistency of logit adjustment, and inconsistency of weight normalisation (§4.2).
Revised: This veriﬁes the consistency of logit adjustment, and inconsistency of weight normalisation (§4.2).

Original: 6. R ESULTS ON REAL - WORLD DATASETS
Revised: 6. R ESULTS ON REAL - WORLD DATASETS

Original: We present results on the CIFAR-10, CIFAR-100, ImageNet and iNaturalist 2018 datasets.
Revised: We present results on the CIFAR-10, CIFAR-100, ImageNet and iNaturalist 2018 datasets.

Original: Following prior work, we create “long-tailed versions” of the CIFAR datasets by suitably downsampling examples per label following the E XP proﬁle of Cui et al.
Revised: Following prior work, we create “long-tailed versions” of the CIFAR datasets by suitably downsampling examples per label following the E XP proﬁle of Cui et al.

Original: ; Cao et al.
Revised: ; Cao et al.

Original: with imbalance ratio ρ = max y P ( y ) / min y P ( y ) = 100 .
Revised: with imbalance ratio ρ = max y P ( y ) / min y P ( y ) = 100 .

Original: Similarly, we use the long-tailed version of ImageNet produced by Liu et al.
Revised: Similarly, we use the long-tailed version of ImageNet produced by Liu et al.

Original: We employ a ResNet-32 for CIFAR, and a ResNet-50 for ImageNet and iNaturalist.
Revised: We employ a ResNet-32 for CIFAR, and a ResNet-50 for ImageNet and iNaturalist.

Original: All models are trained using SGD with momentum; see Appendix C for more details.
Revised: All models are trained using SGD with momentum; see Appendix C for more details.

Original: See also Appendix D.1 for results on CIFAR under the S TEP proﬁle  considered in the literature.
Revised: See also Appendix D.4 for results on CIFAR under the S TEP proﬁle also considered in the literature.

Original: Baselines .
Revised: Baselines .

Original: We consider: (i) empirical risk minimisation (ERM) on the long-tailed data, (ii) post-hoc weight normalisation (Kang et al., 2020) per (3) (using ν y = 
Revised: We consider several representative baselines: (i) empirical risk minimisation (ERM) on the long-tailed data, (ii) post-hoc weight normalisation (Kang et al., 2020) per (3) (using ν y = k w y k 2 ) applied to ERM, (iii) the class-balanced loss of Cui et al.

Original: k w y k 2 and τ = 1 ) applied to ERM, (iii) the adaptive margin loss (Cao et al., 2019) per (5), including with the “deferred reweighting” (DRW) method of training, and (iv) the equalised loss (Tan et al., 2020) per (6), with δ y 

Revised: (2019), (iv) the adaptive margin loss (Cao et al., 2019) per (5), including with the “deferred reweighting” (DRW) training scheme, and (v) the equalised loss (Tan et al., 2020) per (6), with δ y

Original: = F ( π y 0 ) for the threshold-based F of Tan et al.
Revised: = F ( π y 0 ) for the threshold-based F of Tan et al.

Original: Where possible, we report numbers for the baselines(which use the same setup as above) from the respective papers.
Revised: Where possible, we report numbers for the baselines from the respective papers.

Original: We compare the above methods against our proposed post-hoc logit adjustment (9), and logit adjusted loss (10).
Revised: We compare the above methods against our proposed post-hoc logit adjustment (9), and logit adjusted loss (10).

Original: For post-hoc logit adjustment, we ﬁx the scalar τ = 1 ; we analyse the effect of tuning this in Figure 3.
Revised: For post-hoc logit adjustment, we ﬁx the scalar τ = 1 for our basic results; we analyse the effect of tuning this in Figure 3.

Original: We additionally evaluate asimple combination of our logitadjusted softmax cross-entropy
Revised: We additionally evaluate a combination of our logit

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  margin of Cao et al.
Revised: adjusted softmax cross-entropy with the adaptive margin of Cao et al.

Original: (2019); this uses (11) with ∆ yy 0 = log( π y 0 /π y )
Revised: (2019); this uses (11) with ∆ yy 0 = log( π y 0 /π y )

Original: + π − 0 .
Revised: + π − 0 .

Original: 25 y .
Revised: 25 y .

Original: We do not perform any further tuning of ourlogit adjustment techniques.
Revised: We do not perform any further tuning of our techniques.

Original: Results and analysis .
Revised: Results and analysis .

Original: Table 2 summarises our results, which demonstrate our proposed logit adjustment techniques consistently outperform existing methods.
Revised: Table 2 demonstrates that our proposed logit adjustment techniques consistently outperform existing methods.

Original: Indeed,while weight normalisation offers gains over ERM, these are improved signiﬁcantly by post-hoc logit adjustment (e.g., 8% relative reduction on CIFAR-10).
Revised: Indeed, weight normalisation is generally improved signiﬁcantly by post-hoc logit adjustment (e.g., 8% relative reduction on CIFAR-10).

Original: Similarly loss correction techniques are generally outperformed by our logit adjusted softmax cross-entropy (e.g., 6% relative reduction on iNaturalist).
Revised: Similarly, loss correction techniques are generally outperformed by our logit adjusted softmax cross-entropy (e.g., 6% relative reduction on iNaturalist).

Revised: We remark that Cao et al.

Revised: (2019) observed that their loss beneﬁts from a deferred reweighting scheme (DRW), wherein class-weighting is applied after a ﬁxed number of epochs.

Revised: Table 2 indicates this is consistently outperformed by suitable variants of logit adjustment.

Original: While both logit adjustment techniques perform similarly, there is a slight advantage to the loss function version.
Revised: While both logit adjustment techniques perform similarly, there is a slight advantage to the loss function version.

Original: Nonetheless, the strong performance of post-hoc logit adjustment corroborates the ability to decouple representation and classiﬁer learning in long-tail settings (Zhang et al., 2019).
Revised: Nonetheless, the strong performance of post-hoc logit adjustment corroborates the ability to decouple representation and classiﬁer learning in long-tail settings (Zhang et al., 2019).

Original:  Figure 3 studies the effect of tuningthe scaling parameter τ > 0 afforded by post-hoc weight normalisation (using ν y =
Revised: Note that Table 2 only reports the results for scalar τ = 1 . In practice, tuning this parameter can signiﬁcantly improve performance further. Figure 3 studies the effect of tuning τ for post-hoc weight normalisation (using ν y =

Original: k w y k 2 ) and post-hoc logit adjustment.
Revised: k w y k 2 ) and post-hoc logit adjustment.

Original: Even without any scaling, post-hoc logit adjustment generally offers superior performance to the best result from weight normalisation (cf.
Revised: Even without any scaling, post-hoc logit adjustment generally offers superior performance to the best result from weight normalisation (cf.

Original: Table 2); with scaling, this is further improved.
Revised: Table 2); with scaling, this is further improved.

Original: See Appendix D.4 for a plot on ImageNet-LT.
Revised: See Appendix D.4 for a plot on ImageNet-LT.

Original:  Figure 4 reports errors on a per-group basis, where following Kang et al.
Revised: To better understand the gains, Figure 4 reports errors on a per-group basis, where following Kang et al.

Original: we construct three groups of classes — “Many”, “Medium”, and “Few” — comprising those with ≥ 100, between (20 , 100) , and ≤ 20 training examples respectively.
Revised: we construct three groups of classes — “Many”, “Medium”, and “Few” — comprising those with ≥ 100, between (20 , 100) , and ≤ 20 training examples respectively.

Original: Logit adjustment shows consistent gains over all three groups.
Revised: Logit adjustment shows consistent gains over all three groups.

Original: See Appendix D.3 for a ﬁner-grained breakdown.
Revised: See Appendix D.3 for a ﬁner-grained breakdown.

Original: Discussion and extensions Table 2 shows the advantage of logit adjustment over recent proposals, under standard setups from the literature.
Revised: Discussion and extensions Table 2 shows the advantage of logit adjustment over recent proposals, under standard setups from the literature.

Original: Further improvements are possible by fusing complementary ideas, and we remark on a few such options.
Revised: Further improvements are possible by fusing complementary ideas, and we remark on a few such options.

Original: First, one may use a more complex base architecture; e.g., Kang et al.
Revised: First, one may use a more complex base architecture; e.g., Kang et al.

Original: (2020) found gains by employing a ResNet-152, and training for 200 epochs.
Revised: (2020) found gains by employing a ResNet-152, and training for 200 epochs.

Original: Table(Appendix) conﬁrms that logit adjustment similarly beneﬁts from this choice, achieving a balanced error of 30 .
Revised: Table(Appendix) conﬁrms that logit adjustment similarly beneﬁts from this choice, achieving a balanced error of 30 .

Original: 12 % on iNaturalist, and 28 .
Revised: 12 % on iNaturalist, and 28 .

Original: 02 % when combined with the adaptive margin.
Revised: 02 % when combined with the adaptive margin.

Original: Second, Cao et al. (2019) observed that their loss beneﬁts from a deferred reweighting scheme (DRW), wherein class-weighting is applied after a ﬁxed number of epochs. Table 2 indicates this is outperformed by suitable variants of logit adjustment.
Revised: Second, the DRW training scheme (which applies to any loss) may result in further gains for our techniques. Third, incorporating developments in meta-learning (Wang et al., 2017; Jamal et al., 2020) is also a promising avenue.

Original: Nonetheless, DRW (which applies to any loss) may result in further gains for our techniques.

Original: While further exploring such variants are of empirical interest, we hope to have illustrated the conceptual and empirical value of logit adjustment, and leave this for future work.
Revised: While further exploring such variants are of empirical interest, we hope to have illustrated the conceptual and empirical value of logit adjustment, and leave this for future work.