Original: L ONG - TAIL LEARNING VIA LOGIT ADJUSTMENT
Revised: L ONG - TAIL LEARNING VIA LOGIT ADJUSTMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Real-world classiﬁcation problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels have only a few associated samples.
Revised: Real-world classiﬁcation problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels have only a few associated samples.

Original: This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels.
Revised: This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels.

Original: In this paper, we present a statistical framework that uniﬁes and generalises several recent proposals to cope with these challenges.
Revised: In this paper, we present a statistical framework that uniﬁes and generalises several recent proposals to cope with these challenges.

Original: Our framework revisits the classic idea of logit adjustment based on the label frequencies, which encourages a large relative margin between logits of rare positive versus dominant negative labels.
Revised: Our framework revisits the classic idea of logit adjustment based on the label frequencies, which encourages a large relative margin between logits of rare positive versus dominant negative labels.

Original: This yields two techniques for long-tail learning, where such adjustment is either applied post-hoc to a trained model, or enforced in the loss during training.
Revised: This yields two techniques for long-tail learning, where such adjustment is either applied post-hoc to a trained model, or enforced in the loss during training.

Original: These techniques are statistically grounded, and practically effective on four real-world datasets with long-tailed label distributions.
Revised: These techniques are statistically grounded, and practically effective on four real-world datasets with long-tailed label distributions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Real-world classiﬁcation problems typically exhibit a long-tailed label distribution, wherein most labels are associated with only a few samples (Van Horn & Perona, 2017; Buda et al., 2017; Liu et al., 2019).
Revised: Real-world classiﬁcation problems typically exhibit a long-tailed label distribution, wherein most labels are associated with only a few samples (Van Horn & Perona, 2017; Buda et al., 2017; Liu et al., 2019).

Original: Owing to this paucity of samples, generalisation on such labels is challenging; moreover, naïve learning on such data is susceptible to an undesirable bias towards dominant labels.
Revised: Owing to this paucity of samples, generalisation on such labels is challenging; moreover, naïve learning on such data is susceptible to an undesirable bias towards dominant labels.

Original: This problem has been widely studied in the literature on learning under class imbalance (Kubat et al., 1997; Chawla et al., 2002; He & Garcia, 2009), and the related problem of cost-sensitive learning (Elkan, 2001).
Revised: This problem has been widely studied in the literature on learning under class imbalance (Kubat et al., 1997; Chawla et al., 2002; He & Garcia, 2009), and the related problem of cost-sensitive learning (Elkan, 2001).

Original: Recently, long-tail learning has received renewed interest in the context of neural networks.
Revised: Recently, long-tail learning has received renewed interest in the context of neural networks.

Original: Two active strands of work involve post-hoc normalisation of the classiﬁcation weights (Zhang et al., 2019;
Revised: Two active strands of work involve post-hoc normalisation of the classiﬁcation weights (Zhang et al., 2019;

Original: Kim & Kim, 2019; Kang et al., 2020; Ye et al., 2020), and modiﬁcation of the underlying loss to account for varying class penalties (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al., 2020).
Revised: Kim & Kim, 2019; Kang et al., 2020; Ye et al., 2020), and modiﬁcation of the underlying loss to account for varying class penalties (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al., 2020).

Original: Each of these strands are intuitive, and have proven empirically successful.
Revised: Each of these strands are intuitive, and have proven empirically successful.

Original: However, they are not without limitation: e.g., weight normalisation crucially relies on the weight norms being smaller for rare classes; however, this assumption is sensitive to the choice of optimiser (see §2).
Revised: However, they are not without limitation: e.g., weight normalisation crucially relies on the weight norms being smaller for rare classes; however, this assumption is sensitive to the choice of optimiser (see §2.1).

Original: On the other hand, loss modiﬁcation sacriﬁces the consistency that underpins the canonical softmax cross-entropy (see §5.1).
Revised: On the other hand, loss modiﬁcation sacriﬁces the consistency that underpins the canonical softmax cross-entropy (see §5.1).

Original: Consequently, such techniques may prove suboptimal even in simple settings (see §6.1).
Revised: Consequently, such techniques may prove suboptimal even in simple settings (see §6.1).

Original: In this paper, we establish a statistical framework for long-tail learning that offers a uniﬁed view of post-hoc normalisation and loss modiﬁcation techniques, while overcoming their limitations.
Revised: In this paper, we establish a statistical framework for long-tail learning that offers a uniﬁed view of post-hoc normalisation and loss modiﬁcation techniques, while overcoming their limitations.

Original: Our framework revisits the classic idea of logit adjustment based on label frequencies (Provost, 2000;
Revised: Our framework revisits the classic idea of logit adjustment based on label frequencies (Provost, 2000;

Original: Zhou & Liu, 2006; Collell et al., 2016), which encourages a large relative margin between a pair of rare positive and dominant negative labels.
Revised: Zhou & Liu, 2006; Collell et al., 2016), which encourages a large relative margin between a pair of rare positive and dominant negative labels.

Original: Such adjustment can be achieved by shifting the learned logits post-hoc, or augmenting the softmax cross-entropy with a pairwise label margin (cf.
Revised: Such adjustment can be achieved by shifting the learned logits post-hoc, or augmenting the softmax cross-entropy with a pairwise label margin (cf.

Original: While similar in nature to recent techniques, our logit adjustment approaches additionally have a ﬁrm statistical grounding: they are Fisher consistent for minimising the balanced error (cf.
Revised: While similar in nature to recent techniques, our logit adjustment approaches additionally have a ﬁrm statistical grounding: they are Fisher consistent for minimising the balanced error (cf.

Original: (2)), a common metric in long-tail settings which averages the per-class errors.
Revised: (2)), a common metric in long-tail settings which averages the per-class errors.

Original: This statistical grounding translates into strong empirical performance on four real-world datasets with long-tailed label distributions.
Revised: This statistical grounding translates into strong empirical performance on four real-world datasets with long-tailed label distributions.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: (i) we establish a statistical framework for long-tail learning (§3) based on logit adjustment that provides a uniﬁed view of post-hoc correction and loss modiﬁcation (ii) we present two realisations of logit adjustment, applied either post-hoc (§4.1) or during training (§5.1); unlike recent proposals (Table 1), these are consistent for minimising the balanced error(iii) we conﬁrm the efﬁcacy of the proposed logit adjustment techniques compared to several baselines on four real-world datasets with long-tailed label distributions (§6). [Table] Reference [Table]
Revised: (i) we establish a statistical framework for long-tail learning (§3) based on logit adjustment that provides a uniﬁed view of post-hoc correction and loss modiﬁcation (ii) we present two realisations of logit adjustment, applied either post-hoc (§4.1) or during training (§5.1); unlike recent proposals (Table 1), these are consistent for minimising the balanced error

Original: × (Kang et al., 2020)

Original: [Table]

Original: Softmax with rare +

Original: [Table]
Revised: [Table]

Original: upweighting × (Cao et al., 2019)

Original: [Table]

Original: Softmax

Original: [Table]

Original: downweighting × (Tan et al., 2020)

Original: [Table]

Original: Post

Original: [Table]

Original: hoc logit

Original: [Table]

Original: X

Original: This paper (cf. [Table] Softmax with logit [Table]
Revised: (iii) we conﬁrm the efﬁcacy of the proposed logit adjustment techniques compared to several baselines on four real-world datasets with long-tailed label distributions (§6).

Original: X

Original: This paper (cf.

Original:  P ROBLEM SETUP AND RELATED WORK
Revised:  P ROBLEM SETUP AND RELATED WORK

Original: Consider a multiclass classiﬁcation problem with instances X and labels Y =
Revised: Consider a multiclass classiﬁcation problem with instances X and labels Y =

Original: [ L ] .
Revised: [ L ] .

Original: = { 1 , 2 , . . .
Revised: = { 1 , 2 , . . .

Original: , L } .
Revised: , L } .

Original: Given a sample S
Revised: Given a sample S

Original: =
Revised: =

Original: { ( x n , y n ) } Nn =1 ∼ P N for unknown distribution P over X × Y , our goal is to learn a scorer f : X → R L that minimises the misclassiﬁcation error P x,y {+(cid:0) y / ∈ argmax y 
+}
Revised: { ( x n , y n ) } Nn =1 ∼ P N for unknown distribution P over X × Y , our goal is to learn a scorer f : X → R L that minimises the misclassiﬁcation error P x,y {+(cid:0) y / ∈ argmax y 
+}

Original: [Equation]
Revised: ∈ Y f y 0 ( x ) (cid:1) .

Original: Typically, one minimises a surrogate loss ‘ : Y × R L → R such as the softmax cross-entropy, ‘ ( y, f ( x )) = log 
Revised: Typically, one minimises a surrogate loss ‘ : Y × R L → R such as the softmax cross-entropy, ‘ ( y, f ( x )) = log h X

Original: [Equation]
Revised: [Equation]

Revised: i − f

Revised: [Equation]

Revised: [Equation]

Original:  We may view the resulting softmax probabilities p y ( x ) ∝ e f y ( x ) as estimates of P ( y | x ) .
Revised: ) = log h 1 + X [Equation] We may view the resulting softmax probabilities p y ( x ) ∝ e f y ( x ) as estimates of P ( y | x ) .

Original: The setting of learning under class imbalance or long-tail learning is where the distribution P ( y ) is highly skewed, so that many rare (or “tail”) labels have a low probability of occurrence.
Revised: The setting of learning under class imbalance or long-tail learning is where the distribution P ( y ) is highly skewed, so that many rare (or “tail”) labels have a low probability of occurrence.

Original: Here, the misclassiﬁcation error is not a suitable measure of performance: a trivial predictor which classiﬁes every instance to the majority label will attain a low misclassiﬁcation error.
Revised: Here, the misclassiﬁcation error is not a suitable measure of performance: a trivial predictor which classiﬁes every instance to the majority label will attain a low misclassiﬁcation error.

Original: To cope with this, a natural alternative is the balanced error (Chan & Stolfo, 1998; Brodersen et al., 2010; Menon et al., 2013), which averages each of the per-class error rates: under a uniform label distribution
Revised: To cope with this, a natural alternative is the balanced error (Chan & Stolfo, 1998; Brodersen et al., 2010; Menon et al., 2013), which averages each of the per-class error rates: under a uniform label distribution

Original: [Equation]
Revised: [Equation]

Original: This can be seen as implicitly using a balanced class-probability function P bal ( y | x ) ∝ 1 L · P ( x | y ) , as opposed to the native P ( y | x ) ∝ P ( y ) · P ( x | y ) that is employed in the misclassiﬁcation error.
Revised: This can be seen as implicitly using a balanced class-probability function P bal ( y | x ) ∝ 1 L · P ( x | y ) , as opposed to the native P ( y | x ) ∝ P ( y ) · P ( x | y ) that is employed in the misclassiﬁcation error.

Original: Broadly, extant approaches to coping with class imbalance modify:
Revised: Broadly, extant approaches to coping with class imbalance modify:

Original: (i) the inputs to a model, for example by over- or under-sampling (Kubat & Matwin, 1997; Chawla et al., 2002; Wallace et al., ; Mikolov et al., 
Revised: (i) the inputs to a model, for example by over- or under-sampling (Kubat & Matwin, 1997; Chawla et al., 2002; Wallace et al., 2011; Mikolov et al., 2013; Mahajan et al., 2018;

Original: ; Mahajan et al., 
Revised: Yin et al., 2018); (ii) the outputs of a model, for example by post-hoc correction of the decision threshold (Fawcett & Provost, 1996;

Original: ; Yin et al., 2018); (ii) the outputs of a model, for example by post-hoc correction of the decision threshold (Fawcett & Provost, 1996; Collell et al., 2016) or weights (Kim & Kim, 2019; Kang et al., 2020); or (iii) the training procedure of a model, for example by modifying the loss function (Zhang et al., 2017; Cui et al., 2019; Cao et al., 2019; Tan et al., 2020; Jamal et al., 2020).
Revised: Collell et al., 2016) or weights (Kim & Kim, 2019; Kang et al., 2020); or (iii) the training procedure of a model, for example by modifying the loss function (Zhang et al ; Cui et al. , 2019; Cao et al., 2019; Tan et al., 2020; Jamal et al., 2020).

Original: One may easily combine approaches from the ﬁrst stream with those from the latter two.
Revised: One may easily combine approaches from the ﬁrst stream with those from the latter two.

Original: Consequently, we focus on the latter two in this work, and describe some representative recent examples from each.
Revised: Consequently, we focus on the latter two in this work, and describe some representative recent examples from each.

Original: Post-hoc weight normalisation .
Revised: Post-hoc weight normalisation .

Original: Suppose f y ( x ) = w > y Φ( x ) for classiﬁcation weights w y ∈ R D and representations Φ: X → R D , as learned by a neural network.
Revised: Suppose f y ( x ) = w > y Φ( x ) for classiﬁcation weights w y ∈ R D and representations Φ: X → R D , as learned by a neural network.

Original: (We may add per-label bias terms to f y by adding a constant feature to Φ .)
Revised: (We may add per-label bias terms to f y by adding a constant feature to Φ .)

Original: A fruitful avenue of exploration involves decoupling of representation and classiﬁer learning (Zhang et al., 2019).
Revised: A fruitful avenue of exploration involves decoupling of [Figure] || [Figure]

Revised: y ||

Revised: [Figure]

Revised: [Figure]

Revised: |

Revised: [Figure]

Revised: |

Revised: [Figure]

Revised: representation and classiﬁer learning (Zhang et al., 2019).

Original: Concretely, we ﬁrst learn { w y , Φ } via standard training on the long-tailed training sample S , and then for x ∈ X predict the label
Revised: Concretely, we ﬁrst learn { w y , Φ } via standard training on the long-tailed training sample S , and then for x ∈ X predict the label

Original: [Equation]
Revised: [Equation]

Original: for τ > 0 , where ν y =
Revised: for τ > 0 , where ν y =

Original: P ( y ) in Kim & Kim (2019); Ye et al.
Revised: P ( y ) in Kim & Kim (2019); Ye et al.

Original: (2020) and ν y =
Revised: (2020) and ν y =

Original: k w y k 2 in Kang et al.
Revised: k w y k 2 in Kang et al.

Original: Intuitively, either choice of ν y upweights the contribution of rare labels through weight [Figure] || [Figure]
Revised: Intuitively, either choice of ν y upweights the contribution of rare labels through weight normalisation .

Original: y ||

Original: [Figure]

Original: [Figure]

Original: ||

Original: [Figure]

Original: y ||

Original: [Figure]

Original: normalisation .

Original: The choice ν y = k w y k 2 is motivated by the observations that k w y k 2 tends to correlate with
Revised: The choice ν y = k w y k 2 is motivated by the observations that k w y k 2 tends to correlate with

Original: P ( y ) .
Revised: P ( y ) .

Original: Further to the above, one may enforce k w y k 2 = 1 during training (Kim & Kim, 2019).
Revised: Further to the above, one may enforce k w y k 2 = 1 during training (Kim & Kim, 2019).

Original: Loss modiﬁcation .
Revised: Loss modiﬁcation .

Original: A classic means of coping with class imbalance is to balance the loss, wherein ‘ ( y, f ( x )) is weighted by P ( y ) − 1 (Xie & Manski, 1989; Morik et al., 1999): e.g., applied to (1),
Revised: A classic means of coping with class imbalance is to balance the loss, wherein ‘ ( y, f ( x )) is weighted by P ( y ) − 1 (Xie & Manski, 1989; Morik et al., 1999): e.g., applied to (1),

Original: [Equation]
Revised: [Equation]

Original: While intuitive, balancing has minimal effect in separable settings: solutions that achieve zero training loss will necessarily remain optimal even under weighting (Byrd & Lipton, 2019).
Revised: While intuitive, balancing has minimal effect in separable settings: solutions that achieve zero training loss will necessarily remain optimal even under weighting (Byrd & Lipton, 2019).

Original: Intuitively, one would instead like to shift the separator closer to a dominant class.
Revised: Intuitively, one would instead like to shift the separator closer to a dominant class.

Original: Li et al.
Revised: Li et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: Masnadi-Shirazi & Vasconcelos (2010) thus proposed to add per-class margins into the hinge loss.
Revised: Masnadi-Shirazi & Vasconcelos (2010) thus proposed to add per-class margins into the hinge loss.

Original: Cao et al.
Revised: Cao et al.

Original: (2019) similarly proposed to add a per-class margin into the softmax cross-entropy:
Revised: (2019) similarly proposed to add a per-class margin into the softmax cross-entropy:

Original: [Equation]
Revised: [Equation]

Original: where δ y ∝ P ( y ) − 1 / 4 .
Revised: where δ y ∝ P ( y ) − 1 / 4 .

Original: This upweights rare “positive” y to encourage a larger gap f y ( x ) − f y 0 ( x ) , i.e., the margin between y and any “negative” y 0 

Revised: This upweights rare “positive” y to encourage a larger gap f y ( x ) − f y 0 ( x ) , i.e., the margin between y and any “negative” y 0

Original: = y .
Revised: = y .

Original: Separately, Tan et al.
Revised: Separately, Tan et al.

Original: (2020) proposed
Revised: (2020) proposed

Original: [Equation]
Revised: [Equation]

Original: where δ y 0 ≤ 0 is an non-decreasing transform of P ( y 0 ) .
Revised: where δ y 0 ≤ 0 is a non-decreasing transform of P ( y 0 ) .

Original: Note that in the original softmax cross-entropy with δ y 

Revised: Note that in the original softmax cross-entropy with δ y

Original: = 0 , a rare label often receives a strong inhibitory gradient signal as it disproportionately appear as a negative for dominant labels.
Revised: = 0 , a rare label often receives a strong inhibitory gradient signal as it disproportionately appear as a negative for dominant labels.

Original: This can be modulated by letting δ y 0 (cid:28) 0 .
Revised: This can be modulated by letting δ y 0 (cid:28) 0 .

Original: Limitations of existing approaches .
Revised: 2. L IMITATIONS OF EXISTING APPROACHES

Original: Each of the above methods are intuitive, and have shown strong empirical performance.
Revised: Each of the above methods are intuitive, and have shown strong empirical performance.

Original: However,a closer analysis identiﬁes some subtle limitations.
Revised: However,

Original: Limitations of weight normalisation .
Revised: Limitations of weight normalisation .

Original: Post-hoc weight normalisation with ν y =
Revised: Post-hoc weight normalisation with ν y =

Original: k w y k 2 per Kang et al.
Revised: k w y k 2 per Kang et al.

Original: (2020) is motivated by the observation that the weight norm k w y k 2 tends to correlate with P ( y ) .
Revised: (2020) is motivated by the observation that the weight norm k w y k 2 tends to correlate with P ( y ) .

Original: However, this assumption is highly dependent on the choice of optimiser, as Figure 1 illustrates: for
Revised: However, this assumption is highly dependent on the choice of optimiser, as Figure 1 illustrates: for

Original: ResNet-32 models trained on long-tailed versions of CIFAR-10 and CIFAR-100, when using the
Revised: ResNet-32 models trained on long-tailed versions of CIFAR-10 and CIFAR-100, when using the

Original: Adam optimiser, the norms are either anti -correlated or independent of P ( y ) .
Revised: Adam optimiser, the norms are either anti -correlated or independent of P ( y ) .

Original: Weight normalisation thus cannot achieve the desired effect of boosting rare labels’ scores.
Revised: Weight normalisation thus cannot achieve the desired effect of boosting rare labels’ scores.

Original: One may hope to side-step this by simply using ν y =
Revised: One may hope to side-step this by simply using ν y =

Original: P ( y ) ; unfortunately, this choice has more subtle limitations (see §4.2).
Revised: P ( y ) ; unfortunately, this choice has more subtle limitations (see §4.2).

Original: Limitations of loss modiﬁcation .
Revised: Limitations of loss modiﬁcation .

Original: Enforcing a per-label margin per (5) and (6) is intuitive, as it allows for shifting the decision boundary away from rare classes.
Revised: Enforcing a per-label margin per (5) and (6) is intuitive, as it allows for shifting the decision boundary away from rare classes.

Original: However, when doing so, it is important to ensure Fisher consistency (Lin, 2004) (or classiﬁcation calibration (Bartlett et al., 2006)) of the resulting loss for the balanced error.
Revised: However, when doing so, it is important to ensure Fisher consistency (Lin, 2004) (or classiﬁcation calibration (Bartlett et al., 2006)) of the resulting loss for the balanced error.

Original: That is, the minimiser of the expected loss (equally, the empirical risk in the inﬁnite sample limit) should result in a minimal balanced error.
Revised: That is, the minimiser of the expected loss (equally, the empirical risk in the inﬁnite sample limit) should result in a minimal balanced error.

Original: Unfortunately, both (5) and (6) are not consistent in this sense, even for binary problems; see §5.1, §6.1 for details.
Revised: Unfortunately, both (5) and (6) are not consistent in this sense, even for binary problems; see §5.1, §6.1 for details.

Original:  L OGIT ADJUSTMENT FOR LONG - TAIL LEARNING : A STATISTICAL VIEW
Revised:  L OGIT ADJUSTMENT FOR LONG - TAIL LEARNING : A STATISTICAL VIEW

Original: The above suggests that there is scope for improving performance on long-tail problems, both in terms of post-hoc correction and loss modiﬁcation.
Revised: The above suggests that there is scope for improving performance on long-tail problems, both in terms of post-hoc correction and loss modiﬁcation.

Original: We now show how a statistical perspectiveon the problem suggests simple procedures of each type,  which overcome the limitations discussed above.
Revised: We now show how a statistical perspective suggests simple procedures of each type, both of which overcome the limitations discussed above.

Original: Recall that our goal is to minimise the balanced error (2).
Revised: Recall that our goal is to minimise the balanced error (2).

Original: A classical result is that the best possible or Bayes-optimal scorer for this problem, i.e., f ∗ ∈ argmin f : X → R L BER( f ) , satisﬁes the following (Menon et al., 2013), (Koyejo et al., 2014, Corollary 4), (Collell et al., 2016, Theorem 1):
Revised: A classical result is that the best possible or Bayes-optimal scorer for this problem, i.e., f ∗ ∈ argmin f : X → R L BER( f ) , satisﬁes the following (Menon et al., 2013), (Koyejo et al., 2014, Corollary 4), (Collell et al., 2016, Theorem 1):

Original: [Equation]
Revised: [Equation]

Original: where P bal is the balanced class-probability as per §2.
Revised: where P bal is the balanced class-probability as per §2.

Original: In words, the Bayes-optimal prediction is the label under which the given instance x ∈ X is most likely.
Revised: In words, the Bayes-optimal prediction is the label under which the given instance x ∈ X is most likely.

Original: Consequently, for ﬁxed class-conditionals
Revised: Consequently, for ﬁxed class-conditionals

Original: P ( x | y ) , varying the class priors P ( y ) arbitrarily will not affect the optimal scorers.
Revised: P ( x | y ) , varying the class priors P ( y ) arbitrarily will not affect the optimal scorers.

Original: This is intuitively desirable: the balanced error is agnostic to the level of imbalance in the label distribution.
Revised: This is intuitively desirable: the balanced error is agnostic to the level of imbalance in the label distribution.

Original: To further probe (7), suppose the underlying class-probabilities P ( y | x ) ∝ exp( s ∗ y ( x )) , for scorer s ∗ : X → R L .
Revised: To further probe (7), suppose the underlying class-probabilities P ( y | x ) ∝ exp( s ∗ y ( x )) , for scorer s ∗ : X → R L .

Original: Since by deﬁnition P bal ( y | x ) ∝ P ( y | x ) / P ( y ) , (7) becomes
Revised: Since by deﬁnition P bal ( y | x ) ∝ P ( y | x ) / P ( y ) , (7) becomes

Original: [Equation]
Revised: [Equation]

Original: i.e., we translate the optimal logits based on the class priors.
Revised: i.e., we translate the optimal logits based on the class priors.

Original: While s ∗ is of course unknown in practice, (8)immediately suggests two means of optimising for the balanced error:
Revised: Equation 8 provides the ideal predictions for optimising the balanced error, which necessarily rely on unknown quantities s ∗ y ( x ) , P ( y ) that depend on the underlying distribution. Nonetheless, we may seek to approximate these quantities based on our training sample. Concretely, (8) suggests two means of optimising for the balanced error:

Original: (i) train a model to estimate the standard P ( y | x ) (e.g., by minimising the standard softmax-cross entropy on the long-tailed data), and then explicitly modify its logits post-hoc as per (8) (ii) train a model to estimate the balanced P bal ( y | x ) , whose logits are implicitly modiﬁed as per (8).
Revised: (i) train a model to estimate the standard P ( y | x ) (e.g., by minimising the standard softmax-cross entropy on the long-tailed data), and then explicitly modify its logits post-hoc as per (8) (ii) train a model to estimate the balanced P bal ( y | x ) , whose logits are implicitly modiﬁed as per (8).

Original: Such logit adjustment techniques — which have been a classic approach to class-imbalance (Provost, 2000) — neatly align with the post-hoc and loss modiﬁcation streams discussed in §2.
Revised: Such logit adjustment techniques — which have been a classic approach to class-imbalance (Provost, 2000) — neatly align with the post-hoc and loss modiﬁcation streams discussed in §2.

Original: However, unlike most previous techniques from these streams, logit adjustment is endowed with a clear statistical grounding: by construction, the optimal solution under such adjustment coincides with the Bayes-optimal solution (7) for the balanced error, i.e., it is Fisher consistent for minimising the balanced error.
Revised: However, unlike most previous techniques from these streams, logit adjustment is endowed with a clear statistical grounding: by construction, the optimal solution under such adjustment coincides with the Bayes-optimal solution (7) for the balanced error, i.e., it is Fisher consistent for minimising the balanced error.

Original: We now study each of the techniques (i) and (ii) in turn.
Revised: We now study each of the techniques (i) and (ii) in turn.

Original:  P OST - HOC LOGIT ADJUSTMENT
Revised:  P OST - HOC LOGIT ADJUSTMENT

Original: We now propose a post-hoc logit adjustment scheme for a classiﬁer trained on long-tailed data.
Revised: We now propose a post-hoc logit adjustment scheme for a classiﬁer trained on long-tailed data.

Original: We further show this has subtle advantages over recent weight normalisation schemes.
Revised: We further show this has subtle advantages over recent weight normalisation schemes.

Original: 4. T HE POST - HOC LOGIT ADJUSTMENT PROCEDURE
Revised: 4. T HE POST - HOC LOGIT ADJUSTMENT PROCEDURE

Original: When employing the softmax cross-entropy to train a neural network, we aim to approximate the underlying P ( y | x ) with p y ( x ) ∝ exp( f y ( x )) for logits
Revised: When employing the softmax cross-entropy to train a neural network, we aim to approximate the underlying P ( y | x ) with p y ( x ) ∝ exp( f y ( x )) for logits

Original: f y ( x ) = w > y Φ( x ) .
Revised: f y ( x ) = w > y Φ( x ) .

Original: Given learned { w, Φ } , one typically predicts the label argmax
Revised: Given learned { w, Φ } , one typically predicts the label argmax

Original: y ∈
Revised: y ∈

Original: [ L ] f y ( x ) , i.e., the most likely label under the model’s
Revised: [ L ] f y ( x ) , i.e., the most likely label under the model’s

Original: P ( y | x ) .
Revised: P ( y | x ) .

Original: In post-hoc logit adjustment , we propose to instead predict, for suitable τ > 0 :
Revised: In post-hoc logit adjustment , we propose to instead predict, for suitable τ > 0 :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where π ∈ ∆ L (for simplex ∆ ) are estimates of the class priors P ( y ) , e.g., the empirical frequencies on the training sample S .
Revised: where π ∈ ∆ L (for simplex ∆ ) are estimates of the class priors P ( y ) , e.g., the empirical frequencies on the training sample S .

Original: Effectively, (9) adds a label-dependent offset to each of the logits.
Revised: Effectively, (9) adds a label-dependent offset to each of the logits.

Original: When τ = 1 , this can be seen as applying (8) with a plugin estimate of P ( y | x ) , i.e., p y ( x ) ∝ exp( w > y Φ( x )) .
Revised: When τ = 1 , this can be seen as applying (8) with a plugin estimate of P ( y | x ) , i.e., p y ( x ) ∝ exp( w > y Φ( x )) .

Original: When τ 6 = 1 , this can be seen as applying (8) to temperature scaled estimates ¯ p y ( x ) ∝ exp( τ − 1 · w > y Φ( x )) .
Revised: When τ 6 = 1 , this can be seen as applying (8) to temperature scaled estimates ¯ p y ( x ) ∝ exp( τ − 1 · w > y Φ( x )) .

Original: To unpack this, recall that (8) justiﬁes post-hoc logit thresholding given access to the true probabilities P ( y | x ) .
Revised: To unpack this, recall that (8) justiﬁes post-hoc logit thresholding given access to the true probabilities P ( y | x ) .

Original: In practice, high-capacity neural networks often produce uncalibrated estimates of these probabilities (Guo et al., 2017).
Revised: In practice, high-capacity neural networks often produce uncalibrated estimates of these probabilities (Guo et al., 2017).

Original: Temperature scaling is a means to calibrate the estimates, and is routinely employed for distillation (Hinton et al., 2015).
Revised: Temperature scaling is a means to calibrate the estimates, and is routinely employed for distillation (Hinton et al., 2015).

Original: One may treat τ as a tuning parameter to be chosen based on holdout calibration, e.g., the expected calibration error (Murphy & Winkler, 1987; Guo et al., 2017), probabilistic sharpness (Gneiting et al., ; Kuleshov et al., 2018), or a proper scoring rule such as the log-loss or squared error (Gneiting & Raftery, 2007).
Revised: One may treat τ as a tuning parameter to be chosen based on holdout calibration, e.g., the expected calibration error (Murphy & Winkler, 1987; Guo et al., 2017), probabilistic sharpness (Gneiting et al., 2007; Kuleshov et al., 2018), or a proper scoring rule such as the log-loss or squared error (Gneiting & Raftery, 2007).

Original: One may alternately ﬁx τ = 1 and aim to learn inherently calibrated probabilities, e.g., via label smoothing (Szegedy et al., 2016; Müller et al., 2019).
Revised: One may alternately ﬁx τ = 1 and aim to learn inherently calibrated probabilities, e.g., via label smoothing (Szegedy et al., 2016; Müller et al., 2019).

Original: Post-hoc logit adjustment with τ = 1 is not a new idea in the classical label imbalance literature (Fawcett & Provost, 1996; Provost, 2000; Maloof, 2003; Zhou & Liu, 2006;
Revised: Post-hoc logit adjustment with τ = 1 is not a new idea in the classical label imbalance literature (Fawcett & Provost, 1996; Provost, 2000; Maloof, 2003; Zhou & Liu, 2006;

Original: Collell et al., 2016); however, it has had limited exploration in the recent long-tail learning literature.
Revised: Collell et al., 2016); however, it has had limited exploration in the recent long-tail learning literature.

Original: Further, the case τ 6 = 1 is important in practical usage of neural networks, owing to their typical lack of probabilistic calibration (Guo et al., 2017).
Revised: Further, the case τ 6 = 1 is important in practical usage of neural networks, owing to their typical lack of probabilistic calibration (Guo et al., 2017).

Original: Interestingly, post-hoc logit adjustment also has an important advantage over recently proposed weight normalisation techniques, as we now discuss.
Revised: Interestingly, post-hoc logit adjustment also has an important advantage over recently proposed weight normalisation techniques, as we now discuss.

Original: 4. C OMPARISON TO POST - HOC WEIGHT NORMALISAITON
Revised: 4. C OMPARISON TO POST - HOC WEIGHT NORMALISAITON

Original: Recall that weight normalisation involves learning logits
Revised: Recall that weight normalisation involves learning logits

Original: f y ( x ) = w > y Φ( x ) , and then post-hoc normalising the weights via w y /ν τy for τ > 0 .
Revised: f y ( x ) = w > y Φ( x ) , and then post-hoc normalising the weights via w y /ν τy for τ > 0 .

Original: We demonstrated in §2 that using ν y =
Revised: We demonstrated in §2 that using ν y =

Original: k w y k 2 may be ineffective when using adaptive optimisers.
Revised: k w y k 2 may be ineffective when using adaptive optimisers.

Original: However, even with ν y = π y , there is a subtle contrast to post-hoc logit adjustment: while the former performs a multiplicative update to the logits, the latter performs an additive update.
Revised: However, even with ν y = π y , there is a subtle contrast to post-hoc logit adjustment: while the former performs a multiplicative update to the logits, the latter performs an additive update.

Original: The two techniques may thus yield different orderings over labels, since
Revised: The two techniques may thus yield different orderings over labels, since

Original: [Equation]
Revised: [Equation]

Original: Observe that if a rare label y has  negative score w > y Φ( x ) < 0 , and there is another label with  positive score, then it is impossible for the weight normalisation to give y the highest score.
Revised: Observe that if a rare label y has a negative score w > y Φ( x ) < 0 , and there is another label with a positive score, then it is impossible for the weight normalisation to give y the highest score.

Original: By contrast, under logit adjustment, w > y Φ( x ) − ln π y will be lower for dominant classes, regardless of the original sign.
Revised: By contrast, under logit adjustment, w > y Φ( x ) − ln π y will be lower for dominant classes, regardless of the original sign.

Original: Weight normalisation is thus not consistent for the balanced error, unlike logit adjustment.
Revised: Weight normalisation is thus not consistent for the balanced error, unlike logit adjustment.

Original:  T HE LOGIT ADJUSTED SOFTMAX CROSS - ENTROPY
Revised:  T HE LOGIT ADJUSTED SOFTMAX CROSS - ENTROPY

Original: We now show how to directly encode logit adjustment into the softmax cross-entropy.
Revised: We now show how to directly encode logit adjustment into the softmax cross-entropy.

Original: The resulting approach has an intuitive relation to existing loss modiﬁcation techniques.
Revised: The resulting approach has an intuitive relation to existing loss modiﬁcation techniques.

Original: 5. T HE LOGIT ADJUSTED LOSS
Revised: 5. T HE LOGIT ADJUSTED LOSS

Original: From §3, the second approach to optimising for the balanced error is to directly model P bal ( y | x ) ∝
Revised: From §3, the second approach to optimising for the balanced error is to directly model P bal ( y | x ) ∝

Original: P ( y | x ) / P ( y ) .
Revised: P ( y | x ) / P ( y ) .

Original: To do so, consider the following logit adjusted softmax cross-entropy loss for τ > 0 :
Revised: To do so, consider the following logit adjusted softmax cross-entropy loss for τ > 0 :

Original: [Equation]
Revised: [Equation]

Original: Given a scorer that minimises the above, we now predict argmax y ∈
Revised: Given a scorer that minimises the above, we now predict argmax y ∈

Original: [ L ] f y ( x ) as usual.
Revised: [ L ] f y ( x ) as usual.

Original: Compared to the standard softmax cross-entropy (1), the above applies a label-dependent offset to each logit.
Revised: Compared to the standard softmax cross-entropy (1), the above applies a label-dependent offset to each logit.

Original: Compared to (9), we directly enforce the class prior offset while learning the logits, rather than doing this post-hoc.
Revised: Compared to (9), we directly enforce the class prior offset while learning the logits, rather than doing this post-hoc.

Original: The two approaches have a deeper connection: observe that (10) is equivalent to using a scorer of the form g y ( x ) =
Revised: The two approaches have a deeper connection: observe that (10) is equivalent to using a scorer of the form g y ( x ) =

Original: f y ( x ) + τ · log π y , with argmax y ∈
Revised: f y ( x ) + τ · log π y , with argmax y ∈

Original: [ L ] f y ( x ) = argmax
Revised: [ L ] f y ( x ) = argmax

Original: y ∈
Revised: y ∈

Original: [ L ] g y ( x ) − τ · log π y .
Revised: [ L ] g y ( x ) − τ · log π y .

Original: Consequently, one can equivalently view learning with this loss as learning a standard scorer g ( x ) , and post-hoc adjusting its logits.
Revised: Consequently, one can equivalently view learning with this loss as learning a standard scorer g ( x ) , and post-hoc adjusting its logits.

Original: For non-convex objectives, as encountered in neural networks, the bias endowed by adding τ · log π y to the logits is likely to result in a different local minima, typically with improved performance.
Revised: For non-convex objectives, as encountered in neural networks, the bias endowed by adding τ · log π y to the logits is likely to result in a different local minima, typically with improved performance.

Revised: 5. C OMPARISON TO LOSS MODIFICATION TECHNIQUES

Original: For more insight into the logit adjusted loss, consider the following pairwise margin loss
Revised: For more insight into the logit adjusted loss, consider the following pairwise margin loss

Original: [Equation]
Revised: [Equation]

Original: for label weights α y > 0 , and pairwise label margins ∆ yy 0 representing the desired gap between scores for y and y 0 .
Revised: for label weights α y > 0 , and pairwise label margins ∆ yy 0 representing the desired gap between scores for y and y 0 .

Original: For τ = 1 , our logit adjusted loss (10) corresponds to (11) with α y = 1 and ∆ yy 

Revised: For τ = 1 , our logit adjusted loss (10) corresponds to (11) with α y = 1 and ∆ yy

Original: = log (cid:16) π y 

Revised: = log (cid:16) π y

Original: π y (cid:17) .
Revised: π y (cid:17) .

Original: This demands a larger margin between rare positive ( π y ∼ 0 ) and dominant negative ( π y 0 ∼ 1 ) labels, so that scores for dominant classes do not overwhelm those for rare ones.
Revised: This demands a larger margin between rare positive ( π y ∼ 0 ) and dominant negative ( π y 0 ∼ 1 ) labels, so that scores for dominant classes do not overwhelm those for rare ones.

Original: Existing loss modiﬁcation techniques can be viewed as special cases of (11).
Revised: Existing loss modiﬁcation techniques can be viewed as special cases of (11).

Original: For example, α y = 1 /π y and ∆ yy 0 = 0 yields the balanced loss (4).
Revised: For example, α y = 1 /π y and ∆ yy 0 = 0 yields the balanced loss (4).

Original: When α y = 1 , the choice ∆ yy 

Revised: When α y = 1 , the choice ∆ yy

Original: = π − 1 / 4 y yields (5).
Revised: = π − 1 / 4 y yields (5).

Original: Finally, ∆ yy 

Revised: Finally, ∆ yy

Original: = log F ( π y 0 ) yields (6), where F : [0 , 1] → (0 , 1] is some non-decreasing function.
Revised: = log F ( π y 0 ) yields (6), where F : [0 , 1] → (0 , 1] is some non-decreasing function.

Original: These losses thus either consider the frequency of the positive y or negative y 0 , but not both .
Revised: These losses thus either consider the frequency of the positive y or negative y 0 , but not both .

Original: Remarkably, the speciﬁc choice that leads to our loss in (10) has a ﬁrm statistical grounding: it ensures Fisher consistency (in the sense of, e.g., Bartlett et al.
Revised: Remarkably, the speciﬁc choice that leads to our loss in (10) has a ﬁrm statistical grounding: it ensures Fisher consistency (in the sense of, e.g., Bartlett et al.

Original: for the balanced error.
Revised: for the balanced error.

Revised: Proofs for all results are in the Appendix.

Original: Theorem 1.
Revised: Theorem 1.

Original: For any δ ∈ R L + , the pairwise loss in (11) is Fisher consistent with weights and margins
Revised: For any δ ∈ R L + , the pairwise loss in (11) is Fisher consistent with weights and margins

Original: [Equation]
Revised: [Equation]

Original: Letting δ y
Revised: Letting δ y

Original: =
Revised: =

Original: π y , we immediately deduce that the logit-adjusted loss of (10) is consistent, provided our π y is a consistent estimate of P ( y ) .
Revised: π y , we immediately deduce that the logit-adjusted loss of (10) is consistent, provided our π y is a consistent estimate of P ( y ) .

Original: Similarly, δ y = 1 recovers the classic result that the balanced loss is consistent.
Revised: Similarly, δ y = 1 recovers the classic result that the balanced loss is consistent.

Original: While Theorem 1 only provides a sufﬁcient condition in multi-class setting, one can provide a necessary and sufﬁcient condition that rules out other choices of ∆ in the binary case.
Revised: While Theorem 1 only provides a sufﬁcient condition in multi-class setting, one can provide a necessary and sufﬁcient condition that rules out other choices of ∆ in the binary case.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose Y = {± 1 } .
Revised: Suppose Y = {± 1 } .

Original: Let δ y .
Revised: Let δ y .

Original: =
Revised: =

Original: ∆ y, − y , and σ ( z ) =
Revised: ∆ y, − y , and σ ( z ) =

Original: (1 + exp
Revised: (1 + exp

Original: z ))
Revised: z ))

Original: − 1 .
Revised: − 1 .

Original: Then, the pairwise margin loss in (11) is Fisher consistent for the balanced error iff
Revised: Then, the pairwise margin loss in (11) is Fisher consistent for the balanced error iff

Original: [Equation]
Revised: [Equation]

Original: 5. D ISCUSSION AND EXTENSIONS
Revised: 5. D ISCUSSION AND EXTENSIONS

Original: Our pairwise margin loss in (11) subsumes several existing loss-correction approaches in the literature.
Revised: Our pairwise margin loss in (11) subsumes several existing loss-correction approaches in the literature.

Original: Further, it suggests the exploration of new choices of ∆ .
Revised: Further, it suggests the exploration of new choices of ∆ .

Original: For example, we shall see the efﬁcacy of combining the ∆ implicit in the adaptive loss of Cao et al.
Revised: For example, we shall see the efﬁcacy of combining the ∆ implicit in the adaptive loss of Cao et al.

Original: with our proposed ∆ .
Revised: with our proposed ∆ .

Original: One may also generalise the formulation in Theorem 1, and employ ∆ yy 

Revised: One may also generalise the formulation in Theorem 1, and employ ∆ yy

Original: = τ 1 · log π y − τ 2 · log π y 0 , where τ 1 , τ 2 > 0 .
Revised: = τ 1 · log π y − τ 2 · log π y 0 , where τ 1 , τ 2 > 0 .

Original: This interpolates between our loss ( τ 1 = τ 2 ) and a version of the equalised loss ( τ 1 = 0 ).
Revised: This interpolates between our loss ( τ 1 = τ 2 ) and a version of the equalised loss ( τ 1 = 0 ).

Original: For τ = − 1 , a similar loss to (10) has been considered in the context of negative sampling for scalability (Yi et al., 2019): here, one samples a subset of negatives based on π , and corrects the logits to obtain an unbiased estimate of the loss based on all negatives (Bengio & Senecal, 2008).
Revised: For τ = − 1 , a similar loss to (10) has been considered in the context of negative sampling for scalability (Yi et al., 2019): here, one samples a subset of negatives based on π , and corrects the logits to obtain an unbiased estimate of the loss based on all negatives (Bengio & Senecal, 2008).

Original: Losses of the general form (11) have also been explored for structured prediction (Pletscher et al., 2010).
Revised: Losses of the general form (11) have also been explored for structured prediction (Pletscher et al., 2010).

Original: Cao et al.
Revised: Cao et al.

Original: (2019, Theorem 2) provides a rigorous generalisation bound for the adaptive margin loss under the assumption of separable data with binary labels.
Revised: (2019, Theorem 2) provides a rigorous generalisation bound for the adaptive margin loss under the assumption of separable data with binary labels.

Original: The inconsistency of the loss with respect to the balanced error concerns the more general scenario of non-separable multiclass data, which may occur, e.g., owing to label noise or limited model capacity.
Revised: The inconsistency of the loss with respect to the balanced error concerns the more general scenario of non-separable multiclass data, which may occur, e.g., owing to label noise or limited model capacity.

Original: We shall subsequently demonstrate that encouraging consistency is not merely of theoretical interest, and can lead to gains in practice. 
Revised: We shall subsequently demonstrate that encouraging consistency is not merely of theoretical interest, and can lead to gains in practice. [Figure] L o g i [Figure]

Revised: a

Original: E XPERIMENTAL RESULTS
Revised: [Figure] u [Figure] e

Revised: [Figure]

Revised: [Figure]

Revised: B a l a n c e d e rr o r

Original:  We now present experiments conﬁrming our main claims: (i) on simple binary problems, existing weight normalisation and loss modiﬁcation techniques may not converge to the optimal solution (§6.1); (ii) on real-world datasets, our post-hoc logit adjustment generally outperforms weight normalisation, and one can obtain further gains via our logit adjusted softmax cross-entropy (§6.2).
Revised: [Figure] [Figure]  E XPERIMENTAL RESULTS We now present experiments conﬁrming our main claims: (i) on simple binary problems, existing weight normalisation and loss modiﬁcation techniques may not converge to the optimal solution (§6.1); (ii) on real-world datasets, our post-hoc logit adjustment generally outperforms weight normalisation, and one can obtain further gains via our logit adjusted softmax cross-entropy (§6.2).

Original: 6. R ESULTS ON SYNTHETIC DATASET
Revised: 6. R ESULTS ON SYNTHETIC DATASET

Original: We begin with a binary classiﬁcation task, wherein samples from class y ∈ {± 1 } are drawn from a 2D Gaussian with isotropic covariance and means µ y
Revised: We begin with a binary classiﬁcation task, wherein samples from class y ∈ {± 1 } are drawn from a 2D Gaussian with isotropic covariance and means µ y

Original: =
Revised: =

Original: y · (+1 , +1) .
Revised: y · (+1 , +1) .

Original: We introduce class imbalance by setting P ( y = +1)
Revised: We introduce class imbalance by setting P ( y = +1)

Original: = 5% .
Revised: = 5% .

Original: The Bayes-optimal classiﬁer for the balanced error is (see Appendix F) 
Revised: The Bayes-optimal classiﬁer for the balanced error is (see Appendix F) f ∗ ( x ) =

Original: [Equation]{++
 ⇐⇒ P ( x | y = +1) > P ( x | y = − 1)+}
Revised: {++
 ⇐⇒ P ( x | y = +1) > P ( x | y = − 1)+}

Original:  i.e., it is a linear separator passing through the origin.
Revised: ⇐⇒ ( µ 1 − µ − 1 ) > x > 0 , (12) i.e., it is a linear separator passing through the origin.

Original: We compare this separator against those found by several margin losses based on (11): standard ERM ( ∆ yy 0 = 0 ), the adaptive loss (Cao et al., 2019) ( ∆ yy 

Revised: We compare this separator against those found by several margin losses based on (11): standard ERM ( ∆ yy 0 = 0 ), the adaptive loss (Cao et al., 2019) ( ∆ yy

Original: = π − 1 / 4 y ), an instantiation of the equalised loss (Tan et al., 2020) ( ∆ yy 

Revised: = π − 1 / 4 y ), an instantiation of the equalised loss (Tan et al., 2020) ( ∆ yy

Original: = log π y 0 ), and our logit adjusted loss ( ∆ yy 

Revised: = log π y 0 ), and our logit adjusted loss ( ∆ yy

Original: = log π y 

Revised: = log π y

Original: π y ).
Revised: π y ).

Original: For each loss, we train an afﬁne classiﬁer on a sample of 10 ,instances, and evaluate the balanced error on a test set of 10 , 000 samples over 100 independent trials.[Figure] [Table]
Revised: For each loss, we train an afﬁne classiﬁer on a sample of 10 ,instances, and evaluate the balanced error on a test set of 10 , 000 samples over 100 independent trials.

Original: Figure 2 conﬁrms that the logit adjusted margin loss attains a balanced error close to that of the
Revised: Figure 2 conﬁrms that the logit adjusted margin loss attains a balanced error close to that of the

Original: Bayes-optimal, which is visually reﬂected by its learned separator closely matching that in (12).
Revised: Bayes-optimal, which is visually reﬂected by its learned separator closely matching that in (12).

Original: This is in line with our claim of the logit adjusted margin loss being consistent for the balanced error, unlike other approaches.
Revised: This is in line with our claim of the logit adjusted margin loss being consistent for the balanced error, unlike other approaches.

Original: Figure 2 also compares post-hoc weight normalisation and logit adjustment for varying scaling parameter τ (c.f. (3), (9)).
Revised: Figure 2 also compares post-hoc weight normalisation and logit adjustment for varying scaling parameter τ (c.f. (3), (9)).

Original: Logit adjustment is seen to approach the performance of the Bayes predictor; any weight normalisation is however seen to hamper performance.
Revised: Logit adjustment is seen to approach the performance of the Bayes predictor; any weight normalisation is however seen to hamper performance.

Original: This veriﬁes the consistency of logit adjustment, and inconsistency of weight normalisation (§4.2).
Revised: This veriﬁes the consistency of logit adjustment, and inconsistency of weight normalisation (§4.2).

Original: 6. R ESULTS ON REAL - WORLD DATASETS
Revised: 6. R ESULTS ON REAL - WORLD DATASETS

Original: We present results on the CIFAR-10, CIFAR-100, ImageNet and iNaturalist 2018 datasets.
Revised: We present results on the CIFAR-10, CIFAR-100, ImageNet and iNaturalist 2018 datasets.

Original: Following prior work, we create “long-tailed versions” of the CIFAR datasets by suitably downsampling examples per label following the E XP proﬁle of Cui et al.
Revised: Following prior work, we create “long-tailed versions” of the CIFAR datasets by suitably downsampling examples per label following the E XP proﬁle of Cui et al.

Original: ; Cao et al.
Revised: ; Cao et al.

Original: with imbalance ratio ρ = max y P ( y ) / min y P ( y ) = 100 .
Revised: with imbalance ratio ρ = max y P ( y ) / min y P ( y ) = 100 .

Original: Similarly, we use the long-tailed version of ImageNet produced by Liu et al.
Revised: Similarly, we use the long-tailed version of ImageNet produced by Liu et al.

Original: We employ a ResNet-32 for CIFAR, and a ResNet-50 for ImageNet and iNaturalist.
Revised: We employ a ResNet-32 for CIFAR, and a ResNet-50 for ImageNet and iNaturalist.

Original: All models are trained using SGD with momentum; see Appendix C for more details.
Revised: All models are trained using SGD with momentum; see Appendix C for more details.

Original: See also Appendix D.4 for results on CIFAR under the S TEP proﬁle also considered in the literature.
Revised: See also Appendix D.3 for results on CIFAR under the S TEP proﬁle also considered in the literature.

Original: Baselines .
Revised: Baselines .

Original: We consider several representative baselines: (i) empirical risk minimisation (ERM) on the long-tailed data, (ii) post-hoc weight normalisation (Kang et al., 2020) per (3) (using ν y = k w y k 2 ) applied to ERM, (iii) the class-balanced loss of Cui et al.
Revised: We consider several representative baselines: (i) empirical risk minimisation (ERM) on the long-tailed data, (ii) post-hoc weight normalisation (Kang et al., 2020) per (3) (using ν y = k w y k 2 ) applied to ERM, (iii) the class-balanced loss of Cui et al.

Original: (2019), (iv) the adaptive margin loss (Cao et al., 2019) per (5), including with the “deferred reweighting” (DRW) training scheme, and (v) the equalised loss
Revised: (2019), (iv) the adaptive margin loss (Cao et al., 2019) per (5), including with the “deferred reweighting” (DRW) training scheme, and (v) the equalised loss

Original: (Tan et al., 2020) per (6), with δ y 

Revised: (Tan et al., 2020) per (6), with δ y

Original: = F ( π y 0 ) for the threshold-based F of Tan et al.
Revised: = F ( π y 0 ) for the threshold-based F of Tan et al.

Original: Where possible, we report numbers for the baselines from the respective papers.
Revised: Where possible, we report numbers for the baselines from the respective papers.

Original:  We compare the above methods against our proposed post-hoc logit adjustment (9), and logit adjusted loss (10).
Revised: [Table] [Figure] We compare the above methods against our proposed post-hoc logit adjustment (9), and logit adjusted loss (10).

Original: For post-hoc logit adjustment, we ﬁx the scalar τ = 1 for our basic results; we analyse the effect of tuning this in Figure 3.
Revised: For post-hoc logit adjustment, we ﬁx the scalar τ = 1 for our basic results; we analyse the effect of tuning this in Figure 3.

Original: We additionally evaluate a combination of our logit[Figure] [Figure] adjusted softmax cross-entropy with the adaptive margin of Cao et al.
Revised: We additionally evaluate a combination of our logit adjusted softmax cross-entropy with the adaptive margin of Cao et al.

Original: (2019); this uses (11) with ∆ yy 0 = log( π y 0 /π y )
Revised: (2019); this uses (11) with ∆ yy 0 = log( π y 0 /π y )

Original: + π − 0 .
Revised: + π − 0 .

Original: 25 y .
Revised: 25 y .

Original: We do not perform any further tuning of our techniques.
Revised: We do not perform any further tuning of our techniques.

Revised: For all methods, we report the balanced error on the test set.

Revised: (Note that since the test sets are all balanced for these benchmarks, the balanced error is equivalent to the misclassiﬁcation error.)

Original: Results and analysis .
Revised: Results and analysis .

Original: Table 2 demonstrates that our proposed logit adjustment techniques consistently outperform existing methods.
Revised: Table 2 demonstrates that our proposed logit adjustment techniques consistently outperform existing methods.

Original: Indeed, weight normalisation  is generally improved signiﬁcantly by post-hoc logit adjustment (e.g., 8% relative reduction on CIFAR-10).
Revised: Indeed, weight normalisation with τ = 1 is generally improved signiﬁcantly by post-hoc logit adjustment (e.g., 8% relative reduction on CIFAR-10).

Original: Similarly, loss correction techniques are generally outperformed by our logit adjusted softmax cross-entropy (e.g., 6% relative reduction on iNaturalist).
Revised: Similarly, loss correction techniques are generally outperformed by our logit adjusted softmax cross-entropy (e.g., 6% relative reduction on iNaturalist).

Original: We remark that Cao et al.
Revised: We remark that Cao et al.

Original: (2019) observed that their loss beneﬁts from a deferred reweighting scheme (DRW), wherein class-weighting is applied after a ﬁxed number of epochs.
Revised: (2019) observed that their loss beneﬁts from a deferred reweighting scheme (DRW), wherein class-weighting is applied after a ﬁxed number of epochs.

Original: Table 2 indicates this is consistently outperformed by suitable variants of logit adjustment.
Revised: Table 2 indicates this is consistently outperformed by suitable variants of logit adjustment.

Original: While both logit adjustment techniques perform similarly, there is a slight advantage to the loss function version.
Revised: While both logit adjustment techniques perform similarly, there is a slight advantage to the loss function version.

Original: Nonetheless, the strong performance of post-hoc logit adjustment corroborates the ability to decouple representation and classiﬁer learning in long-tail settings (Zhang et al., 2019).
Revised: Nonetheless, the strong performance of post-hoc logit adjustment corroborates the ability to decouple representation and classiﬁer learning in long-tail settings (Zhang et al., 2019).

Original: Note that Table 2 only reports the results for  scalar τ = 1 .
Revised: Table 2 only reports the results for logit adjustment with scalar τ = 1 .

Original: In practice, tuning this parameter can signiﬁcantly improve performance further.
Revised: In practice, tuning τ can signiﬁcantly improve performance further.

Original: Figure 3 studies the effect of tuning τ for post-hoc weight normalisation (using ν y =
Revised: Figure 3 studies the effect of tuning τ for post-hoc weight normalisation (using ν y =

Original: k w y k 2 ) and post-hoc logit adjustment.
Revised: k w y k 2 ) and post-hoc logit adjustment.

Original: Even without any scaling, post-hoc logit adjustment generally offers superior performance to the best result from weight normalisation (cf.
Revised: Even without any scaling, post-hoc logit adjustment generally offers superior performance to the best result from weight normalisation (cf.

Original: Table 2); with scaling, this is further improved.
Revised: Table 2); with scaling, this is further improved.

Original: See Appendix D.4 for a plot on ImageNet-LT.
Revised: In practice, one may choose τ via cross-validation against the balanced error on the training set. For example, on CIFAR-10-LT, we estimate τ ∗ = 2 .for post-hoc logit adjustment, for which the resulting balanced test error of 18 . 73 % is superior to that of weight normalisation for any τ .

Original: To better understand the gains, Figure 4 reports errors on a per-group basis, where following Kang et al.
Revised: To better understand the gains, Figure 4 reports errors on a per-group basis, where following Kang et al.

Original: we construct three groups of classes — “Many”, “Medium”, and “Few” — comprising  those with ≥ 100, between (20 , 100) , and ≤ 20 training examples respectively.
Revised: we construct three groups of classes — “Many”, “Medium”, and “Few” — comprising [Figure] those with ≥ 100, between (20 , 100) , and ≤ 20 training examples respectively.

Original: Logit adjustment shows consistent gains over all three groups.
Revised: Logit adjustment shows consistent gains over all three groups.

Original: See Appendix D.3 for a ﬁner-grained breakdown.
Revised: See Appendix D.2 for a ﬁner-grained breakdown.

Original: Discussion and extensions Table 2 shows the advantage of logit adjustment over recent proposals, under standard setups from the literature.
Revised: Discussion and extensions Table 2 shows the advantage of logit adjustment over recent proposals, under standard setups from the literature.

Original: Further improvements are possible by fusing complementary ideas, and we remark on a few such options.
Revised: Further improvements are possible by fusing complementary ideas, and we remark on a few such options.

Original: First, one may use a more complex base architecture; e.g., Kang et al.
Revised: First, one may use a more complex base architecture; e.g., Kang et al.

Original: (2020) found gains by employing a ResNet-152, and training for 200 epochs.
Revised: (2020) found gains by employing a ResNet-152, and training for 200 epochs.

Original: Table(Appendix) conﬁrms that logit adjustment similarly beneﬁts from this choice, achieving a balanced error of 30 .
Revised: Table(Appendix) conﬁrms that logit adjustment similarly beneﬁts from this choice, achieving a balanced error of 30 .

Original: 12 % on iNaturalist, and 28 .
Revised: 12 % on iNaturalist, and 28 .

Original: 02 % when combined with the adaptive margin.
Revised: 02 % when combined with the adaptive margin.

Original: Second, the DRW training scheme (which applies to any loss) may result in further gains for our techniques.
Revised: Second, the DRW training scheme (which applies to any loss) may result in further gains for our techniques.

Original: Third, incorporating developments in meta-learning (Wang et al., 2017; Jamal et al., 2020) is also a promising avenue.
Revised: Third, incorporating developments in meta-learning (Wang et al., 2017; Jamal et al., 2020) is also a promising avenue.

Original: While further exploring such variants are of empirical interest, we hope to have illustrated the conceptual and empirical value of logit adjustment, and leave this for future work.
Revised: While further exploring such variants are of empirical interest, we hope to have illustrated the conceptual and empirical value of logit adjustment, and leave this for future work