  Deep models trained on long-tailed datasets exhibit unsatisfactory
performance on tail classes. Existing methods usually modify the classification
loss to increase the learning focus on tail classes, which unexpectedly
sacrifice the performance on head classes. In fact, this scheme leads to a
contradiction between the two goals of long-tailed learning, i.e., learning
generalizable representations and facilitating learning for tail classes. In
this work, we explore knowledge distillation in long-tailed scenarios and
propose a novel distillation framework, named Balanced Knowledge Distillation
(BKD), to disentangle the contradiction between the two goals and achieve both
simultaneously. Specifically, given a vanilla teacher model, we train the
student model by minimizing the combination of an instance-balanced
classification loss and a class-balanced distillation loss. The former benefits
from the sample diversity and learns generalizable representation, while the
latter considers the class priors and facilitates learning mainly for tail
classes. The student model trained with BKD obtains significant performance
gain even compared with its teacher model. We conduct extensive experiments on
several long-tailed benchmark datasets and demonstrate that the proposed BKD is
an effective knowledge distillation framework in long-tailed scenarios, as well
as a new state-of-the-art method for long-tailed learning. Code is available at
https://github.com/EricZsy/BalancedKnowledgeDistillation .
