Original: PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
Revised: PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient

Original: Abstract
Revised: Abstract

Original: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection.
Revised: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection.

Original: However, there is still a lack of study on how to distill between heterogeneous detectors.
Revised: However, there is still a lack of study on how to distill between heterogeneous detectors.

Original: In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although their detection heads and label assignments are different.
Revised: In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although their detection heads and label assignments are different.

Original: However, directly aligning the feature maps to distill detectors suffers from two problems.
Revised: However, directly aligning the feature maps to distill detectors suffers from two problems.

Original: First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints on the student.
Revised: First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints on the student.

Original: Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise.
Revised: Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise.

Original: To address the above issues, we propose to imitate features with Pearson
Revised: To address the above issues, we propose to imitate features with Pearson

Original: Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features.
Revised: Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features.

Original: Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teacher pairs.
Revised: Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teacher pairs.

Original: Furthermore, it converges faster.
Revised: Furthermore, it converges faster.

Original: With a powerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNet and FCOS achieve 41.5% and 43.9% mAP on COCO2017, which are 4.1% and 4.8% higher than the baseline, respectively.
Revised: With a powerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNet and FCOS achieve 41.5% and 43.9% mAP on COCO2017, which are 4.1% and 4.8% higher than the baseline, respectively.

Original: Our implementation is available at https://github.com/open-mmlab/mmrazor.
Revised: Our implementation is available at https://github.com/open-mmlab/mmrazor.

Original:  Introduction
Revised:  Introduction

Original: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection.
Revised: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection.

Original: However, there is still a lack of study on how to distill between heterogeneous detectors.
Revised: However, there is still a lack of study on how to distill between heterogeneous detectors.

Original: Most previous works [38, 33, 9, 19, 12] rely on detector-specific designs and can only be applied to homogeneous detectors.
Revised: Most previous works [39, 34, 9, 19, 12] rely on detector-specific designs and can only be applied to homogeneous detectors.

Original: [41, 46] conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted.
Revised: [43, 48] conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted.

Original: Object detection is developing rapidly and algorithms with better performance are proposed continuously.
Revised: Object detection is developing rapidly and algorithms with better performance are proposed continuously.

Original: Nevertheless, it is not easy to change detectors frequently in terms of stability in practical applications.
Revised: Nevertheless, it is not easy to change detectors frequently in terms of stability in practical applications.

Original: Furthermore, in some scenarios, only detectors with a specific architecture can be deployed due to hardware limitations (e.g., two-stage detectors are hard to deploy), while most powerful teachers belong to different categories.
Revised: Furthermore, in some scenarios, only detectors with a specific architecture can be deployed due to hardware limitations (e.g., two-stage detectors are hard to deploy), while most powerful teachers belong to different categories.

Original: Thus, it is promising if knowledge distillation can be conducted between heterogeneous detector pairs.
Revised: Thus, it is promising if knowledge distillation can be conducted between heterogeneous detector pairs.

Original: In addition, current distillation methods, such as [41, 44], usually introduce several complementary loss functions to further improve their performance, so several hyper-parameters are used to adjust the contribution of each loss function which heavily affects their abilities to transfer to other datasets.
Revised: In addition, current distillation methods, such as [43, 46], usually introduce several complementary loss functions to further improve their performance, so several hyper-parameters are used to adjust the contribution of each loss function which heavily affects their abilities to transfer to other datasets.

Original: In this paper, we first empirically verify that FPN feature imitation does make sense even if the student-teacher detector pairs are heterogeneous.
Revised: In this paper, we first empirically verify that FPN feature imitation does make sense even if the student-teacher detector pairs are heterogeneous.

Original: However, directly minimizing the Mean Square
Revised: However, directly minimizing the Mean Square

Original: Error (MSE) between features of the teacher and the student leads to sub-optimal results, with results shown in Table 4.
Revised: Error (MSE) between features of the teacher and the student leads to sub-optimal results, with results shown in Table 5.

Original: Similar conclusions are drawn in [6, 12, 46, 41].
Revised: Similar conclusions are drawn in [6, 12, 48, 43].

Original: In order to explore the limitations of MSE, we elaborately visualize the FPN feature responses of the teacher and student detection models, as shown in Figure 1.
Revised: In order to explore the limitations of MSE, we elaborately visualize the FPN feature responses of the teacher and student detection models, as shown in Figure 1.

Original: Specifically, for an output feature s l ∈ R C × H × W from l -th FPN stage, we select the maximum value in the dimension C at each pixel and obtain a 2-D matrix.
Revised: Specifically, for an output feature s l ∈ R C × H × W from l -th FPN stage, we select the maximum value in the dimension C at each pixel and obtain a 2-D matrix.

Original: Then we normalize the values to 0-255 according to the maximum and minimum values of l 2-D matrices.
Revised: Then we normalize the values to 0-255 according to the maximum and minimum values of l 2-D matrices.

Original: Through these comparisons, we obtain the following observations:
Revised: Through these comparisons, we obtain the following observations:

Original: (1) The feature value magnitude of the teacher and the student is different, especially for heterogeneous detectors, as shown in Figure 1 (left).
Revised: (1) The feature value magnitudes of the teacher and the student are different, especially for heterogeneous detectors, as shown in Figure 1 (left).

Original: So directly aligning the feature maps between the teacher and the student may enforce overly strict constraints and do harm to the student.
Revised: So directly aligning the feature maps between the teacher and the student may enforce overly strict constraints and do harm to the student.

Original: (2) There are always several FPN stageswhose values are larger than the others, as shown in Figure 1 (right).
Revised: The values of several FPN stages are larger than the others, as shown in Figure 1 (right).

Original: It is obvious that FPN stage ’P6’ of the teacher is less activated than FPN stage ’P3’.
Revised: It is obvious that FPN stage ’P6’ of the teacher is less activated than FPN stage ’P3’

Original: However, for detectors such as RetinaNet [23] and FCOS[35], all FPN stagesshare the same detection head.
Revised: However, for detectors such as RetinaNet [23] and FCOS all FPN stages

Original:  Hence, FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD and lead to sub-optimal results.
Revised: share the same detection head Hence, FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD and lead to sub-optimal results.

Original: (3) There are always a few channels whose values  are significantly larger than the others, as shown in Figure 1 (middle).
Revised: (3) The values of some channels are significantly larger than the others, as shown in Figure(middle).

Original: However, it is mentioned in [33, 45] that the less activated features are still practical for distillation.
Revised: However, it is mentioned in [34, 47] that the less activated features are still practical for distillation.

Original: If they are not correctly balanced, the small gradients produced by these less activated channels can be drowned in the large gradients produced by  dominant ones, thus limiting further refinement.
Revised: If they are not correctly balanced, the small gradients produced by these less activated channels can be drowned in the large gradients produced by the dominant ones, thus limiting further refinement.

Original: Furthermore, from the first column of Figure 1 (right), we observe that there is much noise in the object-irrelevant area . To understand why, recall that we select themaximum value in the dimension C at each pixel, while values of several channels are  larger than others.
Revised: Furthermore, from the first column of Figure 1 ( right) , we observe that there is much noise in the object-irrelevant area, because the values of the pixels in certain channels are significantly larger than those in other channels, thus being visualized in the

Revised: figure.

Original: Noise in these channels is visualized.
Revised: However, these pixels may be noise.

Original: Hence, directly imitating the feature maps may introduce much noise.
Revised: Hence, directly imitating the feature maps may introduce much noise.

Original: According to these observations, we propose K nowledge D istillation via P earson Correlation Coefficient (PKD) shown in Figure 2, which focuses on the linear correlation between features of the teacher and the student.
Revised: According to these observations, we propose K nowledge D istillation via P earson Correlation Coefficient (PKD) shown in Figure 2, which focuses on the linear correlation between features of the teacher and the student.

Original: To remove the negative influences of magnitude difference, dominant FPN stages and channels, we first normalize the feature maps to have zero mean and unit variances and minimize the MSE loss between the normalized features.
Revised: To remove the negative influences of magnitude difference between the teacher-student detector pair and within the detector among different FPN stages and channels, we first normalize the feature maps to have zero mean and unit variances and minimize the MSE loss between the normalized features.

Original: Mathematically, it is equivalent to firstly calculating the Pearson Correlation Coefficient ( r ) [2] between two original feature vectors, and then using 1 − r as the feature imitation loss.
Revised: Mathematically, it is equivalent to firstly calculating the Pearson Correlation Coefficient ( r ) [2] between two original feature vectors, and then using 1 − r as the feature imitation loss.

Original: In addition to addressing the above three issues, PCC also has a beneficial effect on the gradient flow through student detectors by reducing the dependence of gradients on the feature value magnitude.

Original: So the training process is more stable.

Original: Compared with the previous methods, our method offers the following advantages.
Revised: Compared with the previous methods, our method offers the following advantages.

Original: First, as the distillation loss is calculated just on FPN features, it can be easily applied to heterogeneous detector pairs, including models with heterogeneous backbones, heterogeneous detection heads and different training strategies such as label assignment.
Revised: First, as the distillation loss is calculated just on FPN features, it can be easily applied to heterogeneous detector pairs, including models with heterogeneous backbones, heterogeneous detection heads and different training strategies such as label assignment.

Original: Second, as there is no need to forward the teacher’s detection head, the training time can be reduced significantly, especially for those models with cascaded heads.
Revised: Second, as there is no need to forward the teacher’s detection head, the training time can be reduced significantly, especially for those models with cascaded heads.

Original: Besides, PKD converges faster than previous methods.
Revised: Besides, PKD converges faster than previous methods.

Original: Last but not least, it is not sensitive to the only  hyper-parameter - distillation loss weight.
Revised: Last but not least, it has only one hyper-parameter - distillation loss weight and is not sensitive to it.

Original: So it can be easily applied to other datasets.
Revised: So it can be easily applied to other datasets.

Original: We conduct extensive experiments to verify the significant performance boosts brought by our approach on COCO dataset [24].
Revised: We conduct extensive experiments to verify the significant performance boosts brought by our approach on COCO dataset [24].

Original: Using the same two-stage detector as the teacher, ResNet based RetinaNet [23] and FCOS [23] achieve 41.5% mAP and 43.9% mAP , which surpasses the baseline by 4.1% and 4.8% respectively, and it also outperforms the previous state-of-the-art methods by a large margin.
Revised: Using the same two-stage detector as the teacher, ResNet based RetinaNet [23] and FCOS [23] achieve 41.5% mAP and 43.9% mAP , which surpasses the baseline by 4.1% and 4.8% respectively, and it also outperforms the previous state-of-the-art methods by a large margin.

Original: In summary, the contributions of this paper are threefold:
Revised: In summary, the contributions of this paper are threefolds:

Original: • We present that FPN feature imitation does make sense although the student-teacher detector pairs are heterogeneous.
Revised: • We argue that FPN feature imitation can distill knowledge successfully even though the student-teacher detector pairs are heterogeneous.

Original: • We propose to imitate FPN features with PCC to focus on relational information and give more freedom to the student feature’s magnitude.
Revised: • We propose to imitate FPN features with PCC to focus on relational information and relax the distribution constraints of the student feature’s magnitude.

Original: It is capable of distilling knowledge for both homogeneous and heterogeneous detector pairs.
Revised: It is capable of distilling knowledge for both homogeneous and heterogeneous detector pairs.

Original: • We verify the effectiveness of our method on various detectors via extensive experiments on the COCO [24], and  without bells or whistles achieve state-of-the-art performance.
Revised: • We verify the effectiveness of our method on various detectors via extensive experiments on the COCO [24], and achieve state-of-the-art performance without bells or whistles.

Original: Moreover, our method converges faster and is not sensitive to the only  hyper-parameter distillation loss weight. It is simple yet effective.
Revised: Moreover, our method converges faster and is not sensitive to the only one hyper-parameter distillation loss weight, which is simple yet effective.

Original:  Related Works 2.1 Object Detection
Revised:  Related Works 2.1 Object Detection

Original: Object detection is considered  one of the most challenging vision tasks which aims at detecting semantic objects of a certain class in images.
Revised: Object detection is considered as one of the most challenging vision tasks which aims at detecting semantic objects of a certain class in images.

Original: Modern detectors are roughly divided into two-stage detectors [31, 4, 13] and one-stage detectors [23, 35, 21, 11].
Revised: Modern detectors are roughly divided into two-stage detectors [31, 4, 13] and one-stage detectors [23, 36, 21, 11].

Original: In two-stage detectors, a Region
Revised: In two-stage detectors, a Region

Original: Proposal Network(RPN) is usually adopted to generate initial rough predictions refined by a taskspecific detection head.
Revised: Proposal Network(RPN) is usually adopted to generate initial rough predictions refined by a taskspecific detection head.

Original: A typical example is Faster R-CNN [31].
Revised: A typical example is Faster R-CNN [31].

Original: In contrast, One-stage detectors, such as RetinaNet [23] and FCOS [35], can directly and densely predict bounding boxes on the output feature maps.
Revised: In contrast, One-stage detectors, such as RetinaNet [23] and FCOS [36], can directly and densely predict bounding boxes on the output feature maps.

Original: Among these works, multi-scale features are usually adopted to handle objects of various scales, e.g. FPN [22], which is considered  a typical case for our study.
Revised: Among these works, multi-scale features are usually adopted to handle objects of various scales, e.g. FPN [22], which is considered as a typical case for our study.

Original: The proposed PKD only distills the intermediate features and does not rely on detector-specific designs so that it can be used directly on various detectors.
Revised: The proposed PKD only distills the intermediate features and does not rely on detector-specific designs so that it can be used directly on various detectors.

Original: 2.2 Knowledge Distillation
Revised: 2.2 Knowledge Distillation

Original: Knowledge Distillation (KD) is a kind of model compression and acceleration approach aiming at transferring knowledge from a teacher model to a student model.
Revised: Knowledge Distillation (KD) is a kind of model compression and acceleration approach aiming at transferring knowledge from a teacher model to a student model.

Original: It is popularized by [15] and then its effectiveness in image classification has been explored by subsequent works [32, 16, 14, 1, 43, 28, 26, 34, 36].
Revised: It is popularized by [15] and then its effectiveness in image classification has been explored by subsequent works [32, 16, 14, 1, 45, 28, 26, 35, 37].

Original: However, adapting KD to object detectors is nontrivial since minimizing the Kullback–Leibler (KL) divergence between the classification head outputs fails to transfer the spatial information from the teacher and only brings limited performance gain to the student.
Revised: However, adapting KD to object detectors is nontrivial since minimizing the Kullback–Leibler (KL) divergence between the classification head outputs fails to transfer the spatial information from the teacher and only brings limited performance gain to the student.

Original: The following three strategies are usually adopted by previous methods in detection to handle the above challenge.
Revised: The following three strategies are usually adopted by previous methods in detection to handle the above challenge.

Original: First, the distillation is usually conducted among multi-scale intermediate features [20], which provides rich spatial information for detection.
Revised: First, the distillation is usually conducted among multi-scale intermediate features [20], which provides rich spatial information for detection.

Original: Second, different feature selection methods are proposed to overcome the foreground-background imbalance.
Revised: Second, different feature selection methods are proposed to overcome the foreground-background imbalance.

Original: Most of these works can be divided into three kinds according to the feature selection approach [18]: proposal-based [6, 20, 9, 42], rule-based [38, 12] and attention-based [44, 46, 19].
Revised: Most of these works can be divided into three kinds according to the feature selection approach [18]: proposal-based [6, 20, 9, 44], rule-based [39, 12] and attention-based [46, 48, 19].

Original: Third, as the relation between different objects contains valuable information, many previous works try to improve the performance of detectors by enabling detectors to capture and make use of these relations, such as non-local modules [44] and global distillation [41].
Revised: Third, as the relation between different objects contains valuable information, many previous works try to improve the performance of detectors by enabling detectors to capture and make use of these relations, such as non-local modules [46] and global distillation [43].

Original: Unlike the previous works, we consider the magnitude difference, dominant FPN stages and channels as the key problem.
Revised: Unlike the previous works, we consider the magnitude difference, dominant FPN stages and channels as the key problem.

Original: We hope our method could serve as a solid baseline and help ease future research in knowledge distillation for object detectors.
Revised: We hope our method could serve as a solid baseline and help ease future research in knowledge distillation for object detectors.

Original:  Method 3.1 Preliminaries
Revised:  Method 3.1 Preliminaries

Original: In this part, we briefly recap the traditional knowledge distillation for object detection.
Revised: In this part, we briefly recap the traditional knowledge distillation for object detection.

Original: Recently, feature-based distillation over multi-scale features is adopted to deal with rich spatial information for detection.
Revised: Recently, feature-based distillation over multi-scale features is adopted to deal with rich spatial information for detection.

Original: Different imitation masks S are proposed to form an attention mechanism for foreground features and filter away noises in the background.
Revised: Different imitation masks M are proposed to form an attention mechanism for foreground features and filter away noises in the background.

Original: The objective can be formulated as
Revised: The objective can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where L is the number of FPN layers, l represents the l -th FPN layer, i , j are the location of the corresponding feature map with width W and height H .
Revised: where L is the number of FPN layers, l represents the l -th FPN layer, i , j are the location of the corresponding feature map with width W and height H .

Original: N l = (cid:80) Cc (cid:80)
Revised: N l = (cid:80) Cc (cid:80)

Original: Wi (cid:80)
Revised: Wi (cid:80)

Original: Hj S lcij .
Revised: Hj M lcij .

Original: F tl and F sl are the l -th layer of feature of student and teacher detectors, respectively.
Revised: F tl and F sl are the l -th layer of feature of student and teacher detectors, respectively.

Original: Function ϕ adapt is a 1x convolution layer to upsample the number of channels for the student network if the number of channels mismatches between the teacher and the student.
Revised: Function ϕ adapt is a 1x convolution layer to upsample the number of channels for the student network if the number of channels mismatches between the teacher and the student.

Original: The definition of S is different in these methods.
Revised: The definition of M is different in these methods.

Original: For example, FRS [46] uses the aggregated classification score map from the corresponding FPN layer, and FGD [41] considers spatial attention, channel attention, object size and foreground-background information simultaneously.
Revised: For example, FRS [48] uses the aggregated classification score map from the corresponding FPN layer, and FGD [43] considers spatial attention, channel attention, object size and foreground-background information simultaneously.

Original: 3.2 Is FPN feature imitation applicable for heterogeneous detector pairs?
Revised: 3.2 Is FPN feature imitation applicable for heterogeneous detector pairs?

Original: Most of the previous works perform distillation on FPN, as FPN integrates multiple layers of the backbone and provides rich spatial information of multi-scale objects.
Revised: Most of the previous works perform distillation on FPN, as FPN integrates multiple layers of the backbone and provides rich spatial information of multi-scale objects.

Original: It is reasonable to force the student to imitate FPN features from its homogeneous teacher as they have the same detection head and label assignment, and better features could lead to better performance.
Revised: It is reasonable to force the student to imitate FPN features from its homogeneous teacher as they have the same detection head and label assignment, and better features could lead to better performance.

Original: However, there is still a lack of study on how to distill between heterogeneous detectors.
Revised: However, there is still a lack of study on how to distill between heterogeneous detectors.

Original: [41, 46] conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted.
Revised: [43, 48] conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted.

Original: Thus, we are motivated to investigate whether FPN feature imitation still makes sense for these heterogeneous detector pairs. 
Revised: Thus, we are motivated to investigate whether FPN feature imitation still makes sense for these heterogeneous detector pairs. [Table]

Revised: Actually, the main idea of feature-based distillation methods is to directly match the feature activations of the teacher and the student.

Original: We first simulate the extreme case of feature-based knowledge distillation via the following backbone and neck replacement.
Revised: Based on this, we first simulate the extreme case of feature-based knowledge distillation via the following backbone and neck replacement.

Original: Without loss of generality, three three popular detectors, including RetinaNet [23], FCOS [35] and GFL [21] are adopted.
Revised: Without loss of generality, three three popular detectors, including RetinaNet [23], FCOS [36] and GFL [21] are adopted.

Original: Their backbones are the same, while the detection heads and training strategies such as label assignment are different.
Revised: Their backbones are the same, while the detection heads and training strategies such as label assignment are different.

Original: We first train these detectors for 12 epochs, then freeze the pre-trained backbones and necks, and fine-tune the randomly initialized detection heads for 12 epochs.
Revised: We first train these detectors for 12 epochs, then freeze the pre-trained backbones and necks, and fine-tune the randomly initialized detection heads for 12 epochs.

Original: During fine-tuning, the parameters in the backbone and neck are frozen and will not be updated.
Revised: During fine-tuning, the parameters in the backbone and neck are frozen and will not be updated.

Original: By analyzing the results in Table 1, we observe that distillation between heterogeneous detectors is possible, but feature magnitude difference needs to be eliminated first.
Revised: By analyzing the results in Table 1, we observe that distillation between heterogeneous detectors is possible, but feature magnitude difference needs to be eliminated first.

Original: 3.3 Feature Imitation with Pearson Correlation Coefficient
Revised: 3.3 Feature Imitation with Pearson Correlation Coefficient

Original: [Table]

Original: As discussed in Section 3.2, a promising feature distillation approach needs to consider the magnitude difference when constructing them into pairs for imitation.
Revised: As discussed in Section 3.2, a promising feature distillation approach needs to consider the magnitude difference when constructing them into pairs for imitation.

Original: Moreover, by comparing the activation patterns shown in Figure 1, we find the dominant FPN stages and channels can negatively interfere with the training phase of the student and lead to sub-optimal results, which is ignored by previous works.
Revised: Moreover, by comparing the activation patterns shown in Figure 1, we find the dominant FPN stages and channels can negatively interfere with the training phase of the student and lead to sub-optimal results, which is ignored by previous works.

Original: To address the above issues, we propose first normalizing the features of the teacher and the student to have zero mean and unit variances and minimizing the MSE between normalized features.
Revised: To address the above issues, we propose first normalizing the features of the teacher and the student to have zero mean and unit variances and minimizing the MSE between normalized features.

Original: Additionally, we want the normalization to obey the convolutional property - so that different elements of the same feature map, at different locations, are normalized in the same way.
Revised: Additionally, we want the normalization to obey the convolutional property - so that different elements of the same feature map, at different locations, are normalized in the same way.

Original: In other words, features convolved by the same convolution kernel are normalized in the same way.

Original: To accomplish this, we jointly normalize all the activations over all locations in a mini-batch.

Original: Let B be the set of all values in a feature map across both the elements of a mini-batch and spatial locations.
Revised: Let B be the set of all values in a feature map across both the elements of a mini-batch and spatial locations.

Original: So for a mini-batch of size b and feature maps of size h × w , we use the effective mini-batch of size m =
Revised: So for a mini-batch of size b and feature maps of size h × w , we use the effective mini-batch of size m =

Original: ∥ B ∥
Revised: ∥ B ∥

Original: = b · hw .
Revised: = b · hw .

Original: Let s ( c ) ∈
Revised: Let s ( c ) ∈

Original: R m be the c -th channel of a batch of FPN outputs and omit c for clarity.
Revised: R m be the c -th channel of a batch of FPN outputs and omit c for clarity.

Original: Then we get the normalized values
Revised: Then we get the normalized values

Original: ˆ s 1 ...m and ˆ t 1 ...m from the student and the teacher, respectively.
Revised: ˆ s 1 ...m and ˆ t 1 ...m from the student and the teacher, respectively.

Original: Instead of delicately designing imitation masks S in Eq.
Revised: Instead of delicately designing imitation masks M in Eq.

Original: 1 to choose the important features, our PKDonly operates on the full feature map.
Revised: 1 to choose the important features, our PKD operates on the full feature map.

Original: That is, the imitation mask S is filled with the scalar value 1.
Revised: That is, the imitation mask M is filled with the scalar value 1.

Original: Hence we can formulate our distillation loss as the following:
Revised: Hence we can formulate our distillation loss as the following:

Original: [Equation]
Revised: [Equation]

Original: Moving forward, minimizing the loss function above is equivalent to maximizing the PCC between the pre-normalized features of the student and the teacher.
Revised: Moving forward, minimizing the loss function above is equivalent to maximizing the PCC between the pre-normalized features of the student and the teacher.

Original: PCC can be computed as:
Revised: PCC can be computed as:

Original: [Equation]
Revised: [Equation]

Original: As ˆ s , ˆ t ∼ N (0 , 1) , we get
Revised: Since ˆ s , ˆ t ∼ N (0 , 1) , we get

Original: [Equation]
Revised: [Equation]

Original: ˆ

Original: [Equation]
Revised: [Equation]

Original:  2 i = 1 .
Revised: ˆ t 2 i = 1 .

Original: Then, we can reformulate Eq. 2 as:
Revised: Then, we can reformulate Eq. 2 as:

Original: [Equation]
Revised: [Equation]

Revised: The Pearson correlation coefficient is essentially a normalized measurement of the covariance, such that the result always has a value between − 1 and 1 .

Revised: Hence, L FPN = 1 − r always has a value between 0 and 2.

Revised: It focuses on the linear correlation between the features of the teacher and the student, and relaxes constraints on the magnitude of the features.

Revised: Actually, the feature maps s , t ∈ R m can be regarded as m data points ( s i , t i ).

Revised: L FPN = 0 implies that all data points lie on a line for which s increases as t increases.

Original: The gradient of PCC, ∂ L /∂s i , with respect to each FPN output s i , is given by:
Revised: Hence the student is well-trained. And vice versa for L FPN = 2 . A value of 1 implies that there is no linear dependency between the features of the student and the teacher. During training, the gradient of PCC, ∂ L /∂s i , with respect to each FPN output s i , is given by:

Original: [Equation]
Revised: [Equation]

Original: recalling r ( s , t ) is PCC in Eq.
Revised: (5) where r ( s , t ) is PCC in Eq.

Original: 3 and σ s is the sample standard deviation of the student’s feature s .
Revised: 3 and σ s is the sample standard deviation of the student’s feature s .

Original: So PCC also has a beneficial effect on the gradient flow through student detectors by reducing the dependence of gradients on the feature value magnitude.

Original: It makes gradient descent-based training algorithms more stable.

Original: In conclusion, PCC focuses on the relational information from the teacher and gives more freedom to the student feature’s magnitude.
Revised: In conclusion, PCC focuses on the relational information from the teacher and relaxes the distribution constraints on the student feature’s magnitude.

Original: Moreover, it eliminates the negative impacts of the dominant FPN stages and channels, leading to better performance.
Revised: Moreover, it eliminates the negative impacts of the dominant FPN stages and channels, leading to better performance.

Original: As a result, the normalization mechanism bridge the gap between the activation patterns of the student and the teacher (see Figure 2).
Revised: As a result, the normalization mechanism bridge the gap between the activation patterns of the student and the teacher (see Figure 2).

Original: Hence, feature imitation with PCC works for heterogeneous detector pairs.
Revised: Hence, feature imitation with PCC works for heterogeneous detector pairs.

Original: We train the student detector with the total loss as follows:
Revised: We train the student detector with the total loss as follows:

Original: [Equation]
Revised: [Equation]

Original: where L GT is the detection training loss, α is the hyper-parameter to balance the detection training loss and distillation loss.
Revised: where L GT is the detection training loss, α is the hyper-parameter to balance the detection training loss and distillation loss.

Original: 3.4 Connection of PCC and KL divergence
Revised: 3.4 Connection of PCC and KL divergence

Original: As discussed in Section 3.3, the normalization mechanism is the key to addressing the three issues mentioned above.
Revised: As discussed in Section 3.3, the normalization mechanism is the key to addressing the three issues mentioned above.

Original: At the same time, KL divergence has been widely used in distillation.
Revised: In previous works [15, 40, 33], KL divergence has been widely used in distillation.

Original: They first convert activations into a probability distribution with softmax function and then minimize the asymmetry KL divergence of the normalized activation maps:
Revised: They first convert activations into a probability distribution with softmax function and then minimize the asymmetry KL divergence of the normalized activation maps:

Original: [Equation]
Revised: [Equation]

Original:  where T is a hyper-parameter to control the degree of softness of the targets, and ϕ is the softmax function:
Revised: (7) where T is a hyper-parameter to control the degree of softness of the targets, and ϕ is the softmax function:

Original: [Equation]
Revised: [Equation]

Original: Here, we show that minimizing KL divergence between normalized features in the high-temperature limit is equivalent to minimizing MSE between normalized ones, and hence equivalent to maximizing
Revised: Here, we show that minimizing KL divergence between normalized features in the high-temperature limit is equivalent to minimizing MSE between normalized ones, and hence equivalent to maximizing

Original: PCC between original ones.
Revised: PCC between original ones.

Original: Let p i = ϕ
Revised: Let p i = ϕ

Original: (ˆ t i ) and q i = ϕ
Revised: (ˆ t i ) and q i = ϕ

Original: ( ˆ s i ) denote the probabilities from the teacher and the student, respectively.
Revised: ( ˆ s i ) denote the probabilities from the teacher and the student, respectively.

Original: The KL divergence gradient, ∂ L KL /∂
Revised: The KL divergence gradient, ∂ L KL /∂

Original: ˆ s i , with respect to each normalized activation
Revised: ˆ s i , with respect to each normalized activation

Original: ˆ s i of the student is given by:
Revised: ˆ s i of the student is given by:

Original: [Equation]
Revised: [Equation]

Original: If T is high compared with the magnitude of the normalized activations, we can approximate:
Revised: If T is large compared with the magnitude of the normalized activations, we can approximate:

Original: [Equation]
Revised: [Equation]

Original: As ˆ s i and ˆ t i have been zero-meaned, Eq.
Revised: As ˆ s i and ˆ t i have been zero-meaned, Eq.

Original: 10 simplifies to:
Revised: 10 simplifies to:

Original: [Equation]
Revised: [Equation]

Original: Exploiting the relation between MSE and PCC shown in Eq.
Revised: Exploiting the relation between MSE and PCC shown in Eq.

Original: 4 gives the desired result.
Revised: 4 gives the desired result.

Original: Experimental results are provided in Section A.3.
Revised: Experimental results are provided in Section A.3.

Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: In order to verify the effectiveness and robustness of our method, we conduct experiments on different detection frameworks on the COCO [24] dataset.
Revised: In order to verify the effectiveness and robustness of our method [Table] we conduct experiments on different detection frameworks on the COCO [24] dataset.

Original: We choose the default 120ktrain images split for training and 5kval images split for the test.
Revised: We choose the default 120k images split for training and 5k images split for the test.

Original: We use the standard training settings following [41] and report mean Average Precision ( AP ) as an evaluation metric, together with AP under different thresholds and scales, i.e. , AP 50 , AP 75 , AP S , AP M and AP L .
Revised: We use the standard training settings following [43] and report mean Average Precision ( AP ) as an evaluation metric, together with AP under different thresholds and scales, i.e. , AP 50 , AP 75 , AP S , AP M and AP L .

Original: For distillation, the hyper-parameter α is set to 6 when using a two-stage detector as the teacher and 10 when using a one-stage one.
Revised: For distillation, the hyper-parameter α is set to 6 when using a two-stage detector as the teacher and 10 when using a one-stage one.

Original: Feature levels of heterogeneous detector pairs may not be strictly[Table] aligned, e.g. , FasterRCNN constructs the feature pyramid from P 2 to P 6 , while RetinaNet uses
Revised: Feature levels of heterogeneous detector pairs may not be strictly aligned, e.g. , FasterRCNN constructs the feature pyramid from P 2 to P 6 , while RetinaNet uses

Original: P 3 to P 7 .
Revised: P 3 to P 7 .

Original: To address the above issue, we upsample the low-resolution feature maps to have the same spatial size as the high-resolution ones.
Revised: To address the above issue, we upsample the low-resolution feature maps to have the same spatial size as the high-resolution ones.

Original: In addition, the adaptive layer ( ϕ adapt in Eq. 1) is unnecessary for our PKD.
Revised: In addition, the adaptive layer ( ϕ adapt in Eq. 1) is unnecessary for our PKD.

Original: All experiments are performed on 8 Tesla A100 GPUs with two images in each.
Revised: All experiments are performed on 8 Tesla A100 GPUs with two images in each.

Original: Our implementation is based on mmdetection [7] and mmrazor [8] with Pytorch framework [29]. ’1x’ (namely 12epochs), ’2x’ (namely 24 epochs) and ’2x+ms’ (namely 24epochs with multi-scale training) training schedules are used.
Revised: Our implementation is based on mmdetection [7] and mmrazor [8] with Pytorch framework [29]. ’1x’ (namely 12epochs), ’2x’ (namely 24 epochs) and ’2x+ms’ (namely 24epochs with multi-scale training) training schedules are used.

Original: More details are given in the Appendix.
Revised: More details are given in the Appendix.

Original: 4.1 Main Results
Revised: 4.1 Main Results

Original: Our method can be applied to different detection frameworks easily, so we first conduct experiments on five popular detectors, including a two-stage detector (Faster RCNN[31]), two anchor-based one-stage detector (RetinaNet [23], RepPoints [40] and TOOD [11]) and an anchor-free detector (FCOS [35]).
Revised: Our method can be applied to different detection frameworks easily, so we first conduct experiments on five popular detectors, including a two-stage detector (Faster RCNN[31]), two anchor-based one-stage detector (RetinaNet [23], RepPoints [42] and TOOD [11]) and an anchor-free detector (FCOS [36]).

Original: Table 2 shows the comparison of the results of state-of-the-art distillation methods on the COCO.
Revised: Table 2 shows the comparison of the results of state-of-the-art distillation methods on the COCO.

Original: Our distillation method surpasses other state-of-the-art methods.
Revised: Our distillation method surpasses other state-of-the-art methods.

Original: All the student detectors gain significant improvements in AP with the knowledge transferred from teacher detectors, e.g. ,
Revised: All the student detectors gain significant improvements in AP with the knowledge transferred from teacher detectors, e.g. ,

Original: FCOS with ResNet-50 gets a 3.7% mAP improvement on the COCO dataset.
Revised: FCOS with ResNet-50 gets a 3.7% mAP improvement on the COCO dataset.

Original: These results indicate the effectiveness and generality of our method in both one-stage and two-stage detectors.
Revised: These results indicate the effectiveness and generality of our method in both one-stage and two-stage detectors.

Original: 4.2 Distilling More Student Detectors with Stronger Heterogeneous Teachers
Revised: 4.2 Distilling More Student Detectors with Stronger Heterogeneous Teachers

Original: Most of the current methods are designed for homogeneous detector pairs.
Revised: Most of the current methods are designed for homogeneous detector pairs.

Original: PKD is general enough to distill knowledge between both homogeneous and heterogeneous detector pairs.
Revised: PKD is general enough to distill knowledge between both homogeneous and heterogeneous detector pairs.

Original: Here we conduct experiments on more detectors and use stronger heterogeneous teacher detectors, as shown in Table 3.
Revised: Here we conduct experiments on more detectors and use stronger heterogeneous teacher detectors, as shown in Table 3.

Original: Comparing with Table
Revised: Comparing with Table

Original: [Table]
Revised: [Table]

Original: we find that student detectors perform better with stronger teacher detectors, e.g. , Retina-Res50 model achieves 41.5% and 39.6% mAP with Mask RCNN-Swin [27] and RetinaRes101, respectively.
Revised: we find that student detectors perform better with stronger teacher detectors, e.g. , Retina-Res50 model achieves 41.5% and 39.6% mAP with Mask RCNN-Swin [27] and RetinaRes101, respectively.

Original: Results show that mimicking the feature maps of stronger heterogeneous teacher detectors can further boost the student’s performance when applied with our PKD
Revised: Results show that mimicking the feature maps of stronger heterogeneous teacher detectors can further boost the student’s performance when applied with our PKD

Original: [Table] [Table]
Revised: [Table] [Table]

Original: three issues and achieve better performance.

Original: [Figure]
Revised: [Table]

Original:  Other feature -based distillation methods 
Revised: 4.3 Other feature-based distillation methods with the normalized features

Original: with the normalized features
Revised: In object detection

Original: In object detection, some feature-based distillation methods [46, 41] transfer the knowledge within some pre-defined imitation regions, and achieved competitive results.
Revised: [Table] some feature-based distillation methods [48, 43] transfer the knowledge within some pre-defined imitation regions [Table] and achieved competitive results.

Original: While we believe full map feature imitation with our PKD can already outperform them.
Revised: In contrast, we believe full map feature imitation with our PKD can already outperform them.

Original: In order to show the universality of our method, we take FitNet [32], GT Mask, FRS [46] and FGD [41] as baselines and build PKD on them.
Revised: In order to show the generality of our method, we take FitNet [32], GT Mask, FRS [48] and FGD [43] as baselines and build PKD on them.

Original: Recaps.
Revised: Recaps.

Original: Full feature map distillation with MSE loss is used in FitNet.
Revised: Full feature map distillation with MSE loss is used in FitNet.

Original: While the 2nd method only imitates features that overlap with ground truth bounding boxes.
Revised: While the GT Mask only imitates features that overlap with ground truth bounding boxes.

Original: Feature imitationand logit mimicking with FRS uses the aggregated classification score, takes maximum operation in channel direction, of the classification head output as a weighted score mask, to guide the distillation of the FPN and classification head.
Revised: Feature imitation with FRS uses the aggregated classification score, takes maximum operation in channel direction, of the classification head output as a weighted score mask, to guide the distillation of the FPN.

Original: And Focal distillation in FGD forces the student to focus on the teacher’s pixels and channels with large absolute mean values.

Original: Global distillation in FGD rebuilds the relation between different pixels and transfers it from teachers to students through GcBlock [5].
Revised: And global distillation in FGD rebuilds the relation between different pixels and transfers it from teachers to students through GcBlock [5].

Original: Results.
Revised: Results.

Original: For a fair comparison, all the hyper-parameters are copied from the original paper.
Revised: For a fair comparison, all the hyper-parameters are copied from the original paper.

Original: Results in Table 5 show that combining PKD with these feature imitation methods can further improve their performance.
Revised: Results in Table 4 show that combining PKD with these feature imitation methods can further improve their performance.

Original: Besides, we do not combine PKD with FGD focal distillation as focal distillation forces the student to focus on the teacher’s pixels and channels with large absolute mean values.

Original: That is, features with larger values are given larger weights, which seems not to agree with the argument that

Original: "The small gradients produced by less activated features can be drowned into the large gradients produced by highly activated ones, thus limiting further refinement."

Original: in [33, 45] and us.

Original: [Table]
Revised: [Table]

Original: 4.4 Analysis 4.4.1 Effectiveness of Pearson Correlation Coefficient
Revised: 4.4 Analysis 4.4.1 Effectiveness of Pearson Correlation Coefficient

Original: Among most previous works, feature-based distillation over multi-scale features is adopted for distillation.
Revised: Among most previous works, feature-based distillation over multi-scale features is adopted for distillation.

Original: In this study, we argue that the magnitude difference, dominant FPN stages and channels can negatively interfere with the training phase of the student and lead to sub-optimal results.
Revised: In this study, we argue that the magnitude difference, dominant FPN stages and channels can negatively interfere with the training phase of the student and lead to sub-optimal results.

Original: To study this empirically, we conduct the following three experiments to explore the effects of MSE loss which is a widely-used loss function in distillation and suffers from the above three issues simultaneously.
Revised: To study this empirically, we conduct the following three experiments to explore the effects of MSE loss which is a widely-used loss function in distillation and suffers from the above three issues simultaneously.

Original: For a fair comparison, we tune the loss weight of MSE for all experiments.
Revised: For a fair comparison, we tune the loss weight of MSE for all experiments.

Original: More details are listed in
Revised: More details are listed in

Original: Section A.4.
Revised: Section A.4.

Original: For the first pair, FCOS-ResX101 is used as the teacher whose feature magnitude is significantly different from that of the student Retina-Res50 (refer to Fig. 4).
Revised: For the first pair, FCOS-ResX101 is used as the teacher whose feature magnitude is significantly different from that of the student Retina-Res50 (refer to Fig. 4).

Original: In this case, what constitutes the knowledge is better presented by relational information from the teacher’s features than the absolute values.
Revised: In this case, what constitutes the knowledge is better presented by relational information from the teacher’s features than the absolute values.

Original: For the second pair, GFL-Res101 is the teacher and FCOS-Res50 is the student.
Revised: For the second pair, GFL-Res101 is the teacher and FCOS-Res50 is the student.

Original: For both teacher and student, features in FPN stages ’P5’ and ’P6’ are less activated than those in stages ’P3’ and ’P4’ (refer to Fig. 5).
Revised: For both teacher and student, features in FPN stages ’P5’ and ’P6’ are less activated than those in stages ’P3’ and ’P4’ (refer to Fig. 5).

Original: Hence, features in stages ’P3’ and ’P4’ could dominate the gradient of the distillation loss, which will overwhelm the effects of other features.
Revised: Hence, features in stages ’P3’ and ’P4’ could dominate the gradient of the distillation loss, which will overwhelm the effects of other features.

Original: Since larger objects are usually assigned to higher feature levels, the student’s performance for large objects is significantly lower than that of ours when MSE is used as the distillation loss.
Revised: Since larger objects are usually assigned to higher feature levels, the student’s performance for large objects is significantly lower than that of ours when MSE is used as the distillation loss.

Original: For the last pair, Retina-ResX101 is the teacher and Retina-Res50 is the student.
Revised: For the last pair, Retina-ResX101 is the teacher and Retina-Res50 is the student.

Original: There are always a few channels with greater values (refer to Fig.
Revised: There are always a few channels with greater values (refer to Fig.

Original: 6 and Fig. 7) , so directly imitating the feature maps may introduce much noise in dominant channels.
Revised: 6 and Fig. 7) , so directly imitating the feature maps may introduce much noise in dominant channels.

Original: By comparing the results in Table 4, we find that our proposed PKD addresses the above three issues and achieves better performance.
Revised: By comparing the results in Table 5, we find that our proposed PKD addresses the above three issues and achieves better performance.

Original: Hence, an effective distillation loss function should have the ability to handle the above three problems.
Revised: Hence, an effective distillation loss function should have the ability to handle the above three problems.

Original: We hope PKD could serve as a solid baseline and help ease future research in knowledge distillation community.
Revised: We hope PKD could serve as a solid baseline and help ease future research in knowledge distillation community.

Original: 4.4.2 Sensitivity study of loss weight α

Original: [Table]

Original: [Table]

Original: In Eq.

Original: 6, we use the loss weight hyper-parameter α to balance the detection training loss and distillation loss.

Original: Here, we conduct several experiments to investigate the influence of α .

Original: As shown in Table 6, the worst result is just a 0.3 mAP drop compared with the best result, indicating our method is not sensitive to the only hyper-parameter α .

Original: Hence it is easy for practical use.

Original: Moving forward, as the normalization mechanism has a beneficial effect on the gradient flow through student detectors, by reducing the dependence of gradients on the feature value magnitude, we can train different networks with relatively uniform hyper-parameters α .

Original: 4.4.3 Convergence Speed
Revised: 4.4.2 Convergence Speed

Original: In this subsection, we conduct experiments with FCOS to compare the convergence speed of our method with other state-of-the-art distillation methods on the COCO benchmark.
Revised: In this subsection, we conduct experiments with FCOS to compare the convergence speed of our method with other state-of-the-art distillation methods on the COCO benchmark.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, the training convergence can significantly speed up in the early training stage with our PKD.
Revised: 3, the training convergence can significantly speed up in the early training stage with our PKD.

Original: Meanwhile, the final performance of the student detector FCOS-Res50 is about 0.5%-0.6% mAP higher than that of FGD and FRS.
Revised: Meanwhile, the final performance of the student detector FCOS-Res50 is about 0.5%-0.6% mAP higher than that of FGD and FRS.

Original: In addition, there is no need to forward the detection head of the teacher, which reduces the training time significantly, especially for those models with cascaded heads.
Revised: In addition, there is no need to forward the detection head of the teacher, which reduces the training time significantly, especially for those models with cascaded heads.

Original: The Pearson correlation coefficient is essentially a normalized measurement of the covariance, such that the result always has a value between − 1 and 1 .
Revised: [Table]

Original: Hence, L FPN = 1 − r in Eq.4 always has a value between 0 and 2. It focuses on the linear correlation between the features of the teacher and the student, and relaxes constraints on the magnitude of the features.
Revised: In Eq.

Original: Let s c , t c ∈ R bhw denote the c-th channel of the student’s and teacher’s features, and omit c for clarification.
Revised: 6, we use the loss weight hyper-parameter α to balance the detection training loss and distillation loss.

Original: Then we get bhw data points ( s i , t i ).
Revised: Here, we conduct several experiments to investigate the influence of α .

Original: L FPN = 0 implies that all data points lie on a line for which s increases as t increases. Hence the student is well-trained.
Revised: As shown in Table 6, the worst result is just a 0.3 mAP drop compared with the best result, indicating our method is not sensitive to the only hyper-parameter α .

Original: And vice versa for L FPN = 2 .
Revised: In addition, since

Original: A value of 1 implies that there is no linear dependency between the features of the student and the teacher.
Revised: PCC relaxes the distribution constraints of the feature value magnitude, a relatively uniform loss weight α can be used among different teacher-student detector pairs to keep the relative contributions of original and distillation loss remain roughly unchanged, which is easy for practical use.

Original:  Conclusion
Revised:  Conclusion

Original: This paper empirically finds that FPN feature imitation is applicable for heterogeneous detector pairs although their detection heads and label assignments are different.
Revised: This paper empirically finds that FPN feature imitation is applicable for heterogeneous detector pairs although their detection heads and label assignments are different.

Original: Then we propose feature imitation with Pearson Correlation Coefficient to focus on the relational information from the teacher and give more freedom to the student’s feature value magnitude.
Revised: Then we propose feature imitation with Pearson Correlation Coefficient to focus on the relational information from the teacher and relax the distribution constraints of the student’s feature value magnitude.

Original: Furthermore, a general KD framework is proposed, capable of distilling knowledge for both homogeneous and heterogeneous detector pairs.
Revised: Furthermore, a general KD framework is proposed, capable of distilling knowledge for both homogeneous and heterogeneous detector pairs.

Original: It converges faster and only introduces one hyper-parameter, which can easily be applied to other datasets.
Revised: It converges faster and only introduces one hyper-parameter, which can easily be applied to other datasets.

Original: However, our understanding of whether our proposed PKD is capable of other tasks such as text recognition is preliminary and left as future works.
Revised: However, our understanding of whether our proposed PKD is capable of other tasks such as text recognition is preliminary and left as future works.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work is supported in part by the Shanghai Committee of Science and Technology,
Revised: This work is supported in part by the Shanghai Committee of Science and Technology,

Original: China (Grant
Revised: China (Grant

Original: No. 20DZ1100800), NSFC 62273347, the National Key Research and Development Program of
Revised: No. 20DZ1100800), NSFC 62273347, the National Key Research and Development Program of

Original: China (2020AAA0103402), Jiangsu Key Research and Development Plan (No.BE2021012-2) and
Revised: China (2020AAA0103402), Jiangsu Key Research and Development Plan (No.BE2021012-2) and

Original: NSFC 
Revised: NSFC 61876182.

Original: A Appendix A.1 Motivation to build a general KD framework
Revised: A Appendix A.1 Motivation to build a general KD framework

Original: Modern detectors are roughly divided into two-stage detectors [31, 13, 3] and dense prediction detectors ( e.g. , anchor-based one-stage detectors [23, 25, 30] and anchor-free one-stage detectors [10, 35, 40]).
Revised: Modern detectors are roughly divided into two-stage detectors [31, 13, 3] and dense prediction detectors ( e.g. , anchor-based one-stage detectors [23, 25, 30] and anchor-free one-stage detectors [10, 36, 42]).

Original: Each family has its own advantages and weakness.
Revised: Each family has its own advantages and weakness.

Original: In particular, two-stage detectors usually have higher performance, while being slower in inference speed and harder to be deployed due to the region proposal network (RPN) and RCNN head.
Revised: In particular, two-stage detectors usually have higher performance, while being slower in inference speed and harder to be deployed due to the region proposal network (RPN) and RCNN head.

Original: On the other hand, dense prediction detectors are faster than two-stage detectors while being less accurate.
Revised: On the other hand, dense prediction detectors are faster than two-stage detectors while being less accurate.

Original: In practice, it is a natural idea to use two-stage detectors as teachers to enhance dense prediction detectors.
Revised: In practice, it is a natural idea to use two-stage detectors as teachers to enhance dense prediction detectors.

Original: Moreover, knowledge distillation between heterogeneous dense prediction detector pairs is also promising.
Revised: Moreover, knowledge distillation between heterogeneous dense prediction detector pairs is also promising.

Original: In some scenarios, only the detectors with a specific architecture can be deployed due to hardware limitations.
Revised: In some scenarios, only the detectors with a specific architecture can be deployed due to hardware limitations.

Original: For example, compared with Batch Normalization [17] and Instance
Revised: For example, compared with Batch Normalization [17] and Instance

Original: Normalization [37], Group Normalization [39] is hard to deploy.
Revised: Normalization [38], Group Normalization [41] is hard to deploy.

Original: However, the most powerful teachers may belong to different categories.
Revised: However, the most powerful teachers may belong to different categories.

Original: Furthermore, object detection is developing rapidly and algorithms with better performance are proposed continuously.
Revised: Furthermore, object detection is developing rapidly and algorithms with better performance are proposed continuously.

Original: Nevertheless, it is not easy to change detectors frequently in terms of stability and hardware runtime limitations in practical applications.
Revised: Nevertheless, it is not easy to change detectors frequently in terms of stability and hardware runtime limitations in practical applications.

Original: So it is helpful if knowledge distillation can be conducted between the latest high-capacity detectors and the widely-used compact detectors.
Revised: So it is helpful if knowledge distillation can be conducted between the latest high-capacity detectors and the widely-used compact detectors.

Original: Hence, we are motivated to design a general distillation method capable of distilling knowledge between both homogeneous and heterogeneous detector pairs.
Revised: Hence, we are motivated to design a general distillation method capable of distilling knowledge between both homogeneous and heterogeneous detector pairs.

Original: A.2 Details of Training Recipe
Revised: A.2 Details of Training Recipe

Original: We conduct experiments on different detection frameworks, including two-stage models, anchorbased one-stage models and anchor-free one-stage models.
Revised: We conduct experiments on different detection frameworks, including two-stage models, anchorbased one-stage models and anchor-free one-stage models.

Original: [18] proposes inheriting strategy which initializes the student with the teacher’s neck and head parameters and gets better results.
Revised: [18] proposes inheriting strategy which initializes the student with the teacher’s neck and head parameters and gets better results.

Original: Here we use this strategy to initialize the student which has the same head structure as the teacher and find that it helps students converge faster.
Revised: Here we use this strategy to initialize the student which has the same head structure as the teacher and find that it helps students converge faster.

Original: All experiments are performed on 8 Tesla A100 GPUs with 2 images in each.
Revised: All experiments are performed on 8 Tesla A100 GPUs with 2 images in each.

Original: Our implementation is based on mmdetection [7] and mmrazor [8] with Pytorch framework [29]. ’1x’ (namely 12 epochs), ’2x’ (namely 24 epochs) and ’2x+ms’ (namely 24 epochs with multi-scale training) training schedules with SGD optimizer are used.
Revised: Our implementation is based on mmdetection [7] and mmrazor [8] with Pytorch framework [29]. ’1x’ (namely 12 epochs), ’2x’ (namely 24 epochs) and ’2x+ms’ (namely 24 epochs with multi-scale training) training schedules with SGD optimizer are used.

Original: Momentum and weight decay are set to 0.9 and 1e-4.
Revised: Momentum and weight decay are set to 0.9 and 1e-4.

Original: The initial learning rate is set to 0.02 for Faster RCNN and 0.01 for others.
Revised: The initial learning rate is set to 0.02 for Faster RCNN and 0.01 for others.

Original: We train FCOS [35] with tricks including GIoULoss, norm-on-bbox and center-sampling which is the same as FGD [41] and GID [9].
Revised: We train FCOS [36] with tricks including GIoULoss, norm-on-bbox and center-sampling which is the same as FGD [43] and GID [9].

Original: For distillation, only one hyper-parameter α is introduced to balance the supervised learning loss and distillation loss, and it is set to 6 when using a two-stage detector as the teacher and 10 when using a one-stage one as the teacher.
Revised: For distillation, only one hyper-parameter α is introduced to balance the supervised learning loss and distillation loss, and it is set to 6 when using a two-stage detector as the teacher and 10 when using a one-stage one as the teacher.

Original: The teacher network is well-trained previously and fixed during training.
Revised: The teacher network is well-trained previously and fixed during training.

Original: A.3 Connection of PCC and KL divergence
Revised: A.3 Connection of PCC and KL divergence

Original: Section 3.4 in the main text shows the connection between PCC and KL divergence.
Revised: Section 3.4 in the main text shows the connection between PCC and KL divergence.

Original: We conduct two experiments on RetinaNet and GFL to verify this empirically.
Revised: We conduct two experiments on RetinaNet and GFL to verify this empirically.

Original: For both experiments, we set loss weight α = 10 and temperature T = 50 .
Revised: For both experiments, we set loss weight α = 10 and temperature T = 50 .

Original: As shown in Table 7, minimizing KL divergence between post-normalized features in the high-temperature limit can also achieve similar results.
Revised: As shown in Table 7, minimizing KL divergence between post-normalized features in the high-temperature limit can also achieve similar results.

Original: A.4 Details of feature imitation with MSE
Revised: A.4 Details of feature imitation with MSE

Original: As shown in Table 4 in the main text, we compare the results of MSE with our proposed PKD.
Revised: As shown in Table 5 in the main text, we compare the results of MSE with our proposed PKD.

Original: As the gradient of MSE loss depends on feature value magnitude and it is usually different among different detectors, we have to tune the loss weight carefully to achieve relatively good results, as shown in
Revised: As the gradient of MSE loss depends on feature value magnitude and it is usually different among different detectors, we have to tune the loss weight carefully to achieve relatively good results, as shown in

Original: Table 8.
Revised: Table 8.

Original: And we put the best results down in Table 4.
Revised: And we put the best results down in Table 5.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A.5 Effectiveness of Pearson Correlation Coefficient
Revised: A.5 Effectiveness of Pearson Correlation Coefficient

Original: In this paper, we argue that the magnitude difference, dominant FPN stages and channels can negatively interfere with the training phase of the student.
Revised: In this paper, we argue that the magnitude difference, dominant FPN stages and channels can negatively interfere with the training phase of the student.

Original: To clearly show that these three issues do exist, we elaborately visualize the FPN feature responses of the teacher and student detectors before distillation, as shown in Figure 4, Figure 5, Figure 6 and Figure 7.
Revised: To clearly show that these three issues do exist, we elaborately visualize the FPN feature responses of the teacher and student detectors before distillation, as shown in Figure 4, Figure 5, Figure 6 and Figure 7.

Original: We follow the visualization method in Section 1.
Revised: We follow the visualization method in Section 1.

Original: Through these comparisons, we obtain the following three observations:
Revised: Through these comparisons, we obtain the following three observations:

Original: (1) The feature value magnitude of different detectors could be significantly different, especially for heterogeneous detectors, as shown in Figure 4.
Revised: (1) The feature value magnitude of different detectors could be significantly different, especially for heterogeneous detectors, as shown in Figure 4.

Original: Directly aligning the feature maps between the teacher and the student may enforce overly strict constraints and do harm to the student (see Tabletop in the main text).
Revised: Directly aligning the feature maps between the teacher and the student may enforce overly strict constraints and do harm to the student (see Tabletop in the main text).

Original: (2) As shown in Figure 5, compared with features in FPN stage ’P3’ and ’P4’, features in stage ’P5’ and ’P6’ are less activated in both GFL-Res101 and FCOS-Res50.
Revised: (2) As shown in Figure 5, compared with features in FPN stage ’P3’ and ’P4’, features in stage ’P5’ and ’P6’ are less activated in both GFL-Res101 and FCOS-Res50.

Original: And it is a common case among different detector pairs.
Revised: And it is a common case among different detector pairs.

Original: FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD.
Revised: FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD.

Original: (3) As shown in Figure 6 (right), the feature magnitude of the 210 -th channel of MaskRCNN-Swin
Revised: (3) As shown in Figure 6 (right), the feature magnitude of the 210 -th channel of MaskRCNN-Swin

Original: FPN stage ’P6’ is significantly larger than others.
Revised: FPN stage ’P6’ is significantly larger than others.

Original: Similar phenomena also exist in other detectors such as RetinaNet (see Figure 7).
Revised: Similar phenomena also exist in other detectors such as RetinaNet (see Figure 7).

Original: The small gradients produced by those less activated channels can be drowned in the large gradients produced by dominant ones, thus limiting further refinement.
Revised: The small gradients produced by those less activated channels can be drowned in the large gradients produced by dominant ones, thus limiting further refinement.

Original: Furthermore, there is much noise in the object-irrelevant area, as depicted in Figure 6 (left).
Revised: Furthermore, there is much noise in the object-irrelevant area, as depicted in Figure 6 (left).

Original: Directly imitating the feature maps may introduce much noise.
Revised: Directly imitating the feature maps may introduce much noise.

Original: Comparing Table 8 and Table 4 in the main text, we find that our proposed PKD addresses the above three issues and achieves better performance.
Revised: Comparing Table 8 and Table 5 in the main text, we find that our proposed PKD addresses the above three issues and achieves better performance.

Original: Hence, an effective distillation loss function should have the ability to handle the above three problems.
Revised: Hence, an effective distillation loss function should have the ability to handle the above three problems.

Original: We hope PKD could serve as a solid baseline and help ease future research in knowledge distillation community.
Revised: We hope PKD could serve as a solid baseline and help ease future research in knowledge distillation community.

Original: We define channels with a larger number as dominant channels.
Revised: We define channels with a larger number as dominant channels.