Original: PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
Revised: PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Knowledge distillation(KD) is a widely-used technique to train compact models inobject detection.
Revised: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection.

Original: However, there is still a lack of study on how to distill betweenheterogeneous detectors.
Revised: However, there is still a lack of study on how to distill between heterogeneous detectors.

Original: In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although theirdetection heads and label assignments are different.
Revised: In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although their detection heads and label assignments are different.

Original: However, directly aligning thefeature maps to distill detectors suffers from two problems.
Revised: However, directly aligning the feature maps to distill detectors suffers from two problems.

Original: First, the difference infeature magnitude between the teacher and the student could enforce overly strictconstraints on the student.
Revised: First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints on the student.

Original: Second, the FPN stages and channels with large featuremagnitude from the teacher model could dominate the gradient of distillation loss,which will overwhelm the effects of other features in KD and introduce muchnoise.
Revised: Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise.

Original: To address the above issues, we propose to imitate features with Pearson
Revised: To address the above issues, we propose to imitate features with Pearson

Original: Correlation Coefficient to focus on the relational information from the teacherand relax constraints on the magnitude of the features.
Revised: Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features.

Original: Our method consistentlyoutperforms the existing detection KD methods and works for both homogeneousand heterogeneous student-teacher pairs.
Revised: Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teacher pairs.

Original: Furthermore, it converges faster.
Revised: Furthermore, it converges faster.

Original: With apowerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNetand FCOS achieve 41.5% and 43.9% mAP on COCO2017, which are 4.1% and4.8% higher than the baseline, respectively. 
Revised: With a powerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNet and FCOS achieve 41.5% and 43.9% mAP on COCO2017, which are 4.1% and 4.8% higher than the baseline, respectively. Our implementation is available at https://github.com/open-mmlab/mmrazor.

Original:  Introduction
Revised:  Introduction

Original: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection.
Revised: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection.

Original: However, there is still a lack of study on how to distill between heterogeneous detectors.
Revised: However, there is still a lack of study on how to distill between heterogeneous detectors.

Original: Most previousworks [33, 29, 9, 17, 11] rely on detector-specific designs and can only be applied to homogeneousdetectors.
Revised: Most previous works [38, 33, 9, 19, 12] rely on detector-specific designs and can only be applied to homogeneous detectors.

Original: [34, 39] conduct experiments on detectors with heterogeneous backbones, but detectorswith heterogeneous detection heads and different label assignments are always omitted.
Revised: [41, 46] conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted.

Original: Objectdetection is developing rapidly and algorithms with better performance are proposed continuously.
Revised: Object detection is developing rapidly and algorithms with better performance are proposed continuously.

Original: Nevertheless, it is not easy to change detectors frequently in terms of stability in practical application.
Revised: Nevertheless, it is not easy to change detectors frequently in terms of stability in practical applications.

Original: Furthermore, in some scenarios, only detectors with a specific architecture can be deployed due to hardware limitations (e.g., two-stage detectors are hard to deploy), while most powerful teachersbelong to different categories.
Revised: Furthermore, in some scenarios, only detectors with a specific architecture can be deployed due to hardware limitations (e.g., two-stage detectors are hard to deploy), while most powerful teachers belong to different categories.

Original: Thus, it is promising if knowledge distillation can be conducted between heterogeneous detector pairs.
Revised: Thus, it is promising if knowledge distillation can be conducted between heterogeneous detector pairs.

Original: In addition, current distillation methods, such as [34, 37],usually introduce several complementary loss functions to further improve their performance, soseveral hyper-parameters are used to adjust the contribution of each loss function which heavilyaffects their abilities to transfer to other datasets.
Revised: In addition, current distillation methods, such as [41, 44], usually introduce several complementary loss functions to further improve their performance, so several hyper-parameters are used to adjust the contribution of each loss function which heavily affects their abilities to transfer to other datasets.

Original: In this paper, we first empirically verify that FPN feature imitation does make sense even if the student- teacher detector pairs are heterogeneous.
Revised: In this paper, we first empirically verify that FPN feature imitation does make sense even if the student-teacher detector pairs are heterogeneous.

Original: However, directly minimizing the Mean Square Error (MSE) between features of the teacher and the student leads to sub-optimal  results [4, 11, 39, 34].
Revised: However, directly minimizing the Mean Square Error (MSE) between features of the teacher and the student leads to sub-optimal results, with results shown in Table 4. Similar conclusions are drawn in [6, 12, 46, 41].

Original: In order to explore the limitations of MSE, we elaborately visualize the FPN feature responses of the teacher and student detection models, as shown in Figure 1.
Revised: In order to explore the limitations of MSE, we elaborately visualize the FPN feature responses of the teacher and student detection models, as shown in Figure 1.

Original: Specifically, for an output feature s l ∈ R C × H × Wfrom l -th FPN stage, we select the maximum value in the dimension C at each pixel and obtain a 2-Dmatrix.
Revised: Specifically, for an output feature s l ∈ R C × H × W from l -th FPN stage, we select the maximum value in the dimension C at each pixel and obtain a 2-D matrix.

Original: Then we normalize the values to 0-255 according to the maximum and minimum values of l2-D matrices.
Revised: Then we normalize the values to 0-255 according to the maximum and minimum values of l 2-D matrices.

Original: Through these comparisons, we obtain the following observations:
Revised: Through these comparisons, we obtain the following observations:

Original: (1) Feature value magnitude of the teacher and the student is different, especially for heterogeneous detectors, as shown in Figure 1 (left).
Revised: (1) The feature value magnitude of the teacher and the student is different, especially for heterogeneous detectors, as shown in Figure 1 (left).

Original: So directly aligning the feature maps between theteacher and the student may enforce overly strict constraints and do harm to the student.
Revised: So directly aligning the feature maps between the teacher and the student may enforce overly strict constraints and do harm to the student.

Original: (2) There are always several FPN stages whose values are larger than the others, as shown in Figure 1 (right).
Revised: (2) There are always several FPN stages whose values are larger than the others, as shown in Figure 1 (right).

Original: It is obvious that FPN stage ’P6’ of the teacher is less activated than FPN stage’P3’.
Revised: It is obvious that FPN stage ’P6’ of the teacher is less activated than FPN stage ’P3’.

Original: However, for detectors such as RetinaNet [21] and FCOS [31], all FPN stages share the samedetection head.
Revised: However, for detectors such as RetinaNet [23] and FCOS [35], all FPN stages share the same detection head.

Original: Hence, FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD and lead to sub-optimal results.
Revised: Hence, FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD and lead to sub-optimal results.

Original: (3) There are always a few channels whose values are significantly larger than the others, as shown in Figure 1 (middle).
Revised: (3) There are always a few channels whose values are significantly larger than the others, as shown in Figure 1 (middle).

Original: However, it is mentioned in [29, 38] that the less activated features are still practical for distillation.
Revised: However, it is mentioned in [33, 45] that the less activated features are still practical for distillation.

Original: If they are not correctly balanced, the small gradients producedby these less activated channels can be drowned in the large gradients produced by dominant ones,thus limiting further refinement.
Revised: If they are not correctly balanced, the small gradients produced by these less activated channels can be drowned in the large gradients produced by dominant ones, thus limiting further refinement.

Original: Furthermore, from the first column of Figure 1 (right), we observethat there is much noise in the object-irrelevant area .
Revised: Furthermore, from the first column of Figure 1 (right), we observe that there is much noise in the object-irrelevant area .

Original: To understand why, recall that we selectthe maximum value in the dimension C at each pixel, while values of several channels are largerthan others.
Revised: To understand why, recall that we select the maximum value in the dimension C at each pixel, while values of several channels are larger than others.

Original: Noise in these channels is visualized.
Revised: Noise in these channels is visualized.

Original: Hence, directly imitating the feature maps mayintroduce much noise.
Revised: Hence, directly imitating the feature maps may introduce much noise.

Original: According to these observations, we propose K nowledge D istillation via P earson Correlation Coefficient (PKD) shown in Figure 2, which focuses on the linear correlation between features of theteacher and the student.
Revised: According to these observations, we propose K nowledge D istillation via P earson Correlation Coefficient (PKD) shown in Figure 2, which focuses on the linear correlation between features of the teacher and the student.

Original: To remove the negative influences of magnitude difference, dominant FPNstages and channels, we first normalize the feature maps to have zero mean and unit variances andminimize the MSE loss between the normalized features, which is equivalent to maximizing the
Revised: To remove the negative influences of magnitude difference, dominant FPN stages and channels, we first normalize the feature maps to have zero mean and unit variances and minimize the MSE loss between the normalized features.

Original:  Pearson Correlation Coefficient (PCC) [2].
Revised: Mathematically, it is equivalent to firstly calculating the Pearson Correlation Coefficient ( r ) [2] between two original feature vectors, and then using 1 − r as the feature imitation loss.

Original: In addition to addressing the above three issues, PCC alsohas a beneficial effect on the gradient flow through student detectors by reducing the dependence ofgradients on the feature value magnitude.
Revised: In addition to addressing the above three issues, PCC also has a beneficial effect on the gradient flow through student detectors by reducing the dependence of gradients on the feature value magnitude.

Original: So the training process is more stable.
Revised: So the training process is more stable.

Original: Compared with the previous methods, our method offers the following advantages.
Revised: Compared with the previous methods, our method offers the following advantages.

Original: First, as thedistillation loss is calculated just on FPN features, it can be easily applied to heterogeneous detectorpairs, including models with heterogeneous backbones, heterogeneous detection heads and differenttraining strategies such as label assignment.
Revised: First, as the distillation loss is calculated just on FPN features, it can be easily applied to heterogeneous detector pairs, including models with heterogeneous backbones, heterogeneous detection heads and different training strategies such as label assignment.

Original: Second, as there is no need to forward the teacher’sdetection head, the training time can be reduced significantly, especially for those models withcascaded heads.
Revised: Second, as there is no need to forward the teacher’s detection head, the training time can be reduced significantly, especially for those models with cascaded heads.

Original: Besides, PKD converges faster than previous methods.
Revised: Besides, PKD converges faster than previous methods.

Original: Last but not least, it is notsensitive to the only hyper-parameter - distillation loss weight.
Revised: Last but not least, it is not sensitive to the only hyper-parameter - distillation loss weight.

Original: So it can be easily applied to other datasets.
Revised: So it can be easily applied to other datasets.

Original: We conduct extensive experiments to verify the significant performance boosts brought byour approach on COCO dataset [22].
Revised: We conduct extensive experiments to verify the significant performance boosts brought by our approach on COCO dataset [24].

Original: Using the same two-stage detector as the teacher, ResNet50based RetinaNet [21] and FCOS [21] achieve 41.5% mAP and 43.9% mAP , which surpasses thebaseline by 4.1% and 4.8% respectively, and it also outperforms the previous state-of-the-art methodsby a large margin.
Revised: Using the same two-stage detector as the teacher, ResNet based RetinaNet [23] and FCOS [23] achieve 41.5% mAP and 43.9% mAP , which surpasses the baseline by 4.1% and 4.8% respectively, and it also outperforms the previous state-of-the-art methods by a large margin.

Original: In summary, the contributions of this paper are threefold:
Revised: In summary, the contributions of this paper are threefold:

Original: • We present that FPN feature imitation does make sense although the student-teacher detectorpairs are heterogeneous.
Revised: • We present that FPN feature imitation does make sense although the student-teacher detector pairs are heterogeneous.

Original: • We propose to imitate FPN features with PCC to focus on relational information and givemore freedom to the student feature’s magnitude.
Revised: • We propose to imitate FPN features with PCC to focus on relational information and give more freedom to the student feature’s magnitude.

Original: It is capable of distilling knowledge forboth homogeneous and heterogeneous detector pairs.•
Revised: It is capable of distilling knowledge for both homogeneous and heterogeneous detector pairs.

Original:  We verify the effectiveness of our method on various detectors via extensive experimentson the COCO [22], and without bells or whistles achieve state-of-the-art performance.
Revised: • We verify the effectiveness of our method on various detectors via extensive experiments on the COCO [24], and without bells or whistles achieve state-of-the-art performance.

Original: Moreover, our method converges faster and is not sensitive to the only hyper-parameter distillation loss weight.
Revised: Moreover, our method converges faster and is not sensitive to the only hyper-parameter distillation loss weight.

Original: It is simple yet effective.
Revised: It is simple yet effective.

Original:  Related Works2.1 Object Detection
Revised:  Related Works 2.1 Object Detection

Original: Object detection is considered one of the most challenging vision tasks which aims at detectingsemantic objects of a certain class in images.
Revised: Object detection is considered one of the most challenging vision tasks which aims at detecting semantic objects of a certain class in images.

Original: Modern detectors are roughly divided into two-stagedetectors [27, 3, 12] and one-stage detectors [21, 31, 19, 10].
Revised: Modern detectors are roughly divided into two-stage detectors [31, 4, 13] and one-stage detectors [23, 35, 21, 11].

Original: In two-stage detectors, a Region
Revised: In two-stage detectors, a Region

Original: Proposal Network(RPN) is usually adopted to generate initial rough predictions refined by a taskspecific detection head.
Revised: Proposal Network(RPN) is usually adopted to generate initial rough predictions refined by a taskspecific detection head.

Original: A typical example is Faster R-CNN [27].
Revised: A typical example is Faster R-CNN [31].

Original: In contrast, One-stage detectors,such as RetinaNet [21] and FCOS [31], can directly and densely predict bounding boxes on theoutput feature maps.
Revised: In contrast, One-stage detectors, such as RetinaNet [23] and FCOS [35], can directly and densely predict bounding boxes on the output feature maps.

Original: Among these works, multi-scale features are usually adopted to handle objectsof various scales, e.g. FPN [20], which is considered a typical case for our study.
Revised: Among these works, multi-scale features are usually adopted to handle objects of various scales, e.g. FPN [22], which is considered a typical case for our study.

Original: The proposed PKDonly distills the intermediate features and does not rely on detector-specific designs so that it can beused directly on various detectors.
Revised: The proposed PKD only distills the intermediate features and does not rely on detector-specific designs so that it can be used directly on various detectors.

Original: 2.2 Knowledge Distillation
Revised: 2.2 Knowledge Distillation

Original: Knowledge Distillation (KD) is a kind of model compression and acceleration approach aimingat transferring knowledge from a teacher model to a student model.
Revised: Knowledge Distillation (KD) is a kind of model compression and acceleration approach aiming at transferring knowledge from a teacher model to a student model.

Original: It is popularized by [14] andthen its effectiveness in image classification has been explored by subsequent works [28, 15, 13,1, 36, 25, 23, 30, 32].
Revised: It is popularized by [15] and then its effectiveness in image classification has been explored by subsequent works [32, 16, 14, 1, 43, 28, 26, 34, 36].

Original: However, adapting KD to object detectors is nontrivial since minimizingthe Kullback–Leibler (KL) divergence between the classification head outputs fails to transfer thespatial information from the teacher and only brings limited performance gain to the student.
Revised: However, adapting KD to object detectors is nontrivial since minimizing the Kullback–Leibler (KL) divergence between the classification head outputs fails to transfer the spatial information from the teacher and only brings limited performance gain to the student.

Original: Thefollowing three strategies are usually adopted by previous methods in detection to handle the abovechallenge.
Revised: The following three strategies are usually adopted by previous methods in detection to handle the above challenge.

Original: First, the distillation is usually conducted among multi-scale intermediate features [18],which provides rich spatial information for detection.
Revised: First, the distillation is usually conducted among multi-scale intermediate features [20], which provides rich spatial information for detection.

Original: Second, different feature selection methods areproposed to overcome the foreground-background imbalance.
Revised: Second, different feature selection methods are proposed to overcome the foreground-background imbalance.

Original: Most of these works can be dividedinto three kinds according to the feature selection approach [16]: proposal-based [4, 18, 9, 35],rule-based [33, 11] and attention-based [37, 39, 17].
Revised: Most of these works can be divided into three kinds according to the feature selection approach [18]: proposal-based [6, 20, 9, 42], rule-based [38, 12] and attention-based [44, 46, 19].

Original: Third, as the relation between different objectscontains valuable information, many previous works try to improve the performance of detectors byenabling detectors to capture and make use of these relations, such as non-local modules [37] andglobal distillation [34].
Revised: Third, as the relation between different objects contains valuable information, many previous works try to improve the performance of detectors by enabling detectors to capture and make use of these relations, such as non-local modules [44] and global distillation [41].

Original: Unlike the previous works, we consider the magnitude difference, dominant FPN stages and channelsas the key problem.
Revised: Unlike the previous works, we consider the magnitude difference, dominant FPN stages and channels as the key problem.

Original: We hope our method could serve as a solid baseline and help ease future researchin knowledge distillation for object detectors.
Revised: We hope our method could serve as a solid baseline and help ease future research in knowledge distillation for object detectors.

Original:  Method 
Revised:  Method 3.1 Preliminaries

Original: 3.1 Preliminaries

Original: In this part, we briefly recap the traditional knowledge distillation for object detection.
Revised: In this part, we briefly recap the traditional knowledge distillation for object detection.

Original: Recently,feature-based distillation over multi-scale features is adopted to deal with rich spatial information fordetection.
Revised: Recently, feature-based distillation over multi-scale features is adopted to deal with rich spatial information for detection.

Original: Different imitation masks S are proposed to form an attention mechanism for foregroundfeatures and filter away noises in the background.
Revised: Different imitation masks S are proposed to form an attention mechanism for foreground features and filter away noises in the background.

Original: The objective can be formulated as:
Revised: The objective can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where L is the number of FPN layers, l represents the l -th FPN layer, i , j are the location ofcorresponding feature map with width W and height H .
Revised: where L is the number of FPN layers, l represents the l -th FPN layer, i , j are the location of the corresponding feature map with width W and height H .

Original: N l = (cid:80) Cc (cid:80)
Revised: N l = (cid:80) Cc (cid:80)

Original: Wi (cid:80)
Revised: Wi (cid:80)

Original: Hj S lcij .
Revised: Hj S lcij .

Original: F tl and F slare the l -th layer of feature of student and teacher detectors, respectively.
Revised: F tl and F sl are the l -th layer of feature of student and teacher detectors, respectively.

Original: Function ϕ adapt is a 1x1convolution layer to upsample the number of channels for the student network if the number ofchannels mismatches between the teacher and the student.
Revised: Function ϕ adapt is a 1x convolution layer to upsample the number of channels for the student network if the number of channels mismatches between the teacher and the student.

Original: The definition of S is different in these methods.
Revised: The definition of S is different in these methods.

Original: For example, FRS [39] use the aggregatedclassification score map from the corresponding FPN layer, and FGD [34] considers spatial attention,channel attention, object size and foreground-background information simultaneously.
Revised: For example, FRS [46] uses the aggregated classification score map from the corresponding FPN layer, and FGD [41] considers spatial attention, channel attention, object size and foreground-background information simultaneously.

Original: 3.2 Is FPN feature imitation applicable for 
Revised: 3.2 Is FPN feature imitation applicable for heterogeneous detector pairs?

Original: heterogeneous detector pairs?

Original: [Table] Most of the previous works perform distillation on FPN,as FPN integrates multiple layers of the backbone and provides rich spatial information of multi-scale objects.
Revised: Most of the previous works perform distillation on FPN, as FPN integrates multiple layers of the backbone and provides rich spatial information of multi-scale objects.

Original: It isreasonable to force the student to imitate FPN features fromits homogeneous teacher as they have the same detectionhead and label assignment, and better features could lead tobetter performance.
Revised: It is reasonable to force the student to imitate FPN features from its homogeneous teacher as they have the same detection head and label assignment, and better features could lead to better performance.

Original: However, there is still a lack of studyon how to distill between heterogeneous detectors.
Revised: However, there is still a lack of study on how to distill between heterogeneous detectors.

Original: [34, 39]conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection headsand different label assignments are always omitted.
Revised: [41, 46] conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted.

Original: Thus,we are motivated to investigate whether FPN feature imitation still makes sense for these heterogeneous detectorpairs.
Revised: Thus, we are motivated to investigate whether FPN feature imitation still makes sense for these heterogeneous detector pairs.

Revised: We first simulate the extreme case of feature-based knowledge distillation via the following backbone and neck replacement.

Original: We first conduct experiments on three popular detectors, including RetinaNet [21], FCOS [31] and GFL [19].
Revised: Without loss of generality, three three popular detectors, including RetinaNet [23], FCOS [35] and GFL [21] are adopted.

Original: Their backbones are the same, while the detection heads and training strategies such aslabel assignment are different.
Revised: Their backbones are the same, while the detection heads and training strategies such as label assignment are different.

Original: We first train these detectors for 12 epochs, then freeze the pre-trainedbackbones and necks, and fine-tune the randomly initialized detection heads for 12 epochs.
Revised: We first train these detectors for 12 epochs, then freeze the pre-trained backbones and necks, and fine-tune the randomly initialized detection heads for 12 epochs.

Original: Duringfine-tuning, the parameters in the backbone and neck are frozen and will not be updated.
Revised: During fine-tuning, the parameters in the backbone and neck are frozen and will not be updated.

Original: By analyzingthe results in Table 1, we obtain the following two observations: (1) As there is no significantmagnitude difference between features of GFL and FCOS 1 , the pre-trained backbone and neck of GFL-Res50 can be used to fine-tune the FCOS head and achieve higher performance. There isa significant mAP drop when fine-tuning the Retina head with the pre-trained backbone and neckof FCOS-Res50 due to the magnitude difference 2 . Hence, FPN feature imitation is applicable forheterogeneous detectors after removing the negative impacts of magnitude difference.
Revised: By analyzing the results in Table 1, we observe that distillation between heterogeneous detectors is possible, but feature magnitude difference needs to be eliminated first.

Original: 3.3 Feature Imitation with Pearson Correlation Coefficient
Revised: 3.3 Feature Imitation with Pearson Correlation Coefficient

Original:  As discussed in Section 3.2, a promising feature distillation approach needs to consider the magnitudedifference when constructing them into pairs for imitation.
Revised: [Table] As discussed in Section 3.2, a promising feature distillation approach needs to consider the magnitude difference when constructing them into pairs for imitation.

Original: Moreover, by comparing the activationpatterns shown in Figure 1, we find the dominant FPN stages and channels can negatively interferewith the training phase of the student and lead to sub-optimal results, which is ignored by previousworks.
Revised: Moreover, by comparing the activation patterns shown in Figure 1, we find the dominant FPN stages and channels can negatively interfere with the training phase of the student and lead to sub-optimal results, which is ignored by previous works.

Original: To address the above issues, we propose first normalizing the features of the teacher and the studentto have zero mean and unit variances and minimizing the MSE between normalized features.
Revised: To address the above issues, we propose first normalizing the features of the teacher and the student to have zero mean and unit variances and minimizing the MSE between normalized features.

Original: Additionally, we want the normalization to obey the convolutional property - so that different elements ofthe same feature map, at different locations, are normalized in the same way.
Revised: Additionally, we want the normalization to obey the convolutional property - so that different elements of the same feature map, at different locations, are normalized in the same way.

Original: In order words, featuresconvolved by the same convolution kernel are normalized in the same way.
Revised: In other words, features convolved by the same convolution kernel are normalized in the same way.

Original: To accomplish this, wejointly normalize all the activations over all locations in a mini-batch.
Revised: To accomplish this, we jointly normalize all the activations over all locations in a mini-batch.

Original: Let B be the set of all values ina feature map across both the elements of a mini-batch and spatial locations.
Revised: Let B be the set of all values in a feature map across both the elements of a mini-batch and spatial locations.

Original: So for a mini-batch ofsize b and feature maps of size h × w , we use the effective mini-batch of size m =
Revised: So for a mini-batch of size b and feature maps of size h × w , we use the effective mini-batch of size m =

Original: ∥ B ∥
Revised: ∥ B ∥

Original: = b · hw .
Revised: = b · hw .

Original: Let s ( c ) ∈
Revised: Let s ( c ) ∈

Original: R m be the c -th channel of a batch of FPN outputs and omit c for clarity.
Revised: R m be the c -th channel of a batch of FPN outputs and omit c for clarity.

Original: Then we getthe normalized values
Revised: Then we get the normalized values

Original: ˆ s 1 ...m and ˆ t 1 ...m from the student and the teacher, respectively.
Revised: ˆ s 1 ...m and ˆ t 1 ...m from the student and the teacher, respectively.

Original: Instead ofdelicately designing imitation masks S in Eq.
Revised: Instead of delicately designing imitation masks S in Eq.

Original: 1 to choose the important features, we treat foregroundand background equally as distillation via background regions effectively reduces the number ofbackground false positives [11].
Revised: 1 to choose the important features, our PKD only operates on the full feature map.

Original: That is, the imitation mask S is filled with the scalar value 1.
Revised: That is, the imitation mask S is filled with the scalar value 1.

Original: Hencewe can formulate our distillation loss as the following:
Revised: Hence we can formulate our distillation loss as the following:

Original: [Equation]
Revised: [Equation]

Original: Moving forward, minimizing the loss function above is equivalent to maximizing the PCC betweenthe pre-normalized features of the student and the teacher.
Revised: Moving forward, minimizing the loss function above is equivalent to maximizing the PCC between the pre-normalized features of the student and the teacher.

Original: The PCC can be computed as:
Revised: PCC can be computed as:

Original: [Equation]
Revised: [Equation]

Original: As ˆ s , ˆ t ∼ N (0 , 1) , we get
Revised: As ˆ s , ˆ t ∼ N (0 , 1) , we get

Original: [Equation]
Revised: [Equation]

Original: ˆ t 2 i = 1 .
Revised: ˆ [Equation] 2 i = 1 .

Original: Then, we can reformulate Eq. 2 as:
Revised: Then, we can reformulate Eq. 2 as:

Original: [Equation]
Revised: [Equation]

Original: The gradient of PCC, ∂ L /∂s i , with respect to each FPN output s i , is given by:
Revised: The gradient of PCC, ∂ L /∂s i , with respect to each FPN output s i , is given by:

Original: [Equation]
Revised: [Equation]

Original: recalling r ( s , t ) is PCC in Eq.
Revised: recalling r ( s , t ) is PCC in Eq.

Original: 3 and σ s is the sample standard deviation of the student’s feature s .
Revised: 3 and σ s is the sample standard deviation of the student’s feature s .

Original: So PCC also has a beneficial effect on the gradient flow through student detectors by reducing the dependence of gradients on the feature value magnitude.
Revised: So PCC also has a beneficial effect on the gradient flow through student detectors by reducing the dependence of gradients on the feature value magnitude.

Original: It makes gradient descent-based trainingalgorithms more stable.
Revised: It makes gradient descent-based training algorithms more stable.

Original: In conclusion, PCC focuses on the relational information from the teacher and gives more freedom tothe student feature’s magnitude.
Revised: In conclusion, PCC focuses on the relational information from the teacher and gives more freedom to the student feature’s magnitude.

Original: Moreover, PCC eliminates the negative impacts of the dominant FPN stages and channels, leading to better performance.
Revised: Moreover, it eliminates the negative impacts of the dominant FPN stages and channels, leading to better performance.

Original: In addition, the normalization mechanismbridge the gap between the activation patterns of the student and the teacher (see Figure 2).
Revised: As a result, the normalization mechanism bridge the gap between the activation patterns of the student and the teacher (see Figure 2).

Original: Hence,feature imitation with PCC works for heterogeneous detector pairs.
Revised: Hence, feature imitation with PCC works for heterogeneous detector pairs.

Original: We train the student detector withthe total loss as follows:
Revised: We train the student detector with the total loss as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where L GT is the detection training loss, α is the hyper-parameter to balance the detection trainingloss and distillation loss.
Revised: where L GT is the detection training loss, α is the hyper-parameter to balance the detection training loss and distillation loss.

Original: 3.4 Connection of PCC and KL divergence
Revised: 3.4 Connection of PCC and KL divergence

Original: As discussed in Section 3.3, the normalization mechanism is the key to addressing the three issuesmentioned above.
Revised: As discussed in Section 3.3, the normalization mechanism is the key to addressing the three issues mentioned above.

Original: At the same time, KL divergence has been widely used in distillation.
Revised: At the same time, KL divergence has been widely used in distillation.

Original: Theyfirst convert activations into a probability distribution with softmax function and then minimize theasymmetry KL divergence of the normalized activation maps:[Equation]
Revised: They first convert activations into a probability distribution with softmax function and then minimize the asymmetry KL divergence of the normalized activation maps:

Original: [Equation]
Revised: [Equation]

Original: where T is a hyper-parameter to control the degree of softness of the targets, and ϕ is the softmaxfunction:
Revised: where T is a hyper-parameter to control the degree of softness of the targets, and ϕ is the softmax function:

Original: [Equation]
Revised: [Equation]

Original: Here, we show that minimizing KL divergence between post-normalized features in the hightemperature limit is equivalent to minimizing MSE between post-normalized ones and hence equivalent to maximizing PCC between pre-normalized ones.
Revised: Here, we show that minimizing KL divergence between normalized features in the high-temperature limit is equivalent to minimizing MSE between normalized ones, and hence equivalent to maximizing PCC between original ones.

Original: Let p i = ϕ
Revised: Let p i = ϕ

Original: (ˆ t i ) and q i = ϕ
Revised: (ˆ t i ) and q i = ϕ

Original: ( ˆ s i ) denote the probabilities from the teacher and the student, respectively.
Revised: ( ˆ s i ) denote the probabilities from the teacher and the student, respectively.

Original: The KL divergence gradient, ∂ L KL /∂
Revised: The KL divergence gradient, ∂ L KL /∂

Original: ˆ s i , with respect to each normalized activation
Revised: ˆ s i , with respect to each normalized activation

Original: ˆ s i of the studentis given by:
Revised: ˆ s i of the student is given by:

Original: [Equation]
Revised: [Equation]

Original: If T is high compared with the magnitude of the normalized activations, we can approximate:
Revised: If T is high compared with the magnitude of the normalized activations, we can approximate:

Original: [Equation]
Revised: [Equation]

Original: As ˆ s i and ˆ t i have been zero-meaned, Eq.
Revised: As ˆ s i and ˆ t i have been zero-meaned, Eq.

Original: 10 simplifies to:
Revised: 10 simplifies to:

Original: [Equation]
Revised: [Equation]

Original: Exploiting the relation between MSE and PCC shown in Eq.
Revised: Exploiting the relation between MSE and PCC shown in Eq.

Original: 4 gives the desired result.
Revised: 4 gives the desired result.

Original: Experimentalresults are provided in the Appendix.
Revised: Experimental results are provided in Section A.3.

Original:  Experiments
Revised:  Experiments

Original: In order to verify the effectiveness and robustness of our method, we conduct experiments on differentdetection frameworks on the COCO [22] dataset.
Revised: In order to verify the effectiveness and robustness of our method, we conduct experiments on different detection frameworks on the COCO [24] dataset.

Original: We choose the default 120k train images split fortraining and 5k val images split for the test.
Revised: We choose the default 120k train images split for training and 5k val images split for the test.

Original: We use the standard training settings following [34]and report mean Average Precision (AP) as an evaluation metric, together with AP under differentthresholds and scales, i.e. , AP 50 , AP 75 , AP S , AP M and AP L .[Table] [Table] [Table]
Revised: We use the standard training settings following [41] and report mean Average Precision ( AP ) as an evaluation metric, together with AP under different thresholds and scales, i.e. , AP 50 , AP 75 , AP S , AP M and AP L .

Original: [Table] [Table] [Table] [Table]
Revised: For distillation, the hyper-parameter α is set to 6 when using a two-stage detector as the teacher and 10 when using a one-stage one.

Original: [Table] [Table] [Table]
Revised: Feature levels of heterogeneous detector pairs may not be strictly

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: aligned, e.g. , FasterRCNN constructs the feature pyramid from P 2 to P 6 , while RetinaNet uses

Original: [Table] [Table] [Table] [Table]
Revised: P 3 to P 7 .

Original: [Table] [Table] [Table] [Table]
Revised: To address the above issue, we upsample the low-resolution feature maps to have the same spatial size as the high-resolution ones.

Original: [Table] For distillation, the hyper-parameter α is set to 6 when using a two-stage detector as the teacher and10 when using a one-stage one as the teacher. Feature levels of heterogeneous detector pairs may notbe strictly aligned, e.g. , FasterRCNN constructs the feature pyramid from P 2 to P 6 , while RetinaNetuses P 3 to P 7 . To address the above issue, we upsample the low-resolution feature maps to have thesame spatial size as the high-resolution ones.
Revised: In addition, the adaptive layer ( ϕ adapt in Eq. 1) is unnecessary for our PKD.

Original: All experiments are performed on 8 Tesla A100 GPUs with two images in each.
Revised: All experiments are performed on 8 Tesla A100 GPUs with two images in each.

Original: Our implementationis based on mmdetection [6] and mmrazor [7] with Pytorch framework [26]. ’1x’ (namely 12epochs),’2x’ (namely 24 epochs) and ’2x+ms’ (namely 24epochs with multi-scale training) training schedulesare used.
Revised: Our implementation is based on mmdetection [7] and mmrazor [8] with Pytorch framework [29]. ’1x’ (namely 12epochs), ’2x’ (namely 24 epochs) and ’2x+ms’ (namely 24epochs with multi-scale training) training schedules are used.

Original: More details are given in the Appendix.
Revised: More details are given in the Appendix.

Original: 4.1 Main Results
Revised: 4.1 Main Results

Original: Our method can be applied to different detection frameworks easily, so we first conduct experimentson five popular detectors, including a two-stage detector (Faster RCNN[27]), three anchor-basedone-stage detector (RetinaNet [21], GFL [19] and TOOD [10]) and an anchor-free detector (FCOS[31]).
Revised: Our method can be applied to different detection frameworks easily, so we first conduct experiments on five popular detectors, including a two-stage detector (Faster RCNN[31]), two anchor-based one-stage detector (RetinaNet [23], RepPoints [40] and TOOD [11]) and an anchor-free detector (FCOS [35]).

Original: Table 2 shows the comparison of the results of state-of-the-art distillation methods on the COCO.
Revised: Table 2 shows the comparison of the results of state-of-the-art distillation methods on the COCO.

Original: Our distillation method surpasses other state-of-the-art methods.
Revised: Our distillation method surpasses other state-of-the-art methods.

Original: All the student detectorsgain significant improvements in AP with the knowledge transferred from teacher detectors, e.g. ,
Revised: All the student detectors gain significant improvements in AP with the knowledge transferred from teacher detectors, e.g. ,

Original: TOOD with ResNet-50 gets a 3.4% mAP improvement on the COCO dataset.
Revised: FCOS with ResNet-50 gets a 3.7% mAP improvement on the COCO dataset.

Original: These results indicatethe effectiveness and generality of our method in both one-stage and two-stage detectors.
Revised: These results indicate the effectiveness and generality of our method in both one-stage and two-stage detectors.

Original: 4.2 Distilling More Student Detectors with Stronger Heterogeneous Teachers
Revised: 4.2 Distilling More Student Detectors with Stronger Heterogeneous Teachers

Original: Most of the current methods are designed for homogeneous detector pairs.
Revised: Most of the current methods are designed for homogeneous detector pairs.

Original: PKD is general enough todistill knowledge between both homogeneous and heterogeneous detector pairs.
Revised: PKD is general enough to distill knowledge between both homogeneous and heterogeneous detector pairs.

Original: Here we conductexperiments on more detectors and use stronger heterogeneous teacher detectors, as shown in Table3.
Revised: Here we conduct experiments on more detectors and use stronger heterogeneous teacher detectors, as shown in Table 3. Comparing with Table

Original: Comparing with Table 2, we find that student detectors perform better with stronger teacherdetectors, e.g. , Retina-Res50 model achieves 41.5% and 39.9% mAP with Mask RCNN-Swin [24]and Retina-Res 101, respectively.
Revised: [Table] we find that student detectors perform better with stronger teacher detectors, e.g. , Retina-Res50 model achieves 41.5% and 39.6% mAP with Mask RCNN-Swin [27] and RetinaRes101, respectively.

Original: In addition, our method also generalizes well for large detectorslike CascadeRCNN with ResNeXt101-DCN [8] as the backbone: boosting its mAP from 47.3% to49.0% with help of Hybrid Task Cascade (HTC) [5].

Original: Results show thatstudent detectors get betterfeatures by mimicking the feature maps of stronger heterogeneous teacher detectors.
Revised: Results show that mimicking the feature maps of stronger heterogeneous teacher detectors can further boost the student’s performance when applied with our PKD

Original: [Table]
Revised: [Table]

Original: Retina-Res50 (S) 2x 37.
Revised: [Table] three issues and achieve better performance.

Original: PKD (Ours) 2x 41.5 (+4.1) 60.
Revised: [Figure] Other feature -based

Original: Mask RCNN-Swin (T) 3x+ms 48.
Revised: distillation methods

Original: FasterRCNN-Res50 (S) 2x 38.
Revised: with the normalized features In object detection, some feature-based distillation methods [46, 41] transfer the knowledge within some pre-defined imitation regions, and achieved competitive results. While we believe full map feature imitation with our PKD can already outperform them.

Original:  PKD (Ours) 2x 42.2 (+3.8) 62. GFL-Res101 (T) 2x+ms 44. FCOS-Res50 (S) 2x+ms 39.
Revised: In order to show the universality of our method, we take FitNet [32], GT Mask, FRS [46] and FGD [41] as baselines and build PKD on them. Recaps.

Original: PKD (Ours) 2x+ms 43.5 (+4.4) 61.
Revised: Full feature map distillation with MSE loss is used in FitNet. While the 2nd method only imitates features that overlap with ground truth bounding boxes. Feature imitation and logit mimicking with FRS uses the aggregated classification score, takes maximum operation in channel direction, of the classification head output as a weighted score mask, to guide the distillation of the FPN and classification head.

Original: Mask RCNN-Swin (T) 3x+ms 48.
Revised: And Focal distillation in FGD forces the student to focus on the teacher’s pixels and channels with large absolute mean values. Global distillation in FGD rebuilds the relation between different pixels and transfers it from teachers to students through GcBlock [5]. Results. For a fair comparison, all the hyper-parameters are copied from the original paper.

Original: FCOS-Res50 (S) 2x+ms 39. PKD (Ours) 2x+ms 43.9 (+4.8) 62. HTC - ResX101 (T) 20e 50.
Revised: Results in Table 5 show that combining PKD with these feature imitation methods can further improve their performance.

Original: Cascade Mask RCNN - ResX101 (S) 1x 47.
Revised: Besides, we do not combine PKD with FGD focal distillation as focal distillation forces the student to focus on the teacher’s pixels and channels with large absolute mean values. That is, features with larger values are given larger weights, which seems not to agree with the argument that "The small gradients produced by less activated features can be drowned into the large gradients produced by highly activated ones, thus limiting further refinement." in [33, 45] and us.

Original: [Table]
Revised: [Table]

Original: 4.3 Analysis 
Revised: 4.4 Analysis 4.4.1 Effectiveness of Pearson Correlation Coefficient

Original: 4.3.1 Effectiveness of Pearson Correlation Coefficient

Original: Among most previous works, feature-based distillation over multi-scale features is adopted fordistillation.
Revised: Among most previous works, feature-based distillation over multi-scale features is adopted for distillation.

Original: In this study, we argue that the magnitude difference, dominant FPN stages and channelscan negatively interfere with the training phase of the student and lead to sub-optimal results.
Revised: In this study, we argue that the magnitude difference, dominant FPN stages and channels can negatively interfere with the training phase of the student and lead to sub-optimal results.

Original: Tostudy this empirically, we conduct the following three experiments to explore the effects of MSE losswhich is a widely-used loss function  and suffers from the above three issues simultaneously.
Revised: To study this empirically, we conduct the following three experiments to explore the effects of MSE loss which is a widely-used loss function in distillation and suffers from the above three issues simultaneously.

Original: We tunethe loss weight of MSE for all experiments.
Revised: For a fair comparison, we tune the loss weight of MSE for all experiments.

Original: More details are listed in the Appendix.
Revised: More details are listed in Section A.4.

Original: In the first pair,we use FCOS-ResX101  as the teacher whose feature magnitude is significantlydifferent from that of the student Retina-Res50.
Revised: For the first pair, FCOS-ResX101 is used as the teacher whose feature magnitude is significantly different from that of the student Retina-Res50 (refer to Fig. 4).

Original: In this case, what constitutes the knowledge is better presented by relational information from the teacher’s features than the absolute values.
Revised: In this case, what constitutes the knowledge is better presented by relational information from the teacher’s features than the absolute values.

Original: Inthe second pair, GFL-Res101 is the teacher and FCOS-Res50 is the student.
Revised: For the second pair, GFL-Res101 is the teacher and FCOS-Res50 is the student.

Original: For both teacher andstudent, features in FPN stage ’P6’ are less activated than those in stage ’P3’ .
Revised: For both teacher and student, features in FPN stages ’P5’ and ’P6’ are less activated than those in stages ’P3’ and ’P4’ (refer to Fig. 5).

Original: Hence, features instage ’P3’  could dominate the gradient of the distillation loss, which will overwhelm the effectsofother features (see AP S and AP L of  MSE and Ours in Table 4).
Revised: Hence, features in stages ’P3’ and ’P4’ could dominate the gradient of the distillation loss, which will overwhelm the effects of other features. Since larger objects are usually assigned to higher feature levels, the student’s performance for large objects is significantly lower than that of ours when MSE is used as the distillation loss.

Original: In the last pair, Retina-ResX101 isthe teacher and Retina-Res50 is the student.
Revised: For the last pair, Retina-ResX101 is the teacher and Retina-Res50 is the student.

Original: There are always a few channels with greater values, sodirectly imitating the feature maps may introduce much noise in dominant channels.
Revised: There are always a few channels with greater values (refer to Fig. 6 and Fig. 7) , so directly imitating the feature maps may introduce much noise in dominant channels.

Original: By comparing the results in Table 4, we find that our proposed PKD addresses the above three issuesand achieves better performance.
Revised: By comparing the results in Table 4, we find that our proposed PKD addresses the above three issues and achieves better performance.

Original: Hence, an effective distillation loss function should have the abilityto handle the above three problems.
Revised: Hence, an effective distillation loss function should have the ability to handle the above three problems.

Original: We hope PKD could serve as a solid baseline and help easefuture research in knowledge distillation community.[Table]
Revised: We hope PKD could serve as a solid baseline and help ease future research in knowledge distillation community.

Original: The student detector is Retina-Res50.

Original: [Table]

Original: 4.3.2 Sensitivity study of 
Revised: 4.4.2 Sensitivity study of loss weight α

Original: weight

Original: [Table] 
Revised: [Table] [Table]

Original: In Eq.
Revised: In Eq.

Original: 6, we use the loss weight hyper-parameter α to balancethe detection training loss and distillation loss.
Revised: 6, we use the loss weight hyper-parameter α to balance the detection training loss and distillation loss.

Original: Here, we conductseveral experiments to investigate the influence of α .
Revised: Here, we conduct several experiments to investigate the influence of α .

Original: As shownin Table 6, the worst result is just a 0.3 mAP drop comparedwith the best result, indicating our method is not sensitive to thehyper-parameter α .
Revised: As shown in Table 6, the worst result is just a 0.3 mAP drop compared with the best result, indicating our method is not sensitive to the only hyper-parameter α .

Original: In addition, since only one hyper-parameteris introduced in PKD, it can be applied to other datasets easily.
Revised: Hence it is easy for practical use.

Original: Moving forward, as the normalization mechanism has a beneficialeffect on the gradient flow through student detectors, by reducing the dependence of gradients on thefeature value magnitude, a same loss weight α can be used across various detectors.
Revised: Moving forward, as the normalization mechanism has a beneficial effect on the gradient flow through student detectors, by reducing the dependence of gradients on the feature value magnitude, we can train different networks with relatively uniform hyper-parameters α .

Original: 4.3.3 Convergence Speed
Revised: 4.4.3 Convergence Speed

Original: In this subsection, we conduct experiments with RetinaNet to compare the convergence speed of ourmethod with other state-of-the-art distillation methods on the COCO benchmark.
Revised: In this subsection, we conduct experiments with FCOS to compare the convergence speed of our method with other state-of-the-art distillation methods on the COCO benchmark.

Original: As shown in Table5, Retina-Res50 with our PKD achieves 40.3% mAP under 1x schedule and outperforms previousmethods under 2x schedule by a large margin.
Revised: As shown in Fig. 3, the training convergence can significantly speed up in the early training stage with our PKD. Meanwhile, the final performance of the student detector FCOS-Res50 is about 0.5%-0.6% mAP higher than that of FGD and FRS.

Original: In addition, there is no need to forward the detectionhead of the teacher, which reduces the training time significantly, especially for those models withcascaded heads.
Revised: In addition, there is no need to forward the detection head of the teacher, which reduces the training time significantly, especially for those models with cascaded heads.

Original: Conclusion
Revised: 4.4.4 Mathematical properties of Pearson Correlation Coefficient

Revised: The Pearson correlation coefficient is essentially a normalized measurement of the covariance, such that the result always has a value between − 1 and 1 . Hence, L FPN = 1 − r in Eq. 4 always has a value between 0 and 2.

Revised: It focuses on the linear correlation between the features of the teacher and the student, and relaxes constraints on the magnitude of the features.

Revised: Let s c , t c ∈ R bhw denote the c-th channel of the student’s and teacher’s features, and omit c for clarification.

Revised: Then we get bhw data points ( s i , t i ).

Revised: L FPN = 0 implies that all data points lie on a line for which s increases as t increases.

Revised: Hence the student is well-trained.

Revised: And vice versa for L FPN = 2 .

Revised: A value of 1 implies that there is no linear dependency between the features of the student and the teacher.

Original: This paper empirically finds that FPN feature imitation is applicable for heterogeneous detector pairsalthough their detection heads and label assignments are different.
Revised: This paper empirically finds that FPN feature imitation is applicable for heterogeneous detector pairs although their detection heads and label assignments are different.

Original: Then we propose feature imitationwith Pearson Correlation Coefficient to focus on the relational information from the teacher and givemore freedom to the student’s feature value magnitude.
Revised: Then we propose feature imitation with Pearson Correlation Coefficient to focus on the relational information from the teacher and give more freedom to the student’s feature value magnitude.

Original: Furthermore, a general KD framework isproposed, capable of distilling knowledge for both homogeneous and heterogeneous detector pairs.
Revised: Furthermore, a general KD framework is proposed, capable of distilling knowledge for both homogeneous and heterogeneous detector pairs.

Original: It converges faster and only introduces one hyper-parameter, which can easily be applied to otherdatasets.
Revised: It converges faster and only introduces one hyper-parameter, which can easily be applied to other datasets.

Original: However, our understanding of whether our proposed PKD is capable of other tasks such astext recognition is preliminary and left as future works.
Revised: However, our understanding of whether our proposed PKD is capable of other tasks such as text recognition is preliminary and left as future works.

Original: Checklist
Revised: Acknowledgments

Original: For all authors...
Revised: This work is supported in part by the Shanghai Committee of Science and Technology, China (Grant No. 20DZ1100800), NSFC 62273347, the National Key Research and Development Program of China (2020AAA0103402), Jiangsu Key Research and Development Plan (No.BE2021012-2) and

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: NSFC A Appendix A.1 Motivation to build a general KD framework Modern detectors are roughly divided into two-stage detectors [31, 13, 3] and dense prediction detectors ( e.g. , anchor-based one-stage detectors [23, 25, 30] and anchor-free one-stage detectors [10, 35, 40]).

Original: contributions and scope?
Revised: Each family has its own advantages and weakness.

Original: [Yes] See Section 3.3 and Section 3.
Revised: In particular, two-stage detectors usually have higher performance, while being slower in inference speed and harder to be deployed due to the region proposal network (RPN) and RCNN head. On the other hand, dense prediction detectors are faster than two-stage detectors while being less accurate. In practice, it is a natural idea to use two-stage detectors as teachers to enhance dense prediction detectors. Moreover, knowledge distillation between heterogeneous dense prediction detector pairs is also promising.

Original: (b) Did you describe the limitations of your work?
Revised: In some scenarios, only the detectors with a specific architecture can be deployed due to hardware limitations.

Original: [Yes] See Section
Revised: For example, compared with Batch Normalization [17] and Instance Normalization [37], Group Normalization [39] is hard to deploy.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: However, the most powerful teachers may belong to different categories.

Original: [N/A] To thebest of our knowledge, there is no potential negative societal impacts of our work. (d) Have you read the ethics review guidelines and ensured that your paper conforms to 
Revised: Furthermore, object detection is developing rapidly and algorithms with better performance are proposed continuously. Nevertheless, it is not easy to change detectors frequently in terms of stability and hardware runtime limitations in practical applications.

Original: them?
Revised: So it is helpful if knowledge distillation can be conducted between the latest high-capacity detectors and the widely-used compact detectors.

Original: [Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: Hence, we are motivated to design a general distillation method capable of distilling knowledge between both homogeneous and heterogeneous detector pairs.

Original: [Yes] See Section3.3 and Section 3.4(b) Did you include complete proofs of all theoretical results? [Yes] See Section 3.3 andSection 3. If you ran experiments...
Revised: We conduct experiments on different detection frameworks, including two-stage models, anchorbased one-stage models and anchor-free one-stage models.

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or asa URL)? [Yes] Codes andinstructions are provided in the Appendix.
Revised: [18] proposes inheriting strategy which initializes the student with the teacher’s neck and head parameters and gets better results. Here we use this strategy to initialize the student which has the same head structure as the teacher and find that it helps students converge faster.

Original: All datasets used are public datasets.
Revised: All experiments are performed on 8 Tesla A100 GPUs with 2 images in each.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: Our implementation is based on mmdetection [7] and mmrazor [8] with Pytorch framework [29]. ’1x’ (namely 12 epochs), ’2x’ (namely 24 epochs) and ’2x+ms’ (namely 24 epochs with multi-scale training) training schedules with SGD optimizer are used.

Original: how they
Revised: Momentum and weight decay are set to 0.9 and 1e-4. The initial learning rate is set to 0.02 for Faster RCNN and 0.01 for others. We train FCOS [35] with tricks including GIoULoss, norm-on-bbox and center-sampling which is the same as FGD [41] and GID [9]. For distillation, only one hyper-parameter α is introduced to balance the supervised learning loss and distillation loss, and it is set to 6 when using a two-stage detector as the teacher and 10 when using a one-stage one as the teacher.

Original: were chosen)?
Revised: The teacher network is well-trained previously and fixed during training.

Original: [Yes] See Section 4. More details are given in the Appendix.
Revised: Section 3.4 in the main text shows the connection between PCC and KL divergence.

Original: (c) Did you report error bars (e.g., with respect to  the random seed after running experiments multiple times)?
Revised: We conduct two experiments on RetinaNet and GFL to verify this empirically. For both experiments, we set loss weight α = 10 and temperature T = 50 . As shown in Table 7, minimizing KL divergence between post-normalized features in the high-temperature limit can also achieve similar results. A.4 Details of feature imitation with MSE

Original: [No] We follow the common practice used in  our baselinemethods, where no error bars are reported.(d) Did you include the total amount of compute and  the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: As shown in Table 4 in the main text, we compare the results of MSE with our proposed PKD. As the gradient of MSE loss depends on feature value magnitude and it is usually different among different detectors, we have to tune the loss weight carefully to achieve relatively good results, as shown in

Original: [Yes] Details are given in the Appendix.4.
Revised: Table 8. And we put the best results down in Table 4. [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: [Table]

Original: (a) If your work uses existing assets, did you
Revised: A.5 Effectiveness of Pearson Correlation Coefficient

Original: cite the creators?
Revised: In this paper, we argue that the magnitude difference, dominant FPN stages and channels can negatively interfere with the training phase of the student. To clearly show that these three issues do exist, we elaborately visualize the FPN feature responses of the teacher and student detectors before distillation, as shown in Figure 4, Figure 5, Figure 6 and Figure 7.

Original: [Yes](b) Did you mention the license of the assets?
Revised: We follow the visualization method in Section 1.

Original: [N/A](c) Did you include any new assets either in the supplemental material or as a URL?
Revised: Through these comparisons, we obtain the following three observations:

Original: [No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [Yes] All datasets used are public datasets.(e)
Revised: (1) The feature value magnitude of different detectors could be significantly different, especially for heterogeneous detectors, as shown in Figure 4.

Original: Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: Directly aligning the feature maps between the teacher and the student may enforce overly strict constraints and do harm to the student (see Tabletop in the main text).

Original: [N/A]5. If you used crowdsourcing or conducted research with human subjects...
Revised: (2) As shown in Figure 5, compared with features in FPN stage ’P3’ and ’P4’, features in stage ’P5’ and ’P6’ are less activated in both GFL-Res101 and FCOS-Res50. And it is a common case among different detector pairs. FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD.

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A](b)
Revised: (3) As shown in Figure 6 (right), the feature magnitude of the 210 -th channel of MaskRCNN-Swin FPN stage ’P6’ is significantly larger than others.

Original: Did you describe any potential participant risks, with links to Institutional Review
Revised: Similar phenomena also exist in other detectors such as RetinaNet (see Figure 7).

Original: Board (IRB) approvals, if applicable?
Revised: The small gradients produced by those less activated channels can be drowned in the large gradients produced by dominant ones, thus limiting further refinement. Furthermore, there is much noise in the object-irrelevant area, as depicted in Figure 6 (left). Directly imitating the feature maps may introduce much noise.

Original: [N/A](c) Did you include the estimated hourly wage paid to participants and  the total amountspent on participant compensation?
Revised: Comparing Table 8 and Table 4 in the main text, we find that our proposed PKD addresses the above three issues and achieves better performance.

Original: [N/A]
Revised: Hence, an effective distillation loss function should have the ability to handle the above three problems. We hope PKD could serve as a solid baseline and help ease future research in knowledge distillation community. We define channels with a larger number as dominant channels.