Original: L EARNING L IGHTWEIGHT O BJECT D ETECTORS VIA P ROGRESSIVE K NOWLEDGE D ISTILLATION
Revised: L EARNING L IGHTWEIGHT O BJECT D ETECTORS VIA P ROGRESSIVE K NOWLEDGE D ISTILLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Resource-constrained perception systems such as edge computing and vision-forrobotics require vision models to be both accurate and lightweight in computation and memory usage.
Revised: Resource-constrained perception systems such as edge computing and vision-forrobotics require vision models to be both accurate and lightweight in computation and memory usage.

Original: Knowledge distillation is one effective strategy to improve the performance of lightweight classification models, but it is less well-explored for structured outputs such as object detection and instance segmentation, where the variable number of outputs and complex internal network modules complicate the distillation.
Revised: Knowledge distillation is one effective strategy to improve the performance of lightweight classification models, but it is less well-explored for structured outputs such as object detection and instance segmentation, where the variable number of outputs and complex internal network modules complicate the distillation.

Original: In this paper, we propose a simple yet surprisingly effective sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teachers to a given lightweight student.
Revised: In this paper, we propose a simple yet surprisingly effective sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teachers to a given lightweight student.

Original: Our approach is inspired by curriculum learning: To distill knowledge from a highly accurate but complex teacher model, we construct a sequence of teachers to help the student gradually adapt.
Revised: Our approach is inspired by curriculum learning: To distill knowledge from a highly accurate but complex teacher model, we construct a sequence of teachers to help the student gradually adapt.

Original: Our progressive distillation strategy can be easily combined with existing distillation mechanisms to consistently maximize student performance in various settings.
Revised: Our progressive distillation strategy can be easily combined with existing distillation mechanisms to consistently maximize student performance in various settings.

Original: To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students, and unprecedentedly boost the performance of ResNet-50 based RetinaNet from 36.5% to 42.0% AP and Mask R-CNN from 38.2% to 42.5% AP on the MS
Revised: To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students, and unprecedentedly boost the performance of ResNet-50 based RetinaNet from 36.5% to 42.0% AP and Mask R-CNN from 38.2% to 42.5% AP on the MS

Original: COCO benchmark.
Revised: COCO benchmark.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The success of recent deep neural network models generally depends on an elaborate design of architectures with tens or hundreds of millions of model parameters.
Revised: The success of recent deep neural network models generally depends on an elaborate design of architectures with tens or hundreds of millions of model parameters.

Original: However, their huge computational complexity and massive memory/storage requirements make them challenging to be deployed in safety-critical real-time applications, especially on devices with limited resources, such as selfdriving cars or virtual/augmented reality models.
Revised: However, their huge computational complexity and massive memory/storage requirements make them challenging to be deployed in safety-critical real-time applications, especially on devices with limited resources, such as selfdriving cars or virtual/augmented reality models.

Original: Such concerns have spawned a wide body of literature on compression and acceleration techniques.
Revised: Such concerns have spawned a wide body of literature on compression and acceleration techniques.

Original: Many approaches focus on reducing computation demands by sparsifying/pruning networks (Lebedev & Lempitsky, 2016; Han et al., 2016), quantization (Rastegari et al., 2016; Wu et al., 2016), or neural architecture search (Zoph & Le, 2017;
Revised: Many approaches focus on reducing computation demands by sparsifying/pruning networks (Lebedev & Lempitsky, 2016; Han et al., 2016), quantization (Rastegari et al., 2016; Wu et al., 2016), or neural architecture search (Zoph & Le, 2017;

Original: Liu et al., 2019), but reduced computation does not always translate into lower latency because of subtle issues with memory access and caching on GPUs (Tan et al., 2019; Ding et al., 2021).
Revised: Liu et al., 2019), but reduced computation does not always translate into lower latency because of subtle issues with memory access and caching on GPUs (Tan et al., 2019; Ding et al., 2021).

Original: Rather than searching over new architectures, we seek to better train existing lightweight architectures that have already been carefully engineered for efficient memory access.
Revised: Rather than searching over new architectures, we seek to better train existing lightweight architectures that have already been carefully engineered for efficient memory access.

Original: Instead of relying on additional data or human supervision, we follow the large body of work on knowledge distillation (Buciluˇa et al., 2006; Hinton et al., 2014) for compressing the information from a large model into a small model.
Revised: Instead of relying on additional data or human supervision, we follow the large body of work on knowledge distillation (Buciluˇa et al., 2006; Hinton et al., 2014) for compressing the information from a large model into a small model.

Original: While most recent efforts in knowledge distillation focus on image classification, relatively less work exists for distilling object detectors.
Revised: While most recent efforts in knowledge distillation focus on image classification, relatively less work exists for distilling object detectors.

Original: The extension from classification to object detection and instance segmentation is nontrivial due to the complicated outputs of the tasks.
Revised: The extension from classification to object detection and instance segmentation is nontrivial due to the complicated outputs of the tasks.

Original: Most detectors operate with multi-task heads (for classification, and box/mask regression) that can generate variable-length outputs.
Revised: Most detectors operate with multi-task heads (for classification, and box/mask regression) that can generate variable-length outputs.

Original: In the literature of detector distillation, recent work (Zhang & Ma, 2021; Shu et al., 2021; Yang et al., 2022b) mainly focuses on designing advanced distillation loss functions for transferring features from teachers to students.
Revised: In the literature of detector distillation, recent work (Zhang & Ma, 2021; Shu et al., 2021; Yang et al., 2022b) mainly focuses on designing advanced distillation loss functions for transferring features from teachers to students.

Original: However, there are two unsolved challenges: 1) The capacity gap (Cho & Hariharan, 2019; Mirzadeh et al., 2020) between models can result in a sub-optimal distilled student even if the strongest teacher has been employed, which is undesired when optimizing the accuracy-efficiency trade-off of the student.
Revised: However, there are two unsolved challenges: 1) The capacity gap (Cho & Hariharan, 2019; Mirzadeh et al., 2020) between models can result in a sub-optimal distilled student even if the strongest teacher has been employed, which is undesired when optimizing the accuracy-efficiency trade-off of the student.

Original: Moreover, when trying to distill knowledge from Transformer-based teachers (Dosovitskiy et al., 2020; Liu et al., 2021)
Revised: Moreover, when trying to distill knowledge from Transformer-based teachers (Dosovitskiy et al., 2020; Liu et al., 2021)

Original: [Figure]
Revised: [Figure]

Original: to classical convolution-based students, this architectural difference can enlarge the teacher-student gap.
Revised: to classical convolution-based students, this architectural difference can enlarge the teacher-student gap.

Original: 2) Current methods assume that one target teacher has been selected.
Revised: 2) Current methods assume that one target teacher has been selected.

Original: However, this meta-level optimization of teacher selection is neglected in the existing literature of detector distillation.
Revised: However, this meta-level optimization of teacher selection is neglected in the existing literature of detector distillation.

Original: In fact, finding a pool of strong teacher candidates is easy, but trial-and-error may be necessary before determining one most compatible teacher for a specific student.
Revised: In fact, finding a pool of strong teacher candidates is easy, but trial-and-error may be necessary before determining one most compatible teacher for a specific student.

Original: To address these challenges, we propose a framework to learn lightweight detectors through progressive knowledge distillation : 1) We find sequential distillation from multiple teachers arranged into a curriculum significantly improves knowledge distillation and bridges the teacher-student capacity gap.
Revised: To address these challenges, we propose a framework to learn lightweight detectors through progressive knowledge distillation : 1) We find sequential distillation from multiple teachers arranged into a curriculum significantly improves knowledge distillation and bridges the teacher-student capacity gap.

Original: As shown in Figure 1, even with huge architectural difference, our progressive distillation can effectively transfer knowledge from Transformer-based teachers to convolution-based students, while previous methods cannot.
Revised: As shown in Figure 1, even with huge architectural difference, our progressive distillation can effectively transfer knowledge from Transformer-based teachers to convolution-based students, while previous methods cannot.

Original: For the teacher selection problem, we design a heuristic algorithm for a given student and a pool of teacher candidates, to automatically determine the order of teachers to use.
Revised: For the teacher selection problem, we design a heuristic algorithm for a given student and a pool of teacher candidates, to automatically determine the order of teachers to use.

Original: This algorithm is based on the analysis of the representation similarity between models, which does not require knowledge of the specific distillation mechanism to be used.
Revised: This algorithm is based on the analysis of the representation similarity between models, which does not require knowledge of the specific distillation mechanism to be used.

Original: Overall, our progressive distillation is a general meta-level strategy that consistently improves both simple feature-matching distillation and more sophisticated ones.
Revised: Overall, our progressive distillation is a general meta-level strategy that consistently improves both simple feature-matching distillation and more sophisticated ones.

Original: With the help of modern distillation mechanisms and teacher detectors, our progressive distillation learns lightweight RetinaNet and Mask R-CNN students with state-of-the-art accuracy.
Revised: With the help of modern distillation mechanisms and teacher detectors, our progressive distillation learns lightweight RetinaNet and Mask R-CNN students with state-of-the-art accuracy.

Original: Furthermore, by analyzing the training loss dynamics of the student model, we find the improvement is not due to minimizing the training loss better.
Revised: Furthermore, by analyzing the training loss dynamics of the student model, we find the improvement is not due to minimizing the training loss better.

Original: Rather, the knowledge transferred from multiple teachers can lead the student to a flat minimum, and thus help the student generalize better.
Revised: Rather, the knowledge transferred from multiple teachers can lead the student to a flat minimum, and thus help the student generalize better.

Original: To summarize, we transfer knowledge from multiple teachers to progressively distill a student.
Revised: To summarize, we transfer knowledge from multiple teachers to progressively distill a student.

Original: Our main contributions include:
Revised: Our main contributions include:

Original: • We propose a framework for learning lightweight detectors through progressive knowledge distillation, which is simple, general, yet effective.
Revised: • We propose a framework for learning lightweight detectors through progressive knowledge distillation, which is simple, general, yet effective.

Original: We develop a principled way to automatically design a sequence of teachers appropriate for a given student and progressively distill it.
Revised: We develop a principled way to automatically design a sequence of teachers appropriate for a given student and progressively distill it.

Original: • Our progressive distillation is a meta-level strategy that can be easily combined with previous efforts in detection distillation.
Revised: • Our progressive distillation is a meta-level strategy that can be easily combined with previous efforts in detection distillation.

Original: We perform comprehensive empirical evaluation on the challenging MS COCO dataset and observe consistent gains.
Revised: We perform comprehensive empirical evaluation on the challenging MS COCO dataset and observe consistent gains.

Original: • For the first time, we investigate distillation from Transformer-based teacher detectors to a convolution-based student, and find progressive distillation is the key to bridge their gap.
Revised: • For the first time, we investigate distillation from Transformer-based teacher detectors to a convolution-based student, and find progressive distillation is the key to bridge their gap.

Original: • We show the performance gain comes from better generalization rather than better optimization.
Revised: • We show the performance gain comes from better generalization rather than better optimization.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Knowledge Distillation: Knowledge distillation or transfer, an idea of training a shallow student network with supervision from a deep teacher, was originally proposed by Buciluˇa et al.
Revised: Knowledge Distillation: Knowledge distillation or transfer, an idea of training a shallow student network with supervision from a deep teacher, was originally proposed by Buciluˇa et al.

Original: (2006), and later formally popularized by Hinton et al.
Revised: (2006), and later formally popularized by Hinton et al.

Original: Different knowledge can be used, such as response-based knowledge (Hinton et al., 2014), and feature-based knowledge (Romero et al., 2015; Heo et al., 2019).
Revised: Different knowledge can be used, such as response-based knowledge (Hinton et al., 2014), and feature-based knowledge (Romero et al., 2015; Heo et al., 2019).

Original: Several multi-teacher knowledge distillation methods have been proposed (Vongkulbhisal et al., 2019; Sau & Balasubramanian, 2016), which usually use the average of logits and feature representations as the knowledge (You et al., 2017; Fukuda et al., 2017).
Revised: Several multi-teacher knowledge distillation methods have been proposed (Vongkulbhisal et al., 2019; Sau & Balasubramanian, 2016), which usually use the average of logits and feature representations as the knowledge (You et al., 2017; Fukuda et al., 2017).

Original: Mirzadeh et al.
Revised: Mirzadeh et al.

Original: (2020) find that an intermediate teacher assistant, decided by architectural similarities, can bridge the gap between the student and the teacher.
Revised: (2020) find that an intermediate teacher assistant, decided by architectural similarities, can bridge the gap between the student and the teacher.

Original: We find it more effective to use a sequence of teachers instead of their ensemble, and extend Mirzadeh et al.
Revised: We find it more effective to use a sequence of teachers instead of their ensemble, and extend Mirzadeh et al.

Original: to a more general case where teacher models have diverse architectures and their relative ordering is unknown.
Revised: to a more general case where teacher models have diverse architectures and their relative ordering is unknown.

Original: Object Detection and Instance Segmentation: A variety of convolutional neural network (CNN) based object detection frameworks have been proposed, and can be generally divided into singlestage and two-stage detectors.
Revised: Object Detection and Instance Segmentation: A variety of convolutional neural network (CNN) based object detection frameworks have been proposed, and can be generally divided into singlestage and two-stage detectors.

Original: Typical single-stage methods include YOLO (Redmon et al., 2016;
Revised: Typical single-stage methods include YOLO (Redmon et al., 2016;

Original: Redmon & Farhadi, 2018) and RetinaNet (Lin et al., 2017b), and two-stage methods include Faster
Revised: Redmon & Farhadi, 2018) and RetinaNet (Lin et al., 2017b), and two-stage methods include Faster

Original: R-CNN (Ren et al., 2014), and Mask R-CNN (He et al., 2017).
Revised: R-CNN (Ren et al., 2014), and Mask R-CNN (He et al., 2017).

Original: Recently, several multi-stage models are proposed, such as HTC (Chen et al., 2019a) and DetectoRS (Qiao et al., 2021).
Revised: Recently, several multi-stage models are proposed, such as HTC (Chen et al., 2019a) and DetectoRS (Qiao et al., 2021).

Original: These detection frameworks achieve better detection accuracy with better backbone networks as feature extractors and with more complicated heads, which are more computationally expensive.
Revised: These detection frameworks achieve better detection accuracy with better backbone networks as feature extractors and with more complicated heads, which are more computationally expensive.

Original: Knowledge Distillation for detection and segmentation: To reduce the computational cost, knowledge distillation has been used to develop efficient detectors.
Revised: Knowledge Distillation for detection and segmentation: To reduce the computational cost, knowledge distillation has been used to develop efficient detectors.

Original: Chen et al.
Revised: Chen et al.

Original: (2017) first use knowledge distillation to enforce the student detector to mimic the teacher’s predictions.
Revised: (2017) first use knowledge distillation to enforce the student detector to mimic the teacher’s predictions.

Original: More recent efforts usually focus on learning from the teacher’s features, rather than final predictions.
Revised: More recent efforts usually focus on learning from the teacher’s features, rather than final predictions.

Original: Various distillation mechanisms have been proposed to leverage the impact of foreground and background objects (Wang et al., 2019; Guo et al., 2021), relation between individual objects (Zhang & Ma, 2021;
Revised: Various distillation mechanisms have been proposed to leverage the impact of foreground and background objects (Wang et al., 2019; Guo et al., 2021), relation between individual objects (Zhang & Ma, 2021;

Original: Dai et al., 2021), or relation between local and global information (Yang et al., 2022a;b).
Revised: Dai et al., 2021), or relation between local and global information (Yang et al., 2022a;b).

Original: Different from these methods that distill from a single teacher, we study distillation from multiple teachers, where a proper sequence of teachers is required, and we find a very simple feature-matching loss is adequate to significantly boost student performance.
Revised: Different from these methods that distill from a single teacher, we study distillation from multiple teachers, where a proper sequence of teachers is required, and we find a very simple feature-matching loss is adequate to significantly boost student performance.

Original:  A PPROACH
Revised:  A PPROACH

Original: We propose to progressively distill a student model S with a pool of N teachers P =
Revised: We propose to progressively distill a student model S with a pool of N teachers P =

Original: { T i } Ni =1 .
Revised: { T i } Ni =1 .

Original: Typical object detectors are composed of four modules: (1) backbone, which extracts visual features, such as ResNet (He et al., 2016) and ResNeXt (Xie et al., 2017); (2) neck, which extracts multi-level feature maps from various stages of the backbone, such as FPN (Lin et al., 2017a) and
Revised: Typical object detectors are composed of four modules: (1) backbone, which extracts visual features, such as ResNet (He et al., 2016) and ResNeXt (Xie et al., 2017); (2) neck, which extracts multi-level feature maps from various stages of the backbone, such as FPN (Lin et al., 2017a) and

Original: Bi-FPN (Tan et al., 2020); (3) optional region proposal network (RPN), which is used in two-stage detectors; and (4) head, which generates final predictions for object detection and segmentation.
Revised: Bi-FPN (Tan et al., 2020); (3) optional region proposal network (RPN), which is used in two-stage detectors; and (4) head, which generates final predictions for object detection and segmentation.

Original: We denote the output feature maps of the neck as F Net , where Net can be either the student model S or one of the teachers T i ∈ P .
Revised: We denote the output feature maps of the neck as F Net , where Net can be either the student model S or one of the teachers T i ∈ P .

Original: With neck modules like FPN, the feature maps can be multi-level.
Revised: With neck modules like FPN, the feature maps can be multi-level.

Original: We propose a meta-strategy for detector distillation that progressively learns a student using a sequence of teachers.
Revised: We propose a meta-strategy for detector distillation that progressively learns a student using a sequence of teachers.

Original: To examine this meta-strategy without involving sophisticated distillation mechanisms, we introduce a simple feature-matching distillation for a single teacher T i in Section 3.1.
Revised: To examine this meta-strategy without involving sophisticated distillation mechanisms, we introduce a simple feature-matching distillation for a single teacher T i in Section 3.1.

Original: Then we discuss progressive distillation with multiple teachers from P in Section 3.2.
Revised: Then we discuss progressive distillation with multiple teachers from P in Section 3.2.

Original: 3. S INGLE T EACHER D ISTILLATION VIA S IMPLE F EATURE M ATCHING
Revised: 3. S INGLE T EACHER D ISTILLATION VIA S IMPLE F EATURE M ATCHING

Original: In order to learn a efficient student detector S through distillation, we encourage the feature representation of a student to be similar to that of the teacher (Chen et al., 2017; Yang et al., 2021).
Revised: In order to learn a efficient student detector S through distillation, we encourage the feature representation of a student to be similar to that of the teacher (Chen et al., 2017; Yang et al., 2021).

Original: To this end, we minimize the discrepancy between the feature representations of the teacher and the student.
Revised: To this end, we minimize the discrepancy between the feature representations of the teacher and the student.

Original: Without bells and whistles, we simply minimize the L2 distance between F T i and F S :
Revised: Without bells and whistles, we simply minimize the L2 distance between F T i and F S :

Original: [Equation]
Revised: [Equation]

Original: ) where r ( · ) is a function used to match the feature map dimensions of the teacher and the student.
Revised: ) where r ( · ) is a function used to match the feature map dimensions of the teacher and the student.

Original: We define r ( · ) as follows:
Revised: We define r ( · ) as follows:

Original: • (Homogeneous case)
Revised: • (Homogeneous case)

Original: If the numbers of channels and the spatial resolutions are both the same between T i and S , r ( · ) is an identity function.
Revised: If the numbers of channels and the spatial resolutions are both the same between T i and S , r ( · ) is an identity function.

Original: • (Heterogeneous case)
Revised: • (Heterogeneous case)

Original: If the numbers of channels are different but the spatial resolutions are the same, we use 1 × 1 convolutional filters as r ( · ) .
Revised: If the numbers of channels are different but the spatial resolutions are the same, we use 1 × 1 convolutional filters as r ( · ) .

Original: If the spatial resolutions are different but the numbers of channels are the same, we use an upsampling
Revised: If the spatial resolutions are different but the numbers of channels are the same, we use an upsampling

Original: layer as r ( · ) .
Revised: layer as r ( · ) .

Original: Note that the mapping r ( · ) is only required at training time and thus not adding any overhead to the inference.
Revised: Note that the mapping r ( · ) is only required at training time and thus not adding any overhead to the inference.

Original: Overall, our loss function can be written as:
Revised: Overall, our loss function can be written as:

Original: [Equation]
Revised: [Equation]

Original: where λ is a balancing hyper-parameter and L detect is the detection loss based on the ground truth labels.
Revised: where λ is a balancing hyper-parameter and L detect is the detection loss based on the ground truth labels.

Original: Compared to state-of-the-art detection distillation approaches (Zhang & Ma, 2021; Shu et al., 2021; Yang et al., 2022a;b), which introduce more complex designs of the distillation loss, this feature-matching distillation is simpler and does not require running the heads of the teacher model.
Revised: Compared to state-of-the-art detection distillation approaches (Zhang & Ma, 2021; Shu et al., 2021; Yang et al., 2022a;b), which introduce more complex designs of the distillation loss, this feature-matching distillation is simpler and does not require running the heads of the teacher model.

Original: Our distillation loss is illustrated in Figure
Revised: Our distillation loss is illustrated in Figure

Original: 2- Left .
Revised: 2- Left .

Original: [Figure]
Revised: [Figure]

Original: L detect L distill L= λ + 36.
Revised: L detect L distill L= λ + 36.

Original: 39.9 Distillation #
Revised: 39.9 Distillation #

Original: Distillation #
Revised: Distillation #

Original: Student
Revised: Student

Original: DistilledStudent
Revised: DistilledStudent

Original: Teacher
Revised: Teacher

Original: DistilledStudent
Revised: DistilledStudent

Original: Teacher...
Revised: Teacher...

Original: S t
Revised: S t

Original: ud e n t P e r f o r m a n ce ( V a li d a t i o n
Revised: ud e n t P e r f o r m a n ce ( V a li d a t i o n

Original: A P )
Revised: A P )

Original: 3. P ROGRESSIVE D ISTILLATION WITH M ULTIPLE T EACHERS
Revised: 3. P ROGRESSIVE D ISTILLATION WITH M ULTIPLE T EACHERS

Original: The overall aim of knowledge distillation is to make a student mimic a teacher’s output, so that the student is able to obtain similar performance to teacher’s.
Revised: The overall aim of knowledge distillation is to make a student mimic a teacher’s output, so that the student is able to obtain similar performance to teacher’s.

Original: However, the capacity of the student model is limited, making it hard for the student to learn from a highly complex teacher (Cho & Hariharan, 2019).
Revised: However, the capacity of the student model is limited, making it hard for the student to learn from a highly complex teacher (Cho & Hariharan, 2019).

Original: To address this issue, multiple teacher networks are used to provide more supervision to a student (Sau & Balasubramanian, 2016;
Revised: To address this issue, multiple teacher networks are used to provide more supervision to a student (Sau & Balasubramanian, 2016;

Original: You et al., 2017).
Revised: You et al., 2017).

Original: Unlike previous methods which distill knowledge from the ensemble of logits or features simultaneously, we propose to distill featurebased knowledge from multiple teachers sequentially .
Revised: Unlike previous methods which distill knowledge from the ensemble of logits or features simultaneously, we propose to distill featurebased knowledge from multiple teachers sequentially .

Original: Our key insight is that instead of mimicking the ensemble of all feature information together, the student can be distilled more effectively by the knowledge provided by one proper teacher each time.
Revised: Our key insight is that instead of mimicking the ensemble of all feature information together, the student can be distilled more effectively by the knowledge provided by one proper teacher each time.

Original: This progressive knowledge distillation approach can be considered as designing a curriculum (Bengio et al., 2009) offered by a sequence of teachers, as illustrated in Figure
Revised: This progressive knowledge distillation approach can be considered as designing a curriculum (Bengio et al., 2009) offered by a sequence of teachers, as illustrated in Figure

Original: 2- Right .
Revised: 2- Right .

Original: The crucial question is: What is the proper order O of the teachers when distilling the student?
Revised: The crucial question is: What is the proper order O of the teachers when distilling the student?

Original: A brute-force approach might search over all orders and pick the best (that produces a distilled student with the highest validation accuracy).
Revised: A brute-force approach might search over all orders and pick the best (that produces a distilled student with the highest validation accuracy).

Original: However, the space of permutation orders grows exponentially with the number of teachers, making this impractical to scale.
Revised: However, the space of permutation orders grows exponentially with the number of teachers, making this impractical to scale.

Original: Therefore, we propose a principled and efficient approach based on a correlation analysis of each model’s learned feature representation.
Revised: Therefore, we propose a principled and efficient approach based on a correlation analysis of each model’s learned feature representation.

Original: First, we quantify the dissimilarity between each pair of models’ representations, as a proxy for the capacity gap between them.
Revised: First, we quantify the dissimilarity between each pair of models’ representations, as a proxy for the capacity gap between them.

Original: Representation (dis)similarity (Raghu et al., 2017; Wang et al., 2018;
Revised: Representation (dis)similarity (Raghu et al., 2017; Wang et al., 2018;

Original: Kornblith et al., 2019) has been studied to understand the learning capacity of neural models.
Revised: Kornblith et al., 2019) has been studied to understand the learning capacity of neural models.

Original: In our setting, we find a linear regression model is adequate for measuring the representation dissimilarity.
Revised: In our setting, we find a linear regression model is adequate for measuring the representation dissimilarity.

Original: Given two pre-trained detectors A and B, we can freeze their parameters, and thus fixing the feature representations.
Revised: Given two pre-trained detectors A and B, we can freeze their parameters, and thus fixing the feature representations.

Original: Then we can learn a linear mapping r ( · ) , implemented by a 1 × 1 convolutional layer at each feature level, as specified in the heterogeneous case in Section 3.1.
Revised: Then we can learn a linear mapping r ( · ) , implemented by a 1 × 1 convolutional layer at each feature level, as specified in the heterogeneous case in Section 3.1.

Original: r ( · ) is trained to minimize
Revised: r ( · ) is trained to minimize

Original: L distill , so it can transform A’s features to approximate B’s features.
Revised: L distill , so it can transform A’s features to approximate B’s features.

Original: After training r ( · ) , we evaluate it by L distill on the validation set, and denote the validation loss as the adaptation cost C ( A, B ) .
Revised: After training r ( · ) , we evaluate it by L distill on the validation set, and denote the validation loss as the adaptation cost C ( A, B ) .

Original: This metric can be a proxy of the capacity gap between a pair of models: When C ( A, B ) is zero, a linear mapping can transform A’s features to B’s, and there is no additional knowledge from B.
Revised: This metric can be a proxy of the capacity gap between a pair of models: When C ( A, B ) is zero, a linear mapping can transform A’s features to B’s, and there is no additional knowledge from B.

Original: When
Revised: When

Original: C ( A, B ) is large, it is more difficult to adapt A’s representation to B’s.
Revised: C ( A, B ) is large, it is more difficult to adapt A’s representation to B’s.

Original: Note that the adaptation cost is non-symmetric – it is relatively easier to adapt a high-capacity model’s representations to a low-capacity model’s representations, than the other way around.
Revised: Note that the adaptation cost is non-symmetric – it is relatively easier to adapt a high-capacity model’s representations to a low-capacity model’s representations, than the other way around.

Original: We design a heuristic algorithm to acquire a proper distillation order
Revised: We design a heuristic algorithm to acquire a proper distillation order

Original: O automatically (details are shown in Algorithm 1 in the appendix).
Revised: O automatically (details are shown in Algorithm 1 in the appendix).

Original: Suppose the maximum number of teachers to be selected is limited by k (which can be arbitrarily decided according to desired training time), and we aim to find a teacher index sequence α no longer than k .
Revised: Suppose the maximum number of teachers to be selected is limited by k (which can be arbitrarily decided according to desired training time), and we aim to find a teacher index sequence α no longer than k .

Original: We construct the teacher order backwards:
Revised: We construct the teacher order backwards:

Original: The best performing teacher is set as the final target T α k ; before the final teacher, we use another teacher, which has the smallest adaptation cost C ( · , T α k ) to that final teacher, as the penultimate teacher T α k − 1 .
Revised: The best performing teacher is set as the final target T α k ; before the final teacher, we use another teacher, which has the smallest adaptation cost C ( · , T α k ) to that final teacher, as the penultimate teacher T α k − 1 .

Original: We repeat this procedure to find preceding teachers, until: (1) when trying to select
Revised: We repeat this procedure to find preceding teachers, until: (1) when trying to select

Original: T α j , we find the transfer costs from remaining teachers to the next teacher C ( · , T α j +1 ) are all larger than the transfer cost from the student to the next teacher C ( S, T α j +1 ) ; or (2) we reach the given maximum step limit k .
Revised: T α j , we find the transfer costs from remaining teachers to the next teacher C ( · , T α j +1 ) are all larger than the transfer cost from the student to the next teacher C ( S, T α j +1 ) ; or (2) we reach the given maximum step limit k .

Original: Intuitively, the resulting sequence of teachers bridges the gap between the student model and the teacher, with an increasingly difficult curriculum.
Revised: Intuitively, the resulting sequence of teachers bridges the gap between the student model and the teacher, with an increasingly difficult curriculum.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Our algorithm for designing teacher orders is lightweight.
Revised: Our algorithm for designing teacher orders is lightweight.

Original: In fact, the main computation overhead of our algorithm is to optimize a set of tiny linear mappings ( R 256 (cid:55)→ R 256 for FPN-based detectors).
Revised: In fact, the main computation overhead of our algorithm is to optimize a set of tiny linear mappings ( R 256 (cid:55)→ R 256 for FPN-based detectors).

Original: It takes about 3 GPU hours for each student model, which is negligible compared to the distillation process that takes hundreds of GPU hours.
Revised: It takes about 3 GPU hours for each student model, which is negligible compared to the distillation process that takes hundreds of GPU hours.

Original: Since our progressive knowledge distillation is a meta-level strategy, it can be combined with previous designs of distillation mechanisms, without much efforts.
Revised: Since our progressive knowledge distillation is a meta-level strategy, it can be combined with previous designs of distillation mechanisms, without much efforts.

Original: Starting with a student detector and a pool of candidate teachers, we can first select a subset of teachers and design their distillation order.
Revised: Starting with a student detector and a pool of candidate teachers, we can first select a subset of teachers and design their distillation order.

Original: In place of the simple feature matching loss, we then apply a more advanced distillation mechanism with each teacher sequentially to train the student detector.
Revised: In place of the simple feature matching loss, we then apply a more advanced distillation mechanism with each teacher sequentially to train the student detector.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We study the efficacy of our proposed strategy, progressive knowledge distillation, from multiple perspectives.
Revised: We study the efficacy of our proposed strategy, progressive knowledge distillation, from multiple perspectives.

Original: First of all in Section 4.1, we use a controlled experiment to demonstrate that our heuristic Algorithm 1 consistently produces teacher orders that are near-optimal compared to all possibilities.
Revised: First of all in Section 4.1, we use a controlled experiment to demonstrate that our heuristic Algorithm 1 consistently produces teacher orders that are near-optimal compared to all possibilities.

Original: Then in Section 4.2 and 4.3, we apply the progressive distillation strategy along with the simple feature-matching loss (Section 3.1) to show this strategy alone brings significant gains to knowledge distillation.
Revised: Then in Section 4.2 and 4.3, we apply the progressive distillation strategy along with the simple feature-matching loss (Section 3.1) to show this strategy alone brings significant gains to knowledge distillation.

Original: Since our contribution of progressive distillation is orthogonal to previous efforts in designing distillation mechanisms, in Section 4.4 we then combine it with state-of-theart distillation mechanisms to maximize the student performance, and we show our progressive distillation is the key to the success of distillation from Transformer-based teachers to convolutionbased students.
Revised: Since our contribution of progressive distillation is orthogonal to previous efforts in designing distillation mechanisms, in Section 4.4 we then combine it with state-of-theart distillation mechanisms to maximize the student performance, and we show our progressive distillation is the key to the success of distillation from Transformer-based teachers to convolutionbased students.

Original: Finally in Section 4.5, we try to understand the performance gain of progressive distillation by analyzing the training loss dynamics.
Revised: Finally in Section 4.5, we try to understand the performance gain of progressive distillation by analyzing the training loss dynamics.

Original: Student and teacher models: To investigate the impact of different teacher models and their combinations, as shown in Table 1, we construct a variety of teacher-student pairs from a set of widely-used object detection and instance segmentation networks, including RetinaNet (Lin et al., 2017b), Mask
Revised: Student and teacher models: To investigate the impact of different teacher models and their combinations, as shown in Table 1, we construct a variety of teacher-student pairs from a set of widely-used object detection and instance segmentation networks, including RetinaNet (Lin et al., 2017b), Mask

Original: R-CNN (He et al., 2017), FCOS (Tian et al., 2019), HTC (Chen et al., 2019a), and DetectoRS (Qiao et al., 2021).
Revised: R-CNN (He et al., 2017), FCOS (Tian et al., 2019), HTC (Chen et al., 2019a), and DetectoRS (Qiao et al., 2021).

Original: They have a wide range of runtime and detection performance.
Revised: They have a wide range of runtime and detection performance.

Original: We select ResNetbackboned RetinaNet and Mask R-CNN as the student models (Student I & II), due to their low latency, simple structure, and wide application, for single-stage and two-stage object detection respectively.
Revised: We select ResNetbackboned RetinaNet and Mask R-CNN as the student models (Student I & II), due to their low latency, simple structure, and wide application, for single-stage and two-stage object detection respectively.

Original: More advanced models such as DetectoRS have better detection performance, but require much more training/inference time, so we use them as teachers.
Revised: More advanced models such as DetectoRS have better detection performance, but require much more training/inference time, so we use them as teachers.

Original: We mainly consider the RetinaNet and Mask R-CNN as the student models, and lightweight variants with a smaller backbone (Student
Revised: We mainly consider the RetinaNet and Mask R-CNN as the student models, and lightweight variants with a smaller backbone (Student

Original: III), or reduced input resolution (Student IV).
Revised: III), or reduced input resolution (Student IV).

Original: Datasets and evaluation metrics: We mainly evaluate on the challenging object detection dataset
Revised: Datasets and evaluation metrics: We mainly evaluate on the challenging object detection dataset

Original: MS COCO 2017 (Lin et al., 2014), which contains bounding boxes and instance segmentations for 80 common object categories.
Revised: MS COCO 2017 (Lin et al., 2014), which contains bounding boxes and instance segmentations for 80 common object categories.

Original: We train our models on the split of train2017 (118k images) and report results on val2017 (5k images).
Revised: We train our models on the split of train2017 (118k images) and report results on val2017 (5k images).

Original: We report the standard COCO-style Average Precision (AP) metric and end-to-end latency (from images to predictions) as the runtime.
Revised: We report the standard COCO-style Average Precision (AP) metric and end-to-end latency (from images to predictions) as the runtime.

Original: We also evaluate
Revised: We also evaluate

Original: [Table]
Revised: [Table]

Original: Suggested
Revised: Suggested

Original: Student
Revised: Student

Original: [Table]
Revised: [Table]

Original: student Ranking in teacher order
Revised: student Ranking in teacher order

Original: AP
Revised: AP

Original: AP range all orders
Revised: AP range all orders

Original: IV 36.
Revised: IV 36.

Original: III → IV 37.
Revised: III → IV 37.

Original: I → III → IV 37. I → III → IV 37.
Revised: I → III → IV 37. I → III → IV 37.

Original: [36.2, 38.2] 7 /    4 k 36.
Revised: [36.2, 38.2] 7 /    4 k 36.

Original: S t u d e n t
Revised: S t u d e n t

Original: A P
Revised: A P

Original: Our teacher orders All teacher orders on another object detection dataset Argoverse-HD (Chang et al., 2019), and a more challenging evaluation protocol streaming perception (Li et al., 2020a).
Revised: Our teacher orders All teacher orders on another object detection dataset Argoverse-HD (Chang et al., 2019), and a more challenging evaluation protocol streaming perception (Li et al., 2020a).

Original: These results are in Appendix D.
Revised: These results are in Appendix D.

Original: Baselines: Our main contribution is orthogonal to previous methods: We leverage a sequence of teachers to distill the student, instead of designing a sophisticated distillation loss to better transfer knowledge from one single teacher.
Revised: Baselines: Our main contribution is orthogonal to previous methods: We leverage a sequence of teachers to distill the student, instead of designing a sophisticated distillation loss to better transfer knowledge from one single teacher.

Original: Since we are studying a new setting where multiple teachers are available, which is missing in previous literature, we mainly focus on the absolute improvements – the performance of our distilled student models compared with the original student models and with the performance upper-bound of the teacher models.
Revised: Since we are studying a new setting where multiple teachers are available, which is missing in previous literature, we mainly focus on the absolute improvements – the performance of our distilled student models compared with the original student models and with the performance upper-bound of the teacher models.

Original: We find using a sequence of teachers, instead of their ensemble, is more effective.
Revised: We find using a sequence of teachers, instead of their ensemble, is more effective.

Original: Due to limited space, we leave this comparison in Appendix B&C.
Revised: Due to limited space, we leave this comparison in Appendix B&C.

Original: 4. S EARCHING FOR THE N EAR -O PTIMAL T EACHER O RDER
Revised: 4. S EARCHING FOR THE N EAR -O PTIMAL T EACHER O RDER

Original: As we have discussed in Section 3.2, finding the optimal order of teachers for the progressive knowledge distillation takes factorial time complexity.
Revised: As we have discussed in Section 3.2, finding the optimal order of teachers for the progressive knowledge distillation takes factorial time complexity.

Original: To acquire a near-optimal teacher order, we propose the heuristic Algorithm 1.
Revised: To acquire a near-optimal teacher order, we propose the heuristic Algorithm 1.

Original: In this section, we will validate that this algorithm is near-optimal.
Revised: In this section, we will validate that this algorithm is near-optimal.

Original: To achieve this comprehensive comparison, we distill Student I with all orders of teachers from the pool Teacher I-IV.
Revised: To achieve this comprehensive comparison, we distill Student I with all orders of teachers from the pool Teacher I-IV.

Original: We use a reduced training budget: For each teacher, we only train the student for 3 epochs on MS COCO.
Revised: We use a reduced training budget: For each teacher, we only train the student for 3 epochs on MS COCO.

Original: We use the linear learning rate schedule, which has been shown comparably effective in a limited budget setting by Li et al.
Revised: We use the linear learning rate schedule, which has been shown comparably effective in a limited budget setting by Li et al.

Original: (2020b).
Revised: (2020b).

Original: We first measure the adaptation costs among the student and teacher models.
Revised: We first measure the adaptation costs among the student and teacher models.

Original: A visualization of the cost graph is shown in Figure 3.
Revised: A visualization of the cost graph is shown in Figure 3.

Original: Following Algorithm 1, we can construct a sequence of teachers.
Revised: Following Algorithm 1, we can construct a sequence of teachers.

Original: We compare the teacher orders given by our proposed algorithm against all other orders, via the performance of the distilled student’s performance.
Revised: We compare the teacher orders given by our proposed algorithm against all other orders, via the performance of the distilled student’s performance.

Original: As shown in Table 2, teacher orders suggested by Algorithm 1 are consistently near-optimal in this setting.
Revised: As shown in Table 2, teacher orders suggested by Algorithm 1 are consistently near-optimal in this setting.

Original: In the following sections, we will use order provided by Algorithm 1, without brute-force iterating over all possible orders.
Revised: In the following sections, we will use order provided by Algorithm 1, without brute-force iterating over all possible orders.

Original: One may question that the greedy path selection shown in Figure 3 is be inferior to a global optimization algorithm.
Revised: One may question that the greedy path selection shown in Figure 3 is be inferior to a global optimization algorithm.

Original: However, we find the later teachers impact the student performance more profoundly, so we need to greedily select teachers from the sequence tail.
Revised: However, we find the later teachers impact the student performance more profoundly, so we need to greedily select teachers from the sequence tail.

Original: More details and comparison with other heuristics are provided in Appendix A.
Revised: More details and comparison with other heuristics are provided in Appendix A.

Original: We start by distilling RetinaNet and Mask R-CNN with a ResNet-50 backbone (Student I & II).
Revised: We start by distilling RetinaNet and Mask R-CNN with a ResNet-50 backbone (Student I & II).

Original: Here we consider homogeneous teachers where the numbers of channels and the spatial resolutions of feature maps are consistent between the student and teacher.
Revised: Here we consider homogeneous teachers where the numbers of channels and the spatial resolutions of feature maps are consistent between the student and teacher.

Original: For the RetinaNet student, we still consider the pool of Teacher I-IV, the same as Section 4.1.
Revised: For the RetinaNet student, we still consider the pool of Teacher I-IV, the same as Section 4.1.

Original: For the Mask R-CNN student, we should no longer use Teacher I (the student itself) or Teacher II (the single-stage teacher does not outperform the student by a large margin).
Revised: For the Mask R-CNN student, we should no longer use Teacher I (the student itself) or Teacher II (the single-stage teacher does not outperform the student by a large margin).

Original: To compensate for that, we include Teacher V, which can be considered as a hybrid model of DetectoRS backbone/neck and Mask R-CNN head.
Revised: To compensate for that, we include Teacher V, which can be considered as a hybrid model of DetectoRS backbone/neck and Mask R-CNN head.

Original: Thus, the teacher pool for Mask R-CNN includes Teacher III-V. To control the total training time, we limit the number of teachers to be 2 .
Revised: Thus, the teacher pool for Mask R-CNN includes Teacher III-V. To control the total training time, we limit the number of teachers to be 2 .

Original: Thus, we initialize from an off-the-shelf (‘OTS’) student, and sequentially distill it with 2 teachers, in total 24 epochs (equivalent to a 2 × training schedule).
Revised: Thus, we initialize from an off-the-shelf (‘OTS’) student, and sequentially distill it with 2 teachers, in total 24 epochs (equivalent to a 2 × training schedule).

Original: Besides the OTS students, we also compare with two other baselines: 1) students trained with a longer 3 × training schedule, and 2) students directly distilled by the final target teacher, using a 2 × training schedule.
Revised: Besides the OTS students, we also compare with two other baselines: 1) students trained with a longer 3 × training schedule, and 2) students directly distilled by the final target teacher, using a 2 × training schedule.

Original: More architectural details are listed in Table 1.
Revised: More architectural details are listed in Table 1.

Original: Following Section 4.1, we use Algorithm 1 to determine the sequence of teachers to use for each student.
Revised: Following Section 4.1, we use Algorithm 1 to determine the sequence of teachers to use for each student.

Original: For RetinaNet student, our algorithm suggests teacher sequence III → IV.
Revised: For RetinaNet student, our algorithm suggests teacher sequence III → IV.

Original: For Mask R-
Revised: For Mask R-

Original: [Table]
Revised: [Table]

Original: CNN student, our algorithm suggests teacher sequence V → IV.
Revised: CNN student, our algorithm suggests teacher sequence V → IV.

Original: Table 3 shows the results on COCO.
Revised: Table 3 shows the results on COCO.

Original: Additional results, analysis, and ablation studies of Mask R-CNN distillation are in Appendix B.
Revised: Additional results, analysis, and ablation studies of Mask R-CNN distillation are in Appendix B.

Original: 4. D ISTILLATION WITH H OMOGENEOUS T EACHERS
Revised: 4. D ISTILLATION WITH H OMOGENEOUS T EACHERS

Original: Overall performance:
Revised: Overall performance:

Original: Our distilled student models (row 4&8) significantly improves over the ‘OTS’ students (row 1&5).
Revised: Our distilled student models (row 4&8) significantly improves over the ‘OTS’ students (row 1&5).

Original: The box AP of RetinaNet is improved from 36 .
Revised: The box AP of RetinaNet is improved from 36 .

Original: 5% to 39 .
Revised: 5% to 39 .

Original: 9% ( +3 .
Revised: 9% ( +3 .

Original: The box AP of Mask R-CNN is improved from 38 .
Revised: The box AP of Mask R-CNN is improved from 38 .

Original: 2% to 41 .
Revised: 2% to 41 .

Original: 4% ( +3 .
Revised: 4% ( +3 .

Original: 2% ) and the mask AP of Mask
Revised: 2% ) and the mask AP of Mask

Original: R-CNN is improved from 34 .
Revised: R-CNN is improved from 34 .

Original: 7% to 37 .
Revised: 7% to 37 .

Original: 3% ( +2 .
Revised: 3% ( +2 .

Original: After progressive distillation, our resulting
Revised: After progressive distillation, our resulting

Original: Mask R-CNN detector has comparable performance with HTC teacher, but much less runtime (51ms vs. 181ms).
Revised: Mask R-CNN detector has comparable performance with HTC teacher, but much less runtime (51ms vs. 181ms).

Original: Comparison with baselines:
Revised: Comparison with baselines:

Original: First, the performance gain is not merely from a longer training schedule.
Revised: First, the performance gain is not merely from a longer training schedule.

Original: Our distilled student models (row 4&8) consistently outperform original students trained with a 3 × schedule (row 2&6).
Revised: Our distilled student models (row 4&8) consistently outperform original students trained with a 3 × schedule (row 2&6).

Original: Second, progressive distillation using a curriculum of teachers (row 4&8) is better than direct distillation from a strong teacher (row 3&7), even if the total training time is the same.
Revised: Second, progressive distillation using a curriculum of teachers (row 4&8) is better than direct distillation from a strong teacher (row 3&7), even if the total training time is the same.

Original: It is worth noting that our detection performance for large objects receives the most gain (about 6% AP L improvement for both models).
Revised: It is worth noting that our detection performance for large objects receives the most gain (about 6% AP L improvement for both models).

Original: The reason why we emphasize AP L is that, in an efficiency-centric real-world application (e.g. autonomous driving, robot navigation), detecting nearby larger objects is more crucial than others.
Revised: The reason why we emphasize AP L is that, in an efficiency-centric real-world application (e.g. autonomous driving, robot navigation), detecting nearby larger objects is more crucial than others.

Original: From a realistic perspective, better AP L shows better applicability of our approach.
Revised: From a realistic perspective, better AP L shows better applicability of our approach.

Original: 4. D ISTILLATION WITH H ETEROGENEOUS T EACHERS
Revised: 4. D ISTILLATION WITH H ETEROGENEOUS T EACHERS

Original: To validate our progressive distillation approach is general, we now consider a more challenging heterogeneous scenario, where students and teachers have different backbones or input resolutions.
Revised: To validate our progressive distillation approach is general, we now consider a more challenging heterogeneous scenario, where students and teachers have different backbones or input resolutions.

Original: Specifically, Student III, a ResNet-18 Mask R-CNN, is distilled with ResNet-50 teachers; Student
Revised: Specifically, Student III, a ResNet-18 Mask R-CNN, is distilled with ResNet-50 teachers; Student

Original: IV, a model with reduced input resolution, is distilled with teachers trained with larger input resolutions.
Revised: IV, a model with reduced input resolution, is distilled with teachers trained with larger input resolutions.

Original: The results are summarized in Table 4, and additional results are included in Appendix C.
Revised: The results are summarized in Table 4, and additional results are included in Appendix C.

Original: Heterogeneous backbones:
Revised: Heterogeneous backbones:

Original: Student III has a ResNet-18 backbone and about half runtime as its
Revised: Student III has a ResNet-18 backbone and about half runtime as its

Original: ResNet-50 counterpart (Teacher I).
Revised: ResNet-50 counterpart (Teacher I).

Original: We find the proper distillation scheme for Student III is to use the sequence of Teacher I
Revised: We find the proper distillation scheme for Student III is to use the sequence of Teacher I

Original: → V → IV, which significantly improves Student III over the ‘OTS’ model.
Revised: → V → IV, which significantly improves Student III over the ‘OTS’ model.

Original: The box AP of Student III is improved from 33 . 3% to 37 .
Revised: The box AP of Student III is improved from 33 . 3% to 37 .

Original: 0% ( +3 .
Revised: 0% ( +3 .

Original: 7% ), and especially for large objects, AP L is improved from 43 .
Revised: 7% ), and especially for large objects, AP L is improved from 43 .

Original: 6% to 50 .
Revised: 6% to 50 .

Original: 0% ( +6 . 4% ).
Revised: 0% ( +6 . 4% ).

Original: Heterogeneous input resolutions: Although inputs with varying resolutions can be fed into most object detectors without changing the architecture, the performance often degenerates when there is a resolution mismatch between training and evaluation (Tan et al., 2020; Li et al., 2020a).
Revised: Heterogeneous input resolutions: Although inputs with varying resolutions can be fed into most object detectors without changing the architecture, the performance often degenerates when there is a resolution mismatch between training and evaluation (Tan et al., 2020; Li et al., 2020a).

Original: If ultimately we want to apply a detector to low-resolution inputs for fast inference, it is better to use low-resolution inputs during training.
Revised: If ultimately we want to apply a detector to low-resolution inputs for fast inference, it is better to use low-resolution inputs during training.

Original: On the other hand, we conjecture that teachers with highresolution inputs may provide finer details that can assist the student.
Revised: On the other hand, we conjecture that teachers with highresolution inputs may provide finer details that can assist the student.

Original: With our progressive distillation approach, we investigate the improvement of a low-resolution student distilled by a sequence of teachers with high-resolution inputs.
Revised: With our progressive distillation approach, we investigate the improvement of a low-resolution student distilled by a sequence of teachers with high-resolution inputs.

Original: We denote the standard input resolution 1333 × 800 as 1 × , and a reduced resolution 333 × 200 as 0 .
Revised: We denote the standard input resolution 1333 × 800 as 1 × , and a reduced resolution 333 × 200 as 0 .

Original: 25 × .
Revised: 25 × .

Original: We distill Student IV (with 0 .
Revised: We distill Student IV (with 0 .

Original: 25 × resolution) by a sequence of Teacher I variants ( 0 .
Revised: 25 × resolution) by a sequence of Teacher I variants ( 0 .

Original: 5 × → 0 .
Revised: 5 × → 0 .

Original: 75 × → 1 × ) .
Revised: 75 × → 1 × ) .

Original: From Table 4, we can see substantial improvement brought by progressive knowledge distillation: the box AP is improved from 25 . 8% to 31 .
Revised: From Table 4, we can see substantial improvement brought by progressive knowledge distillation: the box AP is improved from 25 . 8% to 31 .

Original: 5% ( +5 .
Revised: 5% ( +5 .

Original: and the mask AP is improved from 23 .
Revised: and the mask AP is improved from 23 .

Original: 0% to 28 .
Revised: 0% to 28 .

Original: 2% ( +5 .
Revised: 2% ( +5 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: students.
Revised: students.

Original: Due to the architectural difference pacity directly distilling with Swin-S backbone does not yield better students than convolution-based teachers in Figure 4.
Revised: Due to the architectural difference pacity directly distilling with Swin-S backbone does not yield better students than convolution-based teachers in Figure 4.

Original: An intermediate Swin-T teacher and progressive distilla- tion solve this issue without increasing training time
Revised: An intermediate Swin-T teacher and progressive distilla- tion solve this issue without increasing training time

Original: Compared to off-the-shelf models, our RetinaNet and Mask R-CNN students improve by 5 .
Revised: Compared to off-the-shelf models, our RetinaNet and Mask R-CNN students improve by 5 .

Original: 5 % AP and 4 .
Revised: 5 % AP and 4 .

Original: 3 % box AP, respectively.
Revised: 3 % box AP, respectively.

Original: 4. C OMBINATION WITH S TATE - OF - THE -A RT D ISTILLATION M ECHANISMS
Revised: 4. C OMBINATION WITH S TATE - OF - THE -A RT D ISTILLATION M ECHANISMS

Original: Our meta-level strategy of using a sequence of teachers to progressively distill a student is independent of choice of the distillation mechanism for each teacher.
Revised: Our meta-level strategy of using a sequence of teachers to progressively distill a student is independent of choice of the distillation mechanism for each teacher.

Original: We have shown progressive distillation can boost the simple distillation based on feature matching above, and in this section, we will combine progressive distillation with state-of-the-art distillation mechanisms for object detection to further improve student accuracy.
Revised: We have shown progressive distillation can boost the simple distillation based on feature matching above, and in this section, we will combine progressive distillation with state-of-the-art distillation mechanisms for object detection to further improve student accuracy.

Original: Distillation protocol: We evaluate our progressive distillation with three most recent works on detector distillation: CWD (Shu et al., 2021), FGD (Yang et al., 2022a), and MGD (Yang et al., 2022b).
Revised: Distillation protocol: We evaluate our progressive distillation with three most recent works on detector distillation: CWD (Shu et al., 2021), FGD (Yang et al., 2022a), and MGD (Yang et al., 2022b).

Original: To ensure fair comparison, we use the same teacher-student pairs as them: RetinaNet/ResNetand RetinaNet/ResNeXt-101 are the single-stage student and final teacher.
Revised: To ensure fair comparison, we use the same teacher-student pairs as them: RetinaNet/ResNetand RetinaNet/ResNeXt-101 are the single-stage student and final teacher.

Original: Mask R-CNN/ResNetand Cascade Mask R-CNN/ResNeXt-101-DCN are the two-stage student and final teacher.
Revised: Mask R-CNN/ResNetand Cascade Mask R-CNN/ResNeXt-101-DCN are the two-stage student and final teacher.

Original: Between them, we insert one medium-capacity teacher to progressively distill the student: RetinaNet/ResNet101 for single-stage and Cascade Mask R-CNN/ResNet50-DCN for two-stage.
Revised: Between them, we insert one medium-capacity teacher to progressively distill the student: RetinaNet/ResNet101 for single-stage and Cascade Mask R-CNN/ResNet50-DCN for two-stage.

Original: Also for fairness, we keep the total training epochs the same .
Revised: Also for fairness, we keep the total training epochs the same .

Original: We set “ 1 × ” training schedule for each teacher, so that the total training time is equivalent to “ 2 × ”, the same as in previous works.
Revised: We set “ 1 × ” training schedule for each teacher, so that the total training time is equivalent to “ 2 × ”, the same as in previous works.

Original: Figure 4 shows our progressive distillation strategy consistently improves students’ final accuracy.
Revised: Figure 4 shows our progressive distillation strategy consistently improves students’ final accuracy.

Original: For example, the performance of FGD-distilled RetinaNet/ResNet-50 improves from 40 .
Revised: For example, the performance of FGD-distilled RetinaNet/ResNet-50 improves from 40 .

Original: 7% to 41 .
Revised: 7% to 41 .

Original: 5% AP ( +0 . 8% ), and this gain is larger than mechanism advance from FGD to MGD ( +0 . 3% ).
Revised: 5% AP ( +0 . 8% ), and this gain is larger than mechanism advance from FGD to MGD ( +0 . 3% ).

Original: We bring performance gains to state-of-the-art detection distillation almost for free .
Revised: We bring performance gains to state-of-the-art detection distillation almost for free .

Original: Next, we investigate how to further maximize the student performance.
Revised: Next, we investigate how to further maximize the student performance.

Original: Swin Transformer (Liu et al., 2021) can act as an even stronger teacher than the convolution-based teachers used in previous works.
Revised: Swin Transformer (Liu et al., 2021) can act as an even stronger teacher than the convolution-based teachers used in previous works.

Original: However compared to convolution-based teachers, directly distill from such a teacher cannot improve the student performance, even if we use the state-of-the-art method MGD.
Revised: However compared to convolution-based teachers, directly distill from such a teacher cannot improve the student performance, even if we use the state-of-the-art method MGD.

Original: For example,
Revised: For example,

Original: RetinaNet/Swin-Small ( 47 .
Revised: RetinaNet/Swin-Small ( 47 .

Original: 1% AP) is much stronger than RetinaNet/ResNeXt-101( 41 . 6% AP), but direct distillation from both yields the same student performance ( 41 . 0% AP).
Revised: 1% AP) is much stronger than RetinaNet/ResNeXt-101( 41 . 6% AP), but direct distillation from both yields the same student performance ( 41 . 0% AP).

Original: To bridge the architectural difference and capacity gap between the ResNet-50 student and Swin-Small teacher, we can utilize an intermediate Swin-Tiny teacher.
Revised: To bridge the architectural difference and capacity gap between the ResNet-50 student and Swin-Small teacher, we can utilize an intermediate Swin-Tiny teacher.

Original: As shown in Table 5, this progressive approach brings the best students: the performance of ResNet-50 based RetinaNet increases to 42 .
Revised: As shown in Table 5, this progressive approach brings the best students: the performance of ResNet-50 based RetinaNet increases to 42 .

Original: 0% AP and Mask
Revised: 0% AP and Mask

Original: R-CNN increases to 42 .
Revised: R-CNN increases to 42 .

Original: 5% AP.
Revised: 5% AP.

Original: 4. U NPACKING THE P ERFORMANCE G AIN : G ENERALIZATION OR O PTIMIZATION ?
Revised: 4. U NPACKING THE P ERFORMANCE G AIN : G ENERALIZATION OR O PTIMIZATION ?

Original: We have shown that our distilled student significantly improves the accuracy on the validation data over the off-the-shelf student.
Revised: We have shown that our distilled student significantly improves the accuracy on the validation data over the off-the-shelf student.

Original: As further demonstrated in Figure 5a, the validation accuracy of the distilled student gradually increases during distillation, and achieves a higher value compared with the student trained without teachers.
Revised: As further demonstrated in Figure 5a, the validation accuracy of the distilled student gradually increases during distillation, and achieves a higher value compared with the student trained without teachers.

Original: A natural question then arises – why is distillation helping?
Revised: A natural question then arises – why is distillation helping?

Original: There are two possible hypotheses: (1) improved optimization: distillation facilitates the optimization procedure, leading to a better local minimum, and (2) improved generalization: the distillation process helps the student generalize to unseen data.
Revised: There are two possible hypotheses: (1) improved optimization: distillation facilitates the optimization procedure, leading to a better local minimum, and (2) improved generalization: the distillation process helps the student generalize to unseen data.

Original: Improved optimization is typically manifested through a better model, a lower training loss and a higher validation accuracy, which is exactly the case for Mask R-CNN, HTC and DetectoRS.
Revised: Improved optimization is typically manifested through a better model, a lower training loss and a higher validation accuracy, which is exactly the case for Mask R-CNN, HTC and DetectoRS.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We compare the validation AP (Figure 5a) and the training detection loss L detect (Figure 5b) of the two students during the training process.
Revised: We compare the validation AP (Figure 5a) and the training detection loss L detect (Figure 5b) of the two students during the training process.

Original: Despite its worse training loss, the distilled student can generalize better on the validation set.
Revised: Despite its worse training loss, the distilled student can generalize better on the validation set.

Original: We also compare the loss landscapes (Li et al., 2018) of the original student (Figure 5c) and the distilled student (Figure 5d)
Revised: We also compare the loss landscapes (Li et al., 2018) of the original student (Figure 5c) and the distilled student (Figure 5d)

Original: Distillation can guide the student to converge to a flatter local minimum.
Revised: Distillation can guide the student to converge to a flatter local minimum.

Original: These observations suggest distillation helps generalization rather than optimization .
Revised: These observations suggest distillation helps generalization rather than optimization .

Original: Consequently, one might think that distillation works in the same way.
Revised: Consequently, one might think that distillation works in the same way.

Original: However, our investigation suggests the opposite – our progressive distillation increases both the validation accuracy and the training loss, and therefore effectively reduces the generalization gap.
Revised: However, our investigation suggests the opposite – our progressive distillation increases both the validation accuracy and the training loss, and therefore effectively reduces the generalization gap.

Original: In Figure 5, we compare the original RetinaNet model and the distilled student, which have the same architecture, the same latency and are trained on the same data, but with different supervision (only ground-truth labels vs. additional knowledge distillation).
Revised: In Figure 5, we compare the original RetinaNet model and the distilled student, which have the same architecture, the same latency and are trained on the same data, but with different supervision (only ground-truth labels vs. additional knowledge distillation).

Original: To eliminate the influence of learning rate changes, we train the original student with a 3 × schedule and restart the learning rate at the same time with the distilled student.
Revised: To eliminate the influence of learning rate changes, we train the original student with a 3 × schedule and restart the learning rate at the same time with the distilled student.

Original: Interestingly, although distillation can improve the student’s validation performance, the training detection loss of the distilled student is higher than the original student.
Revised: Interestingly, although distillation can improve the student’s validation performance, the training detection loss of the distilled student is higher than the original student.

Original: This suggests that distillation does not help the optimization process to find a local minimum with a lower training loss, but rather strengthen the generalizability of the student model.
Revised: This suggests that distillation does not help the optimization process to find a local minimum with a lower training loss, but rather strengthen the generalizability of the student model.

Original: To further support this observation, we also visualize the local loss landscape (Li et al., 2018).
Revised: To further support this observation, we also visualize the local loss landscape (Li et al., 2018).

Original: The distilled student has a flatter loss landscape (Figure 5d) compared to the original one (Figure 5c).
Revised: The distilled student has a flatter loss landscape (Figure 5d) compared to the original one (Figure 5c).

Original: As widely believed in the machine learning literature, flat minima lead to better generalization (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017).
Revised: As widely believed in the machine learning literature, flat minima lead to better generalization (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017).

Original: The observation shown in Figure 5 is illustrated for
Revised: The observation shown in Figure 5 is illustrated for

Original: RetinaNet, but we also have similar observation in other students.
Revised: RetinaNet, but we also have similar observation in other students.

Original: As a conclusion, knowledge distillation, which enforces the student to mimic the teachers’ features, can be considered as an implicit regularization, and helps the student combat overfitting and achieve better generalization.
Revised: As a conclusion, knowledge distillation, which enforces the student to mimic the teachers’ features, can be considered as an implicit regularization, and helps the student combat overfitting and achieve better generalization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a simple yet effective approach to knowledge distillation, which progressively transfers the knowledge of a sequence of teachers to learn a lightweight object detector.
Revised: We present a simple yet effective approach to knowledge distillation, which progressively transfers the knowledge of a sequence of teachers to learn a lightweight object detector.

Original: Our approach automatically arranges multiple teachers into a curriculum, and thus effectively mitigating the capacity gap between the teacher and student.
Revised: Our approach automatically arranges multiple teachers into a curriculum, and thus effectively mitigating the capacity gap between the teacher and student.

Original: We successfully distill knowledge from Transformer-based teachers to convolution-based students, and achieve state-of-the-art performance on the challenging
Revised: We successfully distill knowledge from Transformer-based teachers to convolution-based students, and achieve state-of-the-art performance on the challenging

Original: COCO dataset.
Revised: COCO dataset.

Original: Our analysis also finds distillation improves generalization rather than optimization.
Revised: Our analysis also finds distillation improves generalization rather than optimization.

Original: AA PPENDIX
Revised: A PPENDIX

Original: We summarize the content of this supplementary document as follows.
Revised: We summarize the content of this supplementary document as follows.

Original: Section A includes additional results and analysis of our proposed algorithm for teacher order selection.
Revised: Section A includes additional results and analysis of our proposed algorithm for teacher order selection.

Original: Sections B and C provide additional ablation study on distillation with homogeneous teachers and heterogeneous teachers, respectively.
Revised: Sections B and C provide additional ablation study on distillation with homogeneous teachers and heterogeneous teachers, respectively.

Original: Section D shows the generalizability of our approach, which demonstrate the experimental results of the Argoverse-HD dataset with streaming accuracy metric.
Revised: Section D shows the generalizability of our approach, which demonstrate the experimental results of the Argoverse-HD dataset with streaming accuracy metric.

Revised: Section E compares our work with prior knowledge distillation methods in detail.

Revised: Section F provides some additional experiments on combining our method with state-of-the-art distillation mechanism.

Original: Section E lists our implementation details.
Revised: Section G lists our implementation details.

Original: A M ORE R ESULTS ON S EARCHING FOR THE N EAR -O PTIMAL T EACHER O RDER
Revised: A M ORE R ESULTS ON S EARCHING FOR THE N EAR -O PTIMAL T EACHER O RDER

Original: In this section, we show more detailed results about searching a proper teacher order for progressive knowledge distillation, and validate the approach we propose in the main paper.
Revised: In this section, we show more detailed results about searching a proper teacher order for progressive knowledge distillation, and validate the approach we propose in the main paper.

Original: As described in
Revised: As described in

Original: Section 3.2, we first quantify the adaptation cost C ( · , · ) between every pair of models in our pool, and then use a heuristic method (Algorithm 1) to construct a sequence of teachers.
Revised: Section 3.2, we first quantify the adaptation cost C ( · , · ) between every pair of models in our pool, and then use a heuristic method (Algorithm 1) to construct a sequence of teachers.

Original: We have shown that the teacher order suggested by our algorithm is highly competitive in Table 2.
Revised: We have shown that the teacher order suggested by our algorithm is highly competitive in Table 2.

Original: One might think there should be better choices than a greedy algorithm on a directed graph, such as a shortest-path algorithm.
Revised: One might think there should be better choices than a greedy algorithm on a directed graph, such as a shortest-path algorithm.

Original: To validate our algorithm design, we compare our Algorithm 1 against several other algorithms.
Revised: To validate our algorithm design, we compare our Algorithm 1 against several other algorithms.

Original: Algorithm 1: Determining the Teacher Order
Revised: Algorithm 1: Determining the Teacher Order

Original: Input: Student model S , pool of teacher models P
Revised: Input: Student model S , pool of teacher models P

Original: =
Revised: =

Original: { T i } Ni =1 , teacher models’ performance { Q ( T i ) }
Revised: { T i } Ni =1 , teacher models’ performance { Q ( T i ) }

Original: Ni =1 , maximum number of selected teachers k
Revised: Ni =1 , maximum number of selected teachers k

Original: Output: Sequence of teachers O , len ( O ) ≤ k 1 Pick the best performing teacher: T α k ← arg max T u ∈P Q ( T u ) , O ← [ T α k ] 2 Exclude from pool: P ← P \ { T α k } 3 for j ← k − 1 to 1 do
Revised: Output: Sequence of teachers O , len ( O ) ≤ k 1 Pick the best performing teacher: T α k ← arg max T u ∈P Q ( T u ) , O ← [ T α k ] 2 Exclude from pool: P ← P \ { T α k } 3 for j ← k − 1 to 1 do

Original: Get candidate sub-pool: P j = { T u | T u ∈ P , C ( T u , T α j +1 ) <
Revised: Get candidate sub-pool: P j = { T u | T u ∈ P , C ( T u , T α j +1 ) <

Original: C ( S, T α j +1 ) } if P j ̸ = ∅ then
Revised: C ( S, T α j +1 ) } if P j ̸ = ∅ then

Original: Pick the teacher closest to T α j +1 : T α j ← arg min T u ∈P j C ( T u , T α j +

Original: To begin with, we include the detailed adaptation costs C ( · , · ) among RetinaNet (Student I) and its teachers (Teacher I-IV) in Table 6.
Revised: To begin with, we include the detailed adaptation costs C ( · , · ) among RetinaNet (Student I) and its teachers (Teacher I-IV) in Table 6.

Original: As described in Section 4.1, we have distilled Student I with all possible teacher orders in the pool, using a reduced training budget of 3 epochs for each teacher.
Revised: As described in Section 4.1, we have distilled Student I with all possible teacher orders in the pool, using a reduced training budget of 3 epochs for each teacher.

Original: The results of these mini-budget distillation are summarized in Table 7.
Revised: The results of these mini-budget distillation are summarized in Table 7.

Original: From
Revised: From

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: II 
Revised: II → I

Original: [Table]
Revised: [Table]

Original: IV 
Revised: IV 37.

Revised: 7 / 64 Forward

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: I

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Forward
Revised: Forward

Original: [Table]
Revised: [Table]

Original: [Table]

Revised: IV 37.

Original: [Table]
Revised: [Table]

Original: Our
Revised: Our

Original: [Table]
Revised: [Table]

Original: III
Revised: III

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Our
Revised: Our

Original: [Table]
Revised: [Table]

Original: I → III [Table] IV 
Revised: I → III → IV 37.

Original: [Table] Given the adaptation costs in Table 6, we can construct a directed graph, part of which has been illustrated in Figure 3.
Revised: Given the adaptation costs in Table 6, we can construct a directed graph, part of which has been illustrated in Figure 3.

Original: On the directed graph, we can run several algorithms to select a path.
Revised: On the directed graph, we can run several algorithms to select a path.

Original: Besides our Algorithm 1, one may also propose these algorithms:
Revised: Besides our Algorithm 1, one may also propose these algorithms:

Original: • Shortest-path (sum) : Set the student as the source node, and set the best performing teacher as the target node T λ k .
Revised: • Shortest-path (sum) : Set the student as the source node, and set the best performing teacher as the target node T λ k .

Original: Find a path S
Revised: Find a path S

Original: → T λ 1 → · · · → T λ k that minimizes the sum of adaptation costs along the path: min T λ 1 ,...,T λk − 1 C ( S, T λ 1 )
Revised: → T λ 1 → · · · → T λ k that minimizes the sum of adaptation costs along the path: min T λ 1 ,...,T λk − 1 C ( S, T λ 1 )

Original: + (cid:80) k − 1 j =1 C
Revised: + (cid:80) k − 1 j =1 C

Original: ( T λ j , T λ j +1 ) .
Revised: ( T λ j , T λ j +1 ) .

Original: • Shortest-path (max) : Set the student as the source node, and set the best performing teacher as the target node T λ k .
Revised: • Shortest-path (max) : Set the student as the source node, and set the best performing teacher as the target node T λ k .

Original: Find a path S
Revised: Find a path S

Original: → T λ 1 → · · · → T λ k that minimizes the maximum of adaptation costs along the path: min T λ 1 ,...,T λk − 1 max {C ( S, T λ 1 ) , C ( T λ 1 , T λ 2 ) , . .
Revised: → T λ 1 → · · · → T λ k that minimizes the maximum of adaptation costs along the path: min T λ 1 ,...,T λk − 1 max {C ( S, T λ 1 ) , C ( T λ 1 , T λ 2 ) , . .

Original: , C ( T λ k − 1 , T λ k ) } .
Revised: , C ( T λ k − 1 , T λ k ) } .

Original: • Forward construction : Contrary to Algorithm 1, we may start from the student and choose the nearest teacher from the current one, to construct the sequence: T λ 1 ← arg min T u ∈P C ( S, T u ) , T λ j +1 ← arg min T u ∈P C ( T λ j , T u ) .
Revised: • Forward construction : Contrary to Algorithm 1, we may start from the student and choose the nearest teacher from the current one, to construct the sequence: T λ 1 ← arg min T u ∈P C ( S, T u ) , T λ j +1 ← arg min T u ∈P C ( T λ j , T u ) .

Original: The output teacher sequences and corresponding student performance of these three algorithms are summarized in Table 8.
Revised: The output teacher sequences and corresponding student performance of these three algorithms are summarized in Table 8.

Original: In this setting, our Algorithm 1 can consistently produce a competitive teacher order that leads to a good performance of the distilled student.
Revised: In this setting, our Algorithm 1 can consistently produce a competitive teacher order that leads to a good performance of the distilled student.

Original: Compared to our Algorithm 1, shortest-path (max) can achieve a similar performance, and it is only worse than ours when k = 2 .
Revised: Compared to our Algorithm 1, shortest-path (max) can achieve a similar performance, and it is only worse than ours when k = 2 .

Original: Forward construction performs worst among the four algorithms.
Revised: Forward construction performs worst among the four algorithms.

Original: In summary, a greedy backward construction like Algorithm 1 works the best in our setting, rather than globally optimized shortest-path algorithms.
Revised: In summary, a greedy backward construction like Algorithm 1 works the best in our setting, rather than globally optimized shortest-path algorithms.

Original: The final target teacher has the most profound impact on the distilled student’s performance.
Revised: The final target teacher has the most profound impact on the distilled student’s performance.

Original: In order to fully assist the final teacher, we need to use another teacher with the minimal adaptation cost to the final teacher before it, which is exactly the behavior of Algorithm
Revised: In order to fully assist the final teacher, we need to use another teacher with the minimal adaptation cost to the final teacher before it, which is exactly the behavior of Algorithm

Original: [Table]
Revised: [Table]

Original: B A BLATION S TUDY ON D ISTILLATION WITH H OMOGENEOUS T EACHERS
Revised: B A BLATION S TUDY ON D ISTILLATION WITH H OMOGENEOUS T EACHERS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide more details about distillation with homogeneous teachers (Section 4.2).
Revised: we provide more details about distillation with homogeneous teachers (Section 4.2).

Original: We investigate (1) the impact of each individual teacher; and (2) distillation with teachers simultaneously vs. sequentially.
Revised: We investigate (1) the impact of each individual teacher; and (2) distillation with teachers simultaneously vs. sequentially.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Impact of individual teachers: We first distill Student II with each of the three teachers individually: Teacher III has the same backbone and neck but a more advanced head; Teacher IV has more advanced backbone, neck, and head; Teacher V has the same head but more advanced backbone and neck.
Revised: Impact of individual teachers: We first distill Student II with each of the three teachers individually: Teacher III has the same backbone and neck but a more advanced head; Teacher IV has more advanced backbone, neck, and head; Teacher V has the same head but more advanced backbone and neck.

Original: Table 9 provides the performance of the three teachers, where Teacher IV achieves the best performance (row 1-3).
Revised: Table 9 provides the performance of the three teachers, where Teacher IV achieves the best performance (row 1-3).

Original: From Table 10, we can see that our distilled students (row 2-7) significantly and consistently outperform the off-the-shelf student (row 1), demonstrating the effectiveness of our distillation strategy irrespective of the types of teachers .
Revised: From Table 10, we can see that our distilled students (row 2-7) significantly and consistently outperform the off-the-shelf student (row 1), demonstrating the effectiveness of our distillation strategy irrespective of the types of teachers .

Original: Moreover, the improvement of the student distilled with Teacher V (row 2) over that with Teacher III (row 3) shows that a more powerful teacher generally leads to a better distilled student.
Revised: Moreover, the improvement of the student distilled with Teacher V (row 2) over that with Teacher III (row 3) shows that a more powerful teacher generally leads to a better distilled student.

Original: Interestingly, although Teacher IV is more powerful than Teacher V, Table 10 shows that their distilled students achieve quite similar AP (row 2 vs. row 4).
Revised: Interestingly, although Teacher IV is more powerful than Teacher V, Table 10 shows that their distilled students achieve quite similar AP (row 2 vs. row 4).

Original: This indicates that an even more powerful teacher does not necessarily further improve the distilled student; too large a capacity and structure gap between the teacher and student will limit the effectiveness of distillation.
Revised: This indicates that an even more powerful teacher does not necessarily further improve the distilled student; too large a capacity and structure gap between the teacher and student will limit the effectiveness of distillation.

Original: Also, it is easier to distill from teachers with the same head.
Revised: Also, it is easier to distill from teachers with the same head.

Original: Simultaneous vs. progressive distillation: We now distill Student II with the combined teachers, and we choose the top-performing Teacher IV and Teacher V.
Revised: Simultaneous vs. progressive distillation: We now distill Student II with the combined teachers, and we choose the top-performing Teacher IV and Teacher V.

Original: We investigate two types of combination – simultaneous distillation with a feature matching loss between each teacher and the student (row 5), and sequential distillation with teachers one by one (row 6-7).
Revised: We investigate two types of combination – simultaneous distillation with a feature matching loss between each teacher and the student (row 5), and sequential distillation with teachers one by one (row 6-7).

Original: First, we find that using both teachers simultaneously (row 5) is even worse than our method using a single teacher (row 2-4).
Revised: First, we find that using both teachers simultaneously (row 5) is even worse than our method using a single teacher (row 2-4).

Original: This shows that integrating different types of knowledge from multiple teachers is not a trivial task – simultaneously using the features from multiple teachers might provide conflicting supervisions to the student model and thus hinder its distillation process.
Revised: This shows that integrating different types of knowledge from multiple teachers is not a trivial task – simultaneously using the features from multiple teachers might provide conflicting supervisions to the student model and thus hinder its distillation process.

Original: By contrast, our sequential distillation overcomes this issue and improves the performance irrespective of the order of the teachers (row 6-7 vs. row 1-4).
Revised: By contrast, our sequential distillation overcomes this issue and improves the performance irrespective of the order of the teachers (row 6-7 vs. row 1-4).

Original: Second, the sequential order of the teachers makes a difference.
Revised: Second, the sequential order of the teachers makes a difference.

Original: A curriculum-like progression (row 7), where the teacher with a smaller adaptation cost is used first and that with a larger adaptation cost & a higher performance is used later, leads to the best performance.
Revised: A curriculum-like progression (row 7), where the teacher with a smaller adaptation cost is used first and that with a larger adaptation cost & a higher performance is used later, leads to the best performance.

Original: Overall performance: Our best distillation performance is achieved when we first distill Student
Revised: Overall performance: Our best distillation performance is achieved when we first distill Student

Original: II with a curriculum of teachers (Teacher V → IV).
Revised: II with a curriculum of teachers (Teacher V → IV).

Original: Overall, the box AP is improved from 38 .
Revised: Overall, the box AP is improved from 38 .

Original: 2% to 41 .
Revised: 2% to 41 .

Original: and the mask AP is improved from 34 .
Revised: and the mask AP is improved from 34 .

Original: 7% to 37 .
Revised: 7% to 37 .

Original: Our resulting Mask R-CNN detector has comparable performance with HTC, but much smaller runtime .
Revised: Our resulting Mask R-CNN detector has comparable performance with HTC, but much smaller runtime .

Original: C A BLATION S TUDY ON D ISTILLATION WITH H ETEROGENEOUS T EACHERS
Revised: C A BLATION S TUDY ON D ISTILLATION WITH H ETEROGENEOUS T EACHERS

Original: In this section, we provide more details about distillation with heterogeneous teachers (Section 11).
Revised: In this section, we provide more details about distillation with heterogeneous teachers (Section 11).

Original: We investigate the heterogeneous cases where the backbones or input resolutions are different between the teachers and student.
Revised: We investigate the heterogeneous cases where the backbones or input resolutions are different between the teachers and student.

Original: -50 backbone, requiring additional transfer logic.
Revised: -50 backbone, requiring additional transfer logic.

Original: We report the detection (‘Box’) and segmentation (‘Mask’) APs and runtime, and we compare our distilled student with its teachers, and off- the-shelf (‘OTS’) student.
Revised: We report the detection (‘Box’) and segmentation (‘Mask’) APs and runtime, and we compare our distilled student with its teachers, and off- the-shelf (‘OTS’) student.

Original: Our distilled student significantly improves the APs over the ‘OTS’ student by over 3% .
Revised: Our distilled student significantly improves the APs over the ‘OTS’ student by over 3% .

Original: [Table]
Revised: [Table]

Original: Overall performance:
Revised: Overall performance:

Original: Again, Tables 11 and 12 show that our distillation strategy is consistently effective with respect to all the teachers and their combinations, e.g ., the box AP improves from 33 .
Revised: Again, Tables 11 and 12 show that our distillation strategy is consistently effective with respect to all the teachers and their combinations, e.g ., the box AP improves from 33 .

Original: 3% to 37 .
Revised: 3% to 37 .

Original: 0% and the mask AP improves from 30 .
Revised: 0% and the mask AP improves from 30 .

Original: 5% to 33 .
Revised: 5% to 33 .

Original: Two signature findings in heterogeneous distillation: Compared to the homogeneous case, we find the capacity gap between models is a more important factor, and to bridge this gap a proper teacher order plays a more critical role.
Revised: Two signature findings in heterogeneous distillation: Compared to the homogeneous case, we find the capacity gap between models is a more important factor, and to bridge this gap a proper teacher order plays a more critical role.

Original: Details are explained as follows.
Revised: Details are explained as follows.

Original: The student-teacher capacity gap is more pronounced in heterogeneous distillation.
Revised: The student-teacher capacity gap is more pronounced in heterogeneous distillation.

Original: Among the four teachers, Teacher I shares exactly the same neck and head structure with the student, and has a similar but larger backbone; Teacher V has the same head with the student as well, but has a different backbone and neck; Teacher III has similar backbone and neck, but has a different head; and Teacher IV is the most powerful one with completely different architecture.
Revised: Among the four teachers, Teacher I shares exactly the same neck and head structure with the student, and has a similar but larger backbone; Teacher V has the same head with the student as well, but has a different backbone and neck; Teacher III has similar backbone and neck, but has a different head; and Teacher IV is the most powerful one with completely different architecture.

Original: Table 12 (rows 3-6) summarizes the distillation results with single teachers.
Revised: Table 12 (rows 3-6) summarizes the distillation results with single teachers.

Original: First, directly distilling from the strongest teacher (Teacher IV) does not yield the largest improvement.
Revised: First, directly distilling from the strongest teacher (Teacher IV) does not yield the largest improvement.

Original: Second, a relatively less powerful but more similar teacher (Teacher I) leads to the best distillation performance, improving the APs by 2%, although teachers V, III, and IV are all stronger than Teacher I. One possible reason is that Teacher
Revised: Second, a relatively less powerful but more similar teacher (Teacher I) leads to the best distillation performance, improving the APs by 2%, although teachers V, III, and IV are all stronger than Teacher I. One possible reason is that Teacher

Original: I has the same neck and head as Student III as well as similar but deeper backbone, so the capacity gap between Student III and Teacher I is the smallest.
Revised: I has the same neck and head as Student III as well as similar but deeper backbone, so the capacity gap between Student III and Teacher I is the smallest.

Original: Finally, we find that Teacher III is a strong but not particularly helpful teacher, achieving the worst distillation results.
Revised: Finally, we find that Teacher III is a strong but not particularly helpful teacher, achieving the worst distillation results.

Original: One possible reason is that
Revised: One possible reason is that

Original: Training Student III for more epochs improves its performance, but not as much as progressive distillation with teachers.
Revised: Training Student III for more epochs improves its performance, but not as much as progressive distillation with teachers.

Original: Note that for each distillation we train 12 epochs.
Revised: Note that for each distillation we train 12 epochs.

Original: Our distillation strategy is consistently effective irrespective of the types of teachers .
Revised: Our distillation strategy is consistently effective irrespective of the types of teachers .

Original: Moreover, our sequential distillation with multiple teachers outperforms simultaneous distillation with multiple teachers.
Revised: Moreover, our sequential distillation with multiple teachers outperforms simultaneous distillation with multiple teachers.

Original: Our best distilled student is obtained by progressive distilla- tion, where Student III is first distilled with Teacher I (a most similar teacher with the head and
Revised: Our best distilled student is obtained by progressive distilla- tion, where Student III is first distilled with Teacher I (a most similar teacher with the head and

Original: Student III and a deeper backbone
Revised: Student III and a deeper backbone

Original: , then distilled with
Revised: , then distilled with

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: Teacher III has a very different head from Student III, while not as stand-alone accurate as Teacher
Revised: Teacher III has a very different head from Student III, while not as stand-alone accurate as Teacher

Original: IV, making it unable to provide enough guidance to Student III.
Revised: IV, making it unable to provide enough guidance to Student III.

Original: These observations suggest that a smaller capacity gap between the student and the teacher may facilities knowledge transfer.
Revised: These observations suggest that a smaller capacity gap between the student and the teacher may facilities knowledge transfer.

Original: The sequential order of the teachers plays a more critical role in the heterogeneous setting.
Revised: The sequential order of the teachers plays a more critical role in the heterogeneous setting.

Original: Table(row 7-12) presents representative results with different orders or combinations of the teachers.
Revised: Table(row 7-12) presents representative results with different orders or combinations of the teachers.

Original: Again, a proper progressive distillation (row 12) outperforms simultaneous distillation (row 7-9).
Revised: Again, a proper progressive distillation (row 12) outperforms simultaneous distillation (row 7-9).

Original: Notably, it is necessary to start with Teacher I, since the capacity gap between Student III and
Revised: Notably, it is necessary to start with Teacher I, since the capacity gap between Student III and

Original: Teacher I is minimal, with difference only on the depth of their ResNet backbones.
Revised: Teacher I is minimal, with difference only on the depth of their ResNet backbones.

Original: These results confirm the importance of our curriculum-like progression to best benefit from multiple teachers.
Revised: These results confirm the importance of our curriculum-like progression to best benefit from multiple teachers.

Original: Training a student longer vs. distilling a student: As another sanity check, Table 12 includes results of training Student III with more epochs without distillation (row 2-4).
Revised: Training a student longer vs. distilling a student: As another sanity check, Table 12 includes results of training Student III with more epochs without distillation (row 2-4).

Original: We can see that the first 12 additional epochs improve APs by 1% , but there are no significant improvements even if we train for a longer period.
Revised: We can see that the first 12 additional epochs improve APs by 1% , but there are no significant improvements even if we train for a longer period.

Original: This shows the effectiveness of detector distillation.
Revised: This shows the effectiveness of detector distillation.

Original: Distillation with different model resolutions: In Table 12, we have performed distillation where the student and teacher models operated on the same input image resolution ( e.g ., the standard resolution 1 , 333 × 800 on MS COCO).
Revised: Distillation with different model resolutions: In Table 12, we have performed distillation where the student and teacher models operated on the same input image resolution ( e.g ., the standard resolution 1 , 333 × 800 on MS COCO).

Original: In practice, one way to further reduce the latency/runtime of the student is to operate on lower-resolution images.
Revised: In practice, one way to further reduce the latency/runtime of the student is to operate on lower-resolution images.

Original: However, this poses additional challenges – with a teacher of high input resolution and a student of low input resolution, they become even more heterogeneous.
Revised: However, this poses additional challenges – with a teacher of high input resolution and a student of low input resolution, they become even more heterogeneous.

Original: Moreover, image resolution substantially affects object detection performance (Ashraf et al., 2016).
Revised: Moreover, image resolution substantially affects object detection performance (Ashraf et al., 2016).

Original: Here, we are interested in performing distillation with models trained with images of different resolutions to further investigate the generalizability of our approach.
Revised: Here, we are interested in performing distillation with models trained with images of different resolutions to further investigate the generalizability of our approach.

Original: More specifically, we use high-resolution models as teachers and low-resolution models as students, as shown in Table(row 1-4).
Revised: More specifically, we use high-resolution models as teachers and low-resolution models as students, as shown in Table(row 1-4).

Original: In these experiments, the teacher and student feature maps have different spatial resolution .
Revised: In these experiments, the teacher and student feature maps have different spatial resolution .

Original: To tackle this, we simply upsample the spatial maps of the student and supervise the student with the teachers’ features.
Revised: To tackle this, we simply upsample the spatial maps of the student and supervise the student with the teachers’ features.

Original: Again, Table 13 shows that our approach is effective in this more challenging scenario.
Revised: Again, Table 13 shows that our approach is effective in this more challenging scenario.

Original: Our best performance is achieved by progressively distilling the student with its Teacher
Revised: Our best performance is achieved by progressively distilling the student with its Teacher

Original: I-3, I-2, and I-1.
Revised: I-3, I-2, and I-1.

Original: [Table]
Revised: [Table]

Original: D G ENERALIZABILITY TO O THER D ATASETS AND E VALUATION P ROTOCOLS
Revised: D G ENERALIZABILITY TO O THER D ATASETS AND E VALUATION P ROTOCOLS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we study the generalizability of our approach.
Revised: we study the generalizability of our approach.

Original: As an extension from the goldstandard COCO benchmark, we evaluate our distilled student (trained on COCO) on another dataset,
Revised: As an extension from the goldstandard COCO benchmark, we evaluate our distilled student (trained on COCO) on another dataset,

Original: Argoverse-HD, and with another metric, streaming accuracy, and perform distillation on ArgoverseHD directly.
Revised: Argoverse-HD, and with another metric, streaming accuracy, and perform distillation on ArgoverseHD directly.

Original: [Table]
Revised: [Table]

Original: ) APs and runtime.
Revised: ) APs and runtime.

Original: We compare our distilled student with its teachers, and off-the-shelf (‘OTS’) student.
Revised: We compare our distilled student with its teachers, and off-the-shelf (‘OTS’) student.

Original: Our distilled student significantly improves the APs over the ‘OTS’ student by over 2%.
Revised: Our distilled student significantly improves the APs over the ‘OTS’ student by over 2%.

Original: Notably, our distilled student achieves detection accuracy that is comparable with Teacher A but with only around third of the runtime
Revised: Notably, our distilled student achieves detection accuracy that is comparable with Teacher A but with only around third of the runtime

Original: [Table]
Revised: [Table]

Original: Argoverse-HD is a more challenging dataset than COCO due to higher resolution images and significantly more small objects.
Revised: Argoverse-HD is a more challenging dataset than COCO due to higher resolution images and significantly more small objects.

Original: Constructed from the autonomous driving dataset Argoverse 1.1 (Chang et al., 2019), Argoverse-HD contains RGB video sequences and dense 2D bounding box annotations (1,260k boxes in total).
Revised: Constructed from the autonomous driving dataset Argoverse 1.1 (Chang et al., 2019), Argoverse-HD contains RGB video sequences and dense 2D bounding box annotations (1,260k boxes in total).

Original: It consists of 8 object categories, which are a subset of 80 COCO classes and are directly relevant to autonomous driving: person, bicycle, car, motorcycle, bus, truck, traffic light, and stop sign.
Revised: It consists of 8 object categories, which are a subset of 80 COCO classes and are directly relevant to autonomous driving: person, bicycle, car, motorcycle, bus, truck, traffic light, and stop sign.

Original: There are 38k training images and 15k validation images.
Revised: There are 38k training images and 15k validation images.

Original: We report results on the validation images.
Revised: We report results on the validation images.

Original: We test the distilled models trained on COCO on Argoverse-HD without re-training .
Revised: We test the distilled models trained on COCO on Argoverse-HD without re-training .

Original: Table 14-left shows the generalizability of our approach.
Revised: Table 14-left shows the generalizability of our approach.

Original: Streaming accuracy is a recently proposed metric that simultaneously evaluates both the accuracy and latency of algorithms in an online real-time setting (Li et al., 2020a).
Revised: Streaming accuracy is a recently proposed metric that simultaneously evaluates both the accuracy and latency of algorithms in an online real-time setting (Li et al., 2020a).

Original: The evaluator queries the state of the world at all time instants, forcing algorithms to consider the amount of streaming data that must be ignored while processing the last frame.
Revised: The evaluator queries the state of the world at all time instants, forcing algorithms to consider the amount of streaming data that must be ignored while processing the last frame.

Original: Following the setup proposed in Li et al.
Revised: Following the setup proposed in Li et al.

Original: (2020a), we evaluate streaming AP in the context of real-time object detection for autonomous vehicles.
Revised: (2020a), we evaluate streaming AP in the context of real-time object detection for autonomous vehicles.

Original: Table 14-right shows our approach outperforming the prior results from Li et al.
Revised: Table 14-right shows our approach outperforming the prior results from Li et al.

Original: (2020a) by a dramatic margin.
Revised: (2020a) by a dramatic margin.

Original: We find significant wins by using an exceedingly lightweight network (ResNet18 based Mask R-CNN) that can process full-resolution images without sacrificing latency.
Revised: We find significant wins by using an exceedingly lightweight network (ResNet18 based Mask R-CNN) that can process full-resolution images without sacrificing latency.

Original: Due to much higher quantities of small objects, high-reslution processing is more effective than deeper network structures.
Revised: Due to much higher quantities of small objects, high-reslution processing is more effective than deeper network structures.

Original: In addition, progressive distillation further improves performance.
Revised: In addition, progressive distillation further improves performance.

Original: Direct distillation on Argoverse-HD: After testing the distilled model which is trained on COCO, on the Argoverse-HD dataset (Li et al., 2020a) without re-training, we have shown the generalizability of the already-distilled models.
Revised: Direct distillation on Argoverse-HD: After testing the distilled model which is trained on COCO, on the Argoverse-HD dataset (Li et al., 2020a) without re-training, we have shown the generalizability of the already-distilled models.

Original: Here we directly distill the student model on Argoverse-HD, using Faster R-CNN with a ResNet-18 backbone as the student model.
Revised: Here we directly distill the student model on Argoverse-HD, using Faster R-CNN with a ResNet-18 backbone as the student model.

Original: As shown in Table 15, we use four teachers with ResNet-50 backbones (row 1-4), including Faster R-CNN (Ren et al., 2014)
Revised: As shown in Table 15, we use four teachers with ResNet-50 backbones (row 1-4), including Faster R-CNN (Ren et al., 2014)

Original: (Teacher A), Cascade R-CNN (Cai & Vasconcelos, 2018)
Revised: (Teacher A), Cascade R-CNN (Cai & Vasconcelos, 2018)

Original: (Teacher B), and DetectoRS (Qiao et al., 2021) (Teachers C & D).
Revised: (Teacher B), and DetectoRS (Qiao et al., 2021) (Teachers C & D).

Original: The results are summarized in Table 15.
Revised: The results are summarized in Table 15.

Original: Our best distillation performance is achieved when we first distill the student with a similar teacher (Teacher A), and then progressively distill with more powerful teachers (Teachers B, then C, and finally D).
Revised: Our best distillation performance is achieved when we first distill the student with a similar teacher (Teacher A), and then progressively distill with more powerful teachers (Teachers B, then C, and finally D).

Original: Overall, the bbox mAP is improved from 27 .
Revised: Overall, the bbox mAP is improved from 27 .

Original: 1% to 29 .
Revised: 1% to 29 .

Original: In addition, comparing with Table 14-left, the absolute performance of the teachers and students in Table 15 is lower.
Revised: In addition, comparing with Table 14-left, the absolute performance of the teachers and students in Table 15 is lower.

Original: This is because here we use weaker teachers and student models (Faster RCNN for fast experiments) than the models used in Table 14-left (Mask R-CNN).
Revised: This is because here we use weaker teachers and student models (Faster RCNN for fast experiments) than the models used in Table 14-left (Mask R-CNN).

Original: However, the relative improvement (between the distilled and OTS students) of box AP ( 2 . 1% ) is larger than that in Table 14-left ( 1 . 7% ), indicating that learning distillation directly on Argoverse-HD further improves the performance. 
Revised: However, the relative improvement (between the distilled and OTS students) of box AP ( 2 . 1% ) is larger than that in Table 14-left ( 1 . 7% ), indicating that learning distillation directly on Argoverse-HD further improves the performance. E M ORE C OMPARISON WITH P RIOR K NOWLEDGE D ISTILLATION M ETHODS

Revised: The most profound difference between this work and most of the prior work on knowledge distillation is that prior work mainly focuses on the image classification task, while we address the object detection task.

Revised: The detection task (and the associated model architectures) is much more complicated than the classification task.

Revised: This makes the distillation methods developed in the context of classification often not directly applicable to detection.

Revised: That is why dedicated distillation methods (Chen et al., 2017; Wang et al., 2019;

Original: E I MPLEMENTATION D ETAILS
Revised: Guo et al., 2021; Zhang & Ma, 2021; Dai et al., 2021; Guo et al., 2021; Yang et al., 2022a;b) need to be developed for the detection task in the literature. Here, we discuss the difference between our method and prior method on knowledge distillation in detail:

Revised: Progressive distillation: Mirzadeh et al.

Revised: (2020) is related to our method, in the sense that this work progressively distills a student from multiple teachers (one teacher and several additional teacher assistants (TAs)).

Revised: With one TA, the distillation process in Mirzadeh et al.

Revised: (2020) contains three steps: 1)

Revised: The TA is first distilled from the teacher; 2)

Revised: The student

Revised: is distilled from the

Revised: TA; and 3)

Revised: The student trained by the TA is further distilled from the teacher.

Revised: When there are multiple TAs, the shallower TAs are distilled from deeper ones, so that they form a distillation path.

Revised: However, our work is different from Mirzadeh et al.

Revised: (2020) in three important ways:

Revised: • As mentioned above, Mirzadeh et al.

Revised: (2020) focuses on image classification, while we study progressive distillation in the context of object detection.

Revised: In our case, the transferred knowledge is no longer classification logits but intermediate features or structured predictions.

Revised: This would require additional consideration and algorithmic designs to extend progressive distillation from image classification to object detection.

Revised: • Our strategy to construct the teacher sequence is a novel contribution and is fundamentally different from Mirzadeh et al.

Revised: In our work, we propose a heuristic algorithm (Algorithm 1) based on the representation similarities between different models (Section 3.2), which automatically generates the teacher order.

Revised: In Mirzadeh et al.

Revised: (2020), a series of deep networks with increasing depths act as the student, the TA(s), and the teacher.

Revised: One can intuitively determine the distillation sequence of TA(s) according to their increasing depths (which imply learning capacities).

Revised: However in our case, there is a pool of teachers with diverse architectures and their relative

Revised: ordering is unknown.

Revised: This challenge motivates us to design an algorithm to automatically decide the teacher order based on their representation similarities; importantly, the strategy in Mirzadeh et al.

Revised: (2020) is not applicable in our task.

Revised: • Mirzadeh et al.

Revised: (2020) is more cumbersome and time-consuming.

Revised: An intermediate TA in Mirzadeh et al.

Revised: (2020) needs to be first distilled from the teacher or a deeper TA, so the TAs have to be trained one by one.

Revised: By contrast, all of our teachers (including the intermediate teachers and the final teacher) are trained independently.

Revised: This makes the generation of our teachers parallelizable.

Revised: Multi-teacher distillation: The key difference lies in: These methods (You et al., 2017; Lan et al., 2018; Guo et al., 2020) use an ensemble of multiple teachers simultaneously to guide the student learning, while our work distills from multiple teachers sequentially, and we proposed a novel method to construct the appropriate teacher order.

Revised: Empirically, we compare these two strategies, and demonstrate that our sequential progressive strategy outperforms the simultaneous strategy (via teacher ensemble by taking the average of teacher features) for object detection.

Revised: • This comparison is provided in Appendix B (Table 10) and Appendix C (Table 12).

Revised: For example in Table 12, if we compare experiments with ID 7-9 (simultaneous distillation from teacher ensembles) vs. experiments with ID 10-12 (progressive distillation from teacher sequences), we find that progressive distillation is a better choice.

Revised: • Our performance superiority is because in the object detection task, the teacher’s knowledge is transferred from intermediate features, rather than from final classification predictions.

Revised: Thus, the ensemble of multiple teachers might provide conflicting supervision signals for the student, leading to performance interior to our progressive distillation.

Revised: Online distillation deep mutual

Revised: learning: Although the teacher model is also changing during

Revised: online distillation (Yang et al., 2019a; Guo et al., 2020; Yao & Sun, 2020; Li et al., 2022),

Revised: the

Revised: principle of our

Revised: sequential teachers is significantly different from online distillation for the following reasons:

Revised: • Strictly speaking, in online distillation, there is only one teacher – This teacher’s architecture is fixed, and its weights keep updating in an online manner.

Revised: By contrast, we have multiple teachers – These teachers have different architectures, and their weights are first trained independently, and then frozen in the progressive distillation process; in our progressive distillation, we switch the whole teacher model.

Revised: • The type of discrepancy between the student and the teacher is different for ours and online distillation.

Revised: Online distillation often uses similar or even the same architecture for both the teacher and student models.

Revised: Consequently, their capacities are at the same level, and they can evolve together.

Revised: Our study is quite different: The key question we want to address is the capacity gap between the student and the teacher (the capacity gap is due to the architectural difference between the student and the teacher); and our solution is to progressively distill using other teachers with intermediate capacities.

Revised: Other general distillation mechanisms: These methods (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Ahn et al., 2019) introduce other types of distillation mechanisms, but still consider the setting where only one single fixed teacher is involved.

Revised: Different from these methods, we use multiple teachers to progressively transfer knowledge from them to the student.

Revised: Meanwhile, we share some similarities with Romero et al.

Revised: (2015); Zagoruyko & Komodakis (2017);

Revised: Ahn et al.

Revised: (2019) in that they are distilling knowledge from the activations of intermediate layers.

Revised: Our simple feature-matching loss (Section 3.1) and other recent work in detector distillation ( e.g ., CWD (Shu et al., 2021), FGD (Yang et al., 2022a), and MGD (Yang et al., 2022b)) are based on the “hint” distillation (learning from intermediate layers’ outputs) from Romero et al. (2015).

Revised: F M ORE R ESULTS ON C OMBINATION WITH S TATE - OF - THE -A RT D ISTILLATION M ECHANISMS

Revised: In this section, we include additional experimental results requested by the reviewers.

Revised: Other object detectors: In the main paper, we mainly use RetinaNet and Mask R-CNN as student detectors for a fair comparison with existing methods, because most of the prior work on detector distillation has primarily focused on RetinaNet and Mask R-CNN.

Revised: Being commonly-used in real- world applications, RetinaNet and Mask R-CNN represent the two important families of object detectors (single-stage and two-stage).

Revised: In principle

Revised: [Table]

Revised: our work is general and applicable to different types of detection models, as our progressive distillation strategy is designed without assumption on particular types of detector architectures.

Revised: To demonstrate this

Revised: [Table]

Revised: we have performed an additional experiment on another detection model, RepPoints (Yang et al., 2019b).

Revised: RepPoints is an anchor-free detector, with high efficiencyaccuracy trade-off.

Revised: Meanwhile, RepPoints has been experimented in state-of-the-art detector distillation methods such as MGD (Yang et al., 2022b), so we can provide an informative comparison.

Revised: Therefore, we choose RepPoints as the detection model in this additional experiment.

Revised: The student model is RepPoints/ResNet-50 (38.6 AP on COCO)

Revised: [Table]

Revised: and following state-of-the-art method MGD, we use RepPoints/ResNeXt-101 (44.2 AP) as the final teacher model.

Revised: The intermediate teacher in our progressive distillation is RepPoints/ResNet-101 (40.5 AP).

Revised: The following Table 16 shows the results obtained.

Revised: [Table]

Revised: Our progressive distillation strategy improves the performance of the student detector by 4.3% AP, which is also 0.6% AP better than the previous state-of-the-art MGD.

Revised: This result is achieved by only introducing an intermediate teacher and without increasing the training cost.

Revised: This experiment further demonstrates that our progressive distillation strategy is general and can be applied to various detectors.

Revised: Keep using the intermediate teacher: In the main paper, we have discussed the capacity gap between the teacher and student and how to mitigate this gap via progressive distillation.

Revised: One may question that, since the capacity gap between the intermediate teacher and the student is smaller, keeping using this intermediate teacher throughout the distillation procedure might also lead to a good student.

Revised: Here we provide an exemplary experiment result (in addition to Figure 4-left) as an answer to this question: Only using the intermediate teacher is still suboptimal as compared with our proposed progressive distillation.

Revised: In this experiment, we use state-of-the-art distillation method MGD as the base method.

Revised: We use

Revised: RetinaNet/ResNet-50 (37.4 AP on COCO) as the student model, RetinaNet/ResNet-101 (38.9 AP) as the intermediate teacher model, and RetinaNet/ResNeXt-101 (40.8 AP) as the final teacher model.

Revised: The capacity gap between the intermediate teacher and the student is smaller than that between the final teacher and the student.

Revised: The following Table 17 shows the results.

Revised: [Table]

Revised: Keeping using the intermediate teacher (RetinaNet/ResNet-101) for a longer 2 × training schedule indeed improves the performance from 40.2% AP to 40.7%

Revised: [Table]

Revised: but it is still not better than directly using the final teacher (RetinaNet/ResNeXt-101).

Revised: Our progressive distillation, which first uses the intermediate teacher and then the final teacher for distillation, outperforms both direct distillation schemes and achieves 41.4% AP performance.

Revised: As always, we use the same total training time ( 2 × training schedule) as direct distillation for a fair comparison.

Revised: This experiment supports the conclusions that 1) employing an intermediate teacher throughout the distillation process is not a good option; and 2) our progressive distillation, which uses both the intermediate teacher and the best performing teacher sequentially, leads to the best student performance.

Original: We implement detectors and their distillation using the MMDetection codebase (Chen et al., 2019b).
Revised: We implement detectors and their distillation using the MMDetection codebase (Chen et al., 2019b).

Original: We train on 8 GPUs for 12 epochs for each distillation.
Revised: We train on 8 GPUs for 12 epochs for each distillation.

Original: For MS COCO, we use the standard input resolution of 1 , 333 × 800 , with each GPU hosting 2 images.
Revised: For MS COCO, we use the standard input resolution of 1 , 333 × 800 , with each GPU hosting 2 images.

Original: For Argoverse-HD, we use its much higher native resolution as the input at 1 , 920 × 1 , 200 , with each GPU hosting 1 image.
Revised: For Argoverse-HD, we use its much higher native resolution as the input at 1 , 920 × 1 , 200 , with each GPU hosting 1 image.

Original: We use an initial learning rate of 0 .
Revised: We use an initial learning rate of 0 .

Original: 01 (for RetinaNet students) or 0 .
Revised: 01 (for RetinaNet students) or 0 .

Original: 02 (for Mask R-CNN students).
Revised: 02 (for Mask R-CNN students).

Original: We use stochastic gradient descent and a momentum of 0 . 9 .
Revised: We use stochastic gradient descent and a momentum of 0 . 9 .

Original: For the simple feature-matching loss (see
Revised: For the simple feature-matching loss (see

Original: Section 3.1), we perform a grid search over the hyper-parameter λ .
Revised: Section 3.1), we perform a grid search over the hyper-parameter λ .

Original: While the optimal values are dependent on the architectures of the teacher and student models, we find the performance is not very sensitive to λ between 0 . 3 and 0 . 8 .
Revised: While the optimal values are dependent on the architectures of the teacher and student models, we find the performance is not very sensitive to λ between 0 . 3 and 0 . 8 .

Original: We set λ = 0 .
Revised: We set λ = 0 .

Original: 5 for RetinaNet students and λ = 0 .
Revised: 5 for RetinaNet students and λ = 0 .

Original: 8 for
Revised: 8 for

Original: Mask R-CNN students.
Revised: Mask R-CNN students.

Revised: When we combine our progressive distillation with state-of-the-art distillation mechanisms including CWD (Shu et al., 2021), FGD (Yang et al., 2022a), and MGD (Yang et al., 2022b) (Section 4.4), we strictly follow the publicly available implementation from their authors, and use an intermediate teacher (RetinaNet/ResNet-101 or Cascade Mask R-CNN/ResNet50-DCN) for progressive distillation.

Revised: In the original implementation of FGD and MGD, an inheriting strategy (Kang et al., 2021) is utilized, which initializes the student with the teacher’s neck and head parameters to train the student when they have the same head structure.

Revised: In our progressive distillation, we adopt this inheriting strategy only once for the first teacher.

Revised: For the Transformer-based teachers, we use Swin Transformer backbone, which has a hierarchical architecture and shares the “same feature map resolutions as those of typical convolutional networks ( e.g ., ResNet-50)”

Revised: (Liu et al., 2021).

Revised: Following the original implementation of the Swin Transformer, the backbone is equipped with an FPN neck, so the number of neck feature channels is the same as the student.

Revised: As a result, Swin Transformer based teachers can be used like some other convolution-based teachers without the feature map matching function ( r ( · ) in Section 3.1).