Original: P ROXYLESS KD: D IRECT K NOWLEDGE D ISTILLATION WITH INHERITED CLASSIFIER FOR FACE R ECOGNITION
Revised: P ROXYLESS KD: D IRECT K NOWLEDGE D ISTILLATION WITH I NHERITED C LASSIFIER FOR F ACE R ECOGNITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Knowledge Distillation (KD) refers to transferring knowledge from a large model to a smaller one, which is widely used to enhance model performance in machine learning.
Revised: Knowledge Distillation (KD) refers to transferring knowledge from a large model to a smaller one, which is widely used to enhance model performance in machine learning.

Original: It tries to align embedding spaces generated from the teacher and the student model (i.e. to make images corresponding to the same semantics share the same embedding across different models).
Revised: It tries to align embedding spaces generated from the teacher and the student model (i.e. to make images corresponding to the same semantics share the same embedding across different models).

Original: In this work, we focus on its application in face recognition.
Revised: In this work, we focus on its application in face recognition.

Original: We observe that existing knowledge distillation models optimize the proxy tasks that force the student to mimic the teacher’s behavior, instead of directly optimizing the face recognition accuracy.
Revised: We observe that existing knowledge distillation models optimize the proxy tasks that force the student to mimic the teacher’s behavior, instead of directly optimizing the face recognition accuracy.

Original: Consequently, the obtained student models are not guaranteed to be optimal on the target task or able to beneﬁt from advanced constraints, such as large margin constraint (e.g. margin-based softmax).
Revised: Consequently, the obtained student models are not guaranteed to be optimal on the target task or able to beneﬁt from advanced constraints, such as large margin constraint (e.g. margin-based softmax).

Original: We then propose a novel method named ProxylessKD that directly optimizes face recognition accuracy by inheriting the teacher’s classiﬁer as the student’s classiﬁer to guide the student to learn discriminative embeddings in the teacher’s embedding space.
Revised: We then propose a novel method named ProxylessKD that directly optimizes face recognition accuracy by inheriting the teacher’s classiﬁer as the student’s classiﬁer to guide the student to learn discriminative embeddings in the teacher’s embedding space.

Original: The proposed ProxylessKD is very easy to implement and sufﬁciently generic to be extended to other tasks beyond face recognition.
Revised: The proposed ProxylessKD is very easy to implement and sufﬁciently generic to be extended to other tasks beyond face recognition.

Original: We conduct extensive experiments on standard face recognition benchmarks, and the results demonstrate that ProxylessKD achieves superior performance over existing knowledge distillation methods.
Revised: We conduct extensive experiments on standard face recognition benchmarks, and the results demonstrate that ProxylessKD achieves superior performance over existing knowledge distillation methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Knowledge Distillation (KD) is a process of transferring knowledge from a large model to a smaller one.
Revised: Knowledge Distillation (KD) is a process of transferring knowledge from a large model to a smaller one.

Original: This technique is widely used to enhance model performance in many machine learning tasks such as image classiﬁcation (Hinton et al., 2015), object detection (Chen et al., 2017b) and speech translation (Liu et al., 2019c).
Revised: This technique is widely used to enhance model performance in many machine learning tasks such as image classiﬁcation (Hinton et al., 2015), object detection (Chen et al., 2017b) and speech translation (Liu et al., 2019c).

Original: When applied to face recognition, the embeddings of a gallery are usually extracted by a larger teacher model while the embeddings of the query images are extracted by a smaller student model.
Revised: When applied to face recognition, the embeddings of a gallery are usually extracted by a larger teacher model while the embeddings of the query images are extracted by a smaller student model.

Original: The student model is encouraged to align its embedding space with that of the teacher, so as to improve its recognition capability.
Revised: The student model is encouraged to align its embedding space with that of the teacher, so as to improve its recognition capability.

Original: Previous KD works promote the consistency in ﬁnal predictions (Hinton et al., 2015), or in the activations of the hidden layer between student and teacher (Romero et al., 2014; Zagoruyko &
Revised: Previous KD works promote the consistency in ﬁnal predictions (Hinton et al., 2015), or in the activations of the hidden layer between student and teacher (Romero et al., 2014; Zagoruyko &

Original: Komodakis, 2016).
Revised: Komodakis, 2016).

Original: Such an idea of only optimizing the consistency in predictions or activations brings limited performance boost since the student is often a small model with weaker capacity compared with the teacher.
Revised: Such an idea of only optimizing the consistency in predictions or activations brings limited performance boost since the student is often a small model with weaker capacity compared with the teacher.

Original: Later, Park et al.
Revised: Later, Park et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2019) propose to exploit the correlation between instances to guide the student to mimic feature relationships of the teacher over a batch of input data, which achieves better performance.
Revised: (2019) propose to exploit the correlation between instances to guide the student to mimic feature relationships of the teacher over a batch of input data, which achieves better performance.

Original: However, the above works all aim at guiding the student to mimic the behavior of the teacher, which is not suitable for practical face recognition.
Revised: However, the above works all aim at guiding the student to mimic the behavior of the teacher, which is not suitable for practical face recognition.

Original: In reality, it is very important to directly align embedding spaces between student and teacher, which can enable models across different devices to share the same embedding space for feasible similarity comparison.
Revised: In reality, it is very important to directly align embedding spaces between student and teacher, which can enable models across different devices to share the same embedding space for feasible similarity comparison.

Original: To solve this, a simple and direct method is to directly minimize the L2 distance of embeddings extracted by student and teacher.
Revised: To solve this, a simple and direct method is to directly minimize the L2 distance of embeddings extracted by student and teacher.

Original: However, this method (we call it L2KD) only considers minimizing the intra-class distance and ignores maximizing the inter-class distance, and is unable to beneﬁt from some powerful loss functions with large margin (e.g. Cosface loss (Wang et al., 2018a), Arcface loss (Deng et al., 2019)) constraint to further improve the performance.
Revised: However, this method (we call it L2KD) only considers minimizing the intra-class distance and ignores maximizing the inter-class distance, and is unable to beneﬁt from some powerful loss functions with large margin (e.g. Cosface loss (Wang et al., 2018a), Arcface loss (Deng et al., 2019a)) constraint to further improve the performance.

Original: (a)L2KD (b) ProxylessKD 1.
Revised: [Figure]

Original: In this work, we propose an effective knowledge distillation method named ProxylessKD.
Revised: In this work, we propose an effective knowledge distillation method named ProxylessKD.

Original: According to Ranjan et al.
Revised: According to Ranjan et al.

Original: (2017), the classiﬁer neurons in a recognition model can be viewed as the approximate embedding centers of each class.
Revised: (2017), the classiﬁer neurons in a recognition model can be viewed as the approximate embedding centers of each class.

Original: This can be used to guide the embedding learning as in this way, the classiﬁer can encourage the embedding to align with the approximate embedding centers corresponding to the label of the image.
Revised: This can be used to guide the embedding learning as in this way, the classiﬁer can encourage the embedding to align with the approximate embedding centers corresponding to the label of the image.

Original: Inspired by this, we propose to initialize the weight of the student’s classiﬁer with the weight of the teacher’s classiﬁer and ﬁx it during the distillation process, which forces the student to produce an embedding space as consistent with that of the teacher as possible.
Revised: Inspired by this, we propose to initialize the weight of the student’s classiﬁer with the weight of the teacher’s classiﬁer and ﬁx it during the distillation process, which forces the student to produce an embedding space as consistent with that of the teacher as possible.

Original: Different from previous knowledge distillation works (Hinton et al., 2015; Zagoruyko & Komodakis, 2016; Romero et al., 2014; Park et al., 2019; Peng et al., 2019) and L2KD, the proposed ProxylessKD not only directly optimizes the target task but also considers minimizing the intra-class distance and maximizing the inter-class distance.
Revised: Different from previous knowledge distillation works (Hinton et al., 2015; Zagoruyko & Komodakis, 2016; Romero et al., 2014; Park et al., 2019; Peng et al., 2019) and L2KD, the proposed ProxylessKD not only directly optimizes the target task but also considers minimizing the intra-class distance and maximizing the inter-class distance.

Original: Meanwhile it can beneﬁt from large margin constraints (e.g. Cosface loss (Wang et al., 2018a) and Arcface loss (Deng et al., 2019)).
Revised: Meanwhile it can beneﬁt from large margin constraints (e.g. Cosface loss (Wang et al., 2018a) and Arcface loss (Deng et al., 2019a)).

Original: As shown in Figure 1, the intra-class distance in
Revised: As shown in Figure 1, the intra-class distance in

Original: ProxylessKD combined with Arcface loss is much closer than L2KD, and the inter-class distance in
Revised: ProxylessKD combined with Arcface loss is much closer than L2KD, and the inter-class distance in

Original: ProxylessKD combined with Arcface loss is much larger than L2KD.
Revised: ProxylessKD combined with Arcface loss is much larger than L2KD.

Original: Thus it can be expected that our ProxylessKD is able to improve the performance of face recognition, which will be experimentally validated.
Revised: Thus it can be expected that our ProxylessKD is able to improve the performance of face recognition, which will be experimentally validated.

Original: The main contributions in this paper are summarized as follows:
Revised: The main contributions in this paper are summarized as follows:

Original: • We analyze the shortcomings of existing knowledge distillation methods: they only optimize the proxy task rather than the target task; and they cannot conveniently integrate with advanced large margin constraints to further lift performance.
Revised: • We analyze the shortcomings of existing knowledge distillation methods: they only optimize the proxy task rather than the target task; and they cannot conveniently integrate with advanced large margin constraints to further lift performance.

Original: • We propose a simple yet effective KD method named ProxylessKD, which directly boosts embedding space alignment and can be easily combined with existing loss functions to achieve better performance.
Revised: • We propose a simple yet effective KD method named ProxylessKD, which directly boosts embedding space alignment and can be easily combined with existing loss functions to achieve better performance.

Original: • We conduct extensive experiments on standard face recognition benchmarks, and the results well demonstrate the effectiveness of the proposed ProxylessKD.
Revised: • We conduct extensive experiments on standard face recognition benchmarks, and the results well demonstrate the effectiveness of the proposed ProxylessKD.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Knowledge distillation.
Revised: Knowledge distillation.

Original: Knowledge distillation aims to transfer the knowledge from the teacher model to a small model.
Revised: Knowledge distillation aims to transfer the knowledge from the teacher model to a small model.

Original: The pioneer work is Bucilu ˇ a et al.
Revised: The pioneer work is Buciluˇa et al.

Original: (2006), and Hinton et al.
Revised: (2006), and Hinton et al.

Original: (2015) popularizes this idea by deﬁning the concept of knowledge distillation (KD) as training the small model (the student) by exploiting the soft targets provided by a cumbersome model (the teacher).
Revised: (2015) popularizes this idea by deﬁning the concept of knowledge distillation (KD) as training the small model (the student) by exploiting the soft targets provided by a cumbersome model (the teacher).

Original: Unlike the one-hot label, the soft targets from the teacher contain rich related information among classes, which can guide the student to better learn the ﬁne-grained distribution of data and thus lift performance.
Revised: Unlike the one-hot label, the soft targets from the teacher contain rich related information among classes, which can guide the student to better learn the ﬁne-grained distribution of data and thus lift performance.

Original: Lots of variants of model distillation strategies have been proposed and widely adopted in the ﬁelds like image classiﬁcation (Chen et al., 2018), object detection (Chen et al., 2017a), semantic segmentation (Liu et al., 2019a; Park & Heo, 2020), etc.
Revised: Lots of variants of model distillation strategies have been proposed and widely adopted in the ﬁelds like image classiﬁcation (Chen et al., 2018), object detection (Chen et al., 2017a), semantic segmentation (Liu et al., 2019a; Park & Heo, 2020), etc.

Original: Concretely, Zagoruyko & Komodakis (2016) proposed a response-based KD model, Attention Transfer (AT), which aims to teach the student to activate the same region as the teacher model.
Revised: Concretely, Zagoruyko & Komodakis (2016) proposed a response-based KD model, Attention Transfer (AT), which aims to teach the student to activate the same region as the teacher model.

Original: Some relation-based distillation methods have also been developed, which encourage the student to mimic the relation of the output in different stages (Yim et al., 2017) and the samples in a batch (Park et al., 2019).
Revised: Some relation-based distillation methods have also been developed, which encourage the student to mimic the relation of the output in different stages (Yim et al., 2017) and the samples in a batch (Park et al., 2019).

Original: The previous works mostly optimize the proxy tasks rather than the target task.
Revised: The previous works mostly optimize the proxy tasks rather than the target task.

Original: In this work, we directly optimize face recognition accuracy by inheriting the teacher’s classiﬁer as the student’s classiﬁer to guide the student to learn discriminative embeddings in the teacher’s embedding space.
Revised: In this work, we directly optimize face recognition accuracy by inheriting the teacher’s classiﬁer as the student’s classiﬁer to guide the student to learn discriminative embeddings in the teacher’s embedding space.

Revised: Different from (Deng et al., 2019b), it only analyzes the recognition performance of the student model by inheriting the teacher’s classiﬁer.

Revised: We consider the face recognition requirements and analyze the shortcomings of existing knowledge distillation works in the actual retrieval business.

Revised: Meanwhile, we analyze the recognition performance of ProxylessKD when combined with different loss functions and different margins in the loss (e.g. Cosface, Arcface), which is more important for the actual face recognition community.

Original: Loss functions used in face recognition.
Revised: Loss functions used in face recognition

Original: Softmax loss is deﬁned as the pipeline combination of the last fully connected layer, softmax function, and cross-entropy loss.
Revised: Softmax loss is deﬁned as the pipeline combination of the last fully connected layer, softmax function, and cross-entropy loss.

Original: Although it can help the network separate categories in a high-dimensional space, for ﬁne-grained classiﬁcation problems like face recognition, it offers limited accuracy due to the considerable inter-class similarity.
Revised: Although it can help the network separate categories in a high-dimensional space, for ﬁne-grained classiﬁcation problems like face recognition, it offers limited accuracy due to the considerable inter-class similarity.

Original: Liu et al.
Revised: Liu et al.

Original: (2017) proposed Sphereface to achieve smaller maximal intra-class distance than minimal inter-class distance, which can directly enhance feature discrimination.
Revised: (2017) proposed Sphereface to achieve smaller maximal intra-class distance than minimal inter-class distance, which can directly enhance feature discrimination.

Original: Compared with SphereFace in which the margin m is multiplied on the angle, Wang et al.
Revised: Compared with SphereFace in which the margin m is multiplied on the angle, Wang et al.

Original: (2018a)
Revised: (2018a)

Original: ; Whitelam et al.
Revised: ; Whitelam et al.

Original: (2017) proposed CosFace, where the margin is directly subtracted from cosine, achieving better performance than SphereFace and relieving the need for joint supervision from the softmax loss.
Revised: (2017) proposed CosFace, where the margin is directly subtracted from cosine, achieving better performance than SphereFace and relieving the need for joint supervision from the softmax loss.

Original: To further improve feature discrimination, Deng et al.
Revised: To further improve feature discrimination, Deng et al.

Original: (2018) proposed the ArcFace that utilizes the arc-cosine function to calculate the angle, i.e. adding an additive angular margin and back again by the cosine function.
Revised: (2018) proposed the ArcFace that utilizes the arc-cosine function to calculate the angle, i.e. adding an additive angular margin and back again by the cosine function.

Original: In this paper, we combine our ProxylessKD with the above loss functions to further lift performance, e.g. Arcface loss function.
Revised: In this paper, we combine our ProxylessKD with the above loss functions to further lift performance, e.g. Arcface loss function.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We ﬁrst revisit popular loss functions in face recognition in Sec. 3.1, and elaborate on our ProxylessKD in Sec. 3.2.
Revised: We ﬁrst revisit popular loss functions in face recognition in Sec. 3.1, and elaborate on our ProxylessKD in Sec. 3.2.

Original: Then we introduce how to combine our method with existing loss functions in
Revised: Then we introduce how to combine our method with existing loss functions in

Original: Sec. 3.3.
Revised: Sec. 3.3.

Original: 3. R EVISIT LOSS FUNCTION IN FACE RECOGNITION
Revised: 3. R EVISIT LOSS FUNCTION IN FACE RECOGNITION

Original: The most classical loss function in classiﬁcation is the Softmax loss, which is represented as follows:
Revised: The most classical loss function in classiﬁcation is the Softmax loss, which is represented as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, w k denotes the weight of the model classiﬁer, where k ∈ { 1 , 2 , ..., K } and K denotes the number of classes.
Revised: Here, w k denotes the weight of the model classiﬁer, where k ∈ { 1 , 2 , ..., K } and K denotes the number of classes.

Original: x i is the embedding of i - th sample and usually normalized with magnitude replaced with a scale parameter of s .
Revised: x i is the embedding of i - th sample and usually normalized with magnitude replaced with a scale parameter of s .

Original: θ w k ,x i denotes the angle between w k and x i .
Revised: θ w k ,x i denotes the angle between w k and x i .

Original: y is the ground truth label for the input embedding x i .
Revised: y is the ground truth label for the input embedding x i .

Original: N is the batch size.
Revised: N is the batch size.

Original: In recent years, several margin-based softmax loss functions (Liu et al., 2017; Wang et al., 2017; 2018a; Deng et al., 2019) have been proposed to boost the embedding discrimination, which is represented as follows:
Revised: In recent years, several margin-based softmax loss functions (Liu et al., 2017; Wang et al., 2017; 2018a; Deng et al., 2019a) have been proposed to boost the embedding discrimination, which is represented as follows:

Original: [Equation]
Revised: [Equation]

Original: In the above equation, f ( m, θ w y ,x i ) is a margin function.
Revised: In the above equation, f ( m, θ w y ,x i ) is a margin function.

Original: Precisely, f ( m, θ w y ,x i ) = cos ( m · θ w y ,x i ) is A-Softmax loss proposed in (Liu et al., 2017), where m is an integer and greater than zero.
Revised: Precisely, f ( m, θ w y ,x i ) = cos ( m · θ w y ,x i ) is A-Softmax loss proposed in (Liu et al., 2017), where m is an integer and greater than zero.

Original: f ( m, θ w y ,x i ) =
Revised: f ( m, θ w y ,x i ) =

Original: cos ( θ w y ,x i ) − m is the AM-Softmax loss proposed in Wang et al.
Revised: cos ( θ w y ,x i ) − m is the AM-Softmax loss proposed in Wang et al.

Original: (2018a)
Revised: (2018a)

Original: and the hyper-parameter m is greater than zero.
Revised: and the hyper-parameter m is greater than zero.

Original: f ( m, θ w y ,x i ) =
Revised: f ( m, θ w y ,x i ) =

Original: cos ( θ w y ,x i + m ) with m > 0 is Arc-Softmax introduced in Deng et al. (2019), which achieves better performance than the former.
Revised: cos ( θ w y ,x i + m ) with m > 0 is Arc-Softmax introduced in Deng et al. (2019a), which achieves better performance than the former.

Original: Fortunately, the proposed ProxylessKD can be combined with the above loss function, conveniently.
Revised: Fortunately, the proposed ProxylessKD can be combined with the above loss function, conveniently.

Original: In this paper, we combine our proposed ProxylessKD method with above loss functions and investigate their performance.
Revised: In this paper, we combine our proposed ProxylessKD method with above loss functions and investigate their performance.

Original: 3. I NHERITED CLASSIFIER K NOWLEDGE DISTILLATION
Revised: 3. I NHERITED CLASSIFIER K NOWLEDGE DISTILLATION

Original: The models trained for different devices are expected to share the same embedding space for similarity comparison.
Revised: The models trained for different devices are expected to share the same embedding space for similarity comparison.

Original: However, most existing knowledge distillation models only optimize proxy tasks, encouraging the student to mimic the teacher’s behavior, instead of directly optimizing the target accuracy.
Revised: However, most existing knowledge distillation models only optimize proxy tasks, encouraging the student to mimic the teacher’s behavior, instead of directly optimizing the target accuracy.

Original: In this paper, we propose to directly optimize the target task by inheriting the teacher’s classiﬁer to encourage better embedding space consistency between the student and the teacher.
Revised: In this paper, we propose to directly optimize the target task by inheriting the teacher’s classiﬁer to encourage better embedding space consistency between the student and the teacher.

Original: Knowledge distillation with single teacher.
Revised: Knowledge distillation with single teacher.

Original: In most of the existing distillation works (Chen et al., 2018; Liu et al., 2019a; Park & Heo, 2020), single large model is utilized as the teacher to guide the student.
Revised: In most of the existing distillation works (Chen et al., 2018; Liu et al., 2019a; Park & Heo, 2020), single large model is utilized as the teacher to guide the student.

Original: Hence, we ﬁrstly introduce our ProxylessKD method under the common single teacher model knowledge distillation, as shown in Figure 2 (a), where E T and E S represent the embedding extracted by the teacher model T and the student model S , respectively.
Revised: Hence, we ﬁrstly introduce our ProxylessKD method under the common single teacher model knowledge distillation, as shown in Figure 2 (a), where E T and E S represent the embedding [Figure] E t E tn

Original:  Unlike previous works (Hinton et al., 2015; Zagoruyko & Komodakis, 2016) that optimize the proxy task, we emphasize on optimizing the target task.
Revised: E T E S Fix weight extracted by the teacher model T and the student model S , respectively. Unlike previous works (Hinton et al., 2015; Zagoruyko & Komodakis, 2016) that optimize the proxy task, we emphasize on optimizing the target task.

Original: To this end, we try to directly align the embedding space between the teacher model and the student model.
Revised: To this end, we try to directly align the embedding space between the teacher model and the student model.

Original: Speciﬁcally, we ﬁrstly train a teacher model with the Arcface loss (Deng et al., 2019), and then initialize the weight of the student’s classiﬁer with the weight of [Figure] Fix [Figure]
Revised: Speciﬁcally, we ﬁrstly train a teacher model with the Arcface loss (Deng et al., 2019a), and then initialize the weight of the student’s classiﬁer with the weight of the teacher’s classiﬁer and ﬁx the weight in the distillation stage.

Original: the teacher’s classiﬁer and ﬁx the weight in the distillation stage.

Original: Using Arcface loss enables our method to beneﬁt from large margin constraints in the distillation procedure.
Revised: Using Arcface loss enables our method to beneﬁt from large margin constraints in the distillation procedure.

Original: The distillation form of Figure 2 (a) can be deﬁned as follows:
Revised: The distillation form of Figure 2 (a) can be deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: f ( m, θ w ty ,x i ) is a margin function, x i is the the embedding of i - th sample in a batch, w ty and w tk are classiﬁer’s weights from the teacher model, y is the class of the x i , k ∈ { 1 , 2 , 3 ..., K } and K is the number of classes in the dataset.
Revised: f ( m, θ w ty ,x i ) is a margin function, x i is the the embedding of i - th sample in a batch, w ty and w tk are classiﬁer’s weights from the teacher model, y is the class of the x i , k ∈ { 1 , 2 , 3 ..., K } and K is the number of classes in the dataset.

Original: θ w t ,x i is the angle between w t and x i .
Revised: θ w t ,x i is the angle between w t and x i .

Original: m denotes the preset hyper-parameter.
Revised: m denotes the preset hyper-parameter.

Original: When we adjust the value of m , the interval among different intre-class samples will be changed.
Revised: When we adjust the value of m , the interval among different intre-class samples will be changed.

Original: Knowledge distillation with multiple teachers: Using an ensemble of teacher models would further boost the performance of knowledge distillation, according to previous work (Asif et al., 2019).
Revised: Knowledge distillation with multiple teachers: Using an ensemble of teacher models would further boost the performance of knowledge distillation, according to previous work (Asif et al., 2019).

Original: Therefore, we here introduce how to implement ProxylessKD with the ensemble of teacher models in Figure 2, which better aligns with a practical face recognition system.
Revised: Therefore, we here introduce how to implement ProxylessKD with the ensemble of teacher models in Figure 2, which better aligns with a practical face recognition system.

Original: To do this, we ﬁrstly train n different teacher models to acquire n embeddings for each input and concatenate the n embeddings to produce a high-dimensional embedding.
Revised: To do this, we ﬁrstly train n different teacher models to acquire n embeddings for each input and concatenate the n embeddings to produce a high-dimensional embedding.

Original: Secondly, we employee a dimensionality reduction layer or the PCA (Principal Component Analysis) method (Wold et al., 1987) to reduce the high-dimensional embedding to adapt to student’s embedding dimensional.
Revised: Secondly, we employee a dimensionality reduction layer or the PCA (Principal Component Analysis) method (Wold et al., 1987) to reduce the high-dimensional embedding to adapt to student’s embedding dimensional.

Original: Finally, we input the embedding after dimensionality reduction into a new classiﬁer and retrain it.
Revised: Finally, we input the embedding after dimensionality reduction into a new classiﬁer and retrain it.

Original: We will do the same operate as the knowledge distillation with single teacher, when we obtain the new classiﬁer.
Revised: We will do the same operate as the knowledge distillation with single teacher, when we obtain the new classiﬁer.

Original: The ensemble of teachers’ classiﬁer can be optimized as
Revised: The ensemble of teachers’ classiﬁer can be optimized as

Original: [Equation]
Revised: [Equation]

Original: E t i , ( i = 1 , 2 , ..., n ) , n ≥ 2 and n ∈ N + , is the embedding of the i - th sample extracted by n teacher models, E T is the dimensionality reduction vector of the i - th sample, concatenation is the operation of concatenating embeddings, ϕ is dimensionality reduction function (i.e., PCA function or the dimensionality reduction layer function).
Revised: E t i , ( i = 1 , 2 , ..., n ) , n ≥ 2 and n ∈ N + , is the embedding of the i - th sample extracted by n teacher models, E T is the dimensionality reduction vector of the i - th sample, concatenation is the operation of concatenating embeddings, ϕ is dimensionality reduction function (i.e., PCA function or the dimensionality reduction layer function).

Original: 3. I NCORPORATING WITH OTHER LOSS FUNCTIONS
Revised: 3. I NCORPORATING WITH OTHER LOSS FUNCTIONS

Original: In Sec. 3.1, we only introduce the classic loss functions (i.e., Equation (1) and (2) in face recognition.
Revised: In Sec. 3.1, we only introduce the classic loss functions (i.e., Equation (1) and (2) in face recognition.

Original: As long as the loss function uses the output of the classiﬁer to calculate the loss (e.g., ArcNegFace (Liu et al., 2019b)) for optimizing the network, the proposed ProxylessKD method can combine with it.
Revised: As long as the loss function uses the output of the classiﬁer to calculate the loss (e.g., ArcNegFace (Liu et al., 2019b)) for optimizing the network, the proposed ProxylessKD method can combine with it.

Original: Therefore, more powerful loss functions can be incorporated into our ProxylessKD to further improve the performance.
Revised: Therefore, more powerful loss functions can be incorporated into our ProxylessKD to further improve the performance.

Original: The uniﬁed form can be deﬁned as follows:
Revised: The uniﬁed form can be deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: x i is the embedding of the i - th sample.
Revised: x i is the embedding of the i - th sample.

Original: w t is the weight of the classiﬁer from the teacher model.
Revised: w t is the weight of the classiﬁer from the teacher model.

Original: N is the number of samples in a batch.
Revised: N is the number of samples in a batch.

Original: We use C ( · ) to represent the loss calculated by various loss function types.
Revised: We use C ( · ) to represent the loss calculated by various loss function types.

Original: Note, it is not restricted to the ﬁeld of face recognition but also applicable to other general classiﬁcation tasks where our ProxylessKD is used for model performance improvement.
Revised: Note, it is not restricted to the ﬁeld of face recognition but also applicable to other general classiﬁcation tasks where our ProxylessKD is used for model performance improvement.

Original: For example, the recent work (Sun et al., 2020) proposed circle loss with excellent results achieved for the ﬁne-grained classiﬁcation tasks.
Revised: For example, the recent work (Sun et al., 2020) proposed circle loss with excellent results achieved for the ﬁne-grained classiﬁcation tasks.

Original: It can be integrated into the proposed ProxylessKD to further boost the performance in the general classiﬁcation tasks.
Revised: It can be integrated into the proposed ProxylessKD to further boost the performance in the general classiﬁcation tasks.

Original:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS

Original: [Table]
Revised: [Table]

Original: Datasets.
Revised: MegaFace 530( [Table] [Table] [Table]

Original:  We adopt the high-quality version namely MS1MV2 reﬁned from MS-Celeb-1M dataset (Guo et al., 2016) by Deng et al.  for training.
Revised: [Table] [Table] We adopt the high-quality version namely MS1MV2 reﬁned from MS-Celeb-1M dataset (Guo et al., 2016) by Deng et al. (2019a) for training.

Original: For testing, we utilize three face veriﬁcation datasets, i.e. LFW (Huang et al., 2008), CPLFW (Zheng & Deng, 2018), CFPFP (Sengupta et al., 2016).
Revised: For testing, we utilize three face veriﬁcation datasets, i.e. LFW (Huang et al., 2008), CPLFW (Zheng & Deng, 2018), CFPFP (Sengupta et al., 2016).

Original: Besides, we also test our proposed method on large-scale image datasets MegaFace (Kemelmacher-Shlizerman et al., 2016), IJB-B (Whitelam et al., 2017) and
Revised: Besides, we also test our proposed method on large-scale image datasets MegaFace (Kemelmacher-Shlizerman et al., 2016), IJB-B (Whitelam et al., 2017) and

Original: IJB-C (Maze et al., 2018)).
Revised: IJB-C (Maze et al., 2018)).

Original: Details about these datasets are shown in Table 1.
Revised: Details about these datasets are shown in Table 1.

Original: Data processing.
Revised: Data processing.

Original: We follow Wang et al.
Revised: We follow Wang et al.

Original: (2018b)
Revised: (2018b)

Original: ; Deng et al. (2019) to generate the normalized face crops ( 112 × 112 ) with ﬁve facial points in the data processing.
Revised: ; Deng et al. (2019a) to generate the normalized face crops ( 112 × 112 ) with ﬁve facial points in the data processing.

Original: All training faces are horizontally ﬂipped with probability 0.5 for data augmentation.
Revised: All training faces are horizontally ﬂipped with probability 0.5 for data augmentation.

Original: Network architecture.
Revised: Network architecture.

Original: In this parper, we set the n=4, i.e., the four models are the ResNet152,
Revised: In this parper, we set the n=4, i.e., the four models are the ResNet152,

Original: ResNet101, AttentionNet92 and DenseNet201 as the ensemble of teacher models, and choose
Revised: ResNet101, AttentionNet92 and DenseNet201 as the ensemble of teacher models, and choose

Original: ResNet18 as the student model.
Revised: ResNet18 as the student model.

Original: After the last convolutional layer, we leverage the FC-BN structure to get the ﬁnal 512-D embedding.
Revised: After the last convolutional layer, we leverage the FC-BN structure to get the ﬁnal 512-D embedding.

Original: In the ensemble procedure of four teacher models, we train again a new dimensionality reduction layer and the classiﬁer layer with the feature that is cascaded from four teacher models as input to acquire a new embedding.
Revised: In the ensemble procedure of four teacher models, we train again a new dimensionality reduction layer and the classiﬁer layer with the feature that is cascaded from four teacher models as input to acquire a new embedding.

Original: This new embedding is applied to knowledge distillation in L2KD, and the new classiﬁer is inherited by student model to do knowledge distillation.
Revised: This new embedding is applied to knowledge distillation in L2KD, and the new classiﬁer is inherited by student model to do knowledge distillation.

Original: Training.
Revised: Training.

Original: All models are trained from scratch with NAG (Nesterov, 1983) and 512 batch size for each teacher training and 1024 for the remaining training procedure.
Revised: All models are trained from scratch with NAG (Nesterov, 1983) and 512 batch size for each teacher training and 1024 for the remaining training procedure.

Original: The momentum is set to 0.
Revised: The momentum is set to 0.

Original: and the weight decay is 4e-5.
Revised: and the weight decay is 4e-5.

Original: The dimension of all embedding is 512.
Revised: The dimension of all embedding is 512.

Original: The initial learning rate is set to 0.1, 0.1, 0.9, 0.35 in the training of the teacher, student, L2KD, and ProxylessKD, respectively.
Revised: The initial learning rate is set to 0.1, 0.1, 0.9, 0.35 in the training of the teacher, student, L2KD, and ProxylessKD, respectively.

Original: The training process for the teacher model is ﬁnished with 8 epochs, and 16 epochs is used for all remaining experiments.
Revised: The training process for the teacher model is ﬁnished with 8 epochs, and 16 epochs is used for all remaining experiments.

Original: We use the cosine decay in all training and Arcface loss as the supervision, in which m = 0.5 and s=64 following Deng et al. 
Revised: We use the cosine decay in all training and Arcface loss as the supervision, in which m = 0.5 and s=64 following Deng et al. (2019a).

Original: All experiments are done on 8x2080Ti
Revised: All experiments are done on 8x2080Ti

Original: GPUs in parallel and implemented by the Mxnet library based on Gluon-Face.
Revised: GPUs in parallel and implemented by the Mxnet library based on Gluon-Face.

Original: Testing.
Revised: Testing.

Original: In practical face recognition or image search, the embeddings of database usually are extracted by a larger model while the embeddings of query images are are extracted by a smaller model.
Revised: In practical face recognition or image search, the embeddings of database usually are extracted by a larger model while the embeddings of query images are are extracted by a smaller model.

Original: Considering this, we should evaluate the consistency of embeddings extracted by the larger model and the smaller model, which represents the performance of different KD methods.
Revised: Considering this, we should evaluate the consistency of embeddings extracted by the larger model and the smaller model, which represents the performance of different KD methods.

Original: Speciﬁcally, in the identiﬁcation task, a large model is used to extract the embeddings of the database, and a small model is used to extract embeddings of the query images.
Revised: Speciﬁcally, in the identiﬁcation task, a large model is used to extract the embeddings of the database, and a small model is used to extract embeddings of the query images.

Original: In the veriﬁcation task, we ﬁrstly calculate the veriﬁcation accuracy using embeddings of image pair extracted by the large and the small model respectively, then calculate the veriﬁcation accuracy using embeddings of image pair extracted by the small and the large model in turn, ﬁnally take the average of them.
Revised: In the veriﬁcation task, we ﬁrstly calculate the veriﬁcation accuracy using embeddings of image pair extracted by the large and the small model respectively, then calculate the veriﬁcation accuracy using embeddings of image pair extracted by the small and the large model in turn, ﬁnally take the average of them.

Original: In particular, we use the same veriﬁcation method on the small datasets (i.e., LFW, CPLFW, and CFP-FP) following
Revised: In particular, we use the same veriﬁcation method on the small datasets (i.e., LFW, CPLFW, and CFP-FP) following

Original: Wang et al.
Revised: Wang et al.

Original: (2018b)
Revised: (2018b)

Original: ; Deng et al. 
Revised: ; Deng et al. (2019a).

Original: We use two kinds of measure methods (i.e., veriﬁcation and identiﬁcation) to test IJB-B, IJB-C dataset and MegaFace dataset.
Revised: We use two kinds of measure methods (i.e., veriﬁcation and identiﬁcation) to test IJB-B, IJB-C dataset and MegaFace dataset.

Original: Note the images in the 1M interference set are extracted by the larger model, and the query images are extracted by the small model in the MegaFace dataset.
Revised: Note the images in the 1M interference set are extracted by the larger model, and the query images are extracted by the small model in the MegaFace dataset.

Original: Meanwhile, we introduce performance when we only use a small model.
Revised: Meanwhile, we introduce performance when we only use a small model.

Original: 4. A BLATION STUDY
Revised: 4. A BLATION STUDY

Original: Results on different loss functions.
Revised: Results on different loss functions.

Original: Our ProxylessKD can be easily combined with the existing loss functions (e.g., L2softmax (Ranjan et al., 2017), Cosface loss (Wang et al., 2018a), Arcface loss (Deng et al., 2019)) to supervise the learning of embedding and achieve better embedding space alignment.
Revised: Our ProxylessKD can be easily combined with the existing loss functions (e.g., L2softmax (Ranjan et al., 2017), Cosface loss (Wang et al., 2018a), Arcface loss (Deng et al., 2019a)) to supervise the learning of embedding and achieve better embedding space alignment.

Original: In Table 2, 3, 4, we show the performance difference of our method under the supervision of different loss functions.
Revised: In Table 2, 3, 4, we show the performance difference of our method under the supervision of different loss functions.

Original: As shown in Table 2, 3, 4, compared with L2softmax and Cosface loss, our method achieves better results with the supervision of the Arcface loss function.
Revised: As shown in Table 2, 3, 4, compared with L2softmax and Cosface loss, our method achieves better results with the supervision of the Arcface loss function.

Original: This proves that our ProxylessKD is able to lift performance by incorporating a powerful loss function.
Revised: This proves that our ProxylessKD is able to lift performance by incorporating a powerful loss function.

Original: Hence, we can foresee that our method will achieve better results with the development of more powerful loss functions in classiﬁcation.
Revised: Hence, we can foresee that our method will achieve better results with the development of more powerful loss functions in classiﬁcation.

Original: [Table]
Revised: [Table]

Original: LFW

Original: [Table]
Revised: [Table]

Original: CFP

Original: [Table]
Revised: [Table]

Original: ProxylessKD

Original: +

Original: [Table]

Original: [Table]

Original: [Table]

Original: ProxylessKD

Original: [Table]

Original: [Table]

Original: [Table]

Original: Results on different margins.
Revised: Results on different margins.

Original: As shown in Figure

Original: [Table] to further illustrate the impact of different margins on different scale datasets [Table] we utilize Arcface loss with different margins as the supervised loss of the proposed ProxylessKD to test its sensitivity to margins.
Revised: As shown in Figure 3, to further illustrate the impact of different margins on different scale datasets, we utilize Arcface loss with different margins as the supervised loss of the proposed ProxylessKD to test its sensitivity to margins.

Original: Red points mark the best results.
Revised: Red points mark the best results.

Original: Speciﬁcally, from Figure 3 (a), (b) [Table] and (c), we observe that margin=0.2 achieves better results than others.
Revised: Speciﬁcally, from Figure 3 (a), (b), and (c), we observe that margin=0.2 achieves better results than others.

Original: Though on the LFW dataset the best result is gained at margin=0.4, the gap is tiny and merely 0.02%, which demonstrates the small margin is more appropriate at the small scale dataset.
Revised: Though on the LFW dataset the best result is gained at margin=0.4, the gap is tiny and merely 0.02%, which demonstrates the small margin is more appropriate at the small scale dataset.

Original: However, on the IJB-B/C and MegaFace that are large scale datasets, we ﬁnd larger margins bring  better results.
Revised: However, on the IJB-B/C and MegaFace that are large scale datasets, we ﬁnd larger margins bring [Figure] better results.

Original: In particular, when the margin is set to 0.5, the same as the setting in training ensemble of teacher’s classiﬁer, the performance is the best.
Revised: In particular, when the margin is set to 0.5, the same as the setting in training ensemble of teacher’s classiﬁer, the performance is the best.

Original: This indicates the performance of ProxylessKD will be better if using a larger margin at a large scale dataset, as shown in Figure 3 (d) ∼ (k).[Figure]
Revised: This indicates the performance of ProxylessKD will be better if using a larger margin at a large scale dataset, as shown in Figure 3 (d) ∼ (k).

Original: 4. C OMPARING WITH OTHER METHODS
Revised: 4. C OMPARING WITH OTHER METHODS

Original: Single and multiple model mode.
Revised: Single and multiple model mode.

Original: We show two evaluation modes in Table 5, 6, 7.
Revised: We show two evaluation modes in Table 5, 6, 7.

Original: L2KD-s and
Revised: L2KD-s and

Original: ProxylessKD-s mean the embeddings are only extracted by the student model (single model mode), and the sufﬁx of “-m” represents the embeddings of database are extracted with a teacher model while the embeddings of query images are extracted by the student model (multiple model mode).
Revised: ProxylessKD-s mean the embeddings are only extracted by the student model (single model mode), and the sufﬁx of “-m” represents the embeddings of database are extracted with a teacher model while the embeddings of query images are extracted by the student model (multiple model mode).

Original: The more detailed information can be found in Sec. 4.1.
Revised: The more detailed information can be found in Sec. 4.1.

Original: [Table]
Revised: [Table]

Original: ) on LFW, CPLFE, CFP-FP datasets. As shown in Table 5, the accuracy on LFW is similar between L2KD and ProxylessKD, but our
Revised: As shown in Table 5, the accuracy on LFW is similar between L2KD and ProxylessKD, but our

Original: ProxylessKD achieves better performance on CPLFW and CFP-FP under two evaluation modes.
Revised: ProxylessKD achieves better performance on CPLFW and CFP-FP under two evaluation modes.

Original: In particular, we achieve 0.91% and 1.32% improvements on CPLFW and CFP-FP under single model mode, and 1.02% and 0.74% better than the L2KD under multiple model mode.
Revised: In particular, we achieve 0.91% and 1.32% improvements on CPLFW and CFP-FP under single model mode, and 1.02% and 0.74% better than the L2KD under multiple model mode.

Original: Note that
Revised: Note that

Original: ProxylessKD is trained with margin=0.2 with Arcface loss and the margin is set to 0.5 in the next experiments. 
Revised: ProxylessKD is trained with margin=0.2 with Arcface loss and the margin is set to 0.5 in the next experiments. [Table]

Original: The MegaFace dataset contains 100K photos of 530 unique individuals from FaceScrub (Ng &
Revised: The MegaFace dataset contains 100K photos of 530 unique individuals from FaceScrub (Ng &

Original: Winkler, 2014) as the probe set and 1M images of 690K different individuals as the gallery set.
Revised: Winkler, 2014) as the probe set and 1M images of 690K different individuals as the gallery set.

Original: On MegaFace, we employ two testing tasks (veriﬁcation and identiﬁcation) under two mode (i.e., single model mode and multiply model mode).
Revised: On MegaFace, we employ two testing tasks (veriﬁcation and identiﬁcation) under two mode (i.e., single model mode and multiply model mode).

Original: In the testing, except the features of gallery images extracted by the teacher model, the features of the images input the gallery each time is also extracted[Table] by the teacher model.
Revised: In the testing, except the features of gallery images extracted by the teacher model, the features of the images input the gallery each time is also extracted by the teacher model.

Original: In Table [Table] we show the performance of L2KD and ProxylessKD.
Revised: In Table 6, we show the performance of L2KD and ProxylessKD.

Original: Though
Revised: Though

Original: MegaFace is a larger scale dataset

Original: [Table] and for a more complex recognition task [Table] our proposed method still achieves better results in Rank-1 and boosts the performance by 1.03% and 0.12% under the single model mode and multiple model mode, respectively.
Revised: MegaFace is a larger scale dataset, and for a more complex recognition task, our proposed method still achieves better results in Rank-1 and boosts the performance by 1.03% and 0.12% under the single model mode and multiple model mode, respectively.

Original: And, in multiple model mode evaluation, it performs better than single model mode
Revised: And, in multiple model mode evaluation, it performs better than single model mode.

Original: [Table]
Revised: [Table]

Original: The IJB-B dataset (Whitelam et al.
Revised: The IJB-B dataset (Whitelam et al.

Original: (2017)) has 1,845 subjects with 21.8K static images and 55 K frames from 7,011 videos.
Revised: (2017)) has 1,845 subjects with 21.8K static images and 55 K frames from 7,011 videos.

Original: There are 12,115 templates with 10,270 authentic matches and 8 M impostor matches in all.
Revised: There are 12,115 templates with 10,270 authentic matches and 8 M impostor matches in all.

Original: The IJB-C dataset (Maze et al., 2018) is the extension of IJB-B, containing 3,531 subjects with 31.3K static images and 117.5K frames from 11.779 videos.
Revised: The IJB-C dataset (Maze et al., 2018) is the extension of IJB-B, containing 3,531 subjects with 31.3K static images and 117.5K frames from 11.779 videos.

Original: There are 23, templates with 19,557 genuine matches and 15,639 impostor matches in all.
Revised: There are 23, templates with 19,557 genuine matches and 15,639 impostor matches in all.

Original: As shown in Table 7, compared with the L2KD, ProxylessKD achieves the best results among all evaluation methods on
Revised: As shown in Table 7, compared with the L2KD, ProxylessKD achieves the best results among all evaluation methods on

Original: IJB-B and IJB-C and more consistent improvement than L2KD in embedding space alignment between teacher and student.
Revised: IJB-B and IJB-C and more consistent improvement than L2KD in embedding space alignment between teacher and student.

Original: We can see the performance of multiple model mode is better than single model mode, which explains the reason why the base database embeddings are extracted by a large model in the practical face recognition.
Revised: We can see the performance of multiple model mode is better than single model mode, which explains the reason why the base database embeddings are extracted by a large model in the practical face recognition.

Original: Especially at @
Revised: Especially at @

Original: FAR = 1 e − 6 on IJB-C, the accuracy of multiple model mode is improved by 10 ∼ 15 % than single model mode.
Revised: FAR = 1 e − 6 on IJB-C, the accuracy of multiple model mode is improved by 10 ∼ 15 % than single model mode.

Original: This explains why we advocate directly aligning the embedding space is more essential due to the base database features are also extracted by a large model in the parctical face recoginition.
Revised: This explains why we advocate directly aligning the embedding space is more essential due to the base database features are also extracted by a large model in the parctical face recoginition.

Original: The experiments prove ProxylessKD is a more effective strategy than the existing practical face recognition method (i.e., L2KD).
Revised: The experiments prove ProxylessKD is a more effective strategy than the existing practical face recognition method (i.e., L2KD).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work we propose a simple yet powerful knowledge distillation method named ProxylessKD, which inherits the teacher’s classiﬁer as the student’s classiﬁer to directly optimize the remaining networks of the student while ﬁxing the classiﬁer’s weights in the training procedure.
Revised: In this work we propose a simple yet powerful knowledge distillation method named ProxylessKD, which inherits the teacher’s classiﬁer as the student’s classiﬁer to directly optimize the remaining networks of the student while ﬁxing the classiﬁer’s weights in the training procedure.

Original: Compared with L2KD, which only considers the intra-class distance and ignores the intra-class distance, our proposed ProxylessKD pays attention to them both.
Revised: Compared with L2KD, which only considers the intra-class distance and ignores the intra-class distance, our proposed ProxylessKD pays attention to them both.

Original: Meanwhile, it can beneﬁt from the large margin of existing constraints, which is new to exiting knowledge distillation research.
Revised: Meanwhile, it can beneﬁt from the large margin of existing constraints, which is new to exiting knowledge distillation research.

Original: Our method can achieve better performance than other distillation methods in most evaluations, proving its effectiveness.
Revised: Our method can achieve better performance than other distillation methods in most evaluations, proving its effectiveness.