Original: D ISCRIMINABILITY D ISTILLATION IN G ROUP R EPRESENTATION L EARNING
Revised: D ISCRIMINABILITY D ISTILLATION IN G ROUP R EPRESENTATION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning group representation is a commonly concerned issue in tasks where the basic unit is a group, set or sequence.
Revised: Learning group representation is a commonly concerned issue in tasks where the basic unit is a group, set or sequence.

Original: Previously, the computer vision community tries to tackle it by aggregating the elements in a group based on an indicator either deﬁned by human such as the quality or saliency of an element, or generated by a black box such as the attention score or output of a RNN.
Revised: Previously, the computer vision community tries to tackle it by aggregating the elements in a group based on an indicator either deﬁned by human such as the quality or saliency of an element, or generated by a black box such as the attention score or output of a RNN.

Original: This article provides a more essential and explicable view.
Revised: This article provides a more essential and explicable view.

Original: We claim the most signiﬁcant indicator to show whether the group representation can be beneﬁted from an element is not the quality, or an inexplicable score, but the discrimiability .
Revised: We claim the most signiﬁcant indicator to show whether the group representation can be beneﬁted from an element is not the quality, or an inexplicable score, but the discrimiability .

Original: Our key insight is to explicitly design the discrimiability using embedded class centroids on a proxy set, and show the discrimiability distribution w.r.t.
Revised: Our key insight is to explicitly design the discrimiability using embedded class centroids on a proxy set, and show the discrimiability distribution w.r.t.

Original: the element space can be distilled by a light-weight auxiliary distillation network (DDNet).
Revised: the element space can be distilled by a light-weight auxiliary distillation network (DDNet).

Original: This procedure is denoted as discriminability distillation learning (DDL).
Revised: This procedure is denoted as discriminability distillation learning (DDL).

Original: We show the proposed DDL can be ﬂexibly plugged into many group based recognition tasks without inﬂuencing the training procedure of the original tasks.
Revised: We show the proposed DDL can be ﬂexibly plugged into many group based recognition tasks without inﬂuencing the training procedure of the original tasks.

Original: Comprehensive experiments on set-to-set face recognition and action recognition valid the advantage of DDL on both accuracy and efﬁciency, and it pushes forward the state-of-the-art results on these tasks by an impressive margin.
Revised: Comprehensive experiments on set-to-set face recognition and action recognition valid the advantage of DDL on both accuracy and efﬁciency, and it pushes forward the state-of-the-art results on these tasks by an impressive margin.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the rapid development of deep learning and the easy access to large-scale group data, recognition tasks using group information have drawn great attention in the computer vision community.
Revised: With the rapid development of deep learning and the easy access to large-scale group data, recognition tasks using group information have drawn great attention in the computer vision community.

Original: The rich information provided by different elements can complement each other to boost the performance of tasks such as face recognition, action recognition, and person re-identiﬁcation (Wang et al., 2017b; Zhong et al., 2018; Girdhar et al., 2017; Simonyan & Zisserman, 2014;
Revised: The rich information provided by different elements can complement each other to boost the performance of tasks such as face recognition, action recognition, and person re-identiﬁcation (Wang et al., 2017b; Zhong et al., 2018; Girdhar et al., 2017; Simonyan & Zisserman, 2014;

Original: Yang et al., 2017; Liu et al., 2019a;
Revised: Yang et al., 2017; Liu et al., 2019a;

Original: Rao et al., 2017b).
Revised: Rao et al., 2017b).

Original: While traditional practice for group-based recognition is to either aggregate the whole set by average (Li et al., 2014; Taigman et al., 2014) or max pooling (Chowdhury et al., 2016), or just sampling randomly (Wang et al., 2016), the fact that certain elements contribute negatively in recognition tasks has been ignored.
Revised: While traditional practice for group-based recognition is to either aggregate the whole set by average (Li et al., 2014; Taigman et al., 2014) or max pooling (Chowdhury et al., 2016), or just sampling randomly (Wang et al., 2016), the fact that certain elements contribute negatively in recognition tasks has been ignored.

Original: Thus, an important issue is to select representatives from sets for efﬁcient group understanding.
Revised: Thus, an important issue is to select representatives from sets for efﬁcient group understanding.

Original: To tackle such cases, previous methods aim at deﬁning the “quality” or “saliency” for each element in a group (Liu et al., 2017c;
Revised: To tackle such cases, previous methods aim at deﬁning the “quality” or “saliency” for each element in a group (Liu et al., 2017c;

Original: Yang et al., 2017;
Revised: Yang et al., 2017;

Original: Rao et al., 2017b; Nikitin et al., 2017).
Revised: Rao et al., 2017b; Nikitin et al., 2017).

Original: The weights for each element can be automatically learned by self-attention.
Revised: The weights for each element can be automatically learned by self-attention.

Original: For example, Liu et al.
Revised: For example, Liu et al.

Original: (2017c) proposes the Quality Aware Network (QAN) to learn quality score for each image inside an image set during network training.
Revised: (2017c) proposes the Quality Aware Network (QAN) to learn quality score for each image inside an image set during network training.

Original: Other works adopt the same idea and extend to speciﬁc tasks such as video-based person re-identiﬁcation
Revised: Other works adopt the same idea and extend to speciﬁc tasks such as video-based person re-identiﬁcation

Original: (Li et al., 2018; Wu et al., 2018) and action recognition (Wang et al., 2018c)
Revised: (Li et al., 2018; Wu et al., 2018) and action recognition (Wang et al., 2018c)

Original: by learning spatial-temporal attentions.
Revised: by learning spatial-temporal attentions.

Original: However, the whole online quality or attention learning procedures are either manually designed or learned through a black box, which lacks explainability.
Revised: However, the whole online quality or attention learning procedures are either manually designed or learned through a black box, which lacks explainability.

Original: In this work, we explore deeper into the underlying mechanism for deﬁning effective elements instead of relying on self-learned attention.
Revised: In this work, we explore deeper into the underlying mechanism for deﬁning effective elements instead of relying on self-learned attention.

Original: Assuming that a base network has already been trained for element-based recognition using class labels, we deﬁne the “discriminability” of one sample by how difﬁcult it is for the network to discriminate its class.
Revised: Assuming that a base network has already been trained for element-based recognition using class labels, we deﬁne the “discriminability” of one sample by how difﬁcult it is for the network to discriminate its class.

Original: As pointed out by Liu et al.
Revised: As pointed out by Liu et al.

Original: (2018) that the feature embedding of elements lies close to the centroid of their corresponding class are the representatives, while features far away or closer to other classes are the confusing ones which are not discriminatative enough.
Revised: (2018) that the feature embedding of elements lies close to the centroid of their corresponding class are the representatives, while features far away or closer to other classes are the confusing ones which are not discriminatative enough.

Original: Inspired by this observation, we identify a successful discriminability indicator by measuring one embedding’s distance with class centroids and compute the ratio of between positive and hardest-negative, where the positive is its distance with its class’s corresponding centroid and the hardest-negative is the closest counterpart.
Revised: Inspired by this observation, we identify a successful discriminability indicator by measuring one embedding’s distance with class centroids and compute the ratio of between positive and hardest-negative, where the positive is its distance with its class’s corresponding centroid and the hardest-negative is the closest counterpart.

Original: This indicator is deﬁned as the discriminability distillation regulation (DDR).
Revised: This indicator is deﬁned as the discriminability distillation regulation (DDR).

Original: Armed with recent theories on the homogeneity between class centroids and projection weights of classiﬁers (Wang et al., 2017a; Liu et al., 2017b; 2018; Deng et al., 2019a), the entire distancemeasuring procedure can be easily accomplished by simply encoding all elements in one group.
Revised: Armed with recent theories on the homogeneity between class centroids and projection weights of classiﬁers (Wang et al., 2017a; Liu et al., 2017b; 2018; Deng et al., 2019a), the entire distancemeasuring procedure can be easily accomplished by simply encoding all elements in one group.

Original: Thus, the DDR scores can be assessed for each element after the training of the base network.
Revised: Thus, the DDR scores can be assessed for each element after the training of the base network.

Original: This assessing procedure is highly ﬂexible without human supervision nor re-training the base network, so it can be adapted to any existing base.
Revised: This assessing procedure is highly ﬂexible without human supervision nor re-training the base network, so it can be adapted to any existing base.

Original: With our explicitly designed discriminability indicator on the training set, the distillation of such discriminability can be successfully performed with a lightweight discriminability distillation network (DDNet), which shows the superiority of our proposed indicator.
Revised: With our explicitly designed discriminability indicator on the training set, the distillation of such discriminability can be successfully performed with a lightweight discriminability distillation network (DDNet), which shows the superiority of our proposed indicator.

Original: We call the whole procedure uniformly as discriminability distillation learning (DDL).
Revised: We call the whole procedure uniformly as discriminability distillation learning (DDL).

Original: The next step is towards ﬁnding a better aggregation policy.
Revised: The next step is towards ﬁnding a better aggregation policy.

Original: At the test phase, all elements are ﬁrstly sent to the light-weight DDNet.
Revised: At the test phase, all elements are ﬁrstly sent to the light-weight DDNet.

Original: Then element features will be weighted aggregated by their
Revised: Then element features will be weighted aggregated by their

Original: DDR score into group representation.
Revised: DDR score into group representation.

Original: Moreover, in order to achieve the trade-off between accuracy and efﬁciency, we can ﬁlter elements by DDR score and only extract element features of high score.
Revised: Moreover, in order to achieve the trade-off between accuracy and efﬁciency, we can ﬁlter elements by DDR score and only extract element features of high score.

Original: Since the base model tends to be heavy, the ﬁlter can save much computation consumption.
Revised: Since the base model tends to be heavy, the ﬁlter can save much computation consumption.

Original: We evaluate the effectiveness of our proposed DDL on several classical yet challenging tasks.
Revised: We evaluate the effectiveness of our proposed DDL on several classical yet challenging tasks.

Original: Comprehensive experiments show the advantage of our method on both recognition accuracy and computation efﬁciency.
Revised: Comprehensive experiments show the advantage of our method on both recognition accuracy and computation efﬁciency.

Original: We achieve state-of-the-art results without modifying the base networks.
Revised: We achieve state-of-the-art results without modifying the base networks.

Original: We highlight our contributions as follows: (1) We deﬁne the discriminability of one element within a group from a more essential and explicable view, and propose an efﬁcient indicator.
Revised: We highlight our contributions as follows: (1) We deﬁne the discriminability of one element within a group from a more essential and explicable view, and propose an efﬁcient indicator.

Original: (2) We verify that a light-weight network has the capacity of distilling discriminability from the assessed elements.
Revised: (2) We verify that a light-weight network has the capacity of distilling discriminability from the assessed elements.

Original: Combining the post-processing with the network, the great computation burden can be saved comparing with existing methods.
Revised: Combining the post-processing with the network, the great computation burden can be saved comparing with existing methods.

Original: (3) We validate the effectiveness of DDL for both efﬁciency and accuracy on set-to-set face recognition and action recognition through extensive studies.
Revised: (3) We validate the effectiveness of DDL for both efﬁciency and accuracy on set-to-set face recognition and action recognition through extensive studies.

Original: State-ofthe-art results can be achieved.
Revised: State-ofthe-art results can be achieved.

Original:  R ELATED WORK 2. S ET - TO - SET RECOGNITION
Revised:  R ELATED WORK 2. S ET - TO - SET RECOGNITION

Original: Set-to-set recognition which utilizes a group of data of a same class, has been proved efﬁcient on various tasks.
Revised: Set-to-set recognition which utilizes a group of data of a same class, has been proved efﬁcient on various tasks.

Original: Set-to-set recognition has drawn much attention these years since the more videos and group datasets are available.
Revised: Set-to-set recognition has drawn much attention these years since the more videos and group datasets are available.

Original: Compared with recognition with a single image, set-to-set recognition can further explore the complementary information among set elements and beneﬁt from it.
Revised: Compared with recognition with a single image, set-to-set recognition can further explore the complementary information among set elements and beneﬁt from it.

Original: Particularly in this paper, we care for the basic task of face and action recognition.
Revised: Particularly in this paper, we care for the basic task of face and action recognition.

Original: Face Recognition.
Revised: Face Recognition.

Original: To tackle set-to-set face recognition problem.
Revised: To tackle set-to-set face recognition problem.

Original: (Wolf et al., 2011; Kalka et al., 2018; Beveridge et al., 2013; Klare et al., 2015), traditional methods directly estimate the feature similarity among sets of feature vectors (Arandjelovic et al., 2005; Harandi et al., 2011; Cevikalp & Triggs, 2010).
Revised: (Wolf et al., 2011; Kalka et al., 2018; Beveridge et al., 2013; Klare et al., 2015), traditional methods directly estimate the feature similarity among sets of feature vectors (Arandjelovic et al., 2005; Harandi et al., 2011; Cevikalp & Triggs, 2010).

Original: Other works seek to aggregate element features by simply applying max pooling (Chowdhury et al., 2016) or average pooling (Li et al., 2014; Taigman et al., 2014) among set features to form a compact representation.
Revised: Other works seek to aggregate element features by simply applying max pooling (Chowdhury et al., 2016) or average pooling (Li et al., 2014; Taigman et al., 2014) among set features to form a compact representation.

Original: However, since most set elements are under unconstrained scenes, huge variations on blur, resolution, and occlusion appear, which will degrade the set feature discrimination.
Revised: However, since most set elements are under unconstrained scenes, huge variations on blur, resolution, and occlusion appear, which will degrade the set feature discrimination.

Original: How to design a proper aggregation method for set face representation has been the key problem for this approach.
Revised: How to design a proper aggregation method for set face representation has been the key problem for this approach.

Original: Recently, a few methods explore the quality or attention mechanism to form set representation.
Revised: Recently, a few methods explore the quality or attention mechanism to form set representation.

Original: GhostVLAD (Zhong et al., 2018) improves traditional VLAD and down weight low quality element features.
Revised: GhostVLAD (Zhong et al., 2018) improves traditional VLAD and down weight low quality element features.

Original: While Rao et al. (2017a) combine LSTM and reinforcement learning to discard low quality element features.
Revised: While Rao et al. (2017a) combine LSTM and reinforcement learning to discard low quality element features.

Original: Liu et al.
Revised: Liu et al.

Original: (2017c) and Yang et al.
Revised: (2017c) and Yang et al.

Original: (2017) introduce attention mechanisms to assign quality scores for different elements and aggregate feature vectors by quality weighted sum.
Revised: (2017) introduce attention mechanisms to assign quality scores for different elements and aggregate feature vectors by quality weighted sum.

Original: To predict the quality score, an online attention network module is added and co-optimized by the target set-to-set recognition task.
Revised: To predict the quality score, an online attention network module is added and co-optimized by the target set-to-set recognition task.

Original: However, the deﬁnition of generated ’quality’ score remains unclear and the learning procedures are learned through a black box, which lacks explainability.
Revised: However, the deﬁnition of generated ’quality’ score remains unclear and the learning procedures are learned through a black box, which lacks explainability.

Original: In our work, we claim that the most signiﬁcant indicator to show whether the group representation can be beneﬁted from an element is not the quality or an inexplicable score, but the discriminability.
Revised: In our work, we claim that the most signiﬁcant indicator to show whether the group representation can be beneﬁted from an element is not the quality or an inexplicable score, but the discriminability.

Original: And a novel discriminability distillation learning procedure is proposed.
Revised: And a novel discriminability distillation learning procedure is proposed.

Original: Action recognition.
Revised: Action recognition.

Original: With the advance of multimedia era, millions of hours of videos are uploaded to video platforms every day, so video understanding task like action recognition has become a popular research topic.
Revised: With the advance of multimedia era, millions of hours of videos are uploaded to video platforms every day, so video understanding task like action recognition has become a popular research topic.

Original: Real world videos contain variable frames, so it is not practical to put the whole video to a memory limited GPU.
Revised: Real world videos contain variable frames, so it is not practical to put the whole video to a memory limited GPU.

Original: The most usual approach for video understanding is to sample frames or clips and design late fusion strategy to form video-level prediction.
Revised: The most usual approach for video understanding is to sample frames or clips and design late fusion strategy to form video-level prediction.

Original: Frame-based methods (Yue-Hei Ng et al., 2015; Simonyan & Zisserman, 2014; Girdhar et al., 2017) ﬁrstly extract frame features and aggregate them.
Revised: Frame-based methods (Yue-Hei Ng et al., 2015; Simonyan & Zisserman, 2014; Girdhar et al., 2017) ﬁrstly extract frame features and aggregate them.

Original: Simonyan & Zisserman (2014) propose the twostream network to simultaneously capture the appearance and motion information.
Revised: Simonyan & Zisserman (2014) propose the twostream network to simultaneously capture the appearance and motion information.

Original: Wang et al.
Revised: Wang et al.

Original: (2017b) add attention module and learn to discard unrelated frames.
Revised: (2017b) add attention module and learn to discard unrelated frames.

Original: Frames-based methods are computational efﬁcient, but only aggregate high-level frame feature tends to limit the model ability to handle complex motion and temporal relation.
Revised: Frames-based methods are computational efﬁcient, but only aggregate high-level frame feature tends to limit the model ability to handle complex motion and temporal relation.

Original: Clip-based method (Tran et al., 2015; 2018; Feichtenhofer et al., 2018) use 3D convolutional neural network to jointly capture spatio-temporal features, which perform better on action recognition.
Revised: Clip-based method (Tran et al., 2015; 2018; Feichtenhofer et al., 2018) use 3D convolutional neural network to jointly capture spatio-temporal features, which perform better on action recognition.

Original: However, clip-based methods highly rely on the dense sample strategy, which introduces huge computational consumption and makes it unpractical to application.
Revised: However, clip-based methods highly rely on the dense sample strategy, which introduces huge computational consumption and makes it unpractical to application.

Original: In this article, we show that by combing our DDL, the clip-based methods can achieve both excellent performance and computation efﬁciency.
Revised: In this article, we show that by combing our DDL, the clip-based methods can achieve both excellent performance and computation efﬁciency.

Original:  D ISCRIMINABILITY D ISTILLATION L EARNING
Revised:  D ISCRIMINABILITY D ISTILLATION L EARNING

Original: In this section, we ﬁrst formulate the problem of group representation learning in section 3.1 and then deﬁne the discriminability distillation regulation (DDR) in section 3.2.
Revised: In this section, we ﬁrst formulate the problem of group representation learning in section 3.1 and then deﬁne the discriminability distillation regulation (DDR) in section 3.2.

Original: Next, we introduce the whole discriminability distillation learning (DDL) procedure in section 3.3.
Revised: Next, we introduce the whole discriminability distillation learning (DDL) procedure in section 3.3.

Original: In sections 3.4 and 3.5, we discuss the aggregation method and the advantage of our DDL, respectively.
Revised: In sections 3.4 and 3.5, we discuss the aggregation method and the advantage of our DDL, respectively.

Original: 3. F ORMULATION OF G ROUP R EPRESENTATION L EARNING
Revised: 3. F ORMULATION OF G ROUP R EPRESENTATION L EARNING

Original: Group representation learning focuses on formulating a uniform representation for a whole set of elements.
Revised: Group representation learning focuses on formulating a uniform representation for a whole set of elements.

Original: The core of either veriﬁcation task or classiﬁcation task is how to aggregate features of a given element group.
Revised: The core of either veriﬁcation task or classiﬁcation task is how to aggregate features of a given element group.

Original: Deﬁne f
Revised: Deﬁne f

Original: i as the embedded feature of element
Revised: i as the embedded feature of element

Original: I i in a group I S , then the uniform feature representation of the whole group is
Revised: I i in a group I S , then the uniform feature representation of the whole group is

Original: [Equation]
Revised: [Equation]

Original: where G indicates the feature aggregation module.
Revised: where G indicates the feature aggregation module.

Original: While previous research has revealed that conducting G with quality scores (Liu et al., 2017c) has priority over simple aggregation, this kind of methods are not explainable.
Revised: While previous research has revealed that conducting G with quality scores (Liu et al., 2017c) has priority over simple aggregation, this kind of methods are not explainable.

Original: Different from the general quality learning with attention mechanism or manual annotation, we propose a discriminability distillation learning (DDL) process to generate the discriminability of feature representation.
Revised: Different from the general quality learning with attention mechanism or manual annotation, we propose a discriminability distillation learning (DDL) process to generate the discriminability of feature representation.

Original: Based on discriminability distillation regulation (DDR) formulation, our method is practical for learning group representation and can be easily plugged into popular recognition frameworks.
Revised: Based on discriminability distillation regulation (DDR) formulation, our method is practical for learning group representation and can be easily plugged into popular recognition frameworks.

Original: 3. D ISCRIMINABILITY D ISTILLATION R EGULATION
Revised: 3. D ISCRIMINABILITY D ISTILLATION R EGULATION

Original: Towards learning efﬁcient and accurate G , we design the discriminability distillation regulation (DDR) to generate the discriminability to replace the traditional quality score.
Revised: Towards learning efﬁcient and accurate G , we design the discriminability distillation regulation (DDR) to generate the discriminability to replace the traditional quality score.

Original: In DDR, we jointly consider the feature space distribution and explicitly distill the discriminability by encoding the intra-class distance and inter-class distance with class centroids.
Revised: In DDR, we jointly consider the feature space distribution and explicitly distill the discriminability by encoding the intra-class distance and inter-class distance with class centroids.

Original: Let X denote the training set with the identities C and W j , j ∈ [1 , C ] is the class centroids of the ﬁnal classiﬁcation layer.
Revised: Let X denote the training set with the identities C and W j , j ∈ [1 , C ] is the class centroids of the ﬁnal classiﬁcation layer.

Original: For feature f i , i ∈
Revised: For feature f i , i ∈

Original: [1 , s ] with class c where s denotes the size of X , the intra-class distance and inter-
Revised: [1 , s ] with class c where s denotes the size of X , the intra-class distance and inter-

Original: class
Revised: class

Original: [Figure]
Revised: [Figure]

Original: distance are formulated as
Revised: distance are formulated as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The intra-class distance C ic and inter-class distance C ij are shown in Figure 2.
Revised: The intra-class distance C ic and inter-class distance C ij are shown in Figure 2.

Original: After training the base model on classiﬁcation task, features of the elements from the same class are projected to hyperspace tightly in order to form an explicit decision boundary.
Revised: After training the base model on classiﬁcation task, features of the elements from the same class are projected to hyperspace tightly in order to form an explicit decision boundary.

Original: Furthermore, elements close to the centroid of their corresponding class are the representative ones, while elements far away from their corresponding class or closer to other classes are the confusing ones which are not discriminative enough.
Revised: Furthermore, elements close to the centroid of their corresponding class are the representative ones, while elements far away from their corresponding class or closer to other classes are the confusing ones which are not discriminative enough.

Original: Based on this observation, we deﬁne the discriminability Q i of f i as:
Revised: Based on this observation, we deﬁne the discriminability Q i of f i as:

Original: [Equation]
Revised: [Equation]

Original: i.e., the ratio of feature distance between the centroids of its class and the hardest-negative class.
Revised: i.e., the ratio of feature distance between the centroids of its class and the hardest-negative class.

Original: Considering the variant number of elements in different groups, we further normalize the discriminability by:
Revised: Considering the variant number of elements in different groups, we further normalize the discriminability by:

Original: [Equation]
Revised: [Equation]

Original: where τ ( · ) , µ ( · ) and σ ( · ) denote the sigmoid function, the mean value and the standard deviation value of { Q j | j ∈ [1 , s ] } , respectively.
Revised: where τ ( · ) , µ ( · ) and σ ( · ) denote the sigmoid function, the mean value and the standard deviation value of { Q j | j ∈ [1 , s ] } , respectively.

Original: We denote D i as the discriminability distillation regulation (DDR) score.
Revised: We denote D i as the discriminability distillation regulation (DDR) score.

Original: Cooperated with the feature space distribution, the DDR score D i is more interpretable and reasonable than the quality score in traditional quality learning.
Revised: Cooperated with the feature space distribution, the DDR score D i is more interpretable and reasonable than the quality score in traditional quality learning.

Original: It can discriminate features better by explicitly encoding the intra- and inter-class distances with class centroids.
Revised: It can discriminate features better by explicitly encoding the intra- and inter-class distances with class centroids.

Original: 3. D ISCRIMINABILITY D ISTILLATION L EARNING
Revised: 3. D ISCRIMINABILITY D ISTILLATION L EARNING

Original: From section 3.2, given a training dataset θ and a corresponding model, the DDR score D i of f i can be naturally computed by Eq (2)-(4).
Revised: From section 3.2, given a training dataset θ and a corresponding model, the DDR score D i of f i can be naturally computed by Eq (2)-(4).

Original: However, in group representation learning, D i is unavailable to test set T without W j .
Revised: However, in group representation learning, D i is unavailable to test set T without W j .

Original: In order to better embed D i into the group representation learning, we introduce the discriminability distillation learning (DDL) to generate an approximated DDR score ˆ D i for D i .
Revised: In order to better embed D i to the group representation learning procedure, we introduce the discriminability distillation learning (DDL) to generate an approximated DDR score ˆ D i for D i .

Original: Given an arbitrary CNN architecture N and an image
Revised: Given an arbitrary CNN architecture N and an image

Original: I i , ˆ D i can be approximated by
Revised: I i , ˆ D i can be approximated by

Original: [Equation]
Revised: [Equation]

Original: where θ is the parameter of the light-weight DDNet N .
Revised: where θ is the parameter of the light-weight DDNet N .

Original: At the training stage, we apply mean squared error between ˆ D i and target D i as
Revised: At the training stage, we apply mean squared error between ˆ D i and target D i as

Original: [Equation]
Revised: [Equation]

Original: where N is the batch size.
Revised: where N is the batch size.

Original: 3. F EATURE A GGREGATION G
Revised: 3. F EATURE A GGREGATION G

Original: At the test stage, we can generate ˆ D i via Eq (5) for test element
Revised: At the test stage, we can generate ˆ D i via Eq (5) for test element

Original: I i in the given element set I S .
Revised: I i in the given element set I S .

Original: The feature aggregation G in Eq (1) can be formulated as
Revised: The feature aggregation G in Eq (1) can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where n is the element number of I S , and ˆ R i is the re-scaled DDR score of ˆ D i via
Revised: where n is the element number of I S , and ˆ R i is the re-scaled DDR score of ˆ D i via

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In Eq (8), we normalize the DDR score of element set I S between 0 and 1 to ensure a same range for element sets with different lengths.
Revised: In Eq (8), we normalize the DDR score of element set I S between 0 and 1 to ensure a same range for element sets with different lengths.

Original: K and B are formulated as
Revised: K and B are formulated as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 3. A DVANTAGE OF D ISCRIMINABILITY D ISTILLATION L EARNING
Revised: 3. A DVANTAGE OF D ISCRIMINABILITY D ISTILLATION L EARNING

Original: Different from the subjective quality judgment of an image or quality learning via attention mechanism, we explicitly assign discriminability for each image via the feature space distribution.
Revised: Different from the subjective quality judgment of an image or quality learning via attention mechanism, we explicitly assign discriminability for each image via the feature space distribution.

Original: By jointly considering the inter- and intra-class distances with class centroids, DDL can effectively approximate how discriminative a feature is.
Revised: By jointly considering the inter- and intra-class distances with class centroids, DDL can effectively approximate how discriminative a feature is.

Original: According to aggregating more information from features with high discriminability in an element set, the recognition performance can be signiﬁcantly boosted.
Revised: According to aggregating more information from features with high discriminability in an element set, the recognition performance can be signiﬁcantly boosted.

Original: With the assistant of DDL, both hard samples (the model is easy to fail on them) or low-quality images (the model is easy to give wrong predict result) will be weakened.
Revised: With the assistant of DDL, both hard samples (the model is easy to fail on them) or lowquality images (the model is easy to give wrong predict result) can be solved more easily.

Original: For face recognition, DDL can signiﬁcantly improve the performance by concentrating the discriminative information.
Revised: For face recognition, DDL can signiﬁcantly improve the performance by concentrating the discriminative information.

Original: Forvideo action recognition, DDL can further accelerate the pipeline by eliminating the frames with insufﬁcient information.
Revised: For action recognition, DDL can further accelerate the pipeline by eliminating the frames with insufﬁcient information.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTS 4. S ET - TO -S ET F ACE R ECOGNITION D ATASETS AND P ROTOCOLS
Revised:  E XPERIMENTS 4. S ET - TO -S ET F ACE R ECOGNITION D ATASETS AND P ROTOCOLS

Original: In this section, we evaluate DDL for set-to-set face recognition on four datasets including two video sets: the YouTube Face (YTF)
Revised: In this section, we evaluate DDL for set-to-set face recognition on four datasets including two video sets: the YouTube Face (YTF)

Original: Wolf et al.
Revised: Wolf et al.

Original: (2011), iQIYI-VID-FACE iQIYI (2019); and two templatebased sets: the IARPA Janus Benchmark A (IJB-A)
Revised: (2011), iQIYI-VID-FACE iQIYI (2019); and two templatebased sets: the IARPA Janus Benchmark A (IJB-A)

Original: Klare et al. (2015) and the IARPA Janus Benchmark C (IJB-C).
Revised: Klare et al. (2015) and the IARPA Janus Benchmark C (IJB-C).

Original: YTF :
Revised: YTF :

Original: The YouTube Face dataset includes 3425 videos of 1595 identities with an average of 2.
Revised: The YouTube Face dataset includes 3425 videos of 1595 identities with an average of 2.

Original: videos per identity.
Revised: videos per identity.

Original: The videos vary from 48 frames to 6,070 frames.
Revised: The videos vary from 48 frames to 6,070 frames.

Original: It is evaluated with the openset protocol.
Revised: It is evaluated with the openset protocol.

Original: We report the 1:1 face veriﬁcation rate of the given 5,000 video pairs in our experiments without ﬁne-tuning.
Revised: We report the 1:1 face veriﬁcation rate of the given 5,000 video pairs in our experiments without ﬁne-tuning.

Original: iQIYI-VID-FACE :
Revised: iQIYI-VID-FACE :

Original: The iQIYI-VID-FACE dataset iQIYI (2019) aims to identify the person in entertainment video by face images.
Revised: The iQIYI-VID-FACE dataset iQIYI (2019) aims to identify the person in entertainment video by face images.

Original: It is the largest video face recognition test benchmark so far, containing 643,816 video clips of 10,034 identities.
Revised: It is the largest video face recognition test benchmark so far, containing 643,816 video clips of 10,034 identities.

Original: The test protocol is 1:1 veriﬁcation and the True
Revised: The test protocol is 1:1 veriﬁcation and the True

Original: Positive Rate (TAR) under False Positive Rate (FAR) at 0.01% is reported.
Revised: Positive Rate (TAR) under False Positive Rate (FAR) at 0.01% is reported.

Original: IJB-A : The IARPA Janus Benchmark A (IJB-A)
Revised: IJB-A : The IARPA Janus Benchmark A (IJB-A)

Original: Klare et al.
Revised: Klare et al.

Original: (2015) is an unconstrained face recognition benchmark, containing 25, 813 faces images of 500 identities.
Revised: (2015) is an unconstrained face recognition benchmark, containing 25, 813 faces images of 500 identities.

Original: It is a template-based test benchmark where both still images and video frames are included in templates.
Revised: It is a template-based test benchmark where both still images and video frames are included in templates.

Original: We report 1:1 veriﬁcation results by following ofﬁcial protocol.
Revised: We report 1:1 veriﬁcation results by following ofﬁcial protocol.

Original: IJB-C :
Revised: IJB-C :

Original: The IARPA Janus Benchmark C (IJB-C) is an extended version of IJB-A. It has 140,faces images of 3, 531 subjects, so we can test under more strict False Accept Rates (FAR).
Revised: The IARPA Janus Benchmark C (IJB-C) is an extended version of IJB-A. It has 140,faces images of 3, 531 subjects, so we can test under more strict False Accept Rates (FAR).

Original: Since the images in IJB-C dataset have large variations, it is regarded as a challenging set-to-set face recognition benchmark.
Revised: Since the images in IJB-C dataset have large variations, it is regarded as a challenging set-to-set face recognition benchmark.

Original: T HE D ETAILS OF DISCRIMINABILITY DISTILLATION NETWORK
Revised: T HE D ETAILS OF DISCRIMINABILITY DISTILLATION NETWORK

Original: All set images will be sent to the DDN at the inference stage, so the computation burden for DDN is sensitive.
Revised: All set images will be sent to the DDN at the inference stage, so the computation burden for DDN is sensitive.

Original: To make the inference stage efﬁcient, we design the DDN to be light-weighted.
Revised: To make the inference stage efﬁcient, we design the DDN to be light-weighted.

Original: The typical
Revised: The typical

Original: DDN is a channel reduced ResNet-18 network, which only introduces 81.9 Mﬂops computation for 112 × 112 input.
Revised: DDN is a channel reduced ResNet-18 network, which only introduces 81.9 Mﬂops computation for 112 × 112 input.

Original: Compared to the base feature extract model which often reaches 24 Gﬂops (ResNet101), it is super-efﬁcient.
Revised: Compared with the base feature extract model which often reaches 24 Gﬂops (ResNet-101), it is super-efﬁcient.

Original: More discussion about the DDN architecture can be found in the appendix.
Revised: More discussions on the DDN architecture refer to the appendix.

Original: E VALUATION ON Y OU T UBE F ACE AND IQIYI-VID-FACE
Revised: E VALUATION ON Y OU T UBE F ACE AND IQIYI-VID-FACE

Original: As shown in Table 1, our DDL achieves state-of-the-art performance on the Youtube Face benchmark.
Revised: As shown in Table 1, our DDL achieves state-of-the-art performance on the Youtube Face benchmark.

Original: It outperforms Deng et al.
Revised: It outperforms Deng et al.

Original: (2019a) by 1.17% and other set-to-set face recognition methods byan impressive margin.
Revised: (2019a) by 1.17% and other set-to-set face recognition methods by impressive margins.

Original: Since we train a stronger base model, the baseline average method is higher
Revised: Since we train a stronger base model, the baseline average method is higher

Original: [Table] [Table]
Revised: [Table] [Table]

Original: than Deng et al. (2019a), butfrom the comparison for  different aggregation strategies like average pooling, DDL can boost performance for 0.92%, which indicates DDL has learned a meaningful pattern for discriminability.
Revised: than Deng et al. (2019a) [Table] but for the comparisons with different aggregation strategies like average pooling, DDL can boost performance for 0.92%, which indicates DDL has learned a meaningful pattern for discriminability.

Original: As a post-training module, DDL can cooperate with any pre-trained model.
Revised: As a post-training module, DDL can cooperate with any pre-trained model.

Original: To prove the robustness and good generalization ability for our DDL, we re-implement more baseline and combine them with DDL, the results are shown in the appendix.
Revised: To prove the robustness and good generalization ability of our DDL, we re-implement more baseline and combine them with DDL, the results are shown in the appendix.

Original: Since the results on YouTube Face benchmark tends to be saturated, we test our DDL on a challenging video face veriﬁcation dataset IQIYI-VID-FACE, which is of large scale and more close to industrial application.
Revised: Since the results on YouTube Face benchmark tends to be saturated, we test our DDL on a challenging video face veriﬁcation dataset IQIYI-VID-FACE, which is of large scale and more close to industrial application.

Original: Compared to average pooling, DDL achieves 3.21% performance gain.
Revised: Compared to average pooling, DDL achieves 3.21% performance gain.

Original: And only aggregate the top 1 discriminability score feature can still achieve near all frames average performance, which shows our DDL has selected the most discriminate element in the set.
Revised: And only aggregate the top 1 discriminability score feature can still achieve near all frames average performance, which shows our DDL has selected the most discriminate element in the set.

Original: By combing
Revised: By combing

Original: PolyNet base model, our DDL achieves the state-of-the-arts performance.
Revised: PolyNet base model, our DDL achieves the state-of-the-arts performance.

Original: What’s more, the good performance of only aggregate top 1 discriminability frame shows that, we needn’t extract all frames by a heavy base model, but only extract the most discriminate one during inference.
Revised: What’s more, the good performance of only aggregate top 1 discriminability frame shows that, we needn’t extract all frames by a heavy base model, but only extract the most discriminate one during inference.

Original: Since the computation consumption for the DDN is efﬁcient, usually only one of tenths of the heavy feature extract model, DDL makes a huge difference for industrial video face recognition application.
Revised: Since the computation consumption for the DDN is efﬁcient, usually only one of tenths of the heavy feature extract model, DDL makes a huge difference for industrial video face recognition application.

Original: E VALUATION ON IJB-A AND IJB-C
Revised: E VALUATION ON IJB-A AND IJB-C

Original: Table 3 and Table 4 show the results on the IJB-A and IJB-C dataset for different methods.
Revised: Table 3 and Table 4 show the results on the IJB-A and IJB-C dataset for different methods.

Original: From the two tables, we can see that our DDL improves veriﬁcation performance by a convincing margin with average pooling for both two benchmarks, especially under severe FAR at 0.001% by 1.94% and FAR at 0.0001% by 6.88% on IJB-C. Compared with IJB-A, IJB-C has more images and covers more variations among images, such as pose, blur, resolution, and conditions.
Revised: From the two tables, we can see that our DDL improves veriﬁcation performance by a convincing margin with average pooling for both two benchmarks, especially under severe FAR at 0.001% by 1.94% and FAR at 0.0001% by 6.88% on IJB-C. Compared with IJB-A, IJB-C has more images and covers more variations among images, such as pose, blur, resolution, and conditions.

Original: So the performance gain with DDL is huger.
Revised: So the performance gain with DDL is huger.

Original: Compared with the state-of-the-art methods, our DDL improves IJB-A by 4.83% at FAR =0.001 and
Revised: Compared with the state-of-the-art methods, our DDL improves IJB-A by 4.83% at FAR =0.001 and

Original: IJB-C by 4.89% at FAR = 0.0001%.
Revised: IJB-C by 4.89% at FAR = 0.0001%.

Original: These indicate the effectiveness and robustness of our DDL.
Revised: These indicate the effectiveness and robustness of our DDL.

Original: What’s more, unlike many previous methods that need to ﬁne-tune with the base model on set-to-set recognition training datasets, the supervision for DDL training is the feature discriminability on the same training set with the base model, so the ﬁne-tune process is no need.
Revised: What’s more, unlike many previous methods that need to ﬁne-tune with the base model on set-to-set recognition training datasets, the supervision for DDL training is the feature discriminability on the same training set with the base model, so the ﬁne-tune process is no need.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: To qualitatively evaluate the discriminability pattern learned by our DDL, we visualize the discriminability score distribution for two template images in IJB-C datasets.
Revised: To qualitatively evaluate the discriminability pattern learned by our DDL [Table] we visualize the discriminability score distribution for two template images in IJB-C datasets.

Original: As shown in Figure 4, DDL can effectively identity image discriminability.
Revised: As shown in Figure 4, DDL can effectively identity image discriminability.

Original: Pictures with large pose, visual blur, occlusion
Revised: Pictures with large pose, visual blur, occlusion

Original: [Table]
Revised: [Table]

Original: and incomplete content are regarded to be low discriminability.
Revised: and incomplete content are regarded to be low discriminability.

Original: The efﬁcient discriminability judgment ability for our DDL leading to an extraordinary performance on set-to-set face recognition problem.
Revised: The efﬁcient discriminability judgment ability for our DDL leading to an extraordinary performance on set-to-set face recognition problem.

Original: More examples can be found in the appendix.
Revised: More examples can be found in the appendix.

Original: 4. V IDEO ACTION RECOGNITION D ATASETS
Revised: 4. V IDEO ACTION RECOGNITION D ATASETS

Original: ActivityNet-1.2.
Revised: ActivityNet-1.2.

Original: The ActivityNet-1.2 dataset Caba Heilbron et al.
Revised: The ActivityNet-1.2 dataset Caba Heilbron et al.

Original: (2015) contains 4,819 training videos and 2,383 validation videos for 100 action class.
Revised: (2015) contains 4,819 training videos and 2,383 validation videos for 100 action class.

Original: The duration of those videos varies fromseconds to 4 minutes and the average video length is 2 minutes.
Revised: The duration of those videos varies fromseconds to 4 minutes and the average video length is 2 minutes.

Original: We carefully remove videos with multi-label.
Revised: We carefully remove videos with multi-label.

Original: Frames are extracted from videos and resized the width to 240 pixels.
Revised: Frames are extracted from videos and resized the width to 240 pixels.

Original: It is an untrimmed dataset, so large visual and content variation is there.
Revised: It is an untrimmed dataset, so large visual and content variation is there.

Original: Kinetics-700.
Revised: Kinetics-700.

Original: Kinetics Kay et al.
Revised: Kinetics Kay et al.

Original: (2017) is a well-trimmed and popular action recognition benchmark.
Revised: (2017) is a well-trimmed and popular action recognition benchmark.

Original: Since around 2% videos of Kinetics-600 and 12% of videos of Kinetics-400 can’t get access from youtube, we conduct experiments on the Kinetics-700 datasets.
Revised: Since around 2% videos of Kinetics-600 and 12% of videos of Kinetics-400 can’t get access from youtube, we conduct experiments on the Kinetics-700 datasets.

Original: Kinetics-700 containsclasses and 650k videos, and each action class has at least 600 video clips.
Revised: Kinetics-700 containsclasses and 650k videos, and each action class has at least 600 video clips.

Original: Each clip is human annotated with a single action class and lasts around 10s.
Revised: Each clip is human annotated with a single action class and lasts around 10s.

Original: [Figure]
Revised: [Figure]

Original: T e m p
Revised: T e m p

Original: l
Revised: l

Original: a t e
Revised: a t e

Original: T e m p
Revised: T e m p

Original: l
Revised: l

Original: a t e
Revised: a t e

Original: B ASE MODEL AND TEST PROTOCOL
Revised: B ASE MODEL AND TEST PROTOCOL

Original: To combine and compare with DDL, we choose three clips-based 3D CNN models 3D-ResNetHara et al.
Revised: To combine and compare with DDL, we choose three clips-based 3D CNN models 3D-ResNetHara et al.

Original: , SlowFast-50 Feichtenhofer et al.
Revised: , SlowFast-50 Feichtenhofer et al.

Original: (2018) and R(2+1)D-50 Tran et al.
Revised: (2018) and R(2+1)D-50 Tran et al.

Original: To test a video, they ﬁrst split the whole videos into clips and predict individually, then clips level results will average pooling to the whole video’s prediction.
Revised: To test a video, they ﬁrst split the whole videos into clips and predict individually, then clips level results will average pooling to the whole video’s prediction.

Original: All three methods hugely rely on the dense sampling strategy when testing, which is quite a computation consumed.
Revised: All three methods hugely rely on the dense sampling strategy when testing, which is quite a computation consumed.

Original: For our DDL, we ﬁrst ﬁlter clips by the light-weight DDNet and only extract features and aggregate for the Top-K discriminability clips.
Revised: For our DDL, we ﬁrst ﬁlter clips by the light-weight DDNet and only extract features and aggregate for the Top-K discriminability clips.

Original: The K value selected for Activitiynet-1.2 and Kinetics-700 isand 5 respectively, for that they achieve the best efﬁciency and accuracy trade-off in our experiments.
Revised: The K value selected for Activitiynet-1.2 and Kinetics-700 isand 5 respectively, for that they achieve the best efﬁciency and accuracy trade-off in our experiments.

Original: For comparison, we randomly or uniformly sample K clips in videos.
Revised: For comparison, we randomly or uniformly sample K clips in videos.

Original: Dense sampling is also used to compare.
Revised: Dense sampling is also used to compare.

Original: E VALUATION ON A CTIVITY N ET -1. 
Revised: E VALUATION ON A CTIVITY N ET -1.

Original: As shown in table 5, DDL improves recognition performance for all baseline model.
Revised: As shown in table 5, DDL improves recognition performance for all baseline model.

Original: For the stateof-the-arts clips-based model SlowFast, combing with DDL achieves around 4% accuracy gain compared to randomly or uniformly sample.
Revised: For the stateof-the-arts clips-based model SlowFast, combing with DDL achieves around 4% accuracy gain compared to randomly or uniformly sample.

Original: What’s more, DDL can even outperform dense evaluation along with the whole video by a huge margin.
Revised: What’s more, DDL can even outperform dense evaluation along with the whole video by a huge margin.

Original: It is reasonable that DDL can select and aggregate clips with the highest discriminability, which avoids video clips with visual blur or ambiguous content to pollute the video-level prediction.
Revised: It is reasonable that DDL can select and aggregate clips with the highest discriminability, which avoids video clips with visual blur or ambiguous content to pollute the video-level prediction.

Original: What’s more, by combing DDL, the inference process can hugely speed up.
Revised: What’s more, by combing DDL, the inference process can hugely speed up.

Original: For testing a two minute videos and SlowFast-50 model as example, original dense sampling strategies will introduce
Revised: For testing a two minute videos and SlowFast-50 model as example, original dense sampling strategies will introduce

Original: Glops × 60 computation , but for DDL enhanced, only Top discriminability videos needed to predict by the heavy base model, the total consumption is 1.7 Glops × 60 for extract the discriminability score plus 9 × 17 Gﬂops for predicting the top discriminability clips, which is 4x speed up.
Revised: Glops × 60 computation , but for DDL enhanced, only Top discriminability videos needed to predict by the heavy base model, the total consumption is 1.7 Glops × 60 for extract the discriminability score plus 9 × 17 Gﬂops for predicting the top discriminability clips, which is 4x speed up.

Original: E VALUATION ON K INETICS 700.
Revised: E VALUATION ON K INETICS 700.

Original: As shown in the Table 6, DDL outperforms randomly sample by 1.84 % and uniformly sample by 2.18 %.
Revised: As shown in the Table 6, DDL outperforms randomly sample by 1.84 % and uniformly sample by 2.18 %.

Original: For dense sample, DDL can achieve 0.86% performance gain with 6x speed up.
Revised: For dense sample, DDL can achieve 0.86% performance gain with 6x speed up.

Original: Since
Revised: Since

Original: Kinetics-700 is well trimmed, the improvement is not so large compared to the untrimmed video dataset ActivityNet-1.2.
Revised: Kinetics-700 is well trimmed, the improvement is not so large compared to the untrimmed video dataset ActivityNet-1.2.

Original: However, as a post-training module, DDL can cooperate with any clipbased 3D action recognition model without base model re-trained and achieve a more efﬁcient and accurate inference, which is very suitable for real-world video understanding applications.
Revised: However, as a post-training module, DDL can cooperate with any clipbased 3D action recognition model without base model re-trained and achieve a more efﬁcient and accurate inference, which is very suitable for real-world video understanding applications.

Original: C ROSS DATASET AND MODEL EVALUATION .
Revised: C ROSS DATASET AND MODEL EVALUATION .

Original: In the last two sections, we train specialized DDNet for the different datasets and base models respectively, to demonstrate the generalization ability of our DDL, we do the cross dataset and cross model evolution experiment.
Revised: In the last two sections, we train specialized DDNet for the different datasets and base models respectively, to demonstrate the generalization ability of our DDL, we do the cross dataset and cross model evolution experiment.

Original: Action recognition model R(2+1)D for ActivityNet-1.2 dataset is combined with DDL trained with Kinetics-700 and base model SlowFast, the results are shown in table 7.
Revised: Action recognition model R(2+1)D for ActivityNet-1.2 dataset is combined with DDL trained with Kinetics-700 and base model SlowFast, the results are shown in table 7.

Original: It can be found that through training with the different dataset and base model causes a performance drop, DDL can still improve about 1% accuracy for dense sampling.
Revised: It can be found that through training with the different dataset and base model causes a performance drop, DDL can still improve about 1% accuracy for dense sampling.

Original: These results
Revised: These results

Original: [Table] [Table]
Revised: [Table] [Table]

Original: on the validation set and is the average of top1 and top5 accuracy.
Revised: on the validation set and is the average of top1 and top5 accuracy.

Original: Dense sample strategy sample 10 clips along the temporal axis and random crop 3 clips on the spatial axis.
Revised: Dense sample strategy sample 10 clips along the temporal axis and random crop 3 clips on the spatial axis.

Original: show that the discriminability pattern learned by DDL is robust and well to generalize to other datasets.
Revised: show that the discriminability pattern learned by DDL is robust and well to generalize to other datasets.

Original: What’s more, the generalization ability can avoid the training DDL model for each action model and each dataset, making the DDL more efﬁcient and practical.
Revised: What’s more, the generalization ability can avoid the training DDL model for each action model and each dataset, making the DDL more efﬁcient and practical.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have proposed a novel post-processing module called DDL for all group-based recognition tasks.
Revised: In this paper, we have proposed a novel post-processing module called DDL for all group-based recognition tasks.

Original: We explicitly deﬁne the discriminability with observations on feature embedding, then apply a light-weight network for quality distillation and feature aggregation.
Revised: We explicitly deﬁne the discriminability with observations on feature embedding, then apply a light-weight network for quality distillation and feature aggregation.

Original: We identify the advantage of our proposed methods in the following aspects: (1) The entire discriminability distillation is performed without modifying the pre-trained base network, which is highly ﬂexible comparing with existing quality-aware or attention methods.
Revised: We identify the advantage of our proposed methods in the following aspects: (1) The entire discriminability distillation is performed without modifying the pre-trained base network, which is highly ﬂexible comparing with existing quality-aware or attention methods.

Original: (2) Our distillation network is extremely lightweighted with great computational cost salvage.
Revised: (2) Our distillation network is extremely lightweighted with great computational cost salvage.

Original: (3) With our DDL and feature aggregation, we achieve state-of-the-art results on set-to-set face recognition and action recognition tasks.
Revised: (3) With our DDL and feature aggregation, we achieve state-of-the-art results on set-to-set face recognition and action recognition tasks.

Original: A A PPENDIX I MPLEMENTATION D ETAILS FOR SET - TO - SET FACE RECOGNITION
Revised: A A PPENDIX I MPLEMENTATION D ETAILS FOR SET - TO - SET FACE RECOGNITION

Original: Pre-processing : RetinaFace Deng et al. (2019b) is used to detect faces and their corresponding landmarks for all datasets.
Revised: Pre-processing : RetinaFace Deng et al. (2019b) is used to detect faces and their corresponding landmarks for all datasets.

Original: Images are aligned to 112 × 112 by similarity transformation based on their landmarks.
Revised: Images are aligned to 112 × 112 by similarity transformation based on their landmarks.

Original: Training : We train our base recognition model and DDL on MS-Celeb-1M dataset Guo et al.
Revised: Training : We train our base recognition model and DDL on MS-Celeb-1M dataset Guo et al.

Original: Since the original dataset has been proved to be dirty, we use the cleaned version by Deng et al.
Revised: Since the original dataset has been proved to be dirty, we use the cleaned version by Deng et al.

Original: (2019a), which consists of 5,179,510 pictures from 93,431 identities.
Revised: (2019a), which consists of 5,179,510 pictures from 93,431 identities.

Original: For the base model training, we select the modiﬁed version of ResNet-

Revised: For the base model training, we select the modiﬁed version of ResNet-

Original: He et al.
Revised: He et al.

Original: (2016) introduced by Deng et al. (2019a)
Revised: (2016) introduced by Deng et al. (2019a)

Original: and PolyNet Zhang et al.
Revised: and PolyNet Zhang et al.

Original: (2017b)
Revised: (2017b)

Original: as the backbone network.
Revised: as the backbone network.

Original: The loss function is ArcFaceDeng et al.
Revised: The loss function is ArcFaceDeng et al.

Original: (2019a).
Revised: (2019a).

Original: The learning rate starts from 0.1 and is divided by 10 after 60k, 80k iterations, and we ﬁnish our training at 11k.
Revised: The learning rate starts from 0.1 and is divided by 10 after 60k, 80k iterations, and we ﬁnish our training at 11k.

Original: The base model is trained on 16 GPUs with a total batch size ofand based on Pytorch Paszke et al.
Revised: The base model is trained on 16 GPUs with a total batch size ofand based on Pytorch Paszke et al.

Original: (2017) framework.
Revised: (2017) framework.

Original: After training the base network, we extract features for each picture from the training set and compute its DDR score as discussed in section 3.2.
Revised: After training the base network, we extract features for each picture from the training set and compute its DDR score as discussed in section 3.2.

Original: Then the light-weight DDNet will regress the score by L2 loss with the same training image input.
Revised: Then the light-weight DDNet will regress the score by L2 loss with the same training image input.

Original: The learning rate for training our DDNet starts from 0.1 and is divided by 10 after 30k, 60k, and 80k iterations, and the total number of iterations is 100k.
Revised: The learning rate for training our DDNet starts from 0.1 and is divided by 10 after 30k, 60k, and 80k iterations, and the total number of iterations is 100k.

Original: Linear warm-up with 10k iterations is applied towards stable training.
Revised: Linear warm-up with 10k iterations is applied towards stable training.

Original: The DDNet is trained on 8 GPUs with a total batch size of 512 are used.
Revised: The DDNet is trained on 8 GPUs with a total batch size of 512 are used.

Original: D ISCUSSION OF DDN ET ARCHITECTURE
Revised: D ISCUSSION OF DDN ET ARCHITECTURE

Original: As has discussed in section 4.1, the design principle of our DDNet is computation efﬁcient, to introduce less computation burden.
Revised: As has discussed in section 4.1, the design principle of our DDNet is computation efﬁcient, to introduce less computation burden.

Original: The typical architecture of DDNet is shown in Table 8, which is a channel reduction Resnet-18 network.
Revised: The typical architecture of DDNet is shown in Table 8, which is a channel reduction Resnet-18 network.

Original: We also conduct experiments on deeper DDNet which has 34 layers and wider DDNet which is a standard Resnet-18 network .
Revised: We also conduct experiments on deeper DDNet which has 34 layers and wider DDNet which is a standard Resnet-18 network .

Original: As shown in Table 9, neither the deeper or the wider network leads to better performance, which illustrates that the discriminability of face images can be well learned with limited parameters.
Revised: As shown in Table 9, neither the deeper or the wider network leads to better performance, which illustrates that the discriminability of face images can be well learned with limited parameters.

Original: As for action recognition, we utilize the same DDNet architecture where the 2D convolution kernel is replaced by 3D.
Revised: As for action recognition, we utilize the same DDNet architecture where the 2D convolution kernel is replaced by 3D.

Original: R ESULT OF COMBINING MORE BASELINE WITH DDL ON Y OU T UBE F ACE
Revised: R ESULT OF COMBINING MORE BASELINE WITH DDL ON Y OU T UBE F ACE

Original: The baseline model we select in the main part is trained with ArcFace loss function Deng et al.
Revised: The baseline model we select in the main part is trained with ArcFace loss function Deng et al.

Original: (2019a).
Revised: (2019a).

Original: We also re-implement more baseline model with different loss function like Wang et al.
Revised: We also re-implement more baseline model with different loss function like Wang et al.

Original: (2018b), Liu et al. (2017a).
Revised: (2018b), Liu et al. (2017a).

Original: As shown in Table 10, when enhanced by DDL, all baseline achieve performance above 1% performance gain, which indicates the good robustness and generalization ability of our DDL.
Revised: As shown in Table 10, when enhanced by DDL, all baseline achieve performance above 1% performance gain, which indicates the good robustness and generalization ability of our DDL.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D ISCUSSION ON UNBALANCED DATA CLASS WHEN TRAINING DDL
Revised: D ISCUSSION ON UNBALANCED DATA CLASS WHEN TRAINING DDL

Original: Since the data imbalance between classes and such a long-tail distribution has become concerns for training recognition models Zhang et al.
Revised: Since the data imbalance between classes and such a long-tail distribution has become concerns for training recognition models Zhang et al.

Original: (2017a); Van Horn & Perona (2017)
Revised: (2017a); Van Horn & Perona (2017)

Original: , we conduct experiments with two settings for MS1M dataset Guo et al.
Revised: , we conduct experiments with two settings for MS1M dataset Guo et al.

Original: • Tail data: We use one-tenth of raw data with the minimum sample size in each class.
Revised: • Tail data: We use one-tenth of raw data with the minimum sample size in each class.

Original: It has insufﬁcient data per class and contains 519,379 pics from 29,210 classes.
Revised: It has insufﬁcient data per class and contains 519,379 pics from 29,210 classes.

Original: • Head data: We use one-tenth of raw data with the maximum sample size in each class.
Revised: • Head data: We use one-tenth of raw data with the maximum sample size in each class.

Original: It has enough data per class and contains 544,381 pics from 4,833 classes.
Revised: It has enough data per class and contains 544,381 pics from 4,833 classes.

Original: The comparison results for the two settings on IJB-C datasets are illustrated in Table 13.
Revised: The comparison results for the two settings on IJB-C datasets are illustrated in Table 13.

Original: The performance for Tail data and Head data are almost the same because the formulation of DDR doesn’t explicitly encodes identity information.
Revised: The performance for Tail data and Head data are almost the same because the formulation of DDR doesn’t explicitly encodes identity information.

Original: For the classiﬁcation task, insufﬁcient data for one class may lead to bad representation for that class in subspace, but for our DDL training, the DDR definition is class-unspeciﬁed, which means only the discriminability matters, rather than the identity information.
Revised: For the classiﬁcation task, insufﬁcient data for one class may lead to bad representation for that class in subspace, but for our DDL training, the DDR definition is class-unspeciﬁed, which means only the discriminability matters, rather than the identity information.

Original: [Table]
Revised: [Table]

Original: What’s more, the slight performance drop for training with one of the tens of amount indicates that by increasing the training sample, the performance of DDL can further improve.
Revised: What’s more, the slight performance drop for training with one of the tens of amount indicates that by increasing the training sample, the performance of DDL can further improve.

Original: But around 500, of data are enough for training a good DDL model.
Revised: But around 500, of data are enough for training a good DDL model.

Original: [Table]
Revised: [Table]

Original: R ESULTS OF TRAINING DDL WITH OTHER DATASETS .
Revised: R ESULTS OF TRAINING DDL WITH OTHER DATASETS .

Original: In the main article
Revised: In the main article

Original: [Table]
Revised: [Table]

Original: we train our base model and DDL on the MS1M dataset Guo et al.
Revised: we train our base model and DDL on the MS1M dataset Guo et al.

Original: [Table]
Revised: [Table]

Original: which is the largest open face datasets.
Revised: which is the largest open face datasets.

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we also train our base model and DDL on the IMDB dataset Wang et al.
Revised: we also train our base model and DDL on the IMDB dataset Wang et al.

Original: (2018a), which contains 59k identities and 1.7M images.
Revised: (2018a), which contains 59k identities and 1.7M images.

Original: The results on IJB-A datasets are in Table 12.
Revised: The results on IJB-A datasets are in Table 12.

Original: Training with the smaller dataset, DDL also achieves competitive results and better than average pooling for a huge margin.
Revised: Training with the smaller dataset, DDL also achieves competitive results and better than average pooling for a huge margin.

Original: When FPR=0.01%, compared with average pooling, DDL boosts the performance by 29.64%.
Revised: When FPR=0.01%, compared with average pooling, DDL boosts the performance by 29.64%.

Original: This indicates that a weak model can beneﬁt more from DDL.
Revised: This indicates that a weak model can beneﬁt more from DDL.

Original: [Table]
Revised: [Table]

Original: The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported.
Revised: The True Accept Rates (TAR) vs. False Postive Rate (FAR) are reported.

Original: R ESULTS OF CASCADE TRAINING WITH DDL
Revised: R ESULTS OF CASCADE TRAINING WITH DDL

Original: Applying cascade training to reﬁne a better centroid could be a reasonable idea.
Revised: Applying cascade training to reﬁne a better centroid could be a reasonable idea.

Original: We conduct experiments.
Revised: We conduct experiments.

Original: After training one DDL, we use the DDR scores to weighted reﬁne each class’s centroid and generate new DDR scores for DDNet to regress.
Revised: After training one DDL, we use the DDR scores to weighted reﬁne each class’s centroid and generate new DDR scores for DDNet to regress.

Original: The results are in Table.
Revised: The results are in Table.

Original: By cascade training and reﬁned centroids, DDL boosts performance for 0.4% at FAR=0.0001%, which means good centroids leading to a better discriminability representation.
Revised: By cascade training and reﬁned centroids, DDL boosts performance for 0.4% at FAR=0.0001%, which means good centroids leading to a better discriminability representation.

Original: [Table]
Revised: [Table]

Original: D ISCUSSION OF NORMALIZATION AND SCORE SCALE FOR DDL
Revised: D ISCUSSION OF NORMALIZATION AND SCORE SCALE FOR DDL

Original: In Eq(4), we normalize the discriminability by the mean and standard deviation to deal with a variant number of elements in different groups.
Revised: In Eq(4), we normalize the discriminability by the mean and standard deviation to deal with a variant number of elements in different groups.

Original: In our experiments, if the normalization step is removed, the DDNet can’t converge.
Revised: In our experiments, if the normalization step is removed, the DDNet can’t converge.

Original: As for linear map the minimum and maximum DDR score of the element set to 0 and 1 in
Revised: As for linear map the minimum and maximum DDR score of the element set to 0 and 1 in

Original: Eq(8)(9)(10), this step leading to 0.4% performance gain for YTF and 1.6% performance gain for
Revised: Eq(8)(9)(10), this step leading to 0.4% performance gain for YTF and 1.6% performance gain for

Original: IQIYI-VID-FACE.
Revised: IQIYI-VID-FACE.

Original: But the scale process doesn’t contribute to template-based set-to-set recognition like IJB-A and IJB-C dataset.
Revised: But the scale process doesn’t contribute to template-based set-to-set recognition like IJB-A and IJB-C dataset.

Original: The reason is that in a video, frames are captured in the almost same condition, so the variation of frames tends to be limited.
Revised: The reason is that in a video, frames are captured in the almost same condition, so the variation of frames tends to be limited.

Original: The mapping process will separate the score distribution, making it more focuses on frames with higher discriminability.
Revised: The mapping process will separate the score distribution, making it more focuses on frames with higher discriminability.

Original: While for template-based set-to-set recognition like IJB-A and IJB-C, the variation of elements is large and the discriminability score distribution has been separated well, so there is no obvious gain when applying score re-scale.
Revised: While for template-based set-to-set recognition like IJB-A and IJB-C, the variation of elements is large and the discriminability score distribution has been separated well, so there is no obvious gain when applying score re-scale.

Original: More aggregation strategy scripts can found in our GitHub.
Revised: More aggregation strategy scripts can found in our GitHub.

Original: I MPLEMENTATION D ETAILS FOR ACTION RECONGNTION
Revised: I MPLEMENTATION D ETAILS FOR ACTION RECONGNTION

Original: To train on ActivityNet-1.2 dataset, we pre-trained our model on Kinetics-700 datasets.
Revised: To train on ActivityNet-1.2 dataset, we pre-trained our model on Kinetics-700 datasets.

Original: Video clips with adjacent 64 frames are randomly sampled from raw videos for training and testing.
Revised: Video clips with adjacent 64 frames are randomly sampled from raw videos for training and testing.

Original: For the spatial domain, we randomly crop 112*112 pixels with a shorted side randomly sampled in [128, 160] pixels.
Revised: For the spatial domain, we randomly crop 112*112 pixels with a shorted side randomly sampled in [128, 160] pixels.

Original: 32 NVIDIA V100 GPUS with synchronized SGD training and global BN are used to train and we ﬁnd that result for typical training in one 8 GPU machine is the same.
Revised: 32 NVIDIA V100 GPUS with synchronized SGD training and global BN are used to train and we ﬁnd that result for typical training in one 8 GPU machine is the same.

Original: Similar to the 2D-DDL, we generate the pseudo discriminability label for each clip and use a channel reduction 3D resnet-18 network to regress the score after training the base model.
Revised: Similar to the 2D-DDL, we generate the pseudo discriminability label for each clip and use a channel reduction 3D resnet-18 network to regress the score after training the base model.

Original: The training setting for
Revised: The training setting for

Original: Kinetics is the same with ActivityNet-1.2, except we sample clips with adjacent 32 frames.
Revised: Kinetics is the same with ActivityNet-1.2, except we sample clips with adjacent 32 frames.

Original: A. Q UALITATIVE ANALYSIS
Revised: A. Q UALITATIVE ANALYSIS

Original: In the main part, we visualize two templates from IJB-C datasets.
Revised: In the main part, we visualize two templates from IJB-C datasets.

Original: In this subsection, we visualize more examples from YouTube Face dataset.
Revised: In this subsection, we visualize more examples from YouTube Face dataset.

Original: As we can see in these pictures, our DDL has learnt meaningful discriminative patterns.
Revised: As we can see in these pictures, our DDL has learnt meaningful discriminative patterns.

Original: [Figure]
Revised: [Figure]