Original: B AG OF I NSTANCES A GGREGATION B OOSTS S ELF - SUPERVISED D ISTILLATION
Revised: B AG OF I NSTANCES A GGREGATION B OOSTS S ELF - SUPERVISED D ISTILLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent advances in self-supervised learning have experienced remarkable progress, especially for contrastive learning based methods, which regard each image as well as its augmentations as an individual class and try to distinguish them from all other images.
Revised: Recent advances in self-supervised learning have experienced remarkable progress, especially for contrastive learning based methods, which regard each image as well as its augmentations as an individual class and try to distinguish them from all other images.

Original: However, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization.
Revised: However, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization.

Original: This is especially true for small scale models, which we ﬁnd the performance drops dramatically comparing with its supervised counterpart.
Revised: This is especially true for small scale models, which we ﬁnd the performance drops dramatically comparing with its supervised counterpart.

Original: In this paper, we propose a simple but effective distillation strategy for unsupervised learning.
Revised: In this paper, we propose a simple but effective distillation strategy for unsupervised learning.

Original: The highlight is that the relationship among similar samples counts and can be seamlessly transferred to the student to boost the performance.
Revised: The highlight is that the relationship among similar samples counts and can be seamlessly transferred to the student to boost the performance.

Original: Our method, termed as BINGO, which is short for B ag of I nsta N ces a G gregati O n, targets at transferring the relationship learned by the teacher to the student.
Revised: Our method, termed as BINGO, which is short for B ag of I nsta N ces a G gregati O n, targets at transferring the relationship learned by the teacher to the student.

Original: Here bag of instances indicates a set of similar samples constructed by the teacher and are grouped within a bag, and the goal of distillation is to aggregate compact representations over the student with respect to instances in a bag.
Revised: Here bag of instances indicates a set of similar samples constructed by the teacher and are grouped within a bag, and the goal of distillation is to aggregate compact representations over the student with respect to instances in a bag.

Original: Notably, BINGO achieves new state-of-the-art performance on small scale models, i.e. , 65 . 5% and 68 .
Revised: Notably, BINGO achieves new state-of-the-art performance on small scale models, i.e. , 65 . 5% and 68 .

Original: 9% top-1 accuracies with linear evaluation on ImageNet, using ResNet-18 and ResNet-34 as backbone, respectively, surpassing baselines ( 52 .
Revised: 9% top-1 accuracies with linear evaluation on ImageNet, using ResNet-18 and ResNet-34 as backbone, respectively, surpassing baselines ( 52 .

Original: 5% and 57 .
Revised: 5% and 57 .

Original: 4% top-1 accuracies) by a signiﬁcant margin.
Revised: 4% top-1 accuracies) by a signiﬁcant margin.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional Neural Networks (CNNs) have achieved great success in the ﬁeld of computer vision, including image classiﬁcation (He et al., 2016), object detection (Ren et al., 2015) and semantic segmentation (Chen et al., 2017).
Revised: Convolutional Neural Networks (CNNs) have achieved great success in the ﬁeld of computer vision, including image classiﬁcation (He et al., 2016), object detection (Ren et al., 2015) and semantic segmentation (Chen et al., 2017).

Original: However, most of the time, CNNs cannot succeed without enormous human-annotated data.
Revised: However, most of the time, CNNs cannot succeed without enormous human-annotated data.

Original: Recently, self-supervised learning, typiﬁed by contrastive learning (He et al., 2020; Chen et al., 2020a), has been ﬁghting with the annotation-eager challenge and achieves great success.
Revised: Recently, self-supervised learning, typiﬁed by contrastive learning (He et al., 2020; Chen et al., 2020a), has been ﬁghting with the annotation-eager challenge and achieves great success.

Original: Most current self-supervised methods yet focus on networks with large size, e.g ., ResNet50 (He et al., 2016) with more than 20M parameters, but real-life implementation usually involves computation-limited scenarios, e.g ., mobile/edge devices.
Revised: Most current self-supervised methods yet focus on networks with large size, e.g ., ResNet50 (He et al., 2016) with more than 20M parameters, but real-life implementation usually involves computation-limited scenarios, e.g ., mobile/edge devices.

Original: Due to annotation lacking in unsupervised tasks, learning from unlabeled data becomes challenging.
Revised: Due to annotation lacking in unsupervised tasks, learning from unlabeled data becomes challenging.

Original: Recent contrastive learning methods (He et al., 2020; Chen et al., 2020a) tackle this problem by narrowing gaps between embeddings of different augmentations from the same image.
Revised: Recent contrastive learning methods (He et al., 2020; Chen et al., 2020a) tackle this problem by narrowing gaps between embeddings of different augmentations from the same image.

Original: Techniques like momentum encoder for stable updating, memory bank for storing negative pairs, complicated data augmentation strategies etc ., are proposed to avoid collapse and promote the performance.
Revised: Techniques like momentum encoder for stable updating, memory bank for storing negative pairs, complicated data augmentation strategies etc ., are proposed to avoid collapse and promote the performance.

Original: With the above techniques, contrastive learning methods show promising performance.
Revised: With the above techniques, contrastive learning methods show promising performance.

Original: However, contrastive learning requires discriminating all instances, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization.
Revised: However, contrastive learning requires discriminating all instances, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization.

Original: This issue becomes severe for small scale models, which carry too few parameters to ﬁt the enormous data.
Revised: This issue becomes severe for small scale models, which carry too few parameters to ﬁt the enormous data.

Original: Inspired by supervised learning that knowledge from large models can effectively promote the learning ability of small models with distillation, exploring knowledge distillation on unsupervised small models becomes an important topic.
Revised: Inspired by supervised learning that knowledge from large models can effectively promote the learning ability of small models with distillation, exploring knowledge distillation on unsupervised small models becomes an important topic.

Original: Compress (Fang et al., 2020) and SEED (Fang et al., 2020) are two typical methods for unsupervised distillation, which propose to transfer knowledge from the teacher in terms of similarity distributions among different instances.
Revised: Compress (Fang et al., 2020) and SEED (Fang et al., 2020) are two typical methods for unsupervised distillation, which propose to transfer knowledge from the teacher in terms of similarity distributions among different instances.

Original: However, as the similarity distribution is computed by randomly sampling instances from a dynamically maintained queue, this kind of knowledge is mostly constructed based
Revised: However, as the similarity distribution is computed by randomly sampling instances from a dynamically maintained queue, this kind of knowledge is mostly constructed based

Original: (II% 0RELOH1HWY 5HV1HW 5HV1HW    
Revised: (II% 0RELOH1HWY 5HV1HW 5HV1HW    

Original:  7 RS    $ FF     0R&RY6((' 'LV&R%,1* 42.
Revised:  7 RS    $ FF     0R&RY6((' 'LV&R%,1* 42.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Figure] teacher model /DEHOV /DEHOV      7 RS    $ FF     0R&RY 6((' 'LV&R %,1* 30.
Revised: [Figure] teacher model /DEHOV /DEHOV      7 RS    $ FF     0R&RY 6((' 'LV&R %,1* 30.

Original: [Figure]
Revised: [Figure]

Original: on instances with low relation, which fails to effectively model similarity of those highly related samples.
Revised: on instances with low relation, which fails to effectively model similarity of those highly related samples.

Original: To solve this issue, we propose a new self-supervised distillation method, which transfers knowledge by aggregating bags of related instances, named BINGO .
Revised: To solve this issue, we propose a new self-supervised distillation method, which transfers knowledge by aggregating bags of related instances, named BINGO .

Original: In our empirical studies, transferring knowledge based on highly related samples helps boost performance more effectively compared with previous relation-agnostic methods.
Revised: In our empirical studies, transferring knowledge based on highly related samples helps boost performance more effectively compared with previous relation-agnostic methods.

Original: Speciﬁcally, we select an unsupervised pretrained large model as the teacher.
Revised: Speciﬁcally, we select an unsupervised pretrained large model as the teacher.

Original: First, we map the conventional instance-wise dataset into a bag-wise one.
Revised: First, we map the conventional instance-wise dataset into a bag-wise one.

Original: Each original instance is set as an anchor instance of the bag.
Revised: Each original instance is set as an anchor instance of the bag.

Original: By matching similarities of all the other instances’ embeddings produced by the teacher model, we feed instances which show high similarity with the anchor instance into the bag.
Revised: By matching similarities of all the other instances’ embeddings produced by the teacher model, we feed instances which show high similarity with the anchor instance into the bag.

Original: Then we apply the bagged dataset to the small model distillation process.
Revised: Then we apply the bagged dataset to the small model distillation process.

Original: To this end, we propose a bag-aggregation distillation loss, which consists of two components: inter-sample distillation and intra-sample distillation.
Revised: To this end, we propose a bag-aggregation distillation loss, which consists of two components: inter-sample distillation and intra-sample distillation.

Original: For intra-sample distillation, embeddings of the student and teacher from two augmentations of the same instance are pushed together; for inter-sample distillation, embeddings of all instances in one bag are pushed to be more similar with the anchor one.
Revised: For intra-sample distillation, embeddings of the student and teacher from two augmentations of the same instance are pushed together; for inter-sample distillation, embeddings of all instances in one bag are pushed to be more similar with the anchor one.

Original: Equipped with the two proposed distillation loss, the bag-based knowledge from the teacher can be well transferred to the student, which shows signiﬁcant advantages over previous relation-agnostic ones (Fang et al., 2020; Abbasi Koohpayegani et al., 2020).
Revised: Equipped with the two proposed distillation loss, the bag-based knowledge from the teacher can be well transferred to the student, which shows signiﬁcant advantages over previous relation-agnostic ones (Fang et al., 2020; Abbasi Koohpayegani et al., 2020).

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: • We propose a new self-supervised distillation method, which bags related instances by matching similarities of instance embeddings produced by the teacher.
Revised: • We propose a new self-supervised distillation method, which bags related instances by matching similarities of instance embeddings produced by the teacher.

Original: The bagged dataset can effectively boost small model distillation by aggregating instance embeddings in bags.
Revised: The bagged dataset can effectively boost small model distillation by aggregating instance embeddings in bags.

Original: The proposed relation-guided method shows stronger performance than previous relationagnostic ones.
Revised: The proposed relation-guided method shows stronger performance than previous relationagnostic ones.

Original: • BINGO promotes the performance of both ResNet-18 and -34 to new state-of-the-art (SOTA) ones in unsupervised scenarios.
Revised: • BINGO promotes the performance of both ResNet-18 and -34 to new state-of-the-art (SOTA) ones in unsupervised scenarios.

Original: It is worth noting that the distilled models also present far better performance compared with previous SOTA methods on other tasks,
Revised: It is worth noting that the distilled models also present far better performance compared with previous SOTA methods on other tasks,

Original: i
Revised: i

Original: e ., KNN classiﬁcation and semi-supervised learning.
Revised: e ., KNN classiﬁcation and semi-supervised learning.

Original: • BINGO provides a new paradigm for unsupervised distillation where knowledge between instances with high relation could be more effective than relation-agnostic ones.
Revised: • BINGO provides a new paradigm for unsupervised distillation where knowledge between instances with high relation could be more effective than relation-agnostic ones.

Original: This may be inspiring for further explorations on knowledge transfer in unsupervised scenarios.
Revised: This may be inspiring for further explorations on knowledge transfer in unsupervised scenarios.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-supervised Learning As a generic framework to learn representations with unlabeled data, self-supervised learning has experienced remarkable progress over the past few years.
Revised: Self-supervised Learning As a generic framework to learn representations with unlabeled data, self-supervised learning has experienced remarkable progress over the past few years.

Original: By constructing a series of pretext tasks, self-supervised learning aims at extracting discriminative representations from input data.
Revised: By constructing a series of pretext tasks, self-supervised learning aims at extracting discriminative representations from input data.

Original: Previous methods obtain self-supervised representations mainly via a corrupting and recovering manner, from perspectives of spatial ordering (Noroozi & Favaro, 2016), rotation changes (Komodakis & Gidaris, 2018), in-painting (Pathak et al., 2016), or colorization (Zhang et al., 2016), et al .
Revised: Previous methods obtain self-supervised representations mainly via a corrupting and recovering manner, from perspectives of spatial ordering (Noroozi & Favaro, 2016), rotation changes (Komodakis & Gidaris, 2018), in-painting (Pathak et al., 2016), or colorization (Zhang et al., 2016), et al .

Original: Recently, contrastive learning based methods (He et al., 2020; Chen et al., 2020a) emerge and signiﬁcantly promote the performance of self-supervised learning, which aim at maximizing the mutual information between two augmented views of a image.
Revised: Recently, contrastive learning based methods (He et al., 2020; Chen et al., 2020a) emerge and signiﬁcantly promote the performance of self-supervised learning, which aim at maximizing the mutual information between two augmented views of a image.

Original: A series of subsequent works (Grill et al., 2020; Xu et al., 2020b; Dwibedi et al., 2021) further improve the performance to a very high level.
Revised: A series of subsequent works (Grill et al., 2020; Xu et al., 2020b; Dwibedi et al., 2021) further improve the performance to a very high level.

Original: Khosla et al.
Revised: Khosla et al.

Original: (2020) applies contrastive learning on supervised learning, it selects the positive samples from  same category.
Revised: (2020) applies contrastive learning on supervised learning, which selects the positive samples from the same category.

Original: Caron et al.
Revised: Caron et al.

Original: (2020) proposes to align the distribution of one instance’s different views on other categories.
Revised: (2020) proposes to align the distribution of one instance’s different views on other categories.

Original: However, few of them pay attention to self-supervised learning on small-scale models, which are of critical importance to implement self-supervised models on lightweight devices.
Revised: However, few of them pay attention to self-supervised learning on small-scale models, which are of critical importance to implement self-supervised models on lightweight devices.

Original: We propose an effective method to boost the self-supervised learning of small models, which takes advantage of relation-based knowledge between data and shows superior performance than previous ones.
Revised: We propose an effective method to boost the self-supervised learning of small models, which takes advantage of relation-based knowledge between data and shows superior performance than previous ones.

Original: Knowledge Distillation Knowledge distillation aims to transfer knowledge from a model (teacher) to another one (student), usually from a large to small one, which is commonly used for improving the performance of the lightweight model.
Revised: Knowledge Distillation Knowledge distillation aims to transfer knowledge from a model (teacher) to another one (student), usually from a large to small one, which is commonly used for improving the performance of the lightweight model.

Original: Hinton et al.
Revised: Hinton et al.

Original: ﬁrst proposes knowledge distillation via minimizing the KL-divergence between the student and teacher’s logits, which uses the predicted class probabilities from the teacher as soft labels to guide the student model.
Revised: ﬁrst proposes knowledge distillation via minimizing the KL-divergence between the student and teacher’s logits, which uses the predicted class probabilities from the teacher as soft labels to guide the student model.

Original: Instead of mimicking teacher’s logits, Romero et al.
Revised: Instead of mimicking teacher’s logits, Romero et al.

Original: (2014) transfers the knowledge by minimizing the (cid:96) 2 distance between intermediate outputs of the teacher and student model.
Revised: (2014) transfers the knowledge by minimizing the (cid:96) 2 distance between intermediate outputs of the teacher and student model.

Original: To solve the dimension mismatch, Romero et al.
Revised: To solve the dimension mismatch, Romero et al.

Original: (2014) uses a randomly initialized projection layer to enlarge the dimension of a narrower student model.
Revised: (2014) uses a randomly initialized projection layer to enlarge the dimension of a narrower student model.

Original: Based on Romero et al.
Revised: Based on Romero et al.

Original: Zagoruyko & Komodakis (2016) utilizes knowledge stored in the attention map generated by the teacher model, and pushes the student model to pay attention to the area where the teacher focuses on.
Revised: Zagoruyko & Komodakis (2016) utilizes knowledge stored in the attention map generated by the teacher model, and pushes the student model to pay attention to the area where the teacher focuses on.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2021) improves weighted soft labels to adaptively improve the bias-variance tradeoff of each sample.
Revised: (2021) improves weighted soft labels to adaptively improve the bias-variance tradeoff of each sample.

Original: Besides perspectives of soft labels and intermediate features, relation between samples is also an important knowledge.
Revised: Besides perspectives of soft labels and intermediate features, relation between samples is also an important knowledge.

Original: Park et al.
Revised: Park et al.

Original: (2019) and Liu et al.
Revised: (2019) and Liu et al.

Original: (2019) train student model by aligning the pair-wise similarity graph with the teacher.
Revised: (2019) train student model by aligning the pair-wise similarity graph with the teacher.

Original: Recently, some works extend the above distillation method into self-supervised learning scenarios.
Revised: Recently, some works extend the above distillation method into self-supervised learning scenarios.

Original: Tian et al.
Revised: Tian et al.

Original: (2019) uses the contrastive loss to learn cross-modality consistency.
Revised: (2019) uses the contrastive loss to learn cross-modality consistency.

Original: Xu et al.
Revised: Xu et al.

Original: (2020a),Fang et al.
Revised: (2020a),Fang et al.

Original: (2020) and Abbasi Koohpayegani et al.
Revised: (2020) and Abbasi Koohpayegani et al.

Original: (2020) share a similar methodology with Caron et al.
Revised: (2020) share a similar methodology with Caron et al.

Original: (2020) of aligning feature distribution between views of the same instances.
Revised: (2020) of aligning feature distribution between views of the same instances.

Original: The distribution is computed as the pair-wise similarities between student’s outputs and features stored in memory bank.
Revised: The distribution is computed as the pair-wise similarities between student’s outputs and features stored in memory bank.

Original: However, the above relation-based self-supervised distillation methods only compute the similarity between anchor sample and randomly sampled instances from a maintained queue, which ignores the relation between sampled and anchor instances.
Revised: However, the above relation-based self-supervised distillation methods only compute the similarity between anchor sample and randomly sampled instances from a maintained queue, which ignores the relation between sampled and anchor instances.

Original: Choi et al.
Revised: Choi et al.

Original: (2021) uses the teacher model to produce cluster assignments, and encourage the student model to mimic the output of  trainable teacher model on-the-ﬂy, which achieves promising results.
Revised: (2021) uses the teacher model to produce cluster assignments, and encourages the student model to mimic the output of the trainable teacher model on-the-ﬂy, which achieves promising results.

Original: Gao et al.
Revised: Gao et al.

Original: (2021) strengthens the student model by adding a regularization loss on the original contrastive loss, which aims at minimizing the (cid:96) 2 distance between the student’s and teacher’s embedding.
Revised: (2021) strengthens the student model by adding a regularization loss on the original contrastive loss, which aims at minimizing the (cid:96) 2 distance between the student’s and teacher’s embedding.

Original: We propose to transfer the relation knowledge between models via a new type of dataset, which bags related instances.
Revised: We propose to transfer the relation knowledge between models via a new type of dataset, which bags related instances.

Original: By aggregating the bagged instances, the relation knowledge can be effectively transferred.
Revised: By aggregating the bagged instances, the relation knowledge can be effectively transferred.

Original:  A PPROACH
Revised:  A PPROACH

Original: In this section, we introduce the proposed BINGO in details.
Revised: In this section, we introduce the proposed BINGO in details.

Original: First, we discuss how to bag samples in the instance-wise dataset.
Revised: First, we discuss how to bag samples in the instance-wise dataset.

Original: After the samples are bagged, the bag-aggregation based knowledge distillation is introduced.
Revised: After the samples are bagged, the bag-aggregation based knowledge distillation is introduced.

Original: We also discuss how to compute bag-aggregation loss and how they improve the performance of the lightweight model.
Revised: We also discuss how to compute bag-aggregation loss and how they improve the performance of the lightweight model.

Original: The overall framework is illustrated in Fig.
Revised: The overall framework is illustrated in Fig.

Original: 3. B AGGING I NSTANCES WITH S IMILARITY M ATCHING
Revised: 3. B AGGING I NSTANCES WITH S IMILARITY M ATCHING

Original: Given the unlabeled training set X = { x 1 , x 2 , ..., x N } , we deﬁne the corresponding bag-wise training set as Ω = { Ω 1 , Ω 2 , ..., Ω N } , where each bag Ω i consists of a set of instances.
Revised: Given the unlabeled training set X = { x 1 , x 2 , ..., x N } , we deﬁne the corresponding bag-wise training set as Ω = { Ω 1 , Ω 2 , ..., Ω N } , where each bag Ω i consists of a set of instances.

Original: To transfer the instance-wise dataset to a bag-wise one, we ﬁrst feed X into a pretrained teacher model f T and get the corresponding features V = { v 1 , v 2 , ..., v N } where v i =
Revised: To transfer the instance-wise dataset to a bag-wise one, we ﬁrst feed X into a pretrained teacher model f T and get the corresponding features V = { v 1 , v 2 , ..., v N } where v i =

Original: f T ( x i ) .
Revised: f T ( x i ) .

Original: For each anchor sample x a in the dataset, we ﬁnd positive samples which share high similarity with the anchor sample.
Revised: For each anchor sample x a in the dataset, we ﬁnd positive samples which share high similarity with the anchor sample.

Original: Then the anchor sample as well as the similar samples are combined to form a bag.
Revised: Then the anchor sample as well as the similar samples are combined to form a bag.

Original: The samples in one
Revised: The samples in one

Original: Instance-wise Dataset 𝑋
Revised: Instance-wise Dataset 𝑋

Original: Bagging via
Revised: Bagging via

Original: Feature
Revised: Feature

Original: Similarity …
Revised: Similarity …

Original: Features
Revised: Features

Original: Anchor
Revised: Anchor

Original: Positive
Revised: Positive

Original: Positive
Revised: Positive

Original: Positive
Revised: Positive

Original: Positive
Revised: Positive

Original: Bag Aggregation Teacher
Revised: Bag Aggregation Teacher

Original: Student Bag-wise Dataset 𝛺
Revised: Student Bag-wise Dataset 𝛺

Original: Anchor Instance 𝑋
Revised: Anchor Instance 𝑋

Original: 𝑎
Revised: 𝑎

Original: Positive Instance 𝑋 𝑝 𝑡 𝑎 𝑠 𝑎 𝑠 𝑝 𝑡~𝑇 𝑡~𝑇
Revised: Positive Instance 𝑋 𝑝 𝑡 𝑎 𝑠 𝑎 𝑠 𝑝 𝑡~𝑇 𝑡~𝑇

Original: Details about how to aggregate a bag Teacher
Revised: Details about how to aggregate a bag Teacher

Original: [Figure]
Revised: [Figure]

Original: 𝐿 𝑖𝑛𝑡𝑒𝑟
Revised: 𝐿 𝑖𝑛𝑡𝑒𝑟

Original: Instancesampling bag have a compact representation in the embedding space.
Revised: Instancesampling bag have a compact representation in the embedding space.

Original: Several mapping function can be used to ﬁnd similar samples:
Revised: Several mapping function can be used to ﬁnd similar samples:

Original: K-nearest Neighbors For each anchor sample x a in the instance-wise dataset
Revised: K-nearest Neighbors For each anchor sample x a in the instance-wise dataset

Original: , we ﬁrst compute the pairwise similarity with all samples in the dataset S a = { v a · v i | i = 1 , 2 , ..., N } .
Revised: , we ﬁrst compute the pairwise similarity with all samples in the dataset S a = { v a · v i | i = 1 , 2 , ..., N } .

Original: The bag Ω a corresponding to x a is deﬁned as:
Revised: The bag Ω a corresponding to x a is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where top − rank ( · , K ) returns the indices of top K items in a set.
Revised: where top − rank ( · , K ) returns the indices of top K items in a set.

Original: K-means Clustering Given the training feature set V
Revised: K-means Clustering Given the training feature set V

Original: = { v 1 , v 2 , ..., v N } , we ﬁrst assign a pseudo-label q i to each sample i , where q i ∈ { q 1 , ..., q K } .
Revised: = { v 1 , v 2 , ..., v N } , we ﬁrst assign a pseudo-label q i to each sample i , where q i ∈ { q 1 , ..., q K } .

Original: The clustering process is performed by minimizing the following term,
Revised: The clustering process is performed by minimizing the following term,

Original: [Equation]
Revised: [Equation]

Original: ) where c q i denotes the centering feature of all features belonging to the label q
Revised: ) where c q i denotes the centering feature of all features belonging to the label q

Original: i , i.e ., c q i = (cid:80) q j = q i v
Revised: i , i.e ., c q i = (cid:80) q j = q i v

Original: [Equation]
Revised: [Equation]

Original: ∀
Revised: ∀

Original: [Equation]
Revised: [Equation]

Original: = 1 , ..., N .
Revised: = 1 , ..., N .

Original: The bag Ω a of anchor sample x a is deﬁned as:
Revised: The bag Ω a of anchor sample x a is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Ground Truth Label
Revised: Ground Truth Label

Original: If the ground truth label is available, we can also bag samples with the human-annotated semantic labels.
Revised: If the ground truth label is available, we can also bag samples with the human-annotated semantic labels.

Original: Given the label set Y =
Revised: Given the label set Y =

Original: { y 1 , y 2 , ..., y N } , we can bag related instances of the anchor sample x a via:
Revised: { y 1 , y 2 , ..., y N } , we can bag related instances of the anchor sample x a via:

Original: [Equation]
Revised: [Equation]

Original: In this paper, we use K-nearest neighbors as the bagging strategy.
Revised: In this paper, we use K-nearest neighbors as the bagging strategy.

Original: More details about performance of using the K-means clustering based bagging strategy can be found in Appendix.
Revised: More details about performance of using the K-means clustering based bagging strategy can be found in Appendix.

Original: Note that bagging instances via the ground truth label is just used to measure the upper bound of the proposed method.
Revised: Note that bagging instances via the ground truth label is just used to measure the upper bound of the proposed method.

Original: 3. K NOWLEDGE D ISTILLATION VIA B AG A GGREGATION
Revised: 3. K NOWLEDGE D ISTILLATION VIA B AG A GGREGATION

Original: Once we get the bag-wise dataset Ω utilizing a pretrained teacher model, it can be used for distillation process.
Revised: Once we get the bag-wise dataset Ω utilizing a pretrained teacher model, it can be used for distillation process.

Original: In each feed-forward process, the anchor sample x a and the positive sample x p which belong to the same bag Ω a are sampled together in one batch.
Revised: In each feed-forward process, the anchor sample x a and the positive sample x p which belong to the same bag Ω a are sampled together in one batch.

Original: We propose the bag-aggregation distillation loss including the intra-sample distillation loss L intra and inter-sample distillation loss L inter .
Revised: We propose the bag-aggregation distillation loss including the intra-sample distillation loss L intra and inter-sample distillation loss L inter .

Original: To aggregate the representations within a bag into more compact embeddings, we minimize the following target function:
Revised: To aggregate the representations within a bag into more compact embeddings, we minimize the following target function:

Original: [Equation]
Revised: [Equation]

Original: ) where L is a metric function to measure the distance between two embeddings – there are many metrics can be selected, such as cosine similarity, euclidean distance, etc .
Revised: ) where L is a metric function to measure the distance between two embeddings – there are many metrics can be selected, such as cosine similarity, euclidean distance, etc .

Original: Here we use the normalized cosine similarity, i.e ., the contrastive loss commonly used in self-supervised learning to measure the distance between x i and the anchor sample x a .
Revised: Here we use the normalized cosine similarity, i.e ., the contrastive loss commonly used in self-supervised learning to measure the distance between x i and the anchor sample x a .

Original: The target function in Eq.
Revised: The target function in Eq.

Original: 5 can be divided into two components:
Revised: 5 can be divided into two components:

Original: [Equation]
Revised: [Equation]

Original: Three separate data augmentation operators t 1 , t 2 , t 3 are randomly sampled from the same family of
Revised: Three separate data augmentation operators t 1 , t 2 , t 3 are randomly sampled from the same family of

Original: MoCo-v2 augmentations T , which is also adopted in SEED(Fang et al., 2020) and DisCo (Gao et al., 2021).
Revised: MoCo-v2 augmentations T , which is also adopted in SEED(Fang et al., 2020) and DisCo (Gao et al., 2021).

Original: where the ﬁrst item focuses on pulling different views (augmentations) of the same sample together, and the second item aims at pulling different samples that are within a same bag into more related ones.
Revised: where the ﬁrst item focuses on pulling different views (augmentations) of the same sample together, and the second item aims at pulling different samples that are within a same bag into more related ones.

Original: We term the ﬁrst item as L intra and the second item as L inter .
Revised: We term the ﬁrst item as L intra and the second item as L inter .

Original: Intra-Sample Distillation
Revised: Intra-Sample Distillation

Original: The intra-sample distillation loss is a variant of conventional contrastive loss.
Revised: The intra-sample distillation loss is a variant of conventional contrastive loss.

Original: Contrastive learning aims to learn representations by discriminating the positive key among negative samples.
Revised: Contrastive learning aims to learn representations by discriminating the positive key among negative samples.

Original: Given two augmented views x and x (cid:48) of one input image, MoCo (Chen et al., 2020c) uses a online encoder f q and a momentum encoder f k to generate embeddings of the positive pairs: q = f q ( x ) , k = f k ( x (cid:48) ) .
Revised: Given two augmented views x and x (cid:48) of one input image, MoCo (Chen et al., 2020c) uses a online encoder f q and a momentum encoder f k to generate embeddings of the positive pairs: q = f q ( x ) , k = f k ( x (cid:48) ) .

Original: The contrastive loss is deﬁned as
Revised: The contrastive loss is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: During distillation, we simply replace f q and f k by the student model f S and teacher model f T , while weights of the teacher model f T are pretrained and are not updated during distillation.
Revised: During distillation, we simply replace f q and f k by the student model f S and teacher model f T , while weights of the teacher model f T are pretrained and are not updated during distillation.

Original: The intra-sample distillation loss can be formulated as
Revised: The intra-sample distillation loss can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where τ is the temperature parameter.
Revised: where τ is the temperature parameter.

Original: We select negative samples k − in a memory bank, which is widely used in MoCo (He et al., 2020) and many subsequent contrastive learning methods.
Revised: We select negative samples k − in a memory bank, which is widely used in MoCo (He et al., 2020) and many subsequent contrastive learning methods.

Original: The memory bank is a queue of data embeddings and the queue size is much larger than a typical minibatch size.
Revised: The memory bank is a queue of data embeddings and the queue size is much larger than a typical minibatch size.

Original: After each forward iteration, items in the queue are progressively replaced by the current output of the teacher network.
Revised: After each forward iteration, items in the queue are progressively replaced by the current output of the teacher network.

Original: Inter-Sample Distillation Given the anchor sample x a and a positive sample x p in the bag Ω a , it is natural to map highly related samples to more similar representations.
Revised: Inter-Sample Distillation Given the anchor sample x a and a positive sample x p in the bag Ω a , it is natural to map highly related samples to more similar representations.

Original: In other words, we want the bag ﬁlled with related samples to be more compact.
Revised: In other words, we want the bag ﬁlled with related samples to be more compact.

Original: Inspired by Eq.
Revised: Inspired by Eq.

Original: 8, we deﬁne the inter-sample distillation loss as
Revised: 8, we deﬁne the inter-sample distillation loss as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The intra- and inter-sample distillation loss serve as different roles.
Revised: The intra- and inter-sample distillation loss serve as different roles.

Original: The intra-sample distillation works like conventional distillation (Hinton et al., 2015; Romero et al., 2014), which aims at minimizing distances between outputs of the teacher and student model given the same input.
Revised: The intra-sample distillation works like conventional distillation (Hinton et al., 2015; Romero et al., 2014), which aims at minimizing distances between outputs of the teacher and student model given the same input.

Original: However, the intersample distillation mainly focuses on transferring the data relation knowledge taking the bag-wise dataset as the carrier, which is obtained from the pretrained teacher model. [Table]
Revised: However, the intersample distillation mainly focuses on transferring the data relation knowledge taking the bag-wise dataset as the carrier, which is obtained from the pretrained teacher model. [Table]

Original: on the validation set with 10 nearest neighbors.
Revised: on the validation set with 10 nearest neighbors.

Original: ResNet-50 × 2 is used as the teacher.
Revised: ResNet-50 × 2 is used as the teacher.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the feature representations of the distilled student networks on several widely used benchmarks.
Revised: In this section, we evaluate the feature representations of the distilled student networks on several widely used benchmarks.

Original: We ﬁrst report the performance on ImageNet under the linear evaluation and semi-supervised protocols.
Revised: We ﬁrst report the performance on ImageNet under the linear evaluation and semi-supervised protocols.

Original: Then we conduct evaluation on several downstream tasks including object detection and instance segmentation, as well as some ablation studies to diagnose how each component and parameter affect the performance.
Revised: Then we conduct evaluation on several downstream tasks including object detection and instance segmentation, as well as some ablation studies to diagnose how each component and parameter affect the performance.

Original: 4. P RE - TRAINING D ETAILS
Revised: 4. P RE - TRAINING D ETAILS

Original: Pre-training of Teacher Model Two models are used as teachers: ResNet-50 trained with MoCov2 (Chen et al., 2020c) for 800 epochs and ResNet-50 × 2 trained with SwAV for 400 epochs.
Revised: Pre-training of Teacher Model Two models are used as teachers: ResNet-50 trained with MoCov2 (Chen et al., 2020c) for 800 epochs and ResNet-50 × 2 trained with SwAV for 400 epochs.

Original: The ofﬁcially released weights 1 are used to initialize teacher models during distillation for fair comparisons with other methods.
Revised: The ofﬁcially released weights 1 are used to initialize teacher models during distillation for fair comparisons with other methods.

Original: Self-supervised Distillation of Student Model Two models are used as students: ResNet-18 and
Revised: Self-supervised Distillation of Student Model Two models are used as students: ResNet-18 and

Original: ResNet-34.
Revised: ResNet-34.

Original: Following the settings of MoCo in Chen et al.
Revised: Following the settings of MoCo in Chen et al.

Original: (2020c)
Revised: (2020c)

Original: , we add a 2-layer MLP on top of the last averaged pooling layer to form a 128-d embedding vector.
Revised: , we add a 2-layer MLP on top of the last averaged pooling layer to form a 128-d embedding vector.

Original: During distillation, the model is trained with the SGD optimizer with momentum 0.9 and weight decay 0.0001 for 200 epochs on
Revised: During distillation, the model is trained with the SGD optimizer with momentum 0.9 and weight decay 0.0001 for 200 epochs on

Original: ImageNet (Deng et al., 2009).
Revised: ImageNet (Deng et al., 2009).

Original: The batch size and learning rate are set as 256 and 0.03 for 8 GPUs, which simply follow the hyper-parameter settings as in Chen et al.
Revised: The batch size and learning rate are set as 256 and 0.03 for 8 GPUs, which simply follow the hyper-parameter settings as in Chen et al.

Original: (2020c).
Revised: (2020c).

Original: The learning rate is decayed to 0 by a cosine scheduler during training process.
Revised: The learning rate is decayed to 0 by a cosine scheduler during training process.

Original: The temperature τ and the size of memory bank are set as 0.2 and 65,536 respectively.
Revised: The temperature τ and the size of memory bank are set as 0.2 and 65,536 respectively.

Original: For the bagging strategy, we use K-nearest neighbors strategy unless speciﬁed.
Revised: For the bagging strategy, we use K-nearest neighbors strategy unless speciﬁed.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENTS ON I MAGE N ET
Revised: 4. E XPERIMENTS ON I MAGE N ET

Original: Linear Evaluation
Revised: Linear Evaluation

Original: In order to evaluate the performance of BINGO, we train a linear classiﬁer upon the frozen representation, following the common evaluation protocol in Chen et al. (2020c).
Revised: In order to evaluate the performance of BINGO, we train a linear classiﬁer upon the frozen representation, following the common evaluation protocol in Chen et al. (2020c).

Original: For fair comparisons, we use the same hyper-parameters as Fang et al.
Revised: For fair comparisons, we use the same hyper-parameters as Fang et al.

Original: ; Gao et al.
Revised: ; Gao et al.

Original: (2021) during linear evaluation stage.
Revised: (2021) during linear evaluation stage.

Original: The classiﬁer is trained for 100 epochs, using the SGD optimizer with 30 as initial learning rate.
Revised: The classiﬁer is trained for 100 epochs, using the SGD optimizer with 30 as initial learning rate.

Original: As shown in Table 1, BINGO outperform previous DisCo and SEED with the same teacher and student models.
Revised: As shown in Table 1, BINGO outperform previous DisCo and SEED with the same teacher and student models.

Original: Note that SEED distills for 800 epochs while BINGO runs forepochs with ResNet-50 × 2 teacher, which demonstrates the effectiveness of the proposed method.
Revised: Note that SEED distills for 800 epochs while BINGO runs forepochs with ResNet-50 × 2 teacher, which demonstrates the effectiveness of the proposed method.

Original: KNN
Revised: KNN

Original: Classiﬁcation We also evaluate representation of student model using nearest neighbor classiﬁer with cosine similarity.
Revised: Classiﬁcation We also evaluate representation of student model using nearest neighbor classiﬁer with cosine similarity.

Original: KNN classiﬁer can evaluate the learned feature more directly without any parameter tuning.
Revised: KNN classiﬁer can evaluate the learned feature more directly without any parameter tuning.

Original: Following Caron et al.
Revised: Following Caron et al.

Original: ; Abbasi Koohpayegani et al.
Revised: ; Abbasi Koohpayegani et al.

Original: ; Fang et al.
Revised: ; Fang et al.

Original: (2020), we extract features from center-cropped images after the last averaged pooling layers.
Revised: (2020), we extract features from center-cropped images after the last averaged pooling layers.

Original: For convenient comparisons with other methods, we report the validation accuracy with 10 NN (we use the student model distilled from ResNet-50 × 2).
Revised: For convenient comparisons with other methods, we report the validation accuracy with 10 NN (we use the student model distilled from ResNet-50 × 2).

Original: As shown in Table 2, BINGO achieves 61 . 0% and 64 .
Revised: As shown in Table 2, BINGO achieves 61 . 0% and 64 .

Original: 9% accuracies on ResNet-18/34 models, respectively, which outperforms previous methods by a large margin.
Revised: 9% accuracies on ResNet-18/34 models, respectively, which outperforms previous methods by a large margin.

Original: Semi-supervised Classiﬁcation Following previous works Chen et al.
Revised: Semi-supervised Classiﬁcation Following previous works Chen et al.

Original: (2020a;b), we also evaluate the proposed method by ﬁne-tuning the student model ResNet-18 with 1% and 10% labeled data.
Revised: (2020a;b), we also evaluate the proposed method by ﬁne-tuning the student model ResNet-18 with 1% and 10% labeled data.

Original: We follow the training split settings as in Chen et al.
Revised: We follow the training split settings as in Chen et al.

Original: (2020a)
Revised: (2020a)

Original: for fair comparisons.
Revised: for fair comparisons.

Original: The network is ﬁnetuned for 60 epochs with SGD optimizer.
Revised: The network is ﬁnetuned for 60 epochs with SGD optimizer.

Original: The learning rate of the last randomly initialized fc layer is set as 10.
Revised: The learning rate of the last randomly initialized fc layer is set as 10.

Original: As shown in Table 4, using ResNet-18 student model, BINGO obtains best accuracies with the same ResNet-50 and ResNet-152 teacher when using 1% and 10% labels, respectively, both results beats the previous best performance.
Revised: As shown in Table 4, using ResNet-18 student model, BINGO obtains best accuracies with the same ResNet-50 and ResNet-152 teacher when using 1% and 10% labels, respectively, both results beats the previous best performance.

Original: Transfer to Detection and Instance Segmentation We also evaluate the generalization ability of the student model on object detection and instance segmentation tasks.
Revised: Transfer to Detection and Instance Segmentation We also evaluate the generalization ability of the student model on object detection and instance segmentation tasks.

Original: The COCO (Lin et al., 2014) dataset is used for evaluation.
Revised: The COCO (Lin et al., 2014) dataset is used for evaluation.

Original: Following He et al.
Revised: Following He et al.

Original: (2020), we use Mask R-CNN (He et al., 2017) for object detection and instance segmentation and ﬁne-tune all the parameters of the student model ,
Revised: (2020), we use Mask R-CNN (He et al., 2017) for object detection and instance segmentation and ﬁne-tune all the parameters of the student model ,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e .
Revised: e .

Original: ResNet-34 distilled from the ResNet-152 teacher model, end-to-end.
Revised: ResNet-34 distilled from the ResNet-152 teacher model, end-to-end.

Original: As shown in Table 3, BINGO consistently outperforms models pretrained without distillation, as well as SEED Fang et al.
Revised: As shown in Table 3, BINGO consistently outperforms models pretrained without distillation, as well as SEED Fang et al.

Original: and DisCo Gao et al.
Revised: and DisCo Gao et al.

Original: Transfer to CIFAR-10/CIFAR-100 classiﬁcation Following the evaluation protocol in Fang et al.
Revised: Transfer to CIFAR-10/CIFAR-100 classiﬁcation Following the evaluation protocol in Fang et al.

Original: ; Gao et al.
Revised: ; Gao et al.

Original: (2021), we assess the generalization of BINGO on the CIFAR-10/CIFARdataset.
Revised: (2021), we assess the generalization of BINGO on the CIFAR-10/CIFARdataset.

Original: As shown in Table 5, compared with  previous state-of-the-art method DisCo, BINGO outperforms it by 1 . 5% and 3 . 
Revised: As shown in Table 5, compared with the previous state-of-the-art method DisCo (Gao et al., 2021), BINGO outperforms it by 1 . 5% and 3 . 2% respectively

Original: 2% respectively.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: In this section
Revised: [Table]

Original: [Table]
Revised: A BLATION S TUDY

Revised: In this section

Revised: [Table]

Original: we conduct detailed ablation studies to diagnose how each component affect the performance of the distilled model.
Revised: we conduct detailed ablation studies to diagnose how each component affect the performance of the distilled model.

Original: Unless speciﬁed, all results in this section are based on ResNet-18, and distilled for 200 epochs.
Revised: Unless speciﬁed, all results in this section are based on ResNet-18, and distilled for 200 epochs.

Original: Impact of k in K-nearest Neighbors We inspect the inﬂuence of k in K-nearest neighbors bagging strategy.
Revised: Impact of k in K-nearest Neighbors We inspect the inﬂuence of k in K-nearest neighbors bagging strategy.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, the results are relatively robust for a range of k ( k =1,5,10,20).
Revised: 3, the results are relatively robust for a range of k ( k =1,5,10,20).

Original: In addition, we ﬁnd that the classiﬁcation accuracy decrease with k = 10 , 20 compared with k = 5 , because the noise is introduced when k becomes large.
Revised: In addition, we ﬁnd that the classiﬁcation accuracy decrease with k = 10 , 20 compared with k = 5 , because the noise is introduced when k becomes large.

Original: However, the performance with a relative small k = 1 is no better than k = 5 , we think the diversity is sacriﬁced when we only select the topnearest neighborhood all the time.
Revised: However, the performance with a relative small k = 1 is no better than k = 5 , we think the diversity is sacriﬁced when we only select the topnearest neighborhood all the time.

Original: N  N  N  N  7RSNLQ.1

Revised: N  N  N  N  7RSNLQ.1

Original:   7 RS    DFF    63.
Revised:   7 RS    DFF    63.

Original: Lower and Upper Bound of The Proposed Method As shown in Table 6, using data relation extracted from a random initialized model gives a poor performance of 46 .
Revised: Lower and Upper Bound of The Proposed Method As shown in Table 6, using data relation extracted from a random initialized model gives a poor performance of 46 .

Original: 6% , which can be a lower bound of our method.
Revised: 6% , which can be a lower bound of our method.

Original: Then we try to explore the upper bound performance by bagging instances via a supervised-pretrained model, the performance gets an improvement of 0 .
Revised: Then we try to explore the upper bound performance by bagging instances via a supervised-pretrained model, the performance gets an improvement of 0 .

Original: 8% over using data relation extracted from the unsupervised pretrained teacher model.
Revised: 8% over using data relation extracted from the unsupervised pretrained teacher model.

Original: When we directly use the ground truth labels to bag instances, we get a highest upper bound performance, i.e ., 65 . 8% Top-1 accuracy.
Revised: When we directly use the ground truth labels to bag instances, we get a highest upper bound performance, i.e ., 65 . 8% Top-1 accuracy.

Original: Impacts of Data-Relation and Teacher Parameters.
Revised: Impacts of Data-Relation and Teacher Parameters.

Original: In our experiments, both the data relation and model parameters of teacher model are used to distill student model.
Revised: In our experiments, both the data relation and model parameters of teacher model are used to distill student model.

Original: We diagnose how each component affects the distillation performance.
Revised: We diagnose how each component affects the distillation performance.

Original: As shown in Table 7, model parameters represent whether to load the teacher model parameters into the teacher part in Fig.
Revised: As shown in Table 7, model parameters represent whether to load the teacher model parameters into the teacher part in Fig.

Original: 2; data relation refers to the
Revised: 2; data relation refers to the

Original: L inter loss.
Revised: L inter loss.

Original: Besides, L intra is intrinsically a basic contrastive-learning paradigm on two views of one instance.
Revised: Besides, L intra is intrinsically a basic contrastive-learning paradigm on two views of one instance.

Original: no matter the teacher’s parameters are loaded or not, using the the data relation from pretrained teacher model always gets better results than using data relation from online student model, which veriﬁes the efﬁciency of transferring teacher’s data relation to student model.
Revised: no matter the teacher’s parameters are loaded or not, using the the data relation from pretrained teacher model always gets better results than using data relation from online student model, which veriﬁes the efﬁciency of transferring teacher’s data relation to student model.

Original: Interestingly, we ﬁnd that BINGO even gets good result only utilizing teacher’s data relation (Row 3 of Table 7), which is about 10% higher than model training without distillation.
Revised: Interestingly, we ﬁnd that BINGO even gets good result only utilizing teacher’s data relation (Row 3 of Table 7), which is about 10% higher than model training without distillation.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Compare with Other Distillation Methods.
Revised: Compare with Other Distillation Methods.

Original: We now compare with several other distillation strategies to verify the effectiveness of our method.
Revised: We now compare with several other distillation strategies to verify the effectiveness of our method.

Original: We compare with two distillation schemes: feature-based distillation method and relation-based distillation, which is termed as KD and RKD, respectively.
Revised: We compare with two distillation schemes: feature-based distillation method and relation-based distillation, which is termed as KD and RKD, respectively.

Original: Feature-based distillation method aims at minimizing l 2 -distance of teacher & student’s embeddings. Relation-based distillation method aims at minimizing the difference between inter-sample-similarity graph obtained from teacher and student model.
Revised: Feature-based distillation method aims at minimizing l 2 -distance of teacher & student’s embeddings. Relation-based distillation method aims at minimizing the difference between inter-sample-similarity graph obtained from teacher and student model.

Original: As shown in Table 8,
Revised: As shown in Table 8,

Original: BINGO outperforms all theses alternative methods.
Revised: BINGO outperforms all theses alternative methods.

Original: Computational Complexity.
Revised: Computational Complexity.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the batch size is doubled during each forward propagation.
Revised: 2, the batch size is doubled during each forward propagation.

Original: The computation cost is increased compared with SEED (Fang et al., 2020) and MoCov2 (Chen et al., 2020c).
Revised: The computation cost is increased compared with SEED (Fang et al., 2020) and MoCov2 (Chen et al., 2020c).

Original: As for DisCo (Gao et al., 2021), the computational complexity is also increased due to the multiple forward propagation for one sample: once for the mean student, twice for the online student and twice for the teacher model.
Revised: As for DisCo (Gao et al., 2021), the computational complexity is also increased due to the multiple forward propagation for one sample: once for the mean student, twice for the online student and twice for the teacher model.

Original: The total number of forward propagation is 5, 2.5 × bigger than SEED and MoCo-v2.
Revised: The total number of forward propagation is 5, 2.5 × bigger than SEED and MoCo-v2.

Original: For the above analysis, BINGO has a similar computation cost with DisCo.
Revised: For the above analysis, BINGO has a similar computation cost with DisCo.

Original: To compare with SEED and DisCo under the same cost, we distill ResNet-34 from
Revised: To compare with SEED and DisCo under the same cost, we distill ResNet-34 from

Original: ResNet-152 for 100 and 200 epochs respectively.
Revised: ResNet-152 for 100 and 200 epochs respectively.

Original: We compare results of 100 epochs with SEED and 200 epochs with DisCo.
Revised: We compare results of 100 epochs with SEED and 200 epochs with DisCo.

Original: The results are shown in Table 9 and BINGO still shows signiﬁcantly better performance than the two compared methods with the same training cost.
Revised: The results are shown in Table 9 and BINGO still shows signiﬁcantly better performance than the two compared methods with the same training cost.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This paper proposes a new self-supervised distillation method, named BINGO, which bags related instances by matching embeddings of the teacher.
Revised: This paper proposes a new self-supervised distillation method, named BINGO, which bags related instances by matching embeddings of the teacher.

Original: With the instance-wise dataset mapped into a bag-wise one, the new dataset can be applied to the distillation process for small models.
Revised: With the instance-wise dataset mapped into a bag-wise one, the new dataset can be applied to the distillation process for small models.

Original: The knowledge which represents the relation of bagged instances can be transferred by aggregating the bag, including inter-sample and intra-sample aggregation.
Revised: The knowledge which represents the relation of bagged instances can be transferred by aggregating the bag, including inter-sample and intra-sample aggregation.

Original: Our BINGO follows a relation-guided principle, which shows stronger effectiveness than previous relation-agnostic methods.
Revised: Our BINGO follows a relation-guided principle, which shows stronger effectiveness than previous relation-agnostic methods.

Original: The proposed relation-based distillation is a general strategy for improving unsupervised representation, and we hope it would shed light on new directions for unsupervised learning.
Revised: The proposed relation-based distillation is a general strategy for improving unsupervised representation, and we hope it would shed light on new directions for unsupervised learning.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We will release the source code and the pretrained checkpoints to guarantee the reproducibility of this paper.
Revised: We will release the source code and the pretrained checkpoints to guarantee the reproducibility of this paper.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: The proposed approach seeks to improve the performance of unsupervised learning methods for small scale models.
Revised: The proposed approach seeks to improve the performance of unsupervised learning methods for small scale models.

Original: This will improve the application of unsupervised learning methods in more realistic computing resource-constrained scenarios.
Revised: This will improve the application of unsupervised learning methods in more realistic computing resource-constrained scenarios.

Original: As far as we know, our method does not violate any ethical requirements.
Revised: As far as we know, our method does not violate any ethical requirements.

Original: A A PPENDIX A. R ESULTS OF K- MEANS BAGGING STRATEGY
Revised: A A PPENDIX A. R ESULTS OF K- MEANS BAGGING STRATEGY

Original: We also evaluate the performance of using K-means clustering as the bagging strategy.
Revised: We also evaluate the performance of using K-means clustering as the bagging strategy.

Original: According to
Revised: According to

Original: Eq. 2 in the main text, given the pseudo-label q = { q 1 , q 2 , ..., q N } and the anchor instance x a , the bag associated with x a is deﬁned as
Revised: Eq. 2 in the main text, given the pseudo-label q = { q 1 , q 2 , ..., q N } and the anchor instance x a , the bag associated with x a is deﬁned as

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For implementation, ResNet-18 and ResNet-50 are used as the student and teacher model respectively.
Revised: For implementation, ResNet-18 and ResNet-50 are used as the student and teacher model respectively.

Original: We evaluate the linear classiﬁcation accuracy of the student model on ImageNet-1K. We study various cluster numbers C as shown in Tab. 10.
Revised: We evaluate the linear classiﬁcation accuracy of the student model on ImageNet-1K. We study various cluster numbers C as shown in Tab. 10.

Original: We ﬁnd that a bigger cluster number can bring better results than a smaller one.
Revised: We ﬁnd that a bigger cluster number can bring better results than a smaller one.

Original: Noting that the linear classiﬁcation accuracy of bagging with K-nearest neighbors (where k =5) is slightly better than bagging via K-means clustering (with C =20000), i.e . 64 . 0% vs .
Revised: Noting that the linear classiﬁcation accuracy of bagging with K-nearest neighbors (where k =5) is slightly better than bagging via K-means clustering (with C =20000), i.e . 64 . 0% vs .

Original: Moreover, bagging with KNN is more convenient to implement, so we choose the KNN-based bagging strategy in implementation. A. A NALYSIS AND D ISCUSSIONS
Revised: Moreover, bagging with KNN is more convenient to implement, so we choose the KNN-based bagging strategy in implementation. A. A NALYSIS AND D ISCUSSIONS

Original: We now inspect what the student learns during the distillation.
Revised: We now inspect what the student learns during the distillation.

Original: Firstly we compute the average distance between anchor sample x a and its positive samples x p in a bag Ω a over the whole dataset:
Revised: Firstly we compute the average distance between anchor sample x a and its positive samples x p in a bag Ω a over the whole dataset:

Original: [Equation]
Revised: [Equation]

Original: According to Eq.
Revised: According to Eq.

Original: 11, we compute the averaged distance in the bag using distilled student model.
Revised: 11, we compute the averaged distance in the bag using distilled student model.

Original: As shown in Table 11, the averaged distance in a bag is smallest when the student model is distilled with bag-aggregation loss.
Revised: As shown in Table 11, the averaged distance in a bag is smallest when the student model is distilled with bag-aggregation loss.

Original: We also compute the intra-class distance among all intra-class pairwise samples.
Revised: We also compute the intra-class distance among all intra-class pairwise samples.

Original: As shown in Table 12, the proposed method also aggregate the bag of labels with the same ground truth labels on the unseen validation set.
Revised: As shown in Table 12, the proposed method also aggregate the bag of labels with the same ground truth labels on the unseen validation set.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: We visualize the last embedding feature to understanding the aggregating properties of the proposed method.
Revised: We visualize the last embedding feature to understanding the aggregating properties of the proposed method.

Original: 10 classes are randomly selected from validation set.
Revised: 10 classes are randomly selected from validation set.

Original: We provide the t-sne visualization of the student features.
Revised: We provide the t-sne visualization of the student features.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, the same color denotes features with the same label. It can be seen that BINGO gets more compact representations compared with models without distillation or distilling without pulling related samples in a bag
Revised: 4, the same color denotes features with the same label. It can be seen that BINGO gets more compact representations compared with models without distillation or distilling without pulling related samples in a bag

Original: [Table]
Revised: [Table]

Original:       
Revised:       

Original:     
Revised:     

Original: [Figure]
Revised: [Figure]

Original:       
Revised:       

Original:     
Revised:     

Original: [Figure]
Revised: [Figure]

Original: Distill [Figure]
Revised: Distill [Figure]

Original:       
Revised:       

Original:     
Revised:     

Original: [Figure]
Revised: [Figure]