Original: C ONSISTENT

Original: A SSIGNMENT

Original:  FOR R EPRESENTATION L EARNING
Revised: C ONSISTENT A SSIGNMENT FOR R EPRESENTATION L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce Consistent Assignment for Representation Learning (CARL).
Revised: We introduce Consistent Assignment for Representation Learning (CARL).

Original: An unsupervised learning method to learn visual representations by combining contrastive learning with deep clustering.
Revised: An unsupervised learning method to learn visual representations by combining contrastive learning with deep clustering.

Original: By viewing contrastive learning from a clustering perspective, CARL learns unsupervised representations by learning a set of general prototypes that serve as energy anchors to enforce different views of a given image to be assigned to the same prototype.
Revised: By viewing contrastive learning from a clustering perspective, CARL learns unsupervised representations by learning a set of general prototypes that serve as energy anchors to enforce different views of a given image to be assigned to the same prototype.

Original: Unlike contemporary work on contrastive learning with deep clustering, CARL proposes to learn the set of general prototypes in an online fashion, using gradient descent without the necessity of performing ofﬂine clustering or using side algorithms to solve the cluster assignment problem.
Revised: Unlike contemporary work on contrastive learning with deep clustering, CARL proposes to learn the set of general prototypes in an online fashion, using gradient descent without the necessity of performing ofﬂine clustering or using side algorithms to solve the cluster assignment problem.

Original: CARL achieves comparable results with current state-ofthe-art methods in the CIFAR-10, -100, and STL10 datasets.
Revised: CARL achieves comparable results with current state-ofthe-art methods in the CIFAR-10, -100, and STL10 datasets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unsupervised visual representation learning focuses on creating meaningful representations from data and inductive biases.
Revised: Unsupervised visual representation learning focuses on creating meaningful representations from data and inductive biases.

Original: Lately, methods based on Siamese neural networks (Bromley et al., 1994) and contrastive loss functions (He et al., 2020; Chen et al., 2020a) have signiﬁcantly reduced the accuracy gap between supervised and unsupervised based representations.
Revised: Lately, methods based on Siamese neural networks (Bromley et al., 1994) and contrastive loss functions (He et al., 2020; Chen et al., 2020a) have signiﬁcantly reduced the accuracy gap between supervised and unsupervised based representations.

Original: Indeed, for some downstream tasks, unsupervised-based representations already surpass their supervised counterparts (Caron et al., 2020).
Revised: Indeed, for some downstream tasks, unsupervised-based representations already surpass their supervised counterparts (Caron et al., 2020).

Original: In computer vision, approaches to representation learning can be categorized into three groups: (1) contrastive learning methods using instance discrimination, (2) clustering-based methods, and (3) a mixture of the two.
Revised: In computer vision, approaches to representation learning can be categorized into three groups: (1) contrastive learning methods using instance discrimination, (2) clustering-based methods, and (3) a mixture of the two.

Original: Recent state-of-the-art unsupervised representation learning rely on contrastive learning (Tian et al., 2020;
Revised: Recent state-of-the-art unsupervised representation learning rely on contrastive learning (Tian et al., 2020;

Original: He et al., 2020; Chen et al., 2020c;a; Chen & He, 2020;
Revised: He et al., 2020; Chen et al., 2020c;a; Chen & He, 2020;

Original: Oord et al., 2018; Chen et al., 2020b).
Revised: Oord et al., 2018; Chen et al., 2020b).

Original: These methods optimize an instance discrimination pretext task where each image and its transformations are treated as individual classes.
Revised: These methods optimize an instance discrimination pretext task where each image and its transformations are treated as individual classes.

Original: They compare feature vectors of individual images with the goal of organizing the feature space such that similar concepts are placed closer while moving different ones farther.
Revised: They compare feature vectors of individual images with the goal of organizing the feature space such that similar concepts are placed closer while moving different ones farther.

Original: On the other hand, traditional clustering methods aim to learn the data manifold by comparing groups of features that share semantic structure based on a distance metric.
Revised: On the other hand, traditional clustering methods aim to learn the data manifold by comparing groups of features that share semantic structure based on a distance metric.

Original: When combined with deep learning, clustering methods are often designed as two-step algorithms: ﬁrst, a sizable portion of the dataset is clustered, and then the meta clustering information, e.g., prototypes and pseudolabels, are used as supervised signals in a posterior optimization task (Caron et al., 2018; 2019;
Revised: When combined with deep learning, clustering methods are often designed as two-step algorithms: ﬁrst, a sizeable portion of the dataset is clustered, and then the meta clustering information, e.g., prototypes and pseudolabels, are used as supervised signals in a posterior optimization task (Caron et al., 2018; 2019;

Original: Asano et al., 2019; Yan et al., 2020).
Revised: Asano et al., 2019; Yan et al., 2020)

Original: Recent work has also attempted to combine the beneﬁts of contrastive learning and clustering (Li et al., 2020; Caron et al., 2020).
Revised: Recent work has also attempted to combine the beneﬁts of contrastive learning and clustering (Li et al., 2020; Caron et al., 2020).

Original: In particular, Expectation-Maximization approaches alternate between ﬁnding the clusters and maximizing the mutual information between the embeddings and the cluster centroids (Li et al., 2020).
Revised: In particular, Expectation-Maximization approaches alternate between ﬁnding the clusters and maximizing the mutual information between the embeddings and the cluster centroids (Li et al., 2020).

Original: Inspired by them, our work merges the beneﬁts of both approaches by bridging the gap between clustering and contrastive learning.
Revised: Inspired by them, our work merges the beneﬁts of both approaches by bridging the gap between clustering and contrastive learning.

Original: On the one hand, we use unsupervised clustering dynamics to generate robust prototypes that organize the feature space.
Revised: On the one hand, we use unsupervised clustering dynamics to generate robust prototypes that organize the feature space.

Original: On the other, we use contrastive learning to compare the distributions of the views’ assignments w.r.t.
Revised: On the other, we use contrastive learning to compare the distributions of the views’ assignments w.r.t.

Original: the clusters.
Revised: the clusters.

Original: Our experiments show that by mixing both approaches, we can learn useful visual representations in an unsupervised way that performs on par with existing methods in downstream tasks.
Revised: Our experiments show that by mixing both approaches, we can learn useful visual representations in an unsupervised way that performs on par with existing methods in downstream tasks.

Original: From a clustering perspective, we can think of contrastive learning as learning clustered representations at the image-level.
Revised: From a clustering perspective, we can think of contrastive learning as learning clustered representations at the image-level.

Original: However, given the nature of the task, these clusters fail to capture semantic information from the heterogeneous unknown classes because the learned clusters only comprise representations from synthetic views of an image.
Revised: However, given the nature of the task, these clusters fail to capture semantic information from the heterogeneous unknown classes because the learned clusters only comprise representations from synthetic views of an image.

Original: Moreover, since contrastive learning methods handle different images as negatives in the training process, even if two given samples share the same class information, their representations will be pushed farther apart from each other.
Revised: Moreover, since contrastive learning methods handle different images as negatives in the training process, even if two given observations share the same class information, their representations will be pushed farther apart from each other.

Original: In the end, each image will have its own cluster structure.
Revised: In the end, each image will have its own cluster structure.

Original: We propose an alternative method to learn high-level features by clustering views based on consistent assignments.
Revised: We propose an alternative method to learn high-level features by clustering views based on consistent assignments.

Original: Unlike concurrent work that uses K -Nearest Neighbors (Altman, 1992) or KMeans (Lloyd, 1982) as priors to enforce (learn) a cluster mapping, our method learns the prototypes online.
Revised: Unlike concurrent work that uses K -Nearest Neighbors (Altman, 1992) or KMeans (Lloyd, 1982) as priors to enforce (learn) a cluster mapping, our method learns the prototypes online.

Original: Instead of directly maximizing similarities between image embeddings, we force the distribution of positive views’ assignments to be consistent among a set of ﬁnite learnable prototypes.
Revised: Instead of directly maximizing similarities between image embeddings, we force the distribution of positive views’ assignments to be consistent among a set of ﬁnite learnable prototypes.

Original: If the number of prototypes equals the number of observations in the dataset, we would be forcing each cluster only to contain synthetic views of a given observation.
Revised: If the number of prototypes equals the number of observations in the dataset, we would be forcing each cluster only to contain synthetic views of a given observation.

Original: This is equivalent to contrastive learning with instance discrimination.
Revised: This is equivalent to contrastive learning with instance discrimination.

Original: However, if we set the number of prototypes to be smaller than the number of observations in the dataset, by the pigeonhole principle, the learned prototypes will not only cluster different views of an image together, but it will also contain representations of different images that are similar enough to be assigned to the same cluster.
Revised: However, if we set the number of prototypes to be smaller than the number of observations in the dataset, by the pigeonhole principle, the learned prototypes will not only cluster different views of an image together, but it will also contain representations of different images that are similar enough to be assigned to the same cluster.

Original: Regarding our contributions, (i) we propose a learning framework that leverages current contrastive learning methods with clustering-based algorithms to improve the learned representations.
Revised: Regarding our contributions, (i) we propose a learning framework that leverages current contrastive learning methods with clustering-based algorithms to improve the learned representations.

Original: Unlike contemporary work, our method proposes to learn the clusters’ assignments in an online fashion using gradient descent with no need for pre-clustering steps e.g., K -Means or ofﬂine procedures to solve the clustering assignment problem, such as the Sinkhorn-Knopp algorithm (Cuturi, 2013).
Revised: Unlike contemporary work, our method proposes to learn the clusters’ assignments in an online fashion using gradient descent with no need for pre-clustering steps e.g., K -Means or ofﬂine procedures to solve the clustering assignment problem, such as the Sinkhorn-Knopp algorithm (Cuturi, 2013).

Original: (ii) We contrast high-level structures (the distributions of the views over the cluster assignments) instead of low-level ones (such as the representations).
Revised: (ii) We contrast high-level structures (the distributions of the views over the cluster assignments) instead of low-level ones (such as the representations).

Original: And, (iii)
Revised: And, (iii)

Original: Our learned prototypes do not need to hold the semantics of the data but rather become energy anchors that self-organize the space to learn better representations.
Revised: Our learned prototypes do not need to hold the semantics of the data but rather become energy anchors that self-organize the space to learn better representations.

Original: Moreover, our proposed loss function does not require a more extensive set of negative representations, which avoids the common problem of treating representations for the same class as negatives.
Revised: Moreover, our proposed loss function does not require a more extensive set of negative representations, which avoids the common problem of treating representations for the same class as negatives.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: This work builds on top of two main approaches to unsupervised visual representations learning: deep clustering and un/self- learning.
Revised: This work builds on top of two main approaches to unsupervised visual representations learning: deep clustering and un/self-supervised contrastive learning.

Original: Self-supervised learning concerns the idea of devising pretext tasks that extract supervised signals from the data (Doersch et al., 2015; Noroozi & Favaro, 2016;
Revised: Self-supervised learning concerns the idea of devising pretext tasks that extract supervised signals from the data (Doersch et al., 2015; Noroozi & Favaro, 2016;

Original: Gidaris et al., 2018; Zhang et al., 2016).
Revised: Gidaris et al., 2018; Zhang et al., 2016).

Original: Most of these methods work with the same principle.
Revised: Most of these methods work with the same principle.

Original: They corrupt the input with stochastic random transformations and challenge the network to predict some property of the corrupted input.
Revised: They corrupt the input with stochastic random transformations and challenge the network to predict some property of the corrupted input.

Original: One such pretext task is instance discrimination (Dosovitskiy et al., 2015; Wu et al., 2018).
Revised: One such pretext task is instance discrimination (Dosovitskiy et al., 2015; Wu et al., 2018).

Original: It describes a classiﬁcation task in which each image is treated as a unique class and, therefore, stochastic transformations of the same image, often called views, should belong to the same class.
Revised: It describes a classiﬁcation task in which each image is treated as a unique class and, therefore, stochastic transformations of the same image, often called views, should belong to the same class.

Original: Dosovitskiy et al.
Revised: Dosovitskiy et al.

Original: (2015) proposed to optimize this task by learning a linear classiﬁer where the number of output classes matches the number of observations in the dataset.
Revised: (2015) proposed to optimize this task by learning a linear classiﬁer where the number of output classes matches the number of observations in the dataset.

Original: Following, Wu et al.
Revised: Following, Wu et al.

Original: (2018) proposed to use a Noise-Contrastive Estimation (NCE) approximation of the non-parametric softmax classiﬁer that could scale to large datasets gracefully.
Revised: (2018) proposed to use a Noise-Contrastive Estimation (NCE) approximation of the non-parametric softmax classiﬁer that could scale to large datasets gracefully.

Original: Currently, contrastive learning (Hadsell et al., 2006) methods rely on an NCE based loss function called InfoNCE (Oord et al., 2018; Tian et al., 2019).
Revised: Currently, contrastive learning (Hadsell et al., 2006) methods rely on an NCE based loss function called InfoNCE (Oord et al., 2018; Tian et al., 2019).

Original: Recent work describes optimizing the InfoNCE loss through the lens of maximizing the mutual information between representations of the same image (Hjelm et al., 2018; Henaff, 2020;
Revised: Recent work describes optimizing the InfoNCE loss through the lens of maximizing the mutual information between representations of the same image (Hjelm et al., 2018; Henaff, 2020;

Original: Bachman et al., 2019).
Revised: Bachman et al., 2019).

Original: In practice, the success of InfoNCE requires a high number of negative embeddings.
Revised: In practice, the success of InfoNCE requires a high number of negative embeddings.

Original: Nonetheless, since negatives are usually randomly sampled from the dataset, it often leads to a false-negative problem where representations from images of the same class are treated as negatives (Saunshi et al., 2019).
Revised: Nonetheless, since negatives are usually randomly sampled from the dataset, it often leads to a false-negative problem where representations from images of the same class are treated as negatives (Saunshi et al., 2019).

Original: He et al.
Revised: He et al.

Original: (2020) presented MoCo, a contrastive learning framework that employs an additional momentum encoder to provide consistent instance representations for the InfoNCE loss.
Revised: (2020) presented MoCo, a contrastive learning framework that employs an additional momentum encoder to provide consistent instance representations for the InfoNCE loss.

Original: Chen et al.
Revised: Chen et al.

Original: (2020a) presented SimCLR, a Siamese-based (Bromley et al., 1994) contrastive learning method trained with InfoNCE that relies on large bath sizes to draw a high number of negative samples.
Revised: (2020a) presented SimCLR, a Siamese-based (Bromley et al., 1994) contrastive learning method trained with InfoNCE that relies on large bath sizes to draw a high number of negative samples.

Original: BYOL (Grill et al., 2020) proposes a framework that does not require negative samples and learns visual representations by approximating augmented views of the same data point using an (cid:96) 2 loss in the latent space.
Revised: BYOL (Grill et al., 2020) proposes a framework that does not require negative samples and learns visual representations by approximating augmented views of the same data point using an (cid:96) 2 loss in the latent space.

Original: Unlike contrastive learning, we seek to learn prototype vectors that act as anchors on our embedding space.
Revised: Unlike contrastive learning, we seek to learn prototype vectors that act as anchors on our embedding space.

Original: We use these anchors as energy beacons for the images.
Revised: We use these anchors as energy beacons for the images.

Original: Our goal is to use the energy distributions induced by the similarity of the images w.r.t.
Revised: Our goal is to use the energy distributions induced by the similarity of the images w.r.t.

Original: the prototypes to ﬁnd representations that share similarities in the embedding space.
Revised: the prototypes to ﬁnd representations that share similarities in the embedding space.

Original: Recent work proposed clustering-based methods for deep unsupervised representation learning (Asano et al., 2019; Yan et al., 2020; Caron et al., 2019;
Revised: Recent work proposed clustering-based methods for deep unsupervised representation learning (Asano et al., 2019; Yan et al., 2020; Caron et al., 2019;

Original: Li et al., 2020; Caron et al., 2020).
Revised: Li et al., 2020; Caron et al., 2020).

Original: DeepCluster (Caron et al., 2018) learns representations by predicting cluster assignments.
Revised: DeepCluster (Caron et al., 2018) learns representations by predicting cluster assignments.

Original: One of the limitations of this approach is that the classiﬁcation layer needs to be reinitialized once per clusterization.
Revised: One of the limitations of this approach is that the classiﬁcation layer needs to be reinitialized once per clusterization.

Original: DeeperCluster (Caron et al., 2019) builds on top of (Caron et al., 2018) and presents an algorithm to combine hierarchical clustering with unsupervised feature learning using the rotation prediction pretext task (Gidaris et al., 2018).
Revised: DeeperCluster (Caron et al., 2019) builds on top of (Caron et al., 2018) and presents an algorithm to combine hierarchical clustering with unsupervised feature learning using the rotation prediction pretext task (Gidaris et al., 2018).

Original: Similarly, Prototypical Contrastive Learning (PCL) Li et al.
Revised: Similarly, Prototypical Contrastive Learning (PCL) Li et al.

Original: (2020), formulates a self-supervised visual representation learning framework as an Expectation-Maximization (EM) algorithm.
Revised: (2020), formulates a self-supervised visual representation learning framework as an Expectation-Maximization (EM) algorithm.

Original: Our method utilizes a conceptually distinct methodology.
Revised: Our method utilizes a conceptually distinct methodology.

Original: Unlike (Caron et al., 2018; Li et al., 2020), our method does not require a pre-clusterization step of the entire corpus, which vastly reduces memory and computing power requirements.
Revised: Unlike (Caron et al., 2018; Li et al., 2020), our method does not require a pre-clusterization step of the entire corpus, which vastly reduces memory and computing power requirements.

Original: Moreover, since we do not use K -Means clustering as a proxy to learn an additional task, we do not need to reinitialize any layers during optimization, nor is our method susceptible to limitations and assumptions implied by the K -Means algorithm.
Revised: Moreover, since we do not use K -Means clustering as a proxy to learn an additional task, we do not need to reinitialize any layers during optimization, nor is our method susceptible to limitations and assumptions implied by the K -Means algorithm.

Original: Instead, we propose to learn the prototypes end-to-end by consistently enforcing different views of the same image to be assigned to the same prototypes.
Revised: Instead, we propose to learn the prototypes end-to-end by consistently enforcing different views of the same image to be assigned to the same prototypes.

Original: Lastly, our method does not rely on handcrafted pretext tasks.
Revised: Lastly, our method does not rely on handcrafted pretext tasks.

Original: Similar to our work, Caron et al.
Revised: Similar to our work, Caron et al.

Original: (2020) proposes an online clustering method to learn visual representations by contrasting cluster assignments on a second set of latent variables.
Revised: (2020) proposes an online clustering method to learn visual representations by contrasting cluster assignments on a second set of latent variables.

Original: To avoid collapsing modes where all observations are assigned to a few classes, they use the Sinkhorn-Knopp algorithm (Cuturi, 2013) to solve the cluster assignment problem over the latent variables to guide the encoder during learning.
Revised: To avoid collapsing modes where all observations are assigned to a few classes, they use the Sinkhorn-Knopp algorithm (Cuturi, 2013) to solve the cluster assignment problem over the latent variables to guide the encoder during learning.

Original: Unlike (Caron et al., 2020), our method learns cluster assignments endto-end via gradient descent and avoids trivial solutions by enforcing a regularization term over the cluster assignments of views in a given batch.
Revised: Unlike (Caron et al., 2020), our method learns cluster assignments endto-end via gradient descent and avoids trivial solutions by enforcing a regularization term over the cluster assignments of views in a given batch.

Original: Moreover, we use a simpler representation space that does not need a second set of latent variables to stabilize the clusters.
Revised: Moreover, we use a simpler representation space that does not need a second set of latent variables to stabilize the clusters.

Original: Van Gansbeke et al.
Revised: Van Gansbeke et al.

Original: (2020) presented a two-step algorithm for unsupervised classiﬁcation and proposed the SCAN-Loss (Semantic Clustering by Adopting Nearest neighbors) as part of the learning pipeline.
Revised: (2020) presented a two-step algorithm for unsupervised classiﬁcation and proposed the SCAN-Loss (Semantic Clustering by Adopting Nearest neighbors) as part of the learning pipeline.

Original: Their algorithm extracts a set of nearest neighbors from each observation and uses them as priors to learn a second network for semantic clustering.
Revised: Their algorithm extracts a set of nearest neighbors from each observation and uses them as priors to learn a second network for semantic clustering.

Original: Our implementation builds on top of the
Revised: Our implementation builds on top of the

Original: SCAN-Loss, but unlike Van Gansbeke et al.
Revised: SCAN-Loss, but unlike Van Gansbeke et al.

Original: (2020), we employ a Siamese network architecture to learn representations via cluster assignments, end-to-end, without the necessity of optimizing for a second self-supervised task or mining of nearest neighbors.
Revised: (2020), we employ a Siamese network architecture to learn representations via cluster assignments, end-to-end, without the necessity of optimizing for a second self-supervised task or mining of nearest neighbors.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: Our proposal, Consistent Assignment for Representation Learning (CARL), re-frames contrastive learning through a clustering perspective to learn robust representations (Section 2.1).
Revised: Our proposal, Consistent Assignment for Representation Learning (CARL), re-frames contrastive learning through a clustering perspective to learn robust representations (Section 2.1).

Original: CARL builds distributions of the similarities between the prototypes and the image’s views (Section 2.2).
Revised: CARL builds distributions of the similarities between the prototypes and the image’s views (Section 2.2).

Original: To avoid collapsing the representation in a subset of clusters, we impose an uninformative prior to CARL prototypes (Section 2.3).
Revised: To avoid collapsing the representation in a subset of clusters, we impose an uninformative prior to CARL prototypes (Section 2.3).

Original: CARL then optimizes both the similarity to the learned prototypes and the uninformative prior (with a decay schedule for learning).
Revised: CARL then optimizes both the similarity to the learned prototypes and the uninformative prior (with a decay schedule for learning).

Original: Fig.
Revised: Fig.

Original: 1 illustrates the learning pipeline.
Revised: 1 illustrates the learning pipeline.

Original: 2. C ONTRASTIVE L EARNING FROM A C LUSTERING P ERSPECTIVE
Revised: 2. C ONTRASTIVE L EARNING FROM A C LUSTERING P ERSPECTIVE

Original: Let X = { x 1 , x 2 , . . .
Revised: Let X = { x 1 , x 2 , . . .

Original: , x N } be a dataset containing N unlabeled images.
Revised: , x N } be a dataset containing N unlabeled images.

Original: And, let a view v i
Revised: And, let a view v i

Original: = T ( x i ) of an observation x i as the application of a stochastic function T that is designed to change thecontent of x i subjected to preserving the task-relevant information encoded in it. In practice, we can create as many views as needed by applying the stochastic function T . Contrastive learning methods propose to learn visual embeddings by solving an instance discrimination pretext task that is usually
Revised: = T ( x i ) of an observation x i as the application of a stochastic function T that is designed to change the

Original: [Figure]
Revised: [Figure]

Original: Augmenter

Original: Encoder

Original: Encoder

Original: Assigner

Original: Assigner
Revised: Assigner

Original: L
Revised: L

Original: [Figure]
Revised: [Figure]

Original: ai v + i 
Revised: content of x i subjected to preserving the task-relevant information encoded in it.

Original: [Figure] ai z +
Revised: In practice, we can create as many views as needed by applying the stochastic function T .

Original: [Figure] p ai p + [Figure] optimized using the InfoNCE loss (Oord et al., 2018) deﬁned as
Revised: Contrastive learning methods propose to learn visual embeddings by solving an instance discrimination pretext task that is usually optimized using the InfoNCE loss (Oord et al., 2018) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where z ai and z
Revised: where z ai and z

Original: + i are anchor and positive representations taken from an encoder function f such that z ( · ) i =
Revised: + i are anchor and positive representations taken from an encoder function f such that z ( · ) i =

Original: f (cid:0) v ( · ) i (cid:1) , τ is the temperature parameter, and sim ( · ) is a similarity function, e.g., the cosine similarity.
Revised: f (cid:0) v ( · ) i (cid:1) , τ is the temperature parameter, and sim ( · ) is a similarity function, e.g., the cosine similarity.

Original: If we view contrastive learning from a clustering perspective, the InfoNCE loss (1) is minimized when all possible variants (cid:8) v ji (cid:9) j of an image x i are clustered into the same prototype while representations from within a cluster are far apart from the M negative representations in the denominator of (1).
Revised: If we view contrastive learning from a clustering perspective, the InfoNCE loss (1) is minimized when all possible variants (cid:8) v ji (cid:9) j of an image x i are clustered into the same prototype while representations from within a cluster are far apart from the M negative representations in the denominator of (1).

Original: We propose an approach where, instead of comparing against other instances (He et al., 2020) or prototypes of the classes (Li et al., 2020), we learn a set of K general prototypes C = { c 1 , c 2 , . .
Revised: We propose an approach where, instead of comparing against other instances (He et al., 2020) or prototypes of the classes (Li et al., 2020), we learn a set of K general prototypes C = { c 1 , c 2 , . .

Original: . , c K } ,
Revised: . , c K } ,

Original: K (cid:28) N , against which we compare the views to determine their similarity and to promote consistency and conﬁdence when assigning views to clusters.
Revised: K (cid:28) N , against which we compare the views to determine their similarity and to promote consistency and conﬁdence when assigning views to clusters.

Original: I.e., views must agree with high conﬁdence in their cluster assignments.
Revised: I.e., views must agree with high conﬁdence in their cluster assignments.

Original: 2. C ONSISTENT A SSIGNMENT FOR R EPRESENTATION L EARNING
Revised: 2. C ONSISTENT A SSIGNMENT FOR R EPRESENTATION L EARNING

Original: As with previous methods, we treat augmented versions of a given image as views and use them as positive examples for optimization.
Revised: As with previous methods, we treat augmented versions of a given image as views and use them as positive examples for optimization.

Original: Our objective is to transform two positive samples into a distribution of their likelihood to belong to a set of K clusters.
Revised: Our objective is to transform two positive samples into a distribution of their likelihood to belong to a set of K clusters.

Original: To do so, we encode each view through an encoder function z i =
Revised: To do so, we encode each view through an encoder function z i =

Original: f
Revised: f

Original: ( v i ) ∈ R d .
Revised: ( v i ) ∈ R d .

Original: The encoder f comprises a backbone convolutional neural network, such as a ResNet (He et al., 2016), followed by a non-linear multilayer perceptron (MLP) head.
Revised: The encoder f comprises a backbone convolutional neural network, such as a ResNet (He et al., 2016), followed by a non-linear multilayer perceptron (MLP) head.

Original: Our objective is not to cluster the data in an unsupervised manner but rather to learn a set of prototypes that will serve as anchors to differentiate views.
Revised: Our objective is not to cluster the data in an unsupervised manner but rather to learn a set of prototypes that will serve as anchors to differentiate views.

Original: Our hypothesis is that similar views should have similar assignments w.r.t.
Revised: Our hypothesis is that similar views should have similar assignments w.r.t.

Original: the prototypes.
Revised: the prototypes.

Original: Hence, to convert the representations into these assignments, we ﬁrst compare the representation z i against all the prototypes to obtain an energy distribution
Revised: Hence, to convert the representations into these assignments, we ﬁrst compare the representation z i against all the prototypes to obtain an energy distribution

Original: [Equation]
Revised: [Equation]

Original: where q i [ j ] is the j -th element of the energy distribution
Revised: where q i [ j ] is the j -th element of the energy distribution

Original: q i for the
Revised: q i for the

Original: i -th view.
Revised: i -th view.

Original: We learn the set of prototypes through  a linear layer.
Revised: We learn the set of prototypes through an assigner function represented as a linear layer.

Original: Thus, to get a distribution of a given view over all prototypes, we normalize the energy using the softmax function and obtain the posterior probability distribution, i.e., the probability of assigning the view v i to the cluster k .
Revised: Thus, to get a distribution of a given view over all prototypes, we normalize the energy using the softmax function and obtain the posterior probability distribution, i.e., the probability of assigning the view v i to the general prototypes k .

Original: Hence, our normalized probability for the k -th class given our view v i is
Revised: Hence, our normalized probability for the k -th class given our view v i is

Original: [Equation]
Revised: [Equation]

Original:  where q i [ j ] denotes the j -th element of the i -th un-normalized vector output of the classiﬁer for the respective view.
Revised: assigned to [Equation] where q i [ j ] denotes the j -th element of the i -th un-normalized vector output of the assigner for the respective view.

Original: As we mentioned before, our main objective is to contrast the distributions of two views’ likelihood w.r.t.
Revised: As we mentioned before, our main objective is to contrast the distributions of two views’ likelihood w.r.t.

Original: the clusters.
Revised: the clusters.

Original: To do so, the encoder and assigner operate in a Siamese setup where a pair of views from a given sample is independently transformed in its corresponding distribution, see
Revised: To do so, the encoder and assigner operate in a Siamese setup where a pair of views from a given sample is independently transformed in its corresponding distribution, see

Original: Fig.
Revised: Fig.

Original: To ensure the similarity between the views, we optimize the views’ distributions p ai and p + i over the clusters in C , so that the two distributions are consistent with one another.
Revised: To ensure the similarity between the views, we optimize the views’ distributions p ai and p + i over the clusters in C , so that the two distributions are consistent with one another.

Original: In other words, by learning a consistent assignment of views over the clusters, a given prototype will be invariant to augmented versions of an input sample.
Revised: In other words, by learning a consistent assignment of views over the clusters, a given prototype will be invariant to augmented versions of an input sample.

Original: Moreover, because the number of prototypes is smaller than the number of observations in the dataset, the clusters will also contain different observations that share similarities in the embedding space.
Revised: Moreover, because the number of prototypes is smaller than the number of observations in the dataset, the clusters will also contain different observations that share similarities in the embedding space.

Original: We compute the similarity between the views’ distributions as their dot product
Revised: We compute the similarity between the views’ distributions as their dot product

Original: [Equation]
Revised: [Equation]

Original: 4) where B is the size of a minibatch over which we are aggregating the samples.
Revised: where B is the size of a minibatch over which we are aggregating the samples.

Original: In the ideal case of two one-hot vectors signaling the same perfect assignment, the dot product above yields its maximum value of one, and the negative log is minimized.
Revised: In the ideal case of two one-hot vectors signaling the same perfect assignment, the dot product above yields its maximum value of one, and the negative log is minimized.

Original: 2. P REVENTING T RIVIAL S OLUTIONS
Revised: 2. P REVENTING T RIVIAL S OLUTIONS

Original: Only forcing different views v ai and v + i to have the same cluster assignment using our consistency loss (4) leads the networks to ﬁnd a trivial solution where all representations z i are assigned to the same cluster—cf.
Revised: Only forcing different views v a i and v + i to have the same cluster assignment using our consistency loss (4) leads the networks to ﬁnd a trivial solution where all representations z i are assigned to the same cluster—cf.

Original: Fig.
Revised: Fig.

Original: To prevent such triviality, we force the distribution over the classes, P , to be uninformative by minimizing the Kullback-Leibler divergence w.r.t.
Revised: To prevent such triviality, we force the distribution over the classes, P , to be uninformative by minimizing the Kullback-Leibler divergence w.r.t.

Original: a uniform distribution, U .
Revised: a uniform distribution, U .

Original: Our regularization is
Revised: Our regularization is

Original: [Equation]
Revised: [Equation]

Original: where ˆ p ci is the expected distribution over a minibatch of size B ,
Revised: where ˆ p ci is the expected distribution over a minibatch of size B ,

Original: [Equation]
Revised: [Equation]

Original: In other words, we maximize the Shannon entropy of the average distribution of the predictions.
Revised: In other words, we maximize the Shannon entropy of the average distribution of the predictions.

Original: We can interpret the KL-divergence (5) as regularizing the encoder f to encourage the approximate posterior (3) to be closer to the uniform distribution.
Revised: We can interpret the KL-divergence (5) as regularizing the encoder f to encourage the approximate posterior (3) to be closer to the uniform distribution.

Original: Minimizing the KL-divergence (5) will force the predictions within a batch to be spread across all clusters.
Revised: Minimizing the KL-divergence (5) will force the predictions within a batch to be spread across all clusters.

Original: Since we do not know the underlying class distribution in advance, the KL-divergence (5)  acts as a non-informative prior where we assume that the observations X are uniformly assigned among all K prototypes.
Revised: Since we do not know the underlying class distribution in advance, the KL-divergence (5) [Table] acts as a non-informative prior where we assume that the observations X are uniformly assigned among all K prototypes [Table]

Original: By combining the consistency assignment loss (4) with the KL-divergence regularization (5) we obtain our ﬁnal learning objective
Revised: By combining the consistency assignment loss (4) with the KL-divergence regularization (5) we obtain our ﬁnal learning objective

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: where λ e is an epoch-dependent function that returns a scalar that prevents mode collapse at the beginning of training.
Revised: where λ e is an epoch-dependent function that returns a scalar that prevents mode collapse at the beginning of training.

Original: We observed that training is very susceptible to such collapsing to a single assignment if not regularized.
Revised: We observed that training is very susceptible to such collapsing to a single assignment if not regularized.

Original: However, in practice, we noticed that keeping a large ﬁxed value of λ e during training also prevents the encoder from learning more complex representations.
Revised: However, in practice, we noticed that keeping a large ﬁxed value of λ e during training also prevents the encoder from learning more complex representations.

Original: Thus, we recommend a function λ e that decreases as training progresses.
Revised: Thus, we recommend a function λ e that decreases as training progresses.

Original: In theory, any decay schedule, such as an exponential or cosine, could be used.
Revised: In theory, any decay schedule, such as an exponential or cosine, could be used.

Original: We propose a linear decay schedule
Revised: We propose a linear decay schedule

Original: [Equation]
Revised: [Equation]

Original: where b and a denote the start and ending values of the decay, E represents the number of epochs in which the decay will happen, and e is the epoch counter.
Revised: where b and a denote the start and ending values of the decay, E represents the number of epochs in which the decay will happen, and e is the epoch counter.

Original:  U NSUPERVISED F EATURE E VALUATION
Revised:  U NSUPERVISED F EATURE E VALUATION

Original: We evaluate CARL representations extracted from a ResNet-18 backbone encoder and compare the performance with different state-of-the-art methods using the linear evaluation protocol proposed by He et al.
Revised: We evaluate CARL representations extracted from a ResNet-18 backbone encoder and compare the performance with different state-of-the-art methods using the linear evaluation protocol proposed by He et al.

Original: All the models were trained for 200 epochs with the same cosine learning rate decay (Loshchilov & Hutter, 2016) schedule and equal batch sizes.
Revised: All the models were trained for 200 epochs with the same cosine learning rate decay (Loshchilov & Hutter, 2016) schedule and equal batch sizes.

Original: For more information, refer to Appendix A. Table 1 compares our results with previous approaches on CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), and STL-10 (Coates et al., 2011).
Revised: For more information, refer to Appendix A. Table 1 compares our results with previous approaches on CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), and STL-10 (Coates et al., 2011).

Original: We use MoCo v2 as our primary baseline.
Revised: We use MoCo v2 as our primary baseline.

Original: Note that MoCo v2 alone is a solid baseline since it has been shown to outperform SimCLR on ImageNet linear evaluation.
Revised: Note that MoCo v2 alone is a solid baseline since it has been shown to outperform SimCLR on ImageNet linear evaluation.

Original: Our method was able to outperform our primary baseline (MoCo) on most of the datasets and performs on par (if not superior) with other implementations, including
Revised: Our method was able to outperform our primary baseline (MoCo) on most of the datasets and performs on par (if not superior) with other implementations, including

Original: BYOL and SimCLR.
Revised: BYOL and SimCLR.

Original: Moreover, unlike SimCLR and MoCo v2, our method does not use negative samples, nor does it require a momentum-based target encoder to prevent collapsing (Grill et al., 2020).
Revised: Moreover, unlike SimCLR and MoCo v2, our method does not use negative samples, nor does it require a momentum-based target encoder to prevent collapsing (Grill et al., 2020).

Original: The results on Table 1 also show a different side from the state-of-the-art methods since, usually, they are only trained on large-scale datasets like the ImageNet (Deng et al., 2009), which raises questions regarding how they perform of smaller datasets.
Revised: The results on Table 1 also show a different side from the state-of-the-art methods since, usually, they are only trained on large-scale datasets like the ImageNet (Deng et al., 2009), which raises questions regarding how they perform of smaller datasets.

Original: Additionally, we show an ablation study in the Appendix B.
Revised: Additionally, we show an ablation study in the Appendix B.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we presented Consistent Assignment for Representation Learning (CARL).
Revised: In this work, we presented Consistent Assignment for Representation Learning (CARL).

Original: An unsupervised method that learns visual representations by forcing augmented versions of an observation to be consistently assigned over a ﬁnite set of learnable prototypes.
Revised: An unsupervised method that learns visual representations by forcing augmented versions of an observation to be consistently assigned over a ﬁnite set of learnable prototypes.

Original: Unlike contrastive learning methods, we propose a higher-level pretext task that operates over the distributions of views instead of directly optimizing the view’s embeddings.
Revised: Unlike contrastive learning methods, we propose a higher-level pretext task that operates over the distributions of views instead of directly optimizing the view’s embeddings.

Original: Our method also differs from recent work that merges clustering with contrastive learning since it does not require pre-clusterization steps or algorithms to solve the cluster assignment problem.
Revised: Our method also differs from recent work that merges clustering with contrastive learning since it does not require pre-clusterization steps or algorithms to solve the cluster assignment problem.

Original: Instead, we learn a set of general prototypes that act as energy anchors for the views’ representations, entirely online using gradient descent.
Revised: Instead, we learn a set of general prototypes that act as energy anchors for the views’ representations, entirely online using gradient descent.

Original: We studied some of the main components of CARL and the effects of different conﬁgurations of hyperparameters.
Revised: We studied some of the main components of CARL and the effects of different conﬁgurations of hyperparameters.

Original: Lastly, our results show that representations learned by CARL rival or even surpass current state-of-the-art in contrastive learning without resorting to a large number of negative samples or extra encoders. 
Revised: Lastly, our results show that representations learned by CARL rival or even surpass current state-of-the-art in contrastive learning without resorting to a large number of negative samples or extra encoders. A CKNOWLEDGMENTS

Revised: This study was ﬁnanced in part by the Coordenao de Aperfeioamento de Pessoal de Nvel SuperiorBrasil (CAPES) - Finance Code

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: For all experiments, the encoder function f comprises a ResNet-18 backbone followed by a nonlinear 2-hidden layer fully-connected network as projection head as deﬁned by (Grill et al., 2020).
Revised: For all experiments, the encoder function f comprises a ResNet-18 backbone followed by a nonlinear 2-hidden layer fully-connected network as projection head as deﬁned by (Grill et al., 2020) and an assigner function designed as a linear layer.

Original: For all methods, the function f encodes an image x
Revised: For all methods, the function f encodes an image x

Original: i into a 128 -dim representation and we closely follow the hyperparameters proposed by (He et al., 2020).
Revised: i into a 128 -dim representation and we closely follow the hyperparameters proposed by (He et al., 2020).

Original: Namely, learning rate (0.03) and decay schedule (cosine with no restart), as well as the choice of the optimizer (SGD with momentum) and the batch sizes (256), are consistent among all methods.
Revised: Namely, learning rate (0.03) and decay schedule (cosine with no restart), as well as the choice of the optimizer (SGD with momentum) and the batch sizes (256), are consistent among all methods.

Original: The set of data augmentations used to create the views follow (Chen et al., 2020a).
Revised: The set of data augmentations used to create the views follow (Chen et al., 2020a).

Original: Regarding our implementations of MoCo v2, BYOL, and SimCLR, except for MoCo, in which we used the ofﬁcial code repository, the other two methods are our implementations (closely following the ofﬁcial releases in terms of code design and additional hyperparameters).
Revised: Regarding our implementations of MoCo v2, BYOL, and SimCLR, except for MoCo, in which we used the ofﬁcial code repository, the other two methods are our implementations (closely following the ofﬁcial releases in terms of code design and additional hyperparameters).

Original: Lastly, for feature evaluation, we strictly follow the linear evaluation protocol proposed by (He et al., 2020).
Revised: Lastly, for feature evaluation, we strictly follow the linear evaluation protocol proposed by (He et al., 2020).

Original: B A BLATIONS
Revised: B A BLATIONS

Original: In this section, we evaluate the effects of the main hyperparameters of our method.
Revised: In this section, we evaluate the effects of the main hyperparameters of our method.

Original: When not speciﬁed otherwise, all experiments follow a similar protocol.
Revised: When not speciﬁed otherwise, all experiments follow a similar protocol.

Original: We learn representations using a
Revised: We learn representations using a

Original: ResNet-18 backbone trained for 150 epochs, and the KL weight penalty λ e is linearly decayed over the ﬁrst E = 100 epochs.
Revised: ResNet-18 backbone trained for 150 epochs, and the KL weight penalty λ e is linearly decayed over the ﬁrst E = 100 epochs.

Original: To evaluate the multiple experimental setups, we train linear classiﬁers on top of the encoder’s frozen features following the linear evaluation protocol proposed by He et al.
Revised: To evaluate the multiple experimental setups, we train linear classiﬁers on top of the encoder’s frozen features following the linear evaluation protocol proposed by He et al.

Original: and report average Top- 1 accuracy over three independent runs.
Revised: and report average Top- 1 accuracy over three independent runs.

Original: B. D OES D ECREASING THE KL W EIGHT P ENALTY I MPROVES R EPRESENTATION L EARNING ?
Revised: B. D OES D ECREASING THE KL W EIGHT P ENALTY I MPROVES R EPRESENTATION L EARNING ?

Original: The hyperparameter λ e controls the contribution of the KL regularization (5) to the consistency loss (4).
Revised: The hyperparameter λ e controls the contribution of the KL regularization (5) to the consistency loss (4).

Original: Especially at the beginning of training, a higher contribution for the KL term avoids mode collapsing, where the network optimizes the consistency loss (4) by assigning all observations to the same prototype.
Revised: Especially at the beginning of training, a higher contribution for the KL term avoids mode collapsing, where the network optimizes the consistency loss (4) by assigning all observations to the same prototype.

Original: Van Gansbeke et al.
Revised: Van Gansbeke et al.

Original: (2020) make similar claims for the entropy regularization in their SCAN-loss (Van Gansbeke et al., 2020), and suggest a high (constant) value for the scalar hyperparameter
Revised: (2020) make similar claims for the entropy regularization in their SCAN-loss (Van Gansbeke et al., 2020), and suggest a high (constant) value for the scalar hyperparameter

Original: λ e to avoid such trivialities.
Revised: λ e to avoid such trivialities.

Original: We hypothesize that keeping a high value of λ e over the course of training also prevents the network from learning complex features.
Revised: We hypothesize that keeping a high value of λ e over the course of training also prevents the network from learning complex features.

Original: To verify this hypothesis, we trained CARL on the STL-10 (Coates et al., 2011) unsupervised dataset for 200 epochs.
Revised: To verify this hypothesis, we trained CARL on the STL-10 (Coates et al., 2011) unsupervised dataset for 200 epochs.

Original: We measure the performance by training a linear classiﬁer on top of the frozen features of the ResNet-18 backbone.
Revised: We measure the performance by training a linear classiﬁer on top of the frozen features of the ResNet-18 backbone.

Original: We linearly decay the magnitude of the λ e hyperparameter, following (8), from b = 2 . 0 to a = 1 . 0 over the ﬁrst E = 100 epochs instead of keeping it constant for one of the experiments.
Revised: We linearly decay the magnitude of the λ e hyperparameter, following (8), from b = 2 . 0 to a = 1 . 0 over the ﬁrst E = 100 epochs instead of keeping it constant for one of the experiments.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, we observe that the quality of the representations learned by CARL beneﬁts from decreasing the contribution of the KL regularization.
Revised: 2, we observe that the quality of the representations learned by CARL beneﬁts from decreasing the contribution of the KL regularization.

Original: Also, a smaller value of λ e may guide the encoder to a non-optimal solution at the beginning of training.
Revised: Also, a smaller value of λ e may guide the encoder to a non-optimal solution at the beginning of training.

Original: B. D OES THE N UMBER OF G ENERAL P ROTOTYPES I NFLUENCE THE Q UALITY OF THE R EPRESENTATIONS ?
Revised: B. D OES THE N UMBER OF G ENERAL P ROTOTYPES I NFLUENCE THE Q UALITY OF THE R EPRESENTATIONS ?

Original: To evaluate the effect of learning a different number of prototypes, we trained CARL with ResNet18 backbones on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009) for 150 epochs.
Revised: To evaluate the effect of learning a different number of prototypes, we trained CARL with ResNet18 backbones on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009) for 150 epochs.

Original: For all experiments, the KL penalty function λ e starts as b = 4 and is linearly decreased to a = 1 . 5 over the ﬁrst E = 100 epochs.
Revised: For all experiments, the KL penalty function λ e starts as b = 4 and is linearly decreased to a = 1 .over the ﬁrst E = 100 epochs.

Original: Fig.
Revised: Fig.

Original: 3 suggests that over clustering beneﬁts the quality of the learned  representations.
Revised: 3 suggests that over clustering beneﬁts the quality of the learned [Figure] [Figure] representations.

Original: Moreover, the optimal number of general-prototypes depends on the number of actual classes of the dataset, where the best results are obtained when the number of general prototypes is nearly one order of magnitude larger than the number of actual classes of the data.[Figure] [Figure] [Figure]
Revised: Moreover, the optimal number of general prototypes (mapped by the assigner) depends on the number of actual classes of the dataset, where the best results are obtained when the number of general prototypes is nearly one order of magnitude larger than the number of actual classes of the data.