Original: L EARNING TO CLUSTER IN ORDER TO TRANSFER ACROSS DOMAINS AND TASKS
Revised: L EARNING TO CLUSTER IN ORDER TO TRANSFER ACROSS DOMAINS AND TASKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster.
Revised: This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster.

Original: The key insight is that, in addition to features, we can transfer similarity information and this is sufﬁcient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning.
Revised: The key insight is that, in addition to features, we can transfer similarity information and this is sufﬁcient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning.

Original: We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not (pairwise semantic similarity).
Revised: We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not (pairwise semantic similarity).

Original: This similarity is category-agnostic and can be learned from data in the source domain using a similarity network.
Revised: This similarity is category-agnostic and can be learned from data in the source domain using a similarity network.

Original: We then present two novel approaches for performing transfer learning using this similarity function.
Revised: We then present two novel approaches for performing transfer learning using this similarity function.

Original: First, for unsupervised domain adaptation, we design a new loss function to regularize classiﬁcation with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs.
Revised: First, for unsupervised domain adaptation, we design a new loss function to regularize classiﬁcation with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs.

Original: Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network.
Revised: Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network.

Original: Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches.
Revised: Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches.

Original: Using this method, we ﬁrst show state of the art results for the challenging cross-task problem, applied on Omniglot and
Revised: Using this method, we ﬁrst show state of the art results for the challenging cross-task problem, applied on Omniglot and

Original: ImageNet.
Revised: ImageNet.

Original: Our results show that we can reconstruct semantic clusters with high accuracy.
Revised: Our results show that we can reconstruct semantic clusters with high accuracy.

Original: We then evaluate the performance of cross-domain transfer using images from the Ofﬁce-31 and SVHN-MNIST tasks and present top accuracy on both datasets.
Revised: We then evaluate the performance of cross-domain transfer using images from the Ofﬁce-31 and SVHN-MNIST tasks and present top accuracy on both datasets.

Original: Our approach doesn’t explicitly deal with domain discrepancy.
Revised: Our approach doesn’t explicitly deal with domain discrepancy.

Original: If we combine with a domain adaptation loss, it shows further improvement.
Revised: If we combine with a domain adaptation loss , it shows further improvement.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Supervised learning has made signiﬁcant strides in the past decade, with substantial advancements arising from the use of deep neural networks.
Revised: Supervised learning has made signiﬁcant strides in the past decade, with substantial advancements arising from the use of deep neural networks.

Original: However, a large part of this success has come from the existence of extensive labeled datasets.
Revised: However, a large part of this success has come from the existence of extensive labeled datasets.

Original: In many situations, it is not practical to obtain such data due to the amount of effort required or when the task or data distributions change dynamically.
Revised: In many situations, it is not practical to obtain such data due to the amount of effort required or when the task or data distributions change dynamically.

Original: To deal with these situations, the ﬁelds of transfer learning and domain adaptation have explored how to transfer learned knowledge across tasks or domains.
Revised: To deal with these situations, the ﬁelds of transfer learning and domain adaptation have explored how to transfer learned knowledge across tasks or domains.

Original: Many approaches have focused on cases where the distributions of the features and labels have changed, but the task is the same (e.g., classiﬁcation across datasets with the same categories).
Revised: Many approaches have focused on cases where the distributions of the features and labels have changed, but the task is the same (e.g., classiﬁcation across datasets with the same categories).

Original: Cross-task transfer learning strategies, on the other hand, have been widely adopted especially in the computer vision community where features learned by a deep neural network on a large classiﬁcation task have been applied to a wide variety of other tasks (Donahue et al., 2014).
Revised: Cross-task transfer learning strategies, on the other hand, have been widely adopted especially in the computer vision community where features learned by a deep neural network on a large classiﬁcation task have been applied to a wide variety of other tasks (Donahue et al., 2014).

Original: Most of the prior cross-task transfer learning works, however, require labeled target data to learn classiﬁers for the new task.
Revised: Most of the prior cross-task transfer learning works, however, require labeled target data to learn classiﬁers for the new task.

Original: If labels of the target data are absent, there is little choice other than to apply unsupervised approaches such as clustering on the target data with pre-trained feature representations.
Revised: If labels of the target data are absent, there is little choice other than to apply unsupervised approaches such as clustering on the target data with pre-trained feature representations.

Original: In this paper, we focus on the question of what can be transferred (besides features) to support both cross-domain and cross-task transfer learning.
Revised: In this paper, we focus on the question of what can be transferred (besides features) to support both cross-domain and cross-task transfer learning.

Original: We address it with a learned similarity function as the fundamental component of clustering.
Revised: We address it with a learned similarity function as the fundamental component of clustering.

Original: Clustering can then be realized using a neural network trained using the output of the similarity function, which can be successfully used to achieve both cross-task and cross-domain transfer.
Revised: Clustering can then be realized using a neural network trained using the output of the similarity function, which can be successfully used to achieve both cross-task and cross-domain transfer.

Original: The key idea is to formulate the clustering objective to use a learnable (and transferable) term, which in our proposed work is a similarity prediction function.
Revised: The key idea is to formulate the clustering objective to use a learnable (and transferable) term, which in our proposed work is a similarity prediction function.

Original: Our proposed objective function can be easily combined with deep neural networks and optimized end-to-end.
Revised: Our proposed objective function can be easily combined with deep neural networks and optimized end-to-end.

Original: The features and clustering are optimized jointly, hence taking advantage of such side information in a robust way.
Revised: The features and clustering are optimized jointly, hence taking advantage of such side information in a robust way.

Original: Using this method, we show that unsupervised learning can beneﬁt from learning performed on a distinct task, and demonstrate the ﬂexibility of further combining it with a classiﬁcation loss and domain discrepancy loss.
Revised: Using this method, we show that unsupervised learning can beneﬁt from learning performed on a distinct task, and demonstrate the ﬂexibility of further combining it with a classiﬁcation loss and domain discrepancy loss.

Original: In summary, we make several contributions.
Revised: In summary, we make several contributions.

Original: First, we propose to use predictive pairwise similarity as the knowledge that is transferred and formulate a learnable objective function to utilize the pairwise information in a fashion similar to constrained clustering.
Revised: First, we propose to use predictive pairwise similarity as the knowledge that is transferred and formulate a learnable objective function to utilize the pairwise information in a fashion similar to constrained clustering.

Original: We then provide the methodologies to deploy the objective function in both cross-task and cross-domain scenarios with deep neural networks.
Revised: We then provide the methodologies to deploy the objective function in both cross-task and cross-domain scenarios with deep neural networks.

Original: The experimental results for cross-task learning on Omniglot and ImageNet show that we can achieve state of the art clustering results with predicted similarities.
Revised: The experimental results for cross-task learning on Omniglot and ImageNet show that we can achieve state of the art clustering results with predicted similarities.

Original: On the standard domain adaptation benchmark Ofﬁce-31 dataset, we demonstrate improvements over state-of-art even when not performing any explicit domain adaptation, and further improvements if we do.
Revised: On the standard domain adaptation benchmark Ofﬁce-31 dataset, we demonstrate improvements over state-of-art even when not performing any explicit domain adaptation, and further improvements if we do.

Original: Finally, on another domain adaptation task, SVHN-to-MNIST, our approach using Omniglot as the auxiliary dataset achieves top performance with a large margin.
Revised: Finally, on another domain adaptation task, SVHN-to-MNIST, our approach using Omniglot as the auxiliary dataset achieves top performance with a large margin.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transfer Learning: Transfer learning aims to leverage knowledge from the source domain to help learn in the target domain, while only focusing on the performance on the target domain.
Revised: Transfer Learning: Transfer learning aims to leverage knowledge from the source domain to help learn in the target domain, while only focusing on the performance on the target domain.

Original: The type of transferred knowledge includes training instances, features, model parameters and relational knowledge (Pan & Yang, 2010).
Revised: The type of transferred knowledge includes training instances, features, model parameters and relational knowledge (Pan & Yang, 2010).

Original: Pairwise similarity is the meta-knowledge we propose to transfer, which falls in between the last two types.
Revised: Pairwise similarity is the meta-knowledge we propose to transfer, which falls in between the last two types.

Original: The similarity prediction function is a neural network with learned parameters, while the output is the most simpliﬁed form of relational knowledge, i.e., only considering pairwise semantic similarity.
Revised: The similarity prediction function is a neural network with learned parameters, while the output is the most simpliﬁed form of relational knowledge, i.e., only considering pairwise semantic similarity.

Original: Cross-task Transfer Learning: Features learned when trained for ImageNet classiﬁcation (Russakovsky et al., 2015) have boosted the performance of a variety of vision tasks in a supervised setting.
Revised: Cross-task Transfer Learning: Features learned when trained for ImageNet classiﬁcation (Russakovsky et al., 2015) have boosted the performance of a variety of vision tasks in a supervised setting.

Original: For example, new classiﬁcation tasks (Donahue et al., 2014; Yosinski et al., 2014), object detection (Girshick et al., 2014), semantic segmentation (Long et al., 2015a), and image captioning (Vinyals et al., 2015).
Revised: For example, new classiﬁcation tasks (Donahue et al., 2014; Yosinski et al., 2014), object detection (Girshick et al., 2014), semantic segmentation (Long et al., 2015a), and image captioning (Vinyals et al., 2015).

Original: Translated Learning (Dai et al., 2009) has an unsupervised setting similar to ours, but it again focuses only on transferring features across tasks.
Revised: Translated Learning (Dai et al., 2009) has an unsupervised setting similar to ours, but it again focuses only on transferring features across tasks.

Original: Our work explores how learning could beneﬁt from transferring pairwise similarity in an unsupervised setting.
Revised: Our work explores how learning could beneﬁt from transferring pairwise similarity in an unsupervised setting.

Original: Cross-domain Transfer Learning: Also known as domain adaptation (Pan & Yang, 2010), there has recently been a large body of work dealing with domain shift between image datasets by minimizing domain discrepancy (Tzeng et al., 2014;
Revised: Cross-domain Transfer Learning: Also known as domain adaptation (Pan & Yang, 2010), there has recently been a large body of work dealing with domain shift between image datasets by minimizing domain discrepancy (Tzeng et al., 2014;

Original: Long et al., 2015b; Ganin et al., 2016; Sun et al., 2016;
Revised: Long et al., 2015b; Ganin et al., 2016; Sun et al., 2016;

Original: Long et al., 2016; Sener et al., 2016; Carlucci et al., 2017; Long et al., 2017; Zellinger et al., 2017;
Revised: Long et al., 2016; Sener et al., 2016; Carlucci et al., 2017; Long et al., 2017; Zellinger et al., 2017;

Original: Bousmalis et al., 2017).
Revised: Bousmalis et al., 2017).

Original: We address the problem in a complementary way that transfers extra information from the auxiliary dataset and show a larger performance boost with further gains using an additional domain discrepancy loss.
Revised: We address the problem in a complementary way that transfers extra information from the auxiliary dataset and show a larger performance boost with further gains using an additional domain discrepancy loss.

Original: Constrained Clustering: Constrained clustering algorithms can be categorized by how they utilize constraints (e.g. pairwise information).
Revised: Constrained Clustering: Constrained clustering algorithms can be categorized by how they utilize constraints (e.g. pairwise information).

Original: The ﬁrst set of work use the constraints to learn a distance metric.
Revised: The ﬁrst set of work use the constraints to learn a distance metric.

Original: For example, DML (Xing et al., 2003), ITML (Davis et al., 2007), SKMS (Anand et al., 2014), SKKm (Anand et al., 2014; Amid et al., 2016), and SKLR (Amid et al., 2016).
Revised: For example, DML (Xing et al., 2003), ITML (Davis et al., 2007), SKMS (Anand et al., 2014), SKKm (Anand et al., 2014; Amid et al., 2016), and SKLR (Amid et al., 2016).

Original: This group of approaches closely relates to metric learning and needs a clustering algorithm such as K-means in a separate stage to obtain cluster assignments.
Revised: This group of approaches closely relates to metric learning and needs a clustering algorithm such as K-means in a separate stage to obtain cluster assignments.

Original: The second group of work use constraints to formulate the clustering loss.
Revised: The second group of work use constraints to formulate the clustering loss.

Original: For example, CSP (Wang et al., 2014) and COSC (Rangapuram &
Revised: For example, CSP (Wang et al., 2014) and COSC (Rangapuram &

Original: Hein, 2012).
Revised: Hein, 2012).

Original: The third group uses constraints for both metric learning and the clustering objective, such as MPCKMeans (Bilenko et al., 2004) and CECM (Antoine et al., 2012).
Revised: The third group uses constraints for both metric learning and the clustering objective, such as MPCKMeans (Bilenko et al., 2004) and CECM (Antoine et al., 2012).

Original: The fourth group does not use constraints at all.
Revised: The fourth group does not use constraints at all.

Original: A generic clustering algorithms such as K-means (MacQueen et al., 1967), LSC (Chen & Cai, 2011), and LPNMF (Cai et al., 2009) all belong to this category.
Revised: A generic clustering algorithms such as K-means (MacQueen et al., 1967), LSC (Chen & Cai, 2011), and LPNMF (Cai et al., 2009) all belong to this category.

Original: There is a long list of associated works and they are summarized in survey papers, e.g. Davidson & Basu (2007) and Dinler & Tural (2016).
Revised: There is a long list of associated works and they are summarized in survey papers, e.g. Davidson & Basu [Figure] Inputs [Figure]

Revised: (2007) and Dinler & Tural (2016).

Original: Our proposed clustering strategy belongs to the third group.
Revised: Our proposed clustering strategy belongs to the third group.

Original: The constrained clustering methods above are applied to the semi-supervised setting where the groundtruth constraints are sparsely available.
Revised: The constrained clustering methods above are applied to the semi-supervised setting where the groundtruth constraints are sparsely available.

Original: In our unsupervised setting, the ground-truth is unavailable [Figure] Inputs [Figure]
Revised: In our unsupervised setting, the ground-truth is unavailable but predicted constraints are densely available.

Original: but predicted constraints are densely available.

Original: We include all four groups of algorithms in our comparison and show the advantages of the third group.
Revised: We include all four groups of algorithms in our comparison and show the advantages of the third group.

Original:  T HE TRANSFER LEARNING TASKS
Revised:  T HE TRANSFER LEARNING TASKS

Original: To deﬁne the transfer learning problem addressed in this work, we follow the notations used by Pan & Yang (2010).
Revised: To deﬁne the transfer learning problem addressed in this work, we follow the notations used by Pan & Yang (2010).

Original: The goal is to transfer knowledge from source data S =
Revised: The goal is to transfer knowledge from source data S =

Original: ( X S , Y S ) , where X S is the set of data instances and Y S is the corresponding categorical labels, to a target data noted as
Revised: ( X S , Y S ) , where X S is the set of data instances and Y S is the corresponding categorical labels, to a target data noted as

Original: T =
Revised: T =

Original: ( X T , Y T ) .
Revised: ( X T , Y T ) .

Original: The learning is unsupervised since Y T is unknown.
Revised: The learning is unsupervised since Y T is unknown.

Original: The scenario is divided into two cases.
Revised: The scenario is divided into two cases.

Original: One is { Y T } (cid:54) =
Revised: One is { Y T } (cid:54) =

Original: { Y S } , which means the set of categories are not the same and hence the transfer is across tasks.
Revised: { Y S } , which means the set of categories are not the same and hence the transfer is across tasks.

Original: The second case is { Y T } =
Revised: The second case is { Y T } =

Original: { Y S } , but with a domain shift.
Revised: { Y S } , but with a domain shift.

Original: In other words, the marginal probability distributions of the input data are different, i.e., P ( X T ) (cid:54) =
Revised: In other words, the marginal probability distributions of the input data are different, i.e., P ( X T ) (cid:54) =

Original: P ( X S ) .
Revised: P ( X S ) .

Original: The latter is a cross-domain learning problem also called transductive learning.
Revised: The latter is a cross-domain learning problem also called transductive learning.

Original: The domain adaptation approaches which have gained signiﬁcant attention recently belong to the second scenario.
Revised: The domain adaptation approaches which have gained signiﬁcant attention recently belong to the second scenario.

Original: To align with common benchmarks for evaluating transfer learning performance, we further use the notion of an auxiliary dataset and split the source data into S = S (cid:48) ∪ A . S (cid:48)
Revised: To align with common benchmarks for evaluating transfer learning performance, we further use the notion of an auxiliary dataset and split the source data into S = S (cid:48) ∪ A . S (cid:48)

Original: =
Revised: =

Original: ( X (cid:48) S , Y (cid:48) S ) which is only present in the cross-domain transfer scheme and has { Y (cid:48) S } = { Y T } .
Revised: ( X (cid:48) S , Y (cid:48) S ) which is only present in the cross-domain transfer scheme and has { Y (cid:48) S } = { Y T } .

Original: A =
Revised: A =

Original: ( X A , Y A ) is the auxiliary dataset which has a large amount of labeled data and potentially categories as well, and may or may not contain the categories of Y T .
Revised: ( X A , Y A ) is the auxiliary dataset which has a large amount of labeled data and potentially categories as well, and may or may not contain the categories of Y T .

Original: For the cross task scenario, only A and unlabeled T are included, while cross-domain transfer involves A , S (cid:48) , and T .
Revised: For the cross task scenario, only A and unlabeled T are included, while cross-domain transfer involves A , S (cid:48) , and T .

Original: In the following sections we use the above notations and describe the two transfer learning tasks in detail.
Revised: In the following sections we use the above notations and describe the two transfer learning tasks in detail.

Original: Figure 1 illustrates how our approach relates to both tasks.
Revised: Figure 1 illustrates how our approach relates to both tasks.

Original: Transfer across tasks:
Revised: Transfer across tasks:

Original: If the target task has different categories, we cannot directly transfer the classiﬁer from source to target, and there is no labeled target data to use for ﬁne-tuning transferred features.
Revised: If the target task has different categories, we cannot directly transfer the classiﬁer from source to target, and there is no labeled target data to use for ﬁne-tuning transferred features.

Original: Here we propose to ﬁrst reduce the categorization problem to a surrogate same-task problem.
Revised: Here we propose to ﬁrst reduce the categorization problem to a surrogate same-task problem.

Original: We can directly apply transductive transfer learning (Pan & Yang, 2010) to the transformed task.
Revised: We can directly apply transductive transfer learning (Pan & Yang, 2010) to the transformed task.

Original: The cluster structure in the target data is then reconstructed using the predictions in the transformed task.
Revised: The cluster structure in the target data is then reconstructed using the predictions in the transformed task.

Original: See ﬁgure 2 for an illustration.
Revised: See ﬁgure 2 for an illustration.

Original: The source involves a labeled auxiliary dataset A and an unlabeled target dataset T .
Revised: The source involves a labeled auxiliary dataset A and an unlabeled target dataset T .

Original: Y T is the target that must be inferred.
Revised: Y T is the target that must be inferred.

Original: In this scenario the set of categories { Y A } (cid:54) =
Revised: In this scenario the set of categories { Y A } (cid:54) =

Original: { Y T } , and Y T is unknown.
Revised: { Y T } , and Y T is unknown.

Original: We ﬁrst transform the problem of categorization into a pairwise similarity prediction problem.
Revised: We ﬁrst transform the problem of categorization into a pairwise similarity prediction problem.

Original: In other words, we specify a transformation function R such that R ( A ) =
Revised: In other words, we specify a transformation function R such that R ( A ) =

Original: ( X R A , Y R ) , and X R A
Revised: ( X RA , Y R ) , and X RA

Original: = { ( x A,i , x A,j ) } ∀ i,j contains all pairs of data, where { Y R } = { dissimilar, similar } .
Revised: = { ( x A,i , x A,j ) } ∀ i,j contains all pairs of data, where { Y R } = { dissimilar, similar } .

Original: The transformation on the labeled auxiliary data is straightforward.
Revised: The transformation on the labeled auxiliary data is straightforward.

Original: It will be similar if two data instances are from the same category, and vice versa.
Revised: It will be similar if two data instances are from the same category, and vice versa.

Original: We then use it to learn a pairwise similarity prediction function G ( x i , x j ) =
Revised: We then use it to learn a pairwise similarity prediction function G ( x i , x j ) =

Original: y i,j .
Revised: y i,j .

Original: By applying G on T , we can obtain G ( x T,i , x T,j ) =

Original: y T,i,j .

Original: The last step is to infer Y T from

Original: Y R T = { y T,i,j } ∀ i,j , which can be solved using constrained clustering algorithms.

Original: Note that since the

Original: {Y A } = {a,b,c,d} {Y T } = {p,q,r} {Y R } = {similar, dissimilar}
Revised: {Y A } = {a,b,c,d} {Y T } = {p,q,r} {Y R } = {similar, dissimilar}

Original: Different domain Same task {Y R } = {similar, dissimilar}
Revised: Different domain Same task {Y R } = {similar, dissimilar}

Original: Labeled X A
Revised: Labeled X A

Original: Unlabeled X T
Revised: Unlabeled X T

Original: Task Transformation
Revised: Task Transformation

Original: Cluster Reconstruction
Revised: Cluster Reconstruction

Original: G(x i ,x j )
Revised: G(x i ,x j )

Original: Direction of transferring
Revised: Direction of transferring

Original: G(x i ,x j )
Revised: G(x i ,x j )

Original: [Figure]
Revised: [Figure]

Original: (x i ,x j ) actual Y T  is unknown, the algorithm only infers the indices of categories, which could be in order.
Revised: (x i ,x j ) By applying G on T , we can obtain G ( x T,i , x T,j ) = y T,i,j . The last step is to infer Y T from

Revised: Y RT =

Revised: { y T,i,j } ∀ i,j , which can be solved using constrained clustering algorithms.

Revised: Note that since the actual Y T is unknown, the algorithm only infers the indices of categories, which could be in arbitrary order.

Original: The  clusters are expected to contain coherent  categories.
Revised: The resulting clusters are expected to contain coherent semantic categories.

Original: Transfer across domains: The problem setting we consider here is the same as unsupervised domain adaptation.
Revised: Transfer across domains: The problem setting we consider here is the same as unsupervised domain adaptation.

Original: Following the standard evaluation procedure (Long et al., 2017; Ganin et al., 2016), the labeled datasets A is ImageNet and S (cid:48) is one domain in the Ofﬁce-31 dataset.
Revised: Following the standard evaluation procedure (Long et al., 2017; Ganin et al., 2016), the labeled datasets A is ImageNet and S (cid:48) is one domain in the Ofﬁce-31 dataset.

Original: The unlabeled T is another domain in Ofﬁce-31.
Revised: The unlabeled T is another domain in Ofﬁce-31.

Original: The backbone network is pre-trained using A , then ﬁne-tuned with S (cid:48) to learn the classiﬁers.

Original: The goal is to enhance classiﬁcation performance on T by utilizing A , S (cid:48) , and T together.
Revised: The goal is to enhance classiﬁcation performance on T by utilizing A , S (cid:48) , and T together.

Original:  T HE LEARNABLE CLUSTERING OBJECTIVE (LCO)
Revised:  T HE LEARNABLE CLUSTERING OBJECTIVE (LCO)

Original: The key to our approach is the design of a learning objective that can use (noisy) predicted pairwise similarities, and is inspired from constrained clustering which involves using pairwise information in the loss function.
Revised: The key to our approach is the design of a learning objective that can use (noisy) predicted pairwise similarities, and is inspired from constrained clustering which involves using pairwise information in the loss function.

Original: The pairwise information is called must-link/cannot-link constraints or similar/dissimilar pairs (we use the latter).
Revised: The pairwise information is called must-link/cannot-link constraints or similar/dissimilar pairs (we use the latter).

Original: Note that the information is binarized to one and zero for similar and dissimilar pairs, accordingly.
Revised: Note that the information is binarized to one and zero for similar and dissimilar pairs, accordingly.

Original: Although many constrained clustering algorithms  been developed, few of them are scalable with respect to the number of pairwise relationships.
Revised: Although many constrained clustering algorithms have been developed, few of them are scalable with respect to the number of pairwise relationships.

Original: Further, none of them can be easily integrated into deep neural networks.
Revised: Further, none of them can be easily integrated into deep neural networks.

Original: Inspired by the work of Hsu & Kira (2016), we construct a contrastive loss for clustering on the probability outputs of a softmax classiﬁer.
Revised: Inspired by the work of Hsu & Kira (2016), we construct a contrastive loss for clustering on the probability outputs of a softmax classiﬁer.

Original: However, each output node does not have to map to a ﬁxed category but instead each output node represents a probabilistic assignment of a data point to a cluster.
Revised: However, each output node does not have to map to a ﬁxed category but instead each output node represents a probabilistic assignment of a data point to a cluster.

Original: The assignment between output nodes and clusters are formed stochastically during the optimization and is guided by the pairwise similarity.
Revised: The assignment between output nodes and clusters are formed stochastically during the optimization and is guided by the pairwise similarity.

Original: If there is a similar pair, their output distribution should be similar, and vice-versa.
Revised: If there is a similar pair, their output distribution should be similar, and vice-versa.

Original: Speciﬁcally, we use the pair-wise KL-divergence to evaluate the distance between k cluster assignment distributions of two data instances, and use predicted similarity to construct the contrastive loss.
Revised: Speciﬁcally, we use the pair-wise KL-divergence to evaluate the distance between k cluster assignment distributions of two data instances, and use predicted similarity to construct the contrastive loss.

Original: Given a pair of data x p , x q , their corresponding output distributions are deﬁned as P = f ( x p ) and
Revised: Given a pair of data x p , x q , their corresponding output distributions are deﬁned as P = f ( x p ) and

Original: Q = f ( x q ) , while f is the neural network.
Revised: Q = f ( x q ) , while f is the neural network.

Original: The cost of a similar pair is described as :
Revised: The cost of a similar pair is described as :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Feature
Revised: Feature

Original: Feature

Original: Feature
Revised: Feature

Original: Pair Enumeration
Revised: Pair Enumeration

Original: Feature (1,1) Feature (1,2)
Revised: Feature (1,1) Feature (1,2)

Original: Feature (1,3) Feature (2,1) ...
Revised: Feature (1,3) Feature (2,1) ...

Original: Predict Similarity 1 

Revised: Predict Similarity 1

Original: ID 0.7 0.
Revised: ID 0.7 0.

Original: Dense prediction
Revised: Dense prediction

Original: SimilarDissimilar
Revised: SimilarDissimilar

Original: [Figure]
Revised: [Figure]

Original: The cost L ( x p , x q ) + is symmetric w.r.t.
Revised: The cost L ( x p , x q ) + is symmetric w.r.t.

Original: x p , x q , in which P (cid:63) and Q (cid:63) are alternatively assumed to be constant.
Revised: x p , x q , in which P (cid:63) and Q (cid:63) are alternatively assumed to be constant.

Original: Each KL-divergence factor D KL ( P (cid:63) ||Q ) becomes a unary function whose gradient is simply ∂ D KL ( P (cid:63) ||Q ) /∂ Q .
Revised: Each KL-divergence factor D KL ( P (cid:63) ||Q ) becomes a unary function whose gradient is simply ∂ D KL ( P (cid:63) ||Q ) /∂ Q .

Original: If x p , x q comes from a pair which is dissimilar, their output distributions are expected to be different, which can be deﬁned as a hinge-loss function:
Revised: If x p , x q comes from a pair which is dissimilar, their output distributions are expected to be different, which can be deﬁned as a hinge-loss function:

Original: [Equation]
Revised: [Equation]

Original: Given a pair of data with similarity prediction function G ( x p , x q ) ∈ { 0 , 1 } , which is introduced in section 3, the total loss can be deﬁned as a contrastive loss (where we use integer 1 to represent a similar pair):
Revised: Given a pair of data with similarity prediction function G ( x p , x q ) ∈ { 0 , 1 } , which is introduced in section 3, the total loss can be deﬁned as a contrastive loss (where we use integer 1 to represent a similar pair):

Original: [Equation]
Revised: [Equation]

Original: We refer to equation 5 as LCO.
Revised: We refer to equation 5 as LCO.

Original: Function G is the learnable part that utilizes prior knowledge and is trained with auxiliary dataset A before optimizing LCO.
Revised: Function G is the learnable part that utilizes prior knowledge and is trained with auxiliary dataset A before optimizing LCO.

Original: Two particular characteristics of the clustering criterion are worth mentioning: (1) There is no need to deﬁne cluster centers.
Revised: Two particular characteristics of the clustering criterion are worth mentioning: (1) There is no need to deﬁne cluster centers.

Original: (2) There is no predeﬁned metric applied on the feature representation.
Revised: (2) There is no predeﬁned metric applied on the feature representation.

Original: Instead, the divergence is calculated directly on the cluster assignment; therefore, both feature representation and clustering are jointly optimized using back-propagation through the deep neural networks.
Revised: Instead, the divergence is calculated directly on the cluster assignment; therefore, both feature representation and clustering are jointly optimized using back-propagation through the deep neural networks.

Original: 4. T HE P AIRWISE S IMILARITY P REDICTION N ETWORK
Revised: 4. T HE P AIRWISE S IMILARITY P REDICTION N ETWORK

Original: Although there is no restriction of how G should be constructed, we choose deep convolution neural networks due to their efﬁciency in vision tasks.
Revised: Although there is no restriction of how G should be constructed, we choose deep convolution neural networks due to their efﬁciency in vision tasks.

Original: We design a network architecture inspired from
Revised: We design a network architecture inspired from

Original: Zagoruyko & Komodakis (2015).
Revised: Zagoruyko & Komodakis (2015).

Original: While they use it to predict image patch similarity, we use it to predict image-level semantic similarity.
Revised: While they use it to predict image patch similarity, we use it to predict image-level semantic similarity.

Original: However, the Siamese architecture used in Zagoruyko &
Revised: However, the Siamese architecture used in Zagoruyko &

Original: Komodakis (2015) is not efﬁcient in both training and inference, especially when pairwise information is dense.
Revised: Komodakis (2015) is not efﬁcient in both training and inference, especially when pairwise information is dense.

Original: Therefore, instead of using Siamese architecture, we keep the single column backbone but add a pair-enumeration layer on top of the feature extraction network.
Revised: Therefore, instead of using Siamese architecture, we keep the single column backbone but add a pair-enumeration layer on top of the feature extraction network.

Original: The pair-enumeration layer enumerates all pairs of feature vectors within a mini-batch and concatenates the features.
Revised: The pair-enumeration layer enumerates all pairs of feature vectors within a mini-batch and concatenates the features.

Original: Suppose the input of the layer is 10 × 512 with the mini-batch size 10 and the feature dimension 512 ; then the output of the pair-enumeration layer will be 100 × 1024 (self-pairs included).
Revised: Suppose the input of the layer is 10 × 512 with the mini-batch size 10 and the feature dimension 512 ; then the output of the pair-enumeration layer will be 100 × 1024 (self-pairs included).

Original: The architecture is illustrated in ﬁgure 3.
Revised: The architecture is illustrated in ﬁgure 3.

Original: We add one hidden fully connected layer on top of the enumeration layer and a binary classiﬁer at the end.
Revised: We add one hidden fully connected layer on top of the enumeration layer and a binary classiﬁer at the end.

Original: We use the standard cross-entropy loss and train it end-to-end.
Revised: We use the standard cross-entropy loss and train it end-to-end.

Original: The supervision for training was obtained by converting the ground-truth category labels into binary similarity, i.e., if two samples are from the same class then their label will be similar , otherwise dissimilar .
Revised: The supervision for training was obtained by converting the ground-truth category labels into binary similarity, i.e., if two samples are from the same class then their label will be similar , otherwise dissimilar .

Original: The inference is also end-to-end, and it outputs predictions among all similarity pairs in a mini-batch.
Revised: The inference is also end-to-end, and it outputs predictions among all similarity pairs in a mini-batch.

Original: The output probability g ∈ [0 , 1] with 1 means more similar.
Revised: The output probability g ∈ [0 , 1] with 1 means more similar.

Original: We binarize g at 0 .
Revised: We binarize g at 0 .

Original: 5 to obtain discrete similarity predictions.
Revised: 5 to obtain discrete similarity predictions.

Original: In the following sections, we simpliﬁed the notation of the pairwise similarity prediction network as G .
Revised: In the following sections, we simpliﬁed the notation of the pairwise similarity prediction network as G .

Original: Once G is learned, it then works as a static function in our experiments.
Revised: Once G is learned, it then works as a static function in our experiments.

Original: 4. T HE OBJECTIVE FUNCTION WITH DENSE SIMILARITY PREDICTION
Revised: 4. T HE OBJECTIVE FUNCTION WITH DENSE SIMILARITY PREDICTION

Original: Since the pairwise prediction of a mini-batch can be densely obtained from G , to efﬁciently utilize the pair-wise information without forwarding each data multiple times we also combine the pairenumeration layer described in section 4.1 with equation 5.
Revised: Since the pairwise prediction of a mini-batch can be densely obtained from G , to efﬁciently utilize the pair-wise information without forwarding each data multiple times we also combine the pairenumeration layer described in section 4.1 with equation 5.

Original: In this case, the outputs of softmax are
Revised: In this case, the outputs of softmax are

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: enumerated in pairs.
Revised: enumerated in pairs.

Original: Let  the set of all tuples ( p, q ) ,  p and q are indices of sample in a mini-batch.
Revised: Let D be the set of all tuples ( p, q ) , while p and q are indices of sample in a mini-batch.

Original: The dense clustering loss L d for a mini-batch is calculated by:
Revised: The dense clustering loss L d for a mini-batch is calculated by:

Original: [Equation]
Revised: [Equation]

Original: We use L d standalone with deep neural networks to reconstruct semantic clusters and for transfer learning across tasks (ﬁgure 4).
Revised: We use L d standalone with deep neural networks to reconstruct semantic clusters and for transfer learning across tasks (ﬁgure 4).

Original: We call the architecture the constrained clustering network (CCN).
Revised: We call the architecture the constrained clustering network (CCN).

Original: 4. C OMBINING WITH O THER O BJECTIVES
Revised: 4. C OMBINING WITH O THER O BJECTIVES

Original: In the cross-domain case, we additionally have labeled source data.
Revised: In the cross-domain case, we additionally have labeled source data.

Original: This enables us to use LCO for T while using classiﬁcation loss for S (cid:48) .
Revised: This enables us to use LCO for T while using classiﬁcation loss for S (cid:48) .

Original: The overall training procedure is similar to previous domain adaptation approaches in that both source and target data are mixed in a mini-batch, but different losses are applied.
Revised: The overall training procedure is similar to previous domain adaptation approaches in that both source and target data are mixed in a mini-batch, but different losses are applied.

Original: We denote the source domain images in a mini-batch b as X b S and the target domain images X b T with its set of dense pairs D b T .
Revised: We denote the source domain images in a mini-batch b as X b S and the target domain images X b T with its set of dense pairs D b T .

Original: The loss function L cd for cross-domain transfer in a mini-batch can then be formulated as:
Revised: The loss function L cd for cross-domain transfer in a mini-batch can then be formulated as:

Original: [Equation]
Revised: [Equation]

Original: The L cluster and L cls share the same outputs from network f .
Revised: The L cluster and L cls share the same outputs from network f .

Original: Although L cluster does not force the mapping between clusters and categories, L cls does.
Revised: Although L cluster does not force the mapping between clusters and categories, L cls does.

Original: Therefore the learned classiﬁer can be applied on target data directly and picks the maximum probability for the predicted class.
Revised: Therefore the learned classiﬁer can be applied on target data directly and picks the maximum probability for the predicted class.

Original: Note that in our loss function, there is no term to explicitly match the feature distribution between source and target; it merely transfers more knowledge in the form of constraints to regularize the learning of the classiﬁer.
Revised: Note that in our loss function, there is no term to explicitly match the feature distribution between source and target; it merely transfers more knowledge in the form of constraints to regularize the learning of the classiﬁer.

Original: There is also no requirement for the architecture to utilize hidden layers.
Revised: There is also no requirement for the architecture to utilize hidden layers.

Original: Therefore our approach has the large ﬂexibility to be combined with other domain adaptation strategies.
Revised: Therefore our approach has the large ﬂexibility to be combined with other domain adaptation strategies.

Original: Figure 5 illustrates the architectures CCN + and CCN ++ used in our cross-domain experiments.
Revised: Figure 5 illustrates the architectures CCN + and CCN ++ used in our cross-domain experiments.

Revised:  E XPERIMENTS

Revised: This section contains the evaluations with four image datasets and covers both cross-task and crossdomain schemes.

Revised: The details are described in below, and the differences between experimental settings are illustrated in appendix A.

Original: E XPERIMENTS 5. U NSUPERVISED C ROSS -T ASK T RANSFER L EARNING 5.1. S ETUP
Revised: 5. U NSUPERVISED C ROSS -T ASK T RANSFER L EARNING 5.1. S ETUP

Original: The Omniglot dataset (Lake et al., 2015) contains 1623 different handwritten characters and each of them has 20 images drawn by different people.
Revised: The Omniglot dataset (Lake et al., 2015) contains 1623 different handwritten characters and each of them has 20 images drawn by different people.

Original: The characters are from 50 different alphabets and were separated to 30 background sets Omniglot bg and 20 evaluation sets Omniglot eval by the author.
Revised: The characters are from 50 different alphabets and were separated to 30 background sets Omniglot bg and 20 evaluation sets Omniglot eval by the author.

Original: We use the Omniglot bg as the auxiliary dataset ( A ) and the Omniglot eval as the target data ( T ).
Revised: We use the Omniglot bg as the auxiliary dataset ( A ) and the Omniglot eval as the target data ( T ).

Original: The total number of characters in Omniglot bg is 964, which can be regarded as the number of categories available to learn the semantic similarity.
Revised: The total number of characters in Omniglot bg is 964, which can be regarded as the number of categories available to learn the semantic similarity.

Original: The goal is to cluster Omniglot eval to reconstruct its semantic categories, without ever having any labels.
Revised: The goal is to cluster Omniglot eval to reconstruct its semantic categories, without ever having any labels.

Original: The G function has a backbone neural network with four 3x3 convolution layers followed by 2x max-pooling with stride 2.
Revised: The G function has a backbone neural network with four 3x3 convolution layers followed by 2x max-pooling with stride 2.

Original: All hidden layers are followed by batch normalization (Ioffe & Szegedy, 2015) and rectiﬁed linear unit.
Revised: All hidden layers are followed by batch normalization (Ioffe & Szegedy, 2015) and rectiﬁed linear unit.

Original: To prepare the inputs for training, the images from Omniglot bg were resized to 32x32 and normalized to zero mean with unit standard deviation.
Revised: To prepare the inputs for training, the images from Omniglot bg were resized to 32x32 and normalized to zero mean with unit standard deviation.

Original: Each mini-batch has a size of 100 and is sampled from a random 20 characters to make sure the amount of similar pairs is reasonable.
Revised: Each mini-batch has a size of 100 and is sampled from a random 20 characters to make sure the amount of similar pairs is reasonable.

Original: After pair enumeration, there are 10000 pairs subject to the loss function, which is a two-class cross entropy loss.
Revised: After pair enumeration, there are 10000 pairs subject to the loss function, which is a two-class cross entropy loss.

Original: The ground-truth similarity is obtained by converting the categorical labels.
Revised: The ground-truth similarity is obtained by converting the categorical labels.

Original: The loss of G is optimized by stochastic gradient descent and is the only part trained in a supervised manner in this section.
Revised: The loss of G is optimized by stochastic gradient descent and is the only part trained in a supervised manner in this section.

Original: The Constrained Clustering Network (CCN) is used to reconstruct the semantic clusters in the target dataset using outputs from G .
Revised: The Constrained Clustering Network (CCN) is used to reconstruct the semantic clusters in the target dataset using outputs from G .

Original: The network has four 3x3 convolution layers followed by 2x max-pooling with stride 2, one hidden fully-connected layer, and one cluster assignment layer which is also fully connected.
Revised: The network has four 3x3 convolution layers followed by 2x max-pooling with stride 2, one hidden fully-connected layer, and one cluster assignment layer which is also fully connected.

Original: The number of output nodes in the last layer is equal to the number of potential clusters in the target data.
Revised: The number of output nodes in the last layer is equal to the number of potential clusters in the target data.

Original: The output distribution is enumerated in pairs before sending to
Revised: The output distribution is enumerated in pairs before sending to

Original: LCO.
Revised: LCO.

Original: The network is randomly initialized, trained end-to-end, and optimized by stochastic gradient descent with randomly sampled 100 images per mini-batch.
Revised: The network is randomly initialized, trained end-to-end, and optimized by stochastic gradient descent with randomly sampled 100 images per mini-batch.

Original: Note the G function used by LCO is ﬁxed during the optimization.
Revised: Note the G function used by LCO is ﬁxed during the optimization.

Original: The input data preparation is the same as above, except now the data is from Omniglot eval .
Revised: The input data preparation is the same as above, except now the data is from Omniglot eval .

Original: Speciﬁcally, during the training, the same mini-batch is given to both G and
Revised: Speciﬁcally, during the training, the same mini-batch is given to both G and

Original: CCN.
Revised: CCN.

Original: The dense pairwise similarity predictions from G are sent to LCO and are then fully utilized.
Revised: The dense pairwise similarity predictions from G are sent to LCO and are then fully utilized.

Original: The only hyper-parameter in LCO is σ , and we set it to 2 for all our experiments.
Revised: The only hyper-parameter in LCO is σ , and we set it to 2 for all our experiments.

Original: Omniglot eval contains 20 alphabets and each one can be used as a standalone dataset.
Revised: Omniglot eval contains 20 alphabets and each one can be used as a standalone dataset.

Original: Thetarget datasets contain a varied number (20 to 47) of characters.
Revised: Thetarget datasets contain a varied number (20 to 47) of characters.

Original: Therefore we can evaluate the reconstruction performance under a varied number of ground-truth clusters.
Revised: Therefore we can evaluate the reconstruction performance under a varied number of ground-truth clusters.

Original: We tested the situations when the number of character ( K ) in an alphabet is known and unknown.
Revised: We tested the situations when the number of character ( K ) in an alphabet is known and unknown.

Original: When K is known, we set the target number of clusters in the clustering algorithm equal to the true number of characters.
Revised: When K is known, we set the target number of clusters in the clustering algorithm equal to the true number of characters.

Original: If K is unknown, the common practice is to set it to a large number so that the data from different categories will not be forced to be in the same cluster.
Revised: If K is unknown, the common practice is to set it to a large number so that the data from different categories will not be forced to be in the same cluster.

Original: In the experiment, we merely set K to 100, which is much larger than the largest dataset (47).
Revised: In the experiment, we merely set K to 100, which is much larger than the largest dataset (47).

Original: All constrained clustering algorithms can be used to reconstruct the semantic clusters for our problem.
Revised: All constrained clustering algorithms can be used to reconstruct the semantic clusters for our problem.

Original: Since there are mis-predictions in G , robustness to noise is the most important factor.
Revised: Since there are mis-predictions in G , robustness to noise is the most important factor.

Original: Here we include all four types of constrained clustering algorithms introduced in section 2 as the baselines.
Revised: Here we include all four types of constrained clustering algorithms introduced in section 2 as the baselines.

Original: We provide the full set of pairwise constraints to all algorithms including ours.
Revised: We provide the full set of pairwise constraints to all algorithms including ours.

Original: In other words, given an alphabet of 20 characters, it contains 400 images and 16000 predicted similarities from G (note that G makes predictions for both orders of a pair and for self-pairs).
Revised: In other words, given an alphabet of 20 characters, it contains 400 images and 160000 predicted similarities from G (note that G makes predictions for both orders of a pair and for self-pairs).

Original: The full pairwise similarities were presented to all algorithms in random order, while we empirically found it has no noticeable effect on results.
Revised: The full pairwise similarities were presented to all algorithms in random order, while we empirically found it has no noticeable effect on results.

Original: We pick the baseline approaches based on code availability and scalability concerning the number of pairwise constraints.
Revised: We pick the baseline approaches based on code availability and scalability concerning the number of pairwise constraints.

Original: Therefore we have shown results for K-means (MacQueen et al., 1967), LPNMF 
Revised: Therefore we have shown results for K-means (MacQueen et al., 1967), LPNMF (Cai et al., 2009), LSC (Chen & Cai, 2011), ITML (Davis et al., 2007), SKKm (Anand et al., 2014), SKLR (Amid et al., 2016), SKMS (Anand et al., 2014), CSP (Wang et al., 2014) , and MPCK-means (Bilenko et al., 2004) as our baselines. We use the default parameters for each algorithm provided by

Original: [Table]
Revised: [Table]

Revised: the original author except for K , the number of clusters.

Revised: We use the same normalized images used in

Revised: CCN for all algorithms.

Original: The evaluation uses two clustering metrics.
Revised: The evaluation uses two clustering metrics.

Original: The ﬁrst is normalized-mutual information (NMI) (Strehl & Ghosh, 2002) which is widely used for clustering.
Revised: The ﬁrst is normalized-mutual information (NMI) (Strehl & Ghosh, 2002) which is widely used for clustering.

Original: The second one is the clustering accuracy (ACC)
Revised: The second one is the clustering accuracy (ACC)

Original: (Yang et al., 2010).
Revised: (Yang et al., 2010).

Original: The ACC metric ﬁrst ﬁnds the one-to-one matching between predicted clusters and ground-truth labels [Table] and then calculates the classiﬁcation accuracy based on the mapping.
Revised: The ACC metric ﬁrst ﬁnds the one-to-one matching between predicted clusters and ground-truth labels, and then calculates the classiﬁcation accuracy based on the mapping.

Original: All data outside the matched clusters will be regarded as mis-predictions.
Revised: All data outside the matched clusters will be regarded as mis-predictions.

Original: To get high ACC, the algorithm has to generate coherent clusters where each cluster includes most of the data in a category; otherwise the score drops quickly.
Revised: To get high ACC, the algorithm has to generate coherent clusters where each cluster includes most of the data in a category; otherwise the score drops quickly.

Original: Therefore ACC provides better discrimination to evaluate whether the semantic clusters have been reconstructed well.
Revised: Therefore ACC provides better discrimination to evaluate whether the semantic clusters have been reconstructed well.

Original: 5.1. R ESULTS AND D ISCUSSIONS
Revised: 5.1. R ESULTS AND D ISCUSSIONS

Original: We report the average performance over the 20 alphabets in table 1.
Revised: We report the average performance over the 20 alphabets in table 1.

Original: Our approach achieved the top performance on both metrics.
Revised: Our approach achieved the top performance on both metrics.

Original: The CCN demonstrates strong robustness on the challenging scenario of unknown K .
Revised: The CCN demonstrates strong robustness on the challenging scenario of unknown K .

Original: It achieved 78.1% average accuracy.
Revised: It achieved 78.1% average accuracy.

Original: Compared with 82.4% when K is known,
Revised: Compared with 82.4% when K is known,

Original: CCN has a relatively small drop.
Revised: CCN has a relatively small drop.

Original: Compared to the second best algorithm, CSP, which is 65.4%,
Revised: Compared to the second best algorithm, CSP, which is 65.4%,

Original: CCN outperforms it with a large gap.
Revised: CCN outperforms it with a large gap.

Original: The classical approach MPCK-means works surprisingly well when the number of clusters is known, but its performance dropped dramatically from 81.9% to 53.9% when K = 100 .
Revised: The classical approach MPCK-means works surprisingly well when the number of clusters is known, but its performance dropped dramatically from 81.9% to 53.9% when K = 100 .

Original: In the performance breakdown for the 20 individual alphabets, CCN achieved 94% clustering accuracy on Old Church Slavonic Cyrillic , which has 45 characters (appendix table 3).
Revised: In the performance breakdown for the 20 individual alphabets, CCN achieved 94% clustering accuracy on Old Church Slavonic Cyrillic , which has 45 characters (appendix table 5).

Original: Therefore the results show the feasibility of reconstructing semantic clusters using only noisy similarity predictions.
Revised: Therefore the results show the feasibility of reconstructing semantic clusters using only noisy similarity predictions.

Original: When to use the semantic similarity?
Revised: When to use the semantic similarity?

Original: The experiments in table 1 show a clear trend that utilizing the pairwise constraints jointly for both metric learning and minimizing the clustering loss achieves the best performance, including both MPCK-means and CCN.
Revised: The experiments in table 1 show a clear trend that utilizing the pairwise constraints jointly for both metric learning and minimizing the clustering loss achieves the best performance, including both MPCK-means and CCN.

Original: In the case of unknown number of clusters, where we set K = 100 , the algorithms that use constraints to optimize clustering loss have better robustness, for example, CSP and CCN.
Revised: In the case of unknown number of clusters, where we set K = 100 , the algorithms that use constraints to optimize clustering loss have better robustness, for example, CSP and CCN.

Original: The group that only use constraints for metric learning (ITML, SKMS, SKKm, and SKLR) signiﬁcantly outperform the group that does not use it (K-means, LPNMF, LSC).
Revised: The group that only use constraints for metric learning (ITML, SKMS, SKKm, and SKLR) signiﬁcantly outperform the group that does not use it (K-means, LPNMF, LSC).

Original: However, their performance are still far behind CCN.
Revised: However, their performance are still far behind CCN.

Original: Our results conﬁrm the importance of jointly optimizing the metric and clustering.
Revised: Our results conﬁrm the importance of jointly optimizing the metric and clustering.

Original: The robustnessof the algorithm against noisy similarity prediction is the key factor to enable the cross-task transfer framework.
Revised: The robustness against noisy similarity prediction is the key factor to enable the cross-task transfer framework.

Original: To our best  knowledge, table 1 is the ﬁrst comprehensive robustness comparisons using predicted constraints learned from real data instead of converting from groundtruth labels.
Revised: To the best of our knowledge, table 1 is the ﬁrst comprehensive robustness comparisons using predicted constraints learned from real data instead of converting from ground-truth labels.

Original: The accuracy of G in our experiment is shown in appendix table 5 and demonstrates the reasonable performance of G which is on par with Matching-Net (Vinyals et al., 2016).
Revised: The accuracy of G in our experiment is shown in appendix table 7 and demonstrates the reasonable performance of G which is on par with Matching-Net (Vinyals et al., 2016).

Original: After binarizing the prediction at 0.5 probability, the similar pair precision, similar pair recall, dissimilar pair precision, and dissimilar pair recall among the 659 characters are (0 . 392 , 0 . 927 , 0 . 999 , 0 . 995) , accordingly.
Revised: After binarizing the prediction at 0.5 probability, the similar pair precision, similar pair recall, dissimilar pair precision, and dissimilar pair recall among the 659 characters are (0 . 392 , 0 . 927 , 0 . 999 , 0 . 995) , accordingly.

Original: The binarized predictions are better than uniform random guess (0 . 002 , 0 .
Revised: The binarized predictions are better than uniform random guess (0 . 002 , 0 .

Original: 998 , 0 . 500) , but are still noisy.
Revised: 998 , 0 . 500) , but are still noisy.

Original: Therefore it is very challenging for constrained clustering.
Revised: Therefore it is very challenging for constrained clustering.

Original: The visualization of the robustness range of CCN are provided in appendix D, and shows that the robustness is related to the density of pairs involved in a mini-batch.
Revised: The visualization of the robustness range of CCN are provided in appendix D, and shows that the robustness is related to the density of pairs involved in a mini-batch.

Original: We hypothesize that during the optimization, the gradients from wrongly predicted pairs are canceled out by each other or by the correctly predicted pairs.
Revised: We hypothesize that during the optimization, the gradients from wrongly predicted pairs are canceled out by each other or by the correctly predicted pairs.

Original: Therefore the overall gradient still moves the solution towards a better clustering result.
Revised: Therefore the overall gradient still moves the solution towards a better clustering result.

Original: How to predict K ?
Revised: How to predict K ?

Original: Inferring the number of clusters ( NC ) is a hard problem, but with the pairwise similarity information, it becomes feasible.
Revised: Inferring the number of clusters ( NC ) is a hard problem, but with the pairwise similarity information, it becomes feasible.

Original: For evaluation, we compute the difference between the number of dominant clusters ( NDC ) and the true number of categories ( NC gt ) in a dataset.
Revised: For evaluation, we compute the difference between the number of dominant clusters ( NDC ) and the true number of categories ( NC gt ) in a dataset.

Original: We use a naive deﬁnition for NDC , which is the number of clusters that have a size larger than expected size when data is sampled uniformly.
Revised: We use a naive deﬁnition for NDC , which is the number of clusters that have a size larger than expected size when data is sampled uniformly.

Original: In other words, NDC = (cid:80) Ki =1 [ C i > =
Revised: In other words, NDC = (cid:80) Ki =1 [ C i > =

Original: E [ C i ]] , where [ · ] is an Iverson Bracket and C i is the size of cluster i .
Revised: E [ C i ]] , where [ · ] is an Iverson Bracket and C i is the size of cluster i .

Original: For example, E [ C i ] will be 10 if the alphabet has 1000 images and K = 100 .
Revised: For example, E [ C i ] will be 10 if the alphabet has 1000 images and K = 100 .

Original: Then the average difference ( ADif ) is calculated by
Revised: Then the average difference ( ADif ) is calculated by

Original: ADif = 1 | D | (cid:80) d ∈ D | NDC d − NC gtd | , where d (i.e., alphabet) is a dataset in D .
Revised: ADif = 1 | D | (cid:80) d ∈ D | NDC d − NC gtd | , where d (i.e., alphabet) is a dataset in D .

Original: A smaller ADif indicates a better estimate of K .
Revised: A smaller ADif indicates a better estimate of K .

Original: CCN achieves a score of 6.35 (appendix table 4).
Revised: CCN achieves a score of 6.35 (appendix table 6).

Original: We compare this with the baseline approach SKMS (Anand et al., 2014), which does not require a given K and supports a pipeline to estimate K automatically (therefore we only put it into the column K =in table 1.).
Revised: We compare this with the baseline approach SKMS (Anand et al., 2014), which does not require a given K and supports a pipeline to estimate K automatically (therefore we only put it into the column K =in table 1.).

Original: SKMS gets 16.3.
Revised: SKMS gets 16.3.

Original: Furthermore, 10 out of 20 datasets from CCN’s prediction have a difference between NDC d and NC gtd smaller or equal to 3, which shows the feasibility of estimating
Revised: Furthermore, 10 out of 20 datasets from CCN’s prediction have a difference between NDC d and NC gtd smaller or equal to 3, which shows the feasibility of estimating

Original: K with predicted similarity.
Revised: K with predicted similarity.

Original: 5.1. E XPERIMENTS USING THE I MAGE N ET DATASET
Revised: 5.1. E XPERIMENTS USING THE I MAGE N ET DATASET

Original: To demonstrate the scalability of our approach, we applied the same scheme on the ImageNet dataset.
Revised: To demonstrate the scalability of our approach, we applied the same scheme on the ImageNet dataset.

Original: The 1000-class dataset is separated into 882-class ( ImageNet 882 ) and 118-class ( ImageNet 118 ) subsets as the random split in Vinyals et al.
Revised: The 1000-class dataset is separated into 882-class ( ImageNet 882 ) and 118-class ( ImageNet 118 ) subsets as the random split in Vinyals et al.

Original: We use ImageNet 882 for A and 30 classes ( ∼ 39k images) are randomly sampled from ImageNet 118 for T .
Revised: We use ImageNet 882 for A and 30 classes ( ∼ 39k images) are randomly sampled from ImageNet 118 for T .

Original: The difference from section 5.1. is that here we use Resnet-18 for both G and CCN, and the weights of the backbone are pre-trained with ImageNet 882 .
Revised: The difference from section 5.1. is that here we use Resnet-18 for both G and CCN, and the weights of the backbone are pre-trained with ImageNet 882 .

Original: Since the number of pairs is high and it is not feasible to feed them into other constrained clustering algorithms, we compare CCN with K-means, LSC(Chen & Cai, 2011), and
Revised: Since the number of pairs is high and it is not feasible to feed them into other constrained clustering algorithms, we compare CCN with K-means, LSC(Chen & Cai, 2011), and

Original: LPNMF (Cai et al., 2009).
Revised: LPNMF (Cai et al., 2009).

Original: We use the output from the average pooling layer of Resnet-18 as the input to these clustering algorithms.
Revised: We use the output from the average pooling layer of Resnet-18 as the input to these clustering algorithms.

Original: CCN gives the top performance with average ACC 73.8% when K is known, and 65.2% when the number of clusters is unknown, which outperforms the second (34.5% by K-means) with a large margin.
Revised: CCN gives the top performance with average ACC 73.8% when K is known, and 65.2% when the number of clusters is unknown, which outperforms the second (34.5% by K-means) with a large margin.

Original: The full comparison is in appendix table 6.
Revised: The full comparison is in appendix table 8.

Original: And the performance of G is provided in appendix table 7.
Revised: And the performance of G is provided in appendix table 9.

Original: 5. U NSUPERVISED C ROSS -D OMAIN T RANSFER L EARNING 5.2. S ETUP
Revised: 5. U NSUPERVISED C ROSS -D OMAIN T RANSFER L EARNING 5.2. S ETUP

Original: Ofﬁce-31 (Saenko et al., 2010) has images from 31 categories of ofﬁce objects.
Revised: Ofﬁce-31 (Saenko et al., 2010) has images from 31 categories of ofﬁce objects.

Original: The 4652 images are obtained from three domains: Amazon ( a ), DSLR ( d ), and Webcam ( w ).
Revised: The 4652 images are obtained from three domains: Amazon ( a ), DSLR ( d ), and Webcam ( w ).

Original: The dataset is the standard benchmark for evaluating domain adaptation performance in computer vision.
Revised: The dataset is the standard benchmark for evaluating domain adaptation performance in computer vision.

Original: We experiment with all six combinations (source S (cid:48)
Revised: We experiment with all six combinations (source S (cid:48)

Original: → target T ): a → w , a → d , d → a , d → w , w → a , w → d , and report the average accuracy based on ﬁve random experiments for each setting.
Revised: → target T ): a → w , a → d , d → a , d → w , w → a , w → d , and report the average accuracy based on ﬁve random experiments for each setting.

Original: The G function learns the semantic similarity function from the auxiliary dataset A , which is
Revised: The G function learns the semantic similarity function from the auxiliary dataset A , which is

Original: ImageNet with all 1000 categories.
Revised: ImageNet with all 1000 categories.

Original: The backbone network of G is Resnet-18 and has the weights initialized by ImageNet classiﬁcation.
Revised: The backbone network of G is Resnet-18 and has the weights initialized by ImageNet classiﬁcation.

Original: The training process is the same as section 5.1.1 except the images are resized to 224.
Revised: The training process is the same as section 5.1.1 except the images are resized to 224.

Original: We follow the standard protocols using deep neural networks (Long et al., 2017; Ganin et al., 2016) for unsupervised domain adaptation.
Revised: We follow the standard protocols using deep neural networks (Long et al., 2017; Ganin et al., 2016) for unsupervised domain adaptation.

Original: The backbone network of CCN + is pre-trained with ImageNet.
Revised: The backbone network of CCN + is pre-trained with ImageNet.

Original: Appendix ﬁgure 7 illustrates the scheme.
Revised: Appendix ﬁgure 7 illustrates the scheme.

Original: During the training, all source data and target data are used.
Revised: During the training, all source data and target data are used.

Original: Each mini-batch is constructed by 32 labeled samples from source and 96 unlabeled samples from target.
Revised: Each mini-batch is constructed by 32 labeled samples from source and 96 unlabeled samples from target.

Original: Since the target dataset has no labels and could only be randomly sampled, it is crucial to have sufﬁcient mini-batch size to ensure that similar pairs are sampled.
Revised: Since the target dataset has no labels and could only be randomly sampled, it is crucial to have sufﬁcient mini-batch size to ensure that similar pairs are sampled.

Revised: The loss function used in our

Original: The loss function used in our approach is equation (7) and is optimized by stochastic gradient descent.
Revised: [Table] with [Table] approach is equation (7) and is optimized by stochastic gradient descent.

Original: The CCN +/++ and DANN (RevGrad) with ResNet backbone are implemented with Torch.
Revised: The CCN +/++ and DANN (RevGrad) with ResNet backbone are implemented with Torch.

Original: We use the code from original author for JAN.
Revised: We use the code from original author for JAN.

Original: Both DANN and JAN use a 256-dimension bottleneck feature layer.
Revised: Both DANN and JAN use a 256-dimension bottleneck feature layer.

Original: [Table]
Revised: [Table]

Original: R ESULTS AND D ISCUSSIONS
Revised: R ESULTS AND D ISCUSSIONS

Original: The results are summarized in table 2.
Revised: The results are summarized in table 2.

Original: Our approach (CCN + ) demonstrates a strong performance boost for the unsupervised cross-domain transfer problem.
Revised: Our approach (CCN + ) demonstrates a strong performance boost for the unsupervised cross-domain transfer problem.

Original: It reaches 77.5% average accuracy which gained 6.2 points from the 71.3% source-only baseline.
Revised: It reaches 77.5% average accuracy which gained 6.2 points from the 71.3% source-only baseline.

Original: Although our approach merely transfers more information from the auxiliary dataset, it outperforms the strong approach DANN (75.7%), and state-of-the-art JAN (76.9%).
Revised: Although our approach merely transfers more information from the auxiliary dataset, it outperforms the strong approach DANN (75.7%), and state-of-the-art JAN (76.9%).

Original: When combining ours with DANN (CCN ++ ), the performance is further boosted.
Revised: When combining ours with DANN (CCN ++ ), the performance is further boosted.

Original: This indicates that LCO helps mitigate the transfer problem in a certain way that is orthogonal to minimizing the domain discrepancy.
Revised: This indicates that LCO helps mitigate the transfer problem in a certain way that is orthogonal to minimizing the domain discrepancy.

Original: We observe the same trend when using a deeper backbone network, i [Table] e., ResNet-34.
Revised: We observe the same trend when using a deeper backbone network, i.e., ResNet-34.

Original: In such a case the average accuracy achieved is 77.9%, 81.1% and 82% for source-only, CCN + and CCN ++ , respectively, though we used exactly the same G as before (with ResNet-18 backbone for G ).
Revised: In such a case the average accuracy achieved is 77.9%, 81.1% and 82% for source-only, CCN + and CCN ++ , respectively, though we used exactly the same G as before (with ResNet-18 backbone for G ).

Original: This indicates that the information carried in the similarity predictions is not equivalent to transferring features with deeper networks.
Revised: This indicates that the information carried in the similarity predictions is not equivalent to transferring features with deeper networks.

Original: More discussions are in appendix C and the performance of G is provided in appendix table 9 to show that although the prediction has low precision for similar pairs ( ∼ 0 . 2 ), our approach still beneﬁts from the dense similarity predictions.
Revised: More discussions are in appendix C and the performance of G is provided in appendix table 11 to show that although the prediction has low precision for similar pairs ( ∼ 0 . 2 ), our approach still beneﬁts from the dense similarity predictions.

Original: 5.2. E XPERIMENTS USING SVHN- TO -MNIST
Revised: 5.2. E XPERIMENTS USING SVHN- TO -MNIST

Original: We also evaluated the CCN + on another widely compared scenario, which uses color Street View
Revised: We also evaluated the CCN + on another widely compared scenario, which uses color Street View

Original: House Numbers images (SVHN) (Netzer et al., 2011) as S (cid:48) , the gray-scale hand-written digits (MNIST) (LeCun, 1998) as T .
Revised: House Numbers images (SVHN) (Netzer et al., 2011) as S (cid:48) , the gray-scale hand-written digits (MNIST) (LeCun, 1998) as T .

Original: To learn G , we use the Omniglot bg as A .
Revised: To learn G , we use the Omniglot bg as A .

Original: We train all the networks in this section from scratch.
Revised: We train all the networks in this section from scratch.

Original: Our experimental setting is similar to Sener et al.
Revised: Our experimental setting is similar to Sener et al.

Original: We achieve the top performance with 89.1% accuracy.
Revised: We achieve the top performance with 89.1% accuracy.

Original: The performance gain from source-only in our approach is +37.1%, which wins by a large margin compared to the +23.9% of LTR (Sener et al., 2016).
Revised: The performance gain from source-only in our approach is +37.1%, which wins by a large margin compared to the +23.9% of LTR (Sener et al., 2016).

Original: The full comparison is presented in appendix table 10.
Revised: The full comparison is presented in appendix table 12.

Original:  C ONCLUSION AND O UTLOOK
Revised:  C ONCLUSION AND O UTLOOK

Original: In this paper, we demonstrated the usefulness of transferring information in the form of pairwise similarity predictions.
Revised: In this paper, we demonstrated the usefulness of transferring information in the form of pairwise similarity predictions.

Original: Such information can be transferred as a function and utilized by a loss formulation inspired from constrained clustering, but implemented more robustly within a neural network that can jointly optimize both features and clustering outputs based on these noisy predictions.
Revised: Such information can be transferred as a function and utilized by a loss formulation inspired from constrained clustering, but implemented more robustly within a neural network that can jointly optimize both features and clustering outputs based on these noisy predictions.

Original: The experiments for both cross-task and cross-domain transfer learning show strong beneﬁts of using the semantic similarity predictions resulting in new state of the art results across several datasets.
Revised: The experiments for both cross-task and cross-domain transfer learning show strong beneﬁts of using the semantic similarity predictions resulting in new state of the art results across several datasets.

Original: This is true even without explicit domain adaptation for the cross-domain task, and if we add a domain discrepancy loss the beneﬁts increase further.
Revised: This is true even without explicit domain adaptation for the cross-domain task, and if we add a domain discrepancy loss the beneﬁts increase further

Original: There are two key factors that determine the performance of the proposed framework.
Revised: There are two key factors that determine the performance of the proposed framework.

Original: The ﬁrst is the robustness of the constrained clustering and second is the performance of the similarity prediction function.
Revised: The ﬁrst is the robustness of the constrained clustering and second is the performance of the similarity prediction function.

Original: We show robustness of CCN empirically, but we do not explore situations where learning the similarity function is harder.
Revised: We show robustness of CCN empirically, but we do not explore situations where learning the similarity function is harder.

Original: For example, such cases arise when there are a small number of categories in source or a large domain discrepancy between source and target.
Revised: For example, such cases arise when there are a small number of categories in source or a large domain discrepancy between source and target.

Original: One idea to deal with such a situation is learning G with domain adaptation strategies.
Revised: One idea to deal with such a situation is learning G with domain adaptation strategies.

Original: We leave these aspects for future work.
Revised: We leave these aspects for future work.

Revised: A CKNOWLEDGMENTS

Revised: This work was supported by the National Science Foundation and National Robotics Initiative (grant # IIS-1426998)

Original: A PPENDICES A C OMPARISON OF EXPERIMENTAL SETTINGS
Revised: A PPENDICES A C OMPARISON OF EXPERIMENTAL SETTINGS

Original: [Figure]
Revised: [Table]

Revised: [Table]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B S UPPLEMENTARY FOR U NSUPERVISED C ROSS T ASK T RANSFER L EARNING
Revised: B S UPPLEMENTARY FOR U NSUPERVISED C ROSS T ASK T RANSFER L EARNING

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Kannada
Revised: Kannada

Original: [Table]
Revised: [Table]

Original: Malayalam
Revised: Malayalam

Original: [Table]

Original: Manipuri

Original: [Table]
Revised: [Table]

Original: Mongolian
Revised: Mongolian

Original: [Table]
Revised: [Table]

Original: Old_Church_Slavonic_Cyrillic
Revised: Old_Church_Slavonic_Cyrillic

Original: Oriya
Revised: Oriya

Original: Sylheti
Revised: Sylheti

Original: Syriac_Serto
Revised: Syriac_Serto

Original: Tengwar
Revised: Tengwar

Original: Tibetan
Revised: Tibetan

Original: ULOG
Revised: ULOG

Original: ADif 16.
Revised: ADif 16.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  is an approximation since the number of images in each class is only roughly equal in ImageNet.
Revised: It is an approximation since the number of images in each class is only roughly equal in ImageNet.

Original: We sampled 12M pairs for each set to collect the statistics.
Revised: We sampled 12M pairs for each set to collect the statistics.

Original: Sampling more pairs has no noticeable change to the values. 
Revised: Sampling more pairs has no noticeable change to the values. Set

Revised: Similar Precision

Original: [Table]
Revised: [Table]

Original: C S UPPLEMENTARY FOR U NSUPERVISED C ROSS D OMAIN T RANSFER L EARNING C. M ORE DISCUSSION FOR THE O FFICE -31 EXPERIMENTS
Revised: C S UPPLEMENTARY FOR U NSUPERVISED C ROSS D OMAIN T RANSFER L EARNING C. M ORE DISCUSSION FOR THE O FFICE -31 EXPERIMENTS

Original: Our experiment (table 8) shows that simply using deeper pre-trained networks produces a signiﬁcant performance boost.
Revised: Our experiment (table 10) shows that simply using deeper pre-trained networks produces a signiﬁcant performance boost.

Original: Speciﬁcally, using Resnet-50 increases the performance 8.4 points from Alexnet, which surpasses the 6.2 points gained by using one of the state-of-the-art domain adaptation algorithms (JAN (Long et al., 2017)).
Revised: Speciﬁcally, using Resnet-50 increases the performance 8.4 points from Alexnet, which surpasses the 6.2 points gained by using one of the state-of-the-art domain adaptation algorithms (JAN (Long et al., 2017)).

Original: If we regard pre-training with deeper networks as transferring more information, our approach is similar in this aspect since both transfer more information from the auxiliary dataset.
Revised: If we regard pre-training with deeper networks as transferring more information, our approach is similar in this aspect since both transfer more information from the auxiliary dataset.

Original: In this case, memory limitations precluded the application of LCO to such models, and multi-GPU implementations for this problem is an area of future work
Revised: In this case, memory limitations precluded the application of LCO to such models, and multi-GPU implementations for this problem is an area of future work

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D R OBUSTNESS ANALYSIS OF CONSTRAINED CLUSTERING NETWORK D. S ETUP
Revised: D R OBUSTNESS ANALYSIS OF CONSTRAINED CLUSTERING NETWORK D. S ETUP

Original: To quickly explore the large combination of factors that may affect the clustering, we use a small dataset (MNIST) and a small network which has two convolution layers followed by two fully connected layers.
Revised: To quickly explore the large combination of factors that may affect the clustering, we use a small dataset (MNIST) and a small network which has two convolution layers followed by two fully connected layers.

Original: The MNIST dataset is a dataset of handwritten digits that contains 60k training and 10k testing images with size 28x28.
Revised: The MNIST dataset is a dataset of handwritten digits that contains 60k training and 10k testing images with size 28x28.

Original: Only the training set is used in this section and the raw pixels, which were normalized to zero mean and unit standard deviation, were fed into networks directly.
Revised: Only the training set is used in this section and the raw pixels, which were normalized to zero mean and unit standard deviation, were fed into networks directly.

Original: The networks were randomly initialized and the clustering training was run ﬁve times under each combination of factors; we show the best ﬁnal results, as is usual in the random restart regime.
Revised: The networks were randomly initialized and the clustering training was run ﬁve times under each combination of factors; we show the best ﬁnal results, as is usual in the random restart regime.

Original: The mini-batch size was set to 256, thus up to 65536 pairs were presented to the LCO per mini-batch if using full density (D=1).
Revised: The mini-batch size was set to 256, thus up to 65536 pairs were presented to the LCO per mini-batch if using full density (D=1).

Original: There were 235 mini-batches in an epoch and the optimization proceeded for 15 epochs.
Revised: There were 235 mini-batches in an epoch and the optimization proceeded for 15 epochs.

Original: The clustering loss was minimized by stochastic gradient descent with learning rate 0.1 and momentum 0.9.
Revised: The clustering loss was minimized by stochastic gradient descent with learning rate 0.1 and momentum 0.9.

Original: The predicted cluster was assigned at the end by forwarding samples through the clustering networks.
Revised: The predicted cluster was assigned at the end by forwarding samples through the clustering networks.

Original: The best result in the ﬁve runs was reported.
Revised: The best result in the ﬁve runs was reported.

Original: To simulate different performance of the similarity prediction, the label of pairs were ﬂipped according to the designated recall.
Revised: To simulate different performance of the similarity prediction, the label of pairs were ﬂipped according to the designated recall.

Original: For example, to simulate a 90% recall of similar pair, 10% of the ground truth similar pair in a mini-batch were ﬂipped.
Revised: For example, to simulate a 90% recall of similar pair, 10% of the ground truth similar pair in a mini-batch were ﬂipped.

Original: The precision of similar/dissimilar pairs is a function of the recall of both type of pairs, thus controlling the recall is sufﬁcient for the evaluation.
Revised: The precision of similar/dissimilar pairs is a function of the recall of both type of pairs, thus controlling the recall is sufﬁcient for the evaluation.

Original: The recalls for both similar and dissimilar pairs were gradually reduced from one to zero at intervals of 0.1.
Revised: The recalls for both similar and dissimilar pairs were gradually reduced from one to zero at intervals of 0.1.

Original: D. D ISCUSSION
Revised: D. D ISCUSSION

Original: The resulting performance w.r.t different values of recall, density, and number of clusters is visualized in Figure 8.
Revised: The resulting performance w.r.t different values of recall, density, and number of clusters is visualized in Figure 8.

Original: A bright color means high NMI score and is desired.
Revised: A bright color means high NMI score and is desired.

Original: The larger the bright region, the more robust the clustering is against the noise of similarity prediction.
Revised: The larger the bright region, the more robust the clustering is against the noise of similarity prediction.

Original: The ACC score shows almost the same trend and is thus not shown here.
Revised: The ACC score shows almost the same trend and is thus not shown here.

Original: How does similarity prediction affect clustering?
Revised: How does similarity prediction affect clustering?

Original: Looking at the top-left heat map in ﬁgure 8, which has D = 1 and 10 clusters, it can be observed that the NMI score is very robust to low similar pair recall, even lower than 0.5.
Revised: Looking at the top-left heat map in ﬁgure 8, which has D = 1 and 10 clusters, it can be observed that the NMI score is very robust to low similar pair recall, even lower than 0.5.

Original: For recall of dissimilar pairs, the effect of recall is divided at the 0.
Revised: For recall of dissimilar pairs, the effect of recall is divided at the 0.

Original: value: the clustering performance can be very robust to noise in dissimilar pairs if the recall is greater than 0.5; however, it can completely fail if recall is below 0.5.
Revised: value: the clustering performance can be very robust to noise in dissimilar pairs if the recall is greater than 0.5; however, it can completely fail if recall is below 0.5.

Original: For similar pairs, the clustering works on a wide range of recalls when the recall of dissimilar pairs is high.
Revised: For similar pairs, the clustering works on a wide range of recalls when the recall of dissimilar pairs is high.

Original: In practical terms, robustness to the recall of similar pairs is desirable because it is much easier to predict dissimilar pairs than similar pairs in real scenarios.
Revised: In practical terms, robustness to the recall of similar pairs is desirable because it is much easier to predict dissimilar pairs than similar pairs in real scenarios.

Original: In a dataset with 10 categories e.g.
Revised: In a dataset with 10 categories e.g.

Original: Cifar-10, we can easily get 90% recall for dissimilar pairs with purely random guess if the number of classes is known, while the recall for similar pairs will be 10%.
Revised: Cifar-10, we can easily get 90% recall for dissimilar pairs with purely random guess if the number of classes is known, while the recall for similar pairs will be 10%.

Original: How does the density of the constraints affect clustering?
Revised: How does the density of the constraints affect clustering?

Original: We argue that the density of pairwise relationships is the key factor to improving the robustness of clustering.
Revised: We argue that the density of pairwise relationships is the key factor to improving the robustness of clustering.

Original: The density D = 1 means that every pair in a mini-batch is utilized by the clustering loss.
Revised: The density D = 1 means that every pair in a mini-batch is utilized by the clustering loss.

Original: For density D = 0 .
Revised: For density D = 0 .

Original: 1 , it means onlyout of 10 possible constraints is used.
Revised: 1 , it means onlyout of 10 possible constraints is used.

Original: We could regard the higher density as better utilization of the pairwise information in a mini-batch, thus more learning instances contribute to the gradients at once.
Revised: We could regard the higher density as better utilization of the pairwise information in a mini-batch, thus more learning instances contribute to the gradients at once.

Original: Consider a scenario where there is one sample associated with 5 true similar pairs and 3 false similar pairs.
Revised: Consider a scenario where there is one sample associated with 5 true similar pairs and 3 false similar pairs.

Original: In such a case, the gradients introduced by the false similar pairs have a higher chance to be overridden by true similar pairs within the mini-batch, thus the loss can converge faster and is less
Revised: In such a case, the gradients introduced by the false similar pairs have a higher chance to be overridden by true similar pairs within the mini-batch, thus the loss can converge faster and is less

Original: Dissimilar pair recall
Revised: Dissimilar pair recall

Original: S i m il a r p a i r r eca ll
Revised: S i m il a r p a i r r eca ll

Original: K =
Revised: K =

Original: K = K =
Revised: K = K =

Original: D =
Revised: D =

Original: D = 0. D = 0.
Revised: D = 0. D = 0.

Original: D = 0.
Revised: D = 0.

Original: Number of cluster K
Revised: Number of cluster K

Original: Mini-batch pairwise constraint density D affected by errors.
Revised: Mini-batch pairwise constraint density D affected by errors.

Original: In Figure 8, we could see when density decreases, the size of the bright region shrinks signiﬁcantly.
Revised: In Figure 8, we could see when density decreases, the size of the bright region shrinks signiﬁcantly.

Original: In our implementation, enumerating the full pairwise relationships introduces negligible overhead in computation time using GPU.
Revised: In our implementation, enumerating the full pairwise relationships introduces negligible overhead in computation time using GPU.

Original: Although there is overhead for memory consumption, it is limited because only the vector of predicted distributions has to be enumerated for calculating the clustering loss.
Revised: Although there is overhead for memory consumption, it is limited because only the vector of predicted distributions has to be enumerated for calculating the clustering loss.

Original: The effect of varying the number of Clusters In the MNIST experiments, the number of categories is 10.
Revised: The effect of varying the number of Clusters In the MNIST experiments, the number of categories is 10.

Original: We augment the softmax output number up to 100.
Revised: We augment the softmax output number up to 100.

Original: The rows of ﬁgure 8 show that even when the number of output categories is signiﬁcant larger than the number of true object categories, e.g.
Revised: The rows of ﬁgure 8 show that even when the number of output categories is signiﬁcant larger than the number of true object categories, e.g.

Original: 100 > 10 , the clustering performance NMI score only degrades slightly.
Revised: 100 > 10 , the clustering performance NMI score only degrades slightly.