Paper:
Published as a conference paper at ICLR 2018 LEARNING TO CLUSTER IN ORDER TO TRANSFER ACROSS DOMAINS AND TASKS
This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster. The key insight is that, in addition to features, we can transfer similarity information and this is sufficient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning. We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not (pairwise semantic similarity). This similarity is category-agnostic and can be learned from data in the source domain using a similarity network. We then present two novel approaches for performing transfer learning using this similarity function. First, for unsupervised domain adaptation, we design a new loss function to regularize classification with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs. Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network. Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches. Using this method, we first show state of the art results for the challenging cross-task problem, applied on Omniglot and ImageNet. Our results show that we can reconstruct semantic clusters with high accuracy. We then evaluate the performance of cross-domain transfer using images from the Office-31 and SVHN-MNIST tasks and present top accuracy on both datasets. Our approach doesn't explicitly deal with domain discrepancy. If we combine with a domain adaptation loss, it shows further improvement.

Review:
The authors propose a method for performing transfer learning and domain adaptation via a clustering approach. The primary contribution is the introduction of a Learnable Clustering Objective (LCO) that is trained on an auxiliary set of labeled data to correctly identify whether pairs of data belong to the same class. Once the LCO is trained, it is applied to the unlabeled target data and effectively serves to provide "soft labels" for whether or not pairs of target data belong to the same class. A separate model can then be trained to assign target data to clusters while satisfying these soft labels, thereby ensuring that clusters are made up of similar data points. 
The proposed LCO is novel and seems sound, serving as a way to transfer the general knowledge of what a cluster is without requiring advance knowledge of the specific clusters of interest. The authors also demonstrate a variety of extensions, such as how to handle the case when the number of target categories is unknown, as well as how the model can make use of labeled source data in the setting where the source and target share the same task.
The way the method is presented is quite confusing, and required many more reads than normal to understand exactly what is going on. To point out one such problem point, Section 4 introduces f, a network that classifies each data instance into one of k clusters. However, f seems to be mentioned only in a few times by name, despite seeming like a crucial part of the method. Explaining how f is used to construct the CCN could help in clarifying exactly what role f plays in the final model. Likewise, the introduction of G during the explanation of the LCO is rather abrupt, and the intuition of what purpose G serves and why it must be learned from data is unclear. Additionally, because G is introduced alongside the LCO, I was initially misled into understanding was that G was optimized to minimize the LCO. Further text explaining intuitively what G accomplishes (soft labels transferred from the auxiliary dataset to the target dataset) and perhaps a general diagram of what portions of the model are trained on what datasets (G is trained on A, CCN is trained on T and optionally S') would serve the method section greatly and provide a better overview of how the model works.
The experimental evaluation is very thorough, spanning a variety of tasks and settings. Strong results in multiple settings indicate that the proposed method is effective and generalizable. Further details are provided in a very comprehensive appendix, which provides a mix of discussion and analysis of the provided results. It would be nice to see some examples of the types of predictions and mistakes the model makes to further develop an intuition for how the model works. I'm also curious how well the model works if, you do not make use of the labeled source data in the cross-domain setting, thereby mimicking the cross-task setup.
At times, the experimental details are a little unclear. Consistent use of the A, T, and S' dataset abbreviations would help. Also, the results section seems to switch off between calling the method CCN and LCO interchangeably. Finally, a few of the experimental settings differ from their baselines in nontrivial ways. For the Office experiment, the LCO appears to be trained on ImageNet data. While this seems similar in nature to initializing from a network pre-trained on ImageNet, it's worth noting that this requires one to have the entire ImageNet dataset on hand when training such a model, as opposed to other baselines which merely initialize weights and then fine-tune exclusively on the Office data. Similarly, the evaluation on SVHN-MNIST makes use of auxiliary Omniglot data, which makes the results hard to compare to the existing literature, since they generally do not use additional training data in this setting. In addition to the existing comparison, perhaps the authors can also validate a variant in which the auxiliary data is also drawn from the source so as to serve as a more direct comparison to the existing literature.
Overall, the paper seems to have both a novel contribution and strong technical merit. However, the presentation of the method is lacking, and makes it unnecessarily difficult to understand how the model is composed of its parts and how it is trained. I think a more careful presentation of the intuition behind the method and more consistent use of notation would greatly improve the quality of this submission.
=========================
Update after author rebuttal:
=========================
I have read the author's response and have looked at the changes to the manuscript. I am satisfied with the improvements to the paper and have changed my review to 'accept'.