pwc-archive/papers-with-abstracts huggingface dataset:
Unsupervised Domain Adaptation for Distance Metric Learning
Authors: Kihyuk Sohn, Wenling Shang, Xiang Yu, Manmohan Chandraker
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Domain Adaptation, Face Recognition, Metric Learning, Unsupervised Domain Adaptation

Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain. However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated. Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one. To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space. Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.
INFORMATION MAXIMIZATION AUTO-ENCODING
Authors: Dejiao Zhang, Tianchen Zhao, Laura Balzano
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Decoder, Disentanglement, Informativeness, valid

We propose the Information Maximization Autoencoder (IMAE), an information theoretic approach to simultaneously learn continuous and discrete representations in an unsupervised setting. Unlike the Variational Autoencoder framework, IMAE starts from a stochastic encoder that seeks to map each input data to a hybrid discrete and continuous representation with the objective of maximizing the mutual information between the data and their representations. A decoder is included to approximate the posterior distribution of the data given their representations, where a high fidelity approximation can be achieved by leveraging the informative representations.     
We show that the proposed objective is theoretically valid and provides a principled framework for understanding the tradeoffs regarding informativeness of each representation factor, disentanglement of representations, and decoding quality.
Infinitely Deep Infinite-Width Networks
Authors: Jovana Mitrovic, Peter Wirnsberger, Charles Blundell, Dino Sejdinovic, Yee Whye Teh
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.
Training Variational Auto Encoders with Discrete Latent Representations using Importance Sampling
Authors: Alexander Bartler, Felix Wiewel, Bin Yang, Lukas Mauch
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Representation Learning

The Variational Auto Encoder (VAE) is a popular generative 
latent variable model that is often 
applied for representation learning.
Standard VAEs assume continuous valued 
latent variables and are trained by maximization
of the evidence lower bound (ELBO). Conventional methods obtain a 
differentiable estimate of the ELBO with reparametrized sampling and
optimize it with Stochastic Gradient Descend (SGD). However, this is not possible if 
we want to train VAEs with discrete valued latent variables, 
since reparametrized sampling is not possible. Till now, there
exist no simple solutions to circumvent this problem.
In this paper, we propose an easy method to train VAEs 
with binary or categorically valued latent representations. Therefore, we use a differentiable
estimator for the ELBO which is based on importance sampling. In experiments, we verify the approach and
train two different VAEs architectures with Bernoulli and 
Categorically distributed latent representations on two different benchmark
datasets.
BA-Net: Dense Bundle Adjustment Networks
Authors: Chengzhou Tang, Ping Tan
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable, so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image, and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.
An Exhaustive Analysis of Lazy vs. Eager Learning Methods for Real-Estate Property Investment
Authors: Setareh Rafatirad, Maryam Heidari
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: 4k, Clustering, Prediction

Accurate rent prediction in real estate investment can help in generating capital gains and guaranty a financial success. In this paper, we carry out a comprehensive analysis and study of eleven machine learning algorithms for rent prediction, including Linear Regression, Multilayer Perceptron, Random Forest, KNN, ML-KNN, Locally Weighted Learning, SMO, SVM, J48, lazy Decision Tree (i.e., lazy DT), and KStar algorithms. 
Our contribution in this paper is twofold: (1) We present a comprehensive analysis of internal and external attributes of a real-estate housing dataset and their correlation with rental prices. (2) We use rental prediction as a platform to study and compare the performance of eager vs. lazy machine learning methods using myriad of ML algorithms.  
We train our rent prediction models using a Zillow data set of 4K real estate properties in Virginia State of the US, including three house types of single-family, townhouse, and condo. Each data instance in the dataset has 21 internal attributes (e.g., area space, price, number of bed/bath, rent, school rating, so forth). In addition to Zillow data, external attributes like walk/transit score, and crime rate are collected from online data sources. A subset of the collected features - determined by the PCA technique-  are selected to tune the parameters of the prediction models. We employ a hierarchical clustering approach to cluster the data based on two factors of house type, and average rent estimate of zip codes. We evaluate and compare the efficacy of the tuned prediction models based on two metrics of R-squared and Mean Absolute Error, applied on unseen data. Based on our study, lazy models like KStar lead to higher accuracy and lower prediction error compared to eager methods like J48 and LR. However, it is not necessarily found to be an overarching conclusion drawn from the comparison between all the lazy and eager methods in this work.
Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution
Authors: Min Liu, Fupin Yao, Chiho Choi, Sinha Ayan, Karthik Ramani
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Retrieval

The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on non-euclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.
On Random Deep Weight-Tied Autoencoders: Exact Asymptotic Analysis, Phase Transitions, and Implications to Training
Authors: Ping Li, Phan-Minh Nguyen
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to
any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.
Interpretable Continual Learning
Authors: Tameem Adel, Cuong V. Nguyen, Richard E. Turner, Zoubin Ghahramani, Adrian Weller
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Continual Learning

We present a framework for interpretable continual learning (ICL). We show that explanations of previously performed tasks can be used to improve performance on future tasks. ICL generates a good explanation of a finished task, then uses this to focus attention on what is important when facing a new task. The ICL idea is general and may be applied to many continual learning approaches. Here we focus on the variational continual learning framework to take advantage of its flexibility and efficacy in overcoming catastrophic forgetting. We use saliency maps to provide explanations of performed tasks and propose a new metric to assess their quality. Experiments show that ICL achieves state-of-the-art results in terms of overall continual learning performance as measured by average classification accuracy, and also in terms of its explanations, which are assessed qualitatively and quantitatively using the proposed metric.
REVISTING NEGATIVE TRANSFER USING ADVERSARIAL LEARNING
Authors: Saneem Ahmed Chemmengath, Samarth Bharadwaj, Suranjana Samanta, Karthik Sankaranarayanan
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Attribute, Domain Adaptation

An unintended consequence of feature sharing is the model fitting to correlated tasks within the dataset, termed negative transfer.  In this paper, we revisit the problem of negative transfer in multitask setting and find that its corrosive effects are applicable to a wide range of linear and non-linear models, including neural networks. We first study the effects of negative transfer in a principled way and show that previously proposed counter-measures are insufficient, particularly for trainable features. We propose an adversarial training approach to mitigate the effects of negative transfer by viewing the problem in a domain adaptation setting. Finally, empirical results on attribute prediction multi-task on AWA and CUB datasets further validate the need for correcting negative sharing in an end-to-end manner.
Variational Domain Adaptation
Authors: Hirono Okamoto, Shohei Ohsawa, Itto Higuchi, Haruka Murakami, Mizuki Sango, Zhenghang Cui, Masahiro Suzuki, Hiroshi Kajino, Yutaka Matsuo
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Bayesian Inference, Domain Adaptation, Image Generation, Variational Inference

This paper proposes variational domain adaptation, a unified, scalable, simple framework for learning multiple distributions through variational inference. Unlike the existing methods on domain transfer through deep generative models, such as StarGAN (Choi et al., 2017) and UFDN (Liu et al., 2018), the variational domain adaptation has three advantages. Firstly, the  samples from the target are not required. Instead, the framework requires one known source as a prior $p(x)$ and binary discriminators, $p(\mathcal{D}_i|x)$, discriminating the target domain $\mathcal{D}_i$ from others. Consequently, the framework regards a target as a posterior that can be explicitly formulated through the Bayesian inference, $p(x|\mathcal{D}_i) \propto p(\mathcal{D}_i|x)p(x)$, as exhibited by a further proposed model of dual variational autoencoder (DualVAE). Secondly, the framework is scablable to large-scale domains. As well as VAE encodes a sample $x$ as a mode on a latent space: $\mu(x) \in \mathcal{Z}$, DualVAE encodes a domain $\mathcal{D}_i$ as a mode on the dual latent space $\mu^*(\mathcal{D}_i) \in \mathcal{Z}^*$, named domain embedding. It reformulates the posterior with a natural paring $\langle, \rangle: \mathcal{Z} \times \mathcal{Z}^* \rightarrow \Real$, which can be expanded to uncountable infinite domains such as continuous domains as well as interpolation. Thirdly, DualVAE fastly converges without sophisticated automatic/manual hyperparameter search in comparison to GANs as it requires only one additional parameter to VAE. Through the numerical experiment, we demonstrate the three benefits with multi-domain image generation task on CelebA with up to 60 domains, and exhibits that DualVAE records the state-of-the-art performance outperforming StarGAN and UFDN.
End-to-end learning of pharmacological assays from high-resolution microscopy images
Authors: Markus Hofmarcher, Elisabeth Rumetshofer, Sepp Hochreiter, Günter Klambauer
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Cell Segmentation, Drug Discovery, Segmentation

Predicting the outcome of pharmacological assays based on high-resolution microscopy
images of treated cells is a crucial task in drug discovery which tremendously
increases discovery rates. However, end-to-end learning on these images
with convolutional neural networks (CNNs) has not been ventured for this task
because it has been considered infeasible and overly complex. On the largest
available public dataset, we compare several state-of-the-art CNNs trained in an
end-to-end fashion with models based on a cell-centric approach involving segmentation.
We found that CNNs operating on full images containing hundreds
of cells perform significantly better at assay prediction than networks operating
on a single-cell level. Surprisingly, we could predict 29% of the 209 pharmacological
assays at high predictive performance (AUC > 0.9). We compared a
novel CNN architecture called “GapNet” against four competing CNN architectures
and found that it performs on par with the best methods and at the same time
has the lowest training time. Our results demonstrate that end-to-end learning on
high-resolution imaging data is not only possible but even outperforms cell-centric
and segmentation-dependent approaches. Hence, the costly cell segmentation and
feature extraction steps are not necessary, in fact they even hamper predictive performance.
Our work further suggests that many pharmacological assays could
be replaced by high-resolution microscopy imaging together with convolutional
neural networks.
Meta-Learning with Domain Adaptation for Few-Shot Learning under Domain Shift
Authors: Doyen Sahoo, Hung Le, Chenghao Liu, Steven C. H. Hoi
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Domain Adaptation, Few-Shot Learning, Meta-Learning

Few-Shot Learning (learning with limited labeled data) aims to overcome the limitations of traditional machine learning approaches which require thousands of labeled examples to train an effective model. Considered as a hallmark of human intelligence, the community has recently witnessed several contributions on this topic, in particular through meta-learning, where a model learns how to learn an effective model for few-shot learning. The main idea is to acquire prior knowledge from a set of training tasks, which is then used to perform (few-shot) test tasks. Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks. This is a very strong assumption which restricts the usage of meta-learning strategies in the real world where ample training tasks following the same distribution as test tasks may not be available. In this paper, we propose a novel meta-learning paradigm wherein a few-shot learning model is learnt, which simultaneously overcomes domain shift between the train and test tasks via adversarial domain adaptation. We demonstrate the efficacy the proposed method through extensive experiments.
Variational Autoencoders with Jointly Optimized Latent Dependency Structure
Authors: Jiawei He, Yu Gong, Joseph Marino, Greg Mori, Andreas Lehrmann
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

We propose a method for learning the dependency structure between latent variables in deep latent variable models.  Our general modeling and inference framework combines the complementary strengths of deep generative models and probabilistic graphical models. In particular, we express the latent variable space of a variational autoencoder (VAE) in terms of a Bayesian network with a learned, flexible dependency structure.  The network parameters, variational parameters as well as the latent topology are optimized simultaneously with a single objective.  Inference is formulated via a sampling procedure that produces expectations over latent variable structures and incorporates top-down and bottom-up reasoning over latent variable values.  We validate our framework in extensive experiments on MNIST, Omniglot, and CIFAR-10. Comparisons to state-of-the-art structured variational autoencoder baselines show improvements in terms of the expressiveness of the learned model.
Learning Protein Structure with a Differentiable Simulator
Authors: John Ingraham, Adam Riesselman, Chris Sander, Debora Marks
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Protein Folding

The Boltzmann distribution is a natural model for many systems, from brains to materials and biomolecules, but is often of limited utility for fitting data because Monte Carlo algorithms are unable simulate it in available time. This gap between the expressive capabilities and sampling practicalities of energy-based models is exemplified by the protein folding problem, since energy landscapes underlie contemporary knowledge of protein biophysics but computer simulations are often unable to fold all but the smallest proteins from first-principles. In this work we aim to bridge the gap between the expressive capacity of energy functions and the practical capabilities of their simulators by using an unrolled Monte Carlo simulation as a model for data. We compose a neural energy function with a novel and efficient simulator based on Langevin dynamics to build an end-to-end-differentiable model of atomic protein structure given amino acid sequence information. We introduce techniques for stabilizing backpropagation under long roll-outs and demonstrate the model's capacity to make multimodal predictions and to (sometimes) generalize to unobserved protein fold types when trained on a large corpus of protein structures.
Stability of Stochastic Gradient Method with Momentum for Strongly Convex Loss Functions
Authors: Ali Ramezani-Kebrya, Ashish Khisti, and Ben Liang
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

While momentum-based methods, in conjunction with the stochastic gradient descent, are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In practice, the momentum parameter is often chosen in a heuristic fashion with little theoretical guidance. In this work, we use the framework of algorithmic stability to provide an upper-bound on the generalization error for the class of strongly convex loss functions, under mild technical assumptions. Our bound decays to zero inversely with the size of the training set, and increases as the momentum parameter is increased. We also develop an upper-bound on the expected true risk,  in terms of the number of training steps, the size of the training set, and the momentum parameter.
Better Generalization with On-the-fly Dataset Denoising
Authors: Jiaming Song, Tengyu Ma, Michael Auli, Yann Dauphin
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Denoising, Memorization

Memorization in over-parameterized neural networks can severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are to hard avoid in extremely large datasets. We address this problem using the implicit regularization effect of stochastic gradient descent with large learning rates, which we find to be able to separate clean and mislabeled examples with remarkable success using loss statistics. We leverage this to identify and on-the-fly discard mislabeled examples using a threshold on their losses. This leads to On-the-fly Data Denoising (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead. Empirical results demonstrate the effectiveness of ODD on several datasets containing artificial and real-world mislabeled examples.