pwc-archive/papers-with-abstracts huggingface dataset:
Selective Convolutional Units: Improving CNNs via Channel Selectivity
Authors: Jongheon Jeong, Jinwoo Shin
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Model Compression

Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently. In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., re-distributing its computations to important channels. In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks. During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of (a) pruning unimportant channels, and (b) rewiring the pruned parameters to important channels. The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it. Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures.
Sufficient Conditions for Robustness to Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks
Authors: Yarin Gal, Lewis Smith
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: image-classification, Image Classification

We prove, under two sufficient conditions, that idealised models can have no adversarial examples. We discuss which idealised models satisfy our conditions, and show that idealised Bayesian neural networks (BNNs) satisfy these. We continue by studying near-idealised BNNs using HMC inference, demonstrating the theoretical ideas in practice. We experiment with HMC on synthetic data derived from MNIST for which we know the ground-truth image density, showing that near-perfect epistemic uncertainty correlates to density under image manifold, and that adversarial images lie off the manifold in our setting. This suggests why MC dropout, which can be seen as performing approximate inference, has been observed to be an effective defence against adversarial examples in practice; We highlight failure-cases of non-idealised BNNs relying on dropout, suggesting a new attack for dropout models and a new defence as well. Lastly, we demonstrate the defence on a cats-vs-dogs image classification task with a VGG13 variant.
Provable Defenses against Spatially Transformed Adversarial Inputs: Impossibility and Possibility Results
Authors: Xinyang Zhang, Yifan Huang, Chanh Nguyen, Shouling Ji, Ting Wang
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

One intriguing property of neural networks is their inherent vulnerability to adversarial inputs, which are maliciously crafted samples to trigger target networks to misbehave. The state-of-the-art attacks generate adversarial inputs using either pixel perturbation or spatial transformation. Thus far, several provable defenses have been proposed against pixel perturbation-based attacks; yet, little is known about whether such solutions exist for spatial transformation-based attacks. This paper bridges this striking gap by conducting the first systematic study on provable defenses against spatially transformed adversarial inputs. Our findings convey mixed messages. On the impossibility side, we show that such defenses may not exist in practice: for any given networks, it is possible to find legitimate inputs and imperceptible transformations to generate adversarial inputs that force arbitrarily large errors. On the possibility side, we show that it is still feasible to construct adversarial training methods to significantly improve the resilience of networks against adversarial inputs over empirical datasets. We believe our findings provide insights for designing more effective defenses against spatially transformed adversarial inputs.
On Meaning-Preserving Adversarial Perturbations for Sequence-to-Sequence Models
Authors: Paul Michel, Graham Neubig, Xi-An Li, Juan Miguel Pino
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Adversarial Robustness, Machine Translation, Translation

Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance. However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output. Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general. Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs. Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance.
Multi-way Encoding for Robustness to Adversarial Attacks
Authors: Donghyun Kim, Sarah Adel Bargal, Jianming Zhang, Stan Sclaroff
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: image-classification, Image Classification, object-detection, Object Detection

Deep models are state-of-the-art for many computer vision tasks including image classification and object detection. However, it has been shown that deep models are vulnerable to adversarial examples. We highlight how one-hot encoding directly contributes to this vulnerability and propose breaking away from this widely-used, but highly-vulnerable mapping. We demonstrate that by leveraging a different output encoding, multi-way encoding, we can make models more robust. Our approach makes it more difficult for adversaries to find useful gradients for generating adversarial attacks. We present state-of-the-art robustness results for black-box, white-box attacks, and achieve higher clean accuracy on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN when combined with adversarial training. The strength of our approach is also presented in the form of an attack for model watermarking, raising challenges in detecting stolen models.
CONTROLLING COVARIATE SHIFT USING EQUILIBRIUM NORMALIZATION OF WEIGHTS
Authors: Aaron Defazio
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.
Mental Fatigue Monitoring using Brain Dynamics Preferences
Authors: Yuangang Pan, Avinash K Singh, Ivor W. Tsang, Chin-Teng Lin
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: EEG, Electroencephalogram (EEG), Ordinal Classification, regression

Driver's cognitive state of mental fatigue significantly affects driving performance and more importantly public safety. Previous studies leverage the response time (RT) as the metric for mental fatigue and aim at estimating the exact value of RT using electroencephalogram (EEG) signals within a regression model. However, due to the easily corrupted EEG signals and also non-smooth RTs during data collection, regular regression methods generally suffer from poor generalization performance.  Considering that human response time is the reflection of brain dynamics preference rather than a single value, a novel model called Brain Dynamic ranking (BDrank) has been proposed. BDrank could learn from brain dynamics preferences using EEG data robustly and preserve the ordering corresponding to RTs. BDrank model is based on the regularized alternative ordinal classification comparing to regular regression based practices. Furthermore, a transition matrix is introduced to characterize the reliability of each channel used in EEG data, which helps in learning brain dynamics preferences only from informative EEG channels. In order to handle large-scale EEG signals~and obtain higher generalization, an online-generalized Expectation Maximum (OnlineGEM) algorithm also has been proposed to update BDrank in an online fashion. Comprehensive empirical analysis on EEG signals from 44 participants shows that BDrank together with OnlineGEM achieves substantial improvements in reliability while simultaneously detecting possible less informative and noisy EEG channels.
Gradient-based learning for F-measure and other performance metrics
Authors: Yu Gai, Zheng Zhang, Kyunghyun Cho
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: General Classification

Many important classification performance metrics, e.g. $F$-measure, are non-differentiable and non-decomposable, and are thus unfriendly to gradient descent algorithm.
Consequently, despite their popularity as evaluation metrics, these metrics are rarely optimized as training objectives in neural network community.
In this paper, we propose an empirical utility maximization scheme with provable learning guarantees to address the non-differentiability of these metrics. 
We then derive a strongly consistent gradient estimator to handle non-decomposability.
These innovations enable end-to-end optimization of these metrics with the same computational complexity as optimizing a decomposable and differentiable metric, e.g. cross-entropy loss.
MANIFOLDNET: A DEEP NEURAL NETWORK FOR MANIFOLD-VALUED DATA
Authors: Rudrasis Chakraborty, Jose Bouza, Jonathan Manton, Baba C. Vemuri
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: General Classification, Image Reconstruction, Video Classification

Developing deep neural networks (DNNs) for manifold-valued data sets
has gained much interest of late in the deep learning research
community.  Examples of manifold-valued data include data from
omnidirectional cameras on automobiles, drones etc., diffusion
magnetic resonance imaging, elastography and others. In this paper, we
present a novel theoretical framework for DNNs to cope with
manifold-valued data inputs.  In doing this generalization, we draw
parallels to the widely popular convolutional neural networks (CNNs).
We call our network the ManifoldNet.

As in vector spaces where convolutions are equivalent to computing the
weighted mean of functions, an analogous definition for
manifold-valued data can be constructed involving the computation of
the weighted Fr\'{e}chet Mean (wFM). To this end, we present a
provably convergent recursive computation of the wFM of the given
data, where the weights makeup the convolution mask, to be
learned. Further, we prove that the proposed wFM layer achieves a
contraction mapping and hence the ManifoldNet does not need the
additional non-linear ReLU unit used in standard CNNs. Operations such
as pooling in traditional CNN are no longer necessary in this setting
since wFM is already a pooling type operation. Analogous to the
equivariance of convolution in Euclidean space to translations, we
prove that the wFM is equivariant to the action of the group of
isometries admitted by the Riemannian manifold on which the data
reside. This equivariance property facilitates weight sharing within
the network.  We present experiments, using the ManifoldNet framework,
to achieve video classification and image reconstruction using an
auto-encoder+decoder setting. Experimental results demonstrate the
efficacy of ManifoldNet in the context of classification and
reconstruction accuracy.
REPRESENTATION COMPRESSION AND GENERALIZATION IN DEEP NEURAL NETWORKS
Authors: Ravid Shwartz-Ziv, Amichai Painsky, Naftali Tishby
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Information Plane

Understanding the groundbreaking performance of Deep Neural Networks is one
of the greatest challenges to the scientific community today. In this work, we
introduce an information theoretic viewpoint on the behavior of deep networks
optimization processes and their generalization abilities. By studying the Information
Plane, the plane of the mutual information between the input variable and
the desired label, for each hidden layer. Specifically, we show that the training of
the network is characterized by a rapid increase in the mutual information (MI)
between the layers and the target label, followed by a longer decrease in the MI
between the layers and the input variable. Further, we explicitly show that these
two fundamental information-theoretic quantities correspond to the generalization
error of the network, as a result of introducing a new generalization bound that is
exponential in the representation compression. The analysis focuses on typical
patterns of large-scale problems. For this purpose, we introduce a novel analytic
bound on the mutual information between consecutive layers in the network.
An important consequence of our analysis is a super-linear boost in training time
with the number of non-degenerate hidden layers, demonstrating the computational
benefit of the hidden layers.
Unsupervised Adversarial Image Reconstruction
Authors: Arthur Pajot, Emmanuel de Bezenac, Patrick Gallinari
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Image Reconstruction

We address the problem of recovering an underlying signal from lossy, inaccurate observations in an unsupervised setting. Typically, we consider situations where there is little to no background knowledge on the structure of the underlying signal, no access to signal-measurement pairs, nor even unpaired signal-measurement data. The only available information is provided by the observations and the measurement process statistics. We cast the problem as finding the \textit{maximum a posteriori} estimate of the signal given each measurement, and propose a general framework for the reconstruction problem. We use a formulation of generative adversarial networks, where the generator takes as input a corrupted observation in order to produce realistic reconstructions, and add a penalty term tying the reconstruction to the associated observation. We evaluate our reconstructions on several image datasets with different types of corruptions. The proposed approach yields better results than alternative baselines, and comparable performance with model variants trained with additional supervision.
Distributionally Robust Optimization Leads to Better Generalization: on SGD and Beyond
Authors: Jikai Hou, Kaixuan Huang, Zhihua Zhang
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

In this paper, we adopt distributionally robust optimization (DRO) (Ben-Tal et al., 2013) in hope to achieve a better generalization in deep learning tasks. We establish the generalization guarantees and analyze the localized Rademacher complexity for DRO, and conduct experiments to show that DRO obtains a better performance. We reveal the profound connection between SGD and DRO, i.e., selecting a batch can be viewed as choosing a distribution over the training set. From this perspective, we prove that SGD is prone to escape from bad stationary points and small batch SGD outperforms large batch SGD. We give an upper bound for the robust loss when SGD converges and keeps stable. We propose a novel Weighted SGD (WSGD) algorithm framework, which assigns high-variance weights to the data of the current batch. We devise a practical implement of WSGD that can directly optimize the robust loss. We test our algorithm on CIFAR-10 and CIFAR-100, and WSGD achieves significant improvements over the conventional SGD.
A Priori Estimates of the Generalization Error for Two-layer Neural Networks
Authors: Lei Wu, Chao Ma, Weinan E
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

New estimates for the generalization error are established for a nonlinear regression problem using a two-layer neural network model. These new estimates are a priori in nature in the sense that the bounds depend only on some norms of the underlying functions to be fitted,  not the parameters in the model.  In contrast,  most existing results for neural networks are a posteriori in nature in the sense that the bounds depend on some norms of the model parameters. The error rates are comparable to that of the Monte Carlo method in terms of the size of the dataset.  Moreover, these bounds are equally effective in the over-parametrized regime when the network size is much larger than the size of the dataset.
Online Hyperparameter Adaptation via Amortized Proximal Optimization
Authors: Paul Vicol, Jeffery Z. HaoChen, Roger Grosse
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Effective performance of neural networks depends critically on effective tuning of optimization hyperparameters, especially learning rates (and schedules thereof). We present Amortized Proximal Optimization (APO), which takes the perspective that each optimization step should approximately minimize a proximal objective (similar to the ones used to motivate natural gradient and trust region policy optimization). Optimization hyperparameters are adapted to best minimize the proximal objective after one weight update. We show that an idealized version of APO (where an oracle minimizes the proximal objective exactly) achieves global convergence to stationary point and locally second-order convergence to global optimum for neural networks. APO incurs minimal computational overhead. We experiment with using APO to adapt a variety of optimization hyperparameters online during training, including (possibly layer-specific) learning rates, damping coefficients, and gradient variance exponents. For a variety of network architectures and optimization algorithms (including SGD, RMSprop, and K-FAC), we show that with minimal tuning, APO performs competitively with carefully tuned optimizers.
Adversarial Exploration Strategy for Self-Supervised Imitation Learning
Authors: Zhang-Wei Hong, Tsu-Jui Fu, Tzu-Yun Shann, Yi-Hsiang Chang, Chun-Yi Lee
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Deep Reinforcement Learning, Imitation Learning, OpenAI Gym, Reinforcement Learning

We present an adversarial exploration strategy, a simple yet effective imitation learning scheme that incentivizes exploration of an environment without any extrinsic reward or human demonstration. Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other. The former collects training samples for the latter, and its objective is to maximize the error of the latter. The latter is trained with samples collected by the former, and generates rewards for the former when it fails to predict the actual action taken by the former. In such a competitive setting, the DRL agent learns to generate samples that the inverse dynamics model fails to predict correctly, and the inverse dynamics model learns to adapt to the challenging samples. We further propose a reward structure that ensures the DRL agent collects only moderately hard samples and not overly hard ones that prevent the inverse model from imitating effectively. We evaluate the effectiveness of our method on several OpenAI gym robotic arm and hand manipulation tasks against a number of baseline models. Experimental results show that our method is comparable to that directly trained with expert demonstrations, and superior to the other baselines even without any human priors.
On Learning Heteroscedastic Noise Models within Differentiable Bayes Filters
Authors: Alina Kloss, Jeannette Bohg
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Decision Making, State Estimation

In many robotic applications, it is crucial to maintain a belief about the state of 
a system, like the location of a robot or the pose of an object.
These state estimates serve as input for planning and decision making and 
provide feedback during task execution. 
Recursive Bayesian Filtering algorithms address the state estimation problem,
but they require a model of the process dynamics and the sensory observations as well as 
noise estimates that quantify the accuracy of these models. 
Recently, multiple works have demonstrated that the process and sensor models can be 
learned by end-to-end training through differentiable versions of Recursive Filtering methods.
However, even if the predictive models are known, finding suitable noise models 
remains challenging. Therefore, many practical applications rely on very simplistic noise 
models. 
Our hypothesis is that end-to-end training through differentiable Bayesian 
Filters enables us to learn more complex heteroscedastic noise models for
the system dynamics. We evaluate learning such models with different types of 
filtering algorithms and on two different robotic tasks. Our experiments show that especially 
for sampling-based filters like the Particle Filter, learning heteroscedastic noise 
models can drastically improve the tracking performance in comparison to using 
constant noise models.
Learning Actionable Representations with Goal Conditioned Policies
Authors: Dibya Ghosh, Abhishek Gupta, Sergey Levine
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Decision Making, Hierarchical Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Representation Learning

Representation learning is a central challenge across a range of machine learning areas. In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems. Most prior work on representation learning has focused on generative approaches, learning representations that capture all the underlying factors of variation in the observation space in a more disentangled or well-ordered manner. In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -- that are "actionable". These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, eliminating the need for explicit reconstruction. We show how these learned representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks. We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.