pwc-archive/papers-with-abstracts huggingface dataset:
Critical Points of Linear Neural Networks: Analytical Forms and Landscape Properties
Authors: Yi Zhou, Yingbin Liang
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect. Particularly, the properties of critical points and the landscape around them are of importance to determine the convergence performance of optimization algorithms. In this paper, we provide a necessary and sufficient characterization of the analytical forms for the critical points (as well as global minimizers) of the square loss functions for linear neural networks. We show that the analytical forms of the critical points characterize the values of the corresponding loss functions as well as the necessary and sufficient conditions to achieve global minimum. Furthermore, we exploit the analytical forms of the critical points to characterize the landscape properties for the loss functions of linear neural networks and shallow ReLU networks. One particular conclusion is that: While the loss function of linear networks has no spurious local minimum, the loss function of one-hidden-layer nonlinear networks with ReLU activation function does have local minimum that is not global minimum.
Overcoming Catastrophic Interference using Conceptor-Aided Backpropagation
Authors: Xu He, Herbert Jaeger
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Continual Learning, Permuted-MNIST

Catastrophic interference has been a major roadblock in the research of continual learning. Here we propose a variant of the back-propagation algorithm, "Conceptor-Aided Backprop" (CAB), in which gradients are shielded by conceptors against degradation of previously learned tasks. Conceptors have their origin in reservoir computing, where they have been previously shown to overcome catastrophic forgetting. CAB extends these results to deep feedforward networks. On the disjoint and permuted MNIST tasks, CAB outperforms two other methods for coping with catastrophic interference that have recently been proposed.
Imitation Learning from Visual Data with Multiple Intentions
Authors: Aviv Tamar, Khashayar Rohanimanesh, Yin-Lam Chow, Chris Vigorito, Ben Goodrich, Michael Kahane, Derik Pridmore
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Imitation Learning

Recent advances in learning from demonstrations (LfD) with deep neural networks have enabled learning complex robot skills that involve high dimensional perception such as raw image inputs. 
LfD algorithms generally assume learning from single task demonstrations. In practice, however, it is more efficient for a teacher to demonstrate a multitude of tasks without careful task set up, labeling, and engineering. Unfortunately in such cases, traditional imitation learning techniques fail to represent the multi-modal nature of the data, and often result in sub-optimal behavior. In this paper we present an LfD approach for learning multiple modes of behavior from visual data. Our approach is based on a stochastic deep neural network (SNN), which represents the underlying intention in the demonstration as a stochastic activation in the network. We present an efficient algorithm for training SNNs, and for learning with vision inputs, we also propose an architecture that associates the intention with a stochastic attention module.
We demonstrate our method on real robot visual object reaching tasks, and show that
it can reliably learn the multiple behavior modes in the demonstration data. Video results are available at https://vimeo.com/240212286/fd401241b9.
Exploring Sentence Vectors Through Automatic Summarization
Authors: Adly Templeton, Jugal Kalita
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Sentence, Sentence Embeddings

Vector semantics, especially sentence vectors, have recently been used successfully in many areas of natural language processing. However, relatively little work has explored the internal structure and properties of spaces of sentence vectors. In this paper, we will explore the properties of sentence vectors by studying a particular real-world application: Automatic Summarization. In particular, we show that cosine similarity between sentence vectors and document vectors is strongly correlated with sentence importance and that vector semantics can identify and correct gaps between the sentences chosen so far and the document. In addition, we identify specific dimensions which are linked to effective summaries. To our knowledge, this is the first time specific dimensions of sentence embeddings have been connected to sentence properties. We also compare the features of different methods of sentence embeddings. Many of these insights have applications in uses of sentence embeddings far beyond summarization.
Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates
Authors: Leslie N. Smith, Nicholay Topin
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

In this paper, we show a phenomenon, which we named ``super-convergence'', where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods.   The existence of super-convergence is relevant to understanding why deep networks generalize well.  One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate.  Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a  greater boost in performance relative to standard training when the amount of labeled training data is limited.  We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate.  The architectures to replicate this work will be made available upon publication.
Spontaneous Symmetry Breaking in Deep Neural Networks
Authors: Ricky Fok, Aijun An, Xiaogang Wang
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

We propose a framework to understand the unprecedented performance and robustness of deep neural networks using field theory. Correlations between the weights within the same layer can be described by symmetries in that layer, and networks generalize better if such symmetries are broken to reduce the redundancies of the weights. Using a two parameter field theory, we find that the network can break such symmetries itself towards the end of training in a process commonly known in physics as spontaneous symmetry breaking. This corresponds to a network generalizing itself without any user input layers to break the symmetry, but by communication with adjacent layers. In the layer decoupling limit applicable to residual networks (He et al., 2015), we show that the remnant symmetries that survive the non-linear layers are spontaneously broken based on empirical results. The Lagrangian for the non-linear and weight layers together has striking similarities with the one in quantum field theory of a scalar. Using results from quantum field theory we show that our framework is able to explain many experimentally observed phenomena, such as training on random labels with zero error (Zhang et al., 2017), the information bottleneck and the phase transition out of it (Shwartz-Ziv & Tishby, 2017), shattered gradients (Balduzzi et al., 2017), and many more.
Distributed non-parametric deep and wide networks
Authors: Biswa Sengupta, Yu Qian
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Action Recognition, Gaussian Processes, Temporal Action Localization

In recent work, it was shown that combining multi-kernel based support vector machines (SVMs) can lead to near state-of-the-art performance on an action recognition dataset (HMDB-51 dataset). In the present work, we show that combining distributed Gaussian Processes with multi-stream deep convolutional neural networks (CNN) alleviate the need to augment a neural network with hand-crafted features. In contrast to prior work, we treat each deep neural convolutional network as an expert wherein the individual predictions (and their respective uncertainties) are combined into a Product of Experts (PoE) framework.
Iterative temporal differencing with fixed random feedback alignment support spike-time dependent plasticity in vanilla backpropagation for deep learning
Authors: Aras Dargazany, Kunal Mankodiya
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Deep Learning

In vanilla backpropagation (VBP), activation function matters considerably in terms of non-linearity and differentiability.
Vanishing gradient has been an important problem related to the bad choice of activation function in deep learning (DL).
This work shows that a differentiable activation function is not necessary any more for error backpropagation. 
The derivative of the activation function can be replaced by an iterative temporal differencing (ITD) using fixed random feedback weight alignment (FBA).
Using FBA with ITD, we can transform the VBP into a more biologically plausible approach for learning deep neural network architectures.
We don't claim that ITD works completely the same as the spike-time dependent plasticity (STDP) in our brain but this work can be a step toward the integration of STDP-based error backpropagation in deep learning.
Reinforcement Learning via Replica Stacking of Quantum Measurements for the Training of Quantum Boltzmann Machines
Authors: Anna Levit,  Daniel Crawford, Navid Ghadermarzy, Jaspreet S. Oberoi, Ehsan Zahedinejad, Pooya Ronagh
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Recent theoretical and experimental results suggest the possibility of using current and near-future quantum hardware in challenging sampling tasks. In this paper, we introduce free-energy-based reinforcement learning (FERL) as an application of quantum hardware. We propose a method for processing a quantum annealer’s measured qubit spin configurations in approximating the free energy of a quantum Boltzmann machine (QBM). We then apply this method to perform reinforcement learning on the grid-world problem using the D-Wave 2000Q quantum annealer. The experimental results show that our technique is a promising method for harnessing the power of quantum sampling in reinforcement learning tasks.
Building effective deep neural networks one feature at a time
Authors: Martin Mundt, Tobias Weis, Kishore Konda, Visvanathan Ramesh
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Feature Importance

Successful training of convolutional neural networks is often associated with suffi-
ciently deep architectures composed of high amounts of features. These networks
typically rely on a variety of regularization and pruning techniques to converge
to less redundant states. We introduce a novel bottom-up approach to expand
representations in fixed-depth architectures. These architectures start from just a
single feature per layer and greedily increase width of individual layers to attain
effective representational capacities needed for a specific task. While network
growth can rely on a family of metrics, we propose a computationally efficient
version based on feature time evolution and demonstrate its potency in determin-
ing feature importance and a networks’ effective capacity. We demonstrate how
automatically expanded architectures converge to similar topologies that benefit
from lesser amount of parameters or improved accuracy and exhibit systematic
correspondence in representational complexity with the specified task. In contrast
to conventional design patterns with a typical monotonic increase in the amount of
features with increased depth, we observe that CNNs perform better when there is
more learnable parameters in intermediate, with falloffs to earlier and later layers.
Multi-label Learning for Large Text Corpora using Latent Variable Model with Provable Gurantees
Authors: Sayantan Dasgupta
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Learning Theory, Multi-Label Learning, PAC learning

Here we study the problem of learning labels for large text corpora where each document can be assigned a variable number of labels. The problem is trivial when the label dimensionality is small and can be easily solved by a series of one-vs-all classifiers. However, as the label dimensionality increases, the parameter space of such one-vs-all classifiers becomes extremely large and outstrips the memory. Here we propose a latent variable model to reduce the size of the parameter space, but still efficiently learn the labels. We learn the model using spectral learning and show how to extract the parameters using only three passes through the training dataset. Further, we analyse the sample complexity of our model using PAC learning theory and then demonstrate the performance of our algorithm on several benchmark datasets in comparison with existing algorithms.
Egocentric Spatial Memory Network
Authors: Mengmi Zhang, Keng Teck Ma, Joo Hwee Lim, Shih-Cheng Yen, Qi Zhao, Jiashi Feng
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Navigate, Simultaneous Localization and Mapping

Inspired by neurophysiological discoveries of navigation cells in the mammalian
brain, we introduce the first deep neural network architecture for modeling Egocentric
Spatial Memory (ESM). It learns to estimate the pose of the agent and
progressively construct top-down 2D global maps from egocentric views in a spatially
extended environment. During the exploration, our proposed ESM network
model updates belief of the global map based on local observations using a recurrent
neural network. It also augments the local mapping with a novel external
memory to encode and store latent representations of the visited places based on
their corresponding locations in the egocentric coordinate. This enables the agents
to perform loop closure and mapping correction. This work contributes in the
following aspects: first, our proposed ESM network provides an accurate mapping
ability which is vitally important for embodied agents to navigate to goal locations.
In the experiments, we demonstrate the functionalities of the ESM network in
random walks in complicated 3D mazes by comparing with several competitive
baselines and state-of-the-art Simultaneous Localization and Mapping (SLAM)
algorithms. Secondly, we faithfully hypothesize the functionality and the working
mechanism of navigation cells in the brain. Comprehensive analysis of our model
suggests the essential role of individual modules in our proposed architecture and
demonstrates efficiency of communications among these modules. We hope this
work would advance research in the collaboration and communications over both
fields of computer science and computational neuroscience.
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy
Authors: Gintare Karolina Dziugaite, Daniel M. Roy
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Generalization Bounds, valid

We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound’s prior, violating the hypothesis of the PAC-Bayes theorem that the prior is chosen independently of the data. Indeed, available implementations of Entropy-SGD rapidly obtain zero training error on random labels and the same holds of the Gibbs posterior. In order to obtain a valid generalization bound, we show that an ε-differentially private prior yields a valid PAC-Bayes bound, a straightforward consequence of results connecting generalization with differential privacy. Using stochastic gradient Langevin dynamics (SGLD) to approximate the well-known exponential release mechanism, we observe that generalization error on MNIST (measured on held out data) falls within the (empirically nonvacuous) bounds computed under the assumption that SGLD produces perfect samples. In particular, Entropy-SGLD can be configured to yield relatively tight generalization bounds and still fit real labels, although these same settings do not obtain state-of-the-art performance.
Decoupling the Layers in Residual Networks
Authors: Ricky Fok, Aijun An, Zana Rashidi, Xiaogang Wang
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

We propose a Warped Residual Network (WarpNet) using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network. We apply a perturbation theory on residual networks and decouple the interactions between residual units. The resulting warp operator is a first order approximation of the output over multiple layers. The first order perturbation theory exhibits properties such as binomial path lengths and exponential gradient scaling found experimentally by Veit et al (2016). 
We demonstrate through an extensive performance study that the proposed network achieves comparable predictive performance to the original residual network with the same number of parameters, while achieving a significant speed-up on the total training time. As WarpNet performs model parallelism in residual network training in which weights are distributed over different GPUs, it offers speed-up and capability to train larger networks compared to original residual networks.
ENRICHMENT OF FEATURES FOR CLASSIFICATION USING AN OPTIMIZED LINEAR/NON-LINEAR COMBINATION OF INPUT FEATURES
Authors: Mehran Taghipour-Gorjikolaie, Seyyed Mohammad Razavi, Javad Sadri
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Classification, General Classification

Automatic classification of objects is one of the most important tasks in engineering
and data mining applications. Although using more complex and advanced
classifiers can help to improve the accuracy of classification systems, it can be
done by analyzing data sets and their features for a particular problem. Feature
combination is the one which can improve the quality of the features. In this paper,
a structure similar to Feed-Forward Neural Network (FFNN) is used to generate an
optimized linear or non-linear combination of features for classification. Genetic
Algorithm (GA) is applied to update weights and biases. Since nature of data sets
and their features impact on the effectiveness of combination and classification
system, linear and non-linear activation functions (or transfer function) are used
to achieve more reliable system. Experiments of several UCI data sets and using
minimum distance classifier as a simple classifier indicate that proposed linear and
non-linear intelligent FFNN-based feature combination can present more reliable
and promising results. By using such a feature combination method, there is no
need to use more powerful and complex classifier anymore.
Thinking like a machine — generating visual rationales through latent space optimization
Authors: Jarrel Seah, Jennifer Tang, Andy Kitchen, Jonathan Seah
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Interpretability and small labelled datasets are key issues in the practical application of deep learning, particularly in areas such as medicine. In this paper, we present a semi-supervised technique that addresses both these issues simultaneously. We learn dense representations from large unlabelled image datasets, then use those representations to both learn classifiers from small labeled sets and generate visual rationales explaining the predictions. Using chest radiography diagnosis as a motivating application, we show our method has good generalization ability by learning to represent our chest radiography dataset while training a classifier on an separate set from a different institution. Our method identifies heart failure and other thoracic diseases. For each prediction, we generate visual rationales for positive classifications by optimizing a latent representation to minimize the probability of disease while constrained by a similarity measure in image space. Decoding the resultant latent representation produces an image without apparent disease. The difference between the original and the altered image forms an interpretable visual rationale for the algorithm's prediction. Our method simultaneously produces visual rationales that compare favourably to previous techniques and a classifier that outperforms the current state-of-the-art.
AANN: Absolute Artificial Neural Network
Authors: Animesh Karnewar
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

This research paper describes a simplistic architecture named as AANN: Absolute Artificial Neural Network, which can be used to create highly interpretable representations of the input data. These representations are generated by penalizing the learning of the network in such a way that those learned representations correspond to the respective labels present in the labelled dataset used for supervised training; thereby, simultaneously giving the network the ability to classify the input data. The network can be used in the reverse direction to generate data that closely resembles the input by feeding in representation vectors as required. This research paper also explores the use of mathematical abs (absolute valued) functions as activation functions which constitutes the core part of this neural network architecture. Finally the results obtained on the MNIST dataset by using this technique are presented and discussed in brief.
Dependent Bidirectional RNN with Extended-long Short-term Memory
Authors: Yuanhang Su, Yuzhong Huang, C. -C. Jay Kuo
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

In this work, we first conduct mathematical analysis on the memory, which is
defined as a function that maps an element in a sequence to the current output,
of three RNN cells; namely, the simple recurrent neural network (SRN), the long
short-term memory (LSTM) and the gated recurrent unit (GRU). Based on the
analysis, we propose a new design, called the extended-long short-term memory
(ELSTM), to extend the memory length of a cell. Next, we present a multi-task
RNN model that is robust to previous erroneous predictions, called the dependent
bidirectional recurrent neural network (DBRNN), for the sequence-in-sequenceout
(SISO) problem. Finally, the performance of the DBRNN model with the
ELSTM cell is demonstrated by experimental results.
Make SVM great again with Siamese kernel for few-shot learning
Authors: Bence Tilk
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Few-Shot Learning, General Classification, Metric Learning, One-Shot Learning

While deep neural networks have shown outstanding results in a wide range of applications,
learning from a very limited number of examples is still a challenging
task. Despite the difficulties of the few-shot learning, metric-learning techniques
showed the potential of the neural networks for this task. While these methods
perform well, they don’t provide satisfactory results. In this work, the idea of
metric-learning is extended with Support Vector Machines (SVM) working mechanism,
which is well known for generalization capabilities on a small dataset.
Furthermore, this paper presents an end-to-end learning framework for training
adaptive kernel SVMs, which eliminates the problem of choosing a correct kernel
and good features for SVMs. Next, the one-shot learning problem is redefined
for audio signals. Then the model was tested on vision task (using Omniglot
dataset) and speech task (using TIMIT dataset) as well. Actually, the algorithm
using Omniglot dataset improved accuracy from 98.1% to 98.5% on the one-shot
classification task and from 98.9% to 99.3% on the few-shot classification task.