pwc-archive/papers-with-abstracts huggingface dataset:
Discovery of Predictive Representations With a Network of General Value Functions
Authors: Matthew Schlegel, Andrew Patterson, Adam White, Martha White
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Decision Making, Reinforcement Learning

The ability of an agent to {\em discover} its own learning objectives has long been considered a key ingredient for artificial general intelligence. Breakthroughs in autonomous decision making and reinforcement learning have primarily been in domains where the agent's goal is outlined and clear: such as playing a game to win, or driving safely. Several studies have demonstrated that learning extramural sub-tasks and auxiliary predictions can improve (1) single human-specified task learning, (2) transfer of learning, (3) and the agent's learned representation of the world. In all these examples, the agent was instructed what to learn about. We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent's representation of the world. Specifically, our system maintains a large collection of predictions, continually pruning and replacing predictions. We highlight the importance of considering stability rather than convergence for such a system, and develop an adaptive, regularized algorithm towards that aim. We provide several experiments in computational micro-worlds demonstrating that this simple approach can be effective for discovering useful predictions autonomously.
Generation and Consolidation of Recollections for Efficient Deep Lifelong Learning
Authors: Matt Riemer, Michele Franceschini, and Tim Klinger
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Knowledge Distillation, Lifelong learning

Deep lifelong learning systems need to efficiently manage resources to scale to large numbers of experiences and non-stationary goals. In this paper, we explore the relationship between lossy compression and the resource constrained lifelong learning problem of function transferability. We demonstrate that lossy episodic experience storage can enable efficient function transferability between different architectures and algorithms at a fraction of the storage cost of lossless storage. This is achieved by introducing a generative knowledge distillation strategy that does not store any full training examples. As an important extension of this idea, we show that lossy recollections stabilize deep networks much better than lossless sampling in resource constrained settings of lifelong learning while avoiding catastrophic forgetting. For this setting, we propose a novel dual purpose recollection buffer used to both stabilize the recollection generator itself and an accompanying reasoning model.
The Manifold Assumption and Defenses Against Adversarial Perturbations
Authors: Xi Wu, Uyeong Jang, Lingjiao Chen, Somesh Jha
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

In the adversarial-perturbation problem of neural networks, an adversary starts with a neural network model $F$ and a point $\bfx$ that $F$ classifies correctly, and applies a \emph{small perturbation} to  $\bfx$ to produce another point $\bfx'$ that $F$ classifies \emph{incorrectly}.  In this paper, we propose taking into account \emph{the inherent confidence information} produced by models when studying adversarial perturbations, where a natural measure of ``confidence'' is \|F(\bfx)\|_\infty$ (i.e. how confident $F$ is about its prediction?). Motivated by a thought experiment based on the manifold assumption, we propose a ``goodness property'' of models which states that \emph{confident regions of a good model should be well separated}. We give formalizations of this property and examine existing robust training objectives in view of them. Interestingly, we find that a recent objective by Madry et al. encourages training a model that satisfies well our formal version of the goodness property, but has a weak control of points that are wrong but with low confidence. However, if Madry et al.'s model is indeed a good solution to their objective, then good and bad points are now distinguishable and we can try to embed uncertain points back to the closest confident region to get (hopefully) correct predictions. We thus propose embedding objectives and algorithms, and perform an empirical study using this method. Our experimental results are encouraging: Madry et al.'s model wrapped with our embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining good generalization behavior.
Learning to select examples for program synthesis
Authors: Yewen Pu, Zachery Miranda, Armando Solar-Lezama, Leslie Pack Kaelbling
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Program Synthesis

Program synthesis is a class of regression problems where one seeks a solution, in the form of a source-code program, that maps the inputs to their corresponding outputs exactly. Due to its precise and combinatorial nature, it is commonly formulated as a constraint satisfaction problem, where input-output examples are expressed constraints, and solved with a constraint solver. A key challenge of this formulation is that of scalability: While constraint solvers work well with few well-chosen examples, constraining the entire set of example constitutes a significant overhead in both time and memory. In this paper we address this challenge by constructing a representative subset of examples that is both small and is able to constrain the solver sufficiently. We build the subset one example at a time, using a trained discriminator to predict the probability of unchosen input-output examples conditioned on the chosen input-output examples, adding the least probable example to the subset. Experiment on a diagram drawing domain shows our approach produces subset of examples that are small and representative for the constraint solver.
Neuron as an Agent
Authors: Shohei Ohsawa, Kei Akuzawa, Tatsuya Matsushima, Gustavo Bezerra, Yusuke Iwasawa, Hiroshi Kajino, Seiya Takenaka, Yutaka Matsuo
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: counterfactual, Multi-agent Reinforcement Learning, OpenAI Gym, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Existing multi-agent reinforcement learning (MARL) communication methods have relied on a trusted third party (TTP) to distribute reward to agents, leaving them inapplicable in peer-to-peer environments. This paper proposes reward distribution using {\em Neuron as an Agent} (NaaA) in MARL without a TTP with two key ideas: (i) inter-agent reward distribution and (ii) auction theory. Auction theory is introduced because inter-agent reward distribution is insufficient for optimization. Agents in NaaA maximize their profits (the difference between reward and cost) and, as a theoretical result, the auction mechanism is shown to have agents autonomously evaluate counterfactual returns as the values of other agents. NaaA enables representation trades in peer-to-peer environments, ultimately regarding unit in neural networks as agents. Finally, numerical experiments (a single-agent environment from OpenAI Gym and a multi-agent environment from ViZDoom) confirm that NaaA framework optimization leads to better performance in reinforcement learning.
“Style” Transfer for Musical Audio Using Multiple Time-Frequency Representations
Authors: Shaun Barry, Youngmoo Kim
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Style Transfer, Texture Synthesis

Neural Style Transfer has become a popular technique for
generating images of distinct artistic styles using convolutional neural networks. This
recent success in image style transfer has raised the question of
whether similar methods can be leveraged to alter the “style” of musical
audio. In this work, we attempt long time-scale high-quality audio transfer
and texture synthesis in the time-domain that captures harmonic,
rhythmic, and timbral elements related to musical style, using examples that
may have different lengths and musical keys. We demonstrate the ability
to use randomly initialized convolutional neural networks to transfer
these aspects of musical style from one piece onto another using 3
different representations of audio: the log-magnitude of the Short Time
Fourier Transform (STFT), the Mel spectrogram, and the Constant-Q Transform
spectrogram. We propose using these representations as a way of
generating and modifying perceptually significant characteristics of
musical audio content. We demonstrate each representation's
shortcomings and advantages over others by carefully designing
neural network structures that complement the nature of musical audio. Finally, we show that the most
compelling “style” transfer examples make use of an ensemble of these
representations to help capture the varying desired characteristics of
audio signals.
Model-based imitation learning from state trajectories
Authors: Subhajit Chaudhury, Daiki Kimura, Tadanobu Inoue, Ryuki Tachibana
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Imitation Learning, model, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Imitation learning from demonstrations usually relies on learning a policy from trajectories of optimal states and actions. However, in real life expert demonstrations, often the action information is missing and only state trajectories are available. We present a model-based imitation learning method that can learn environment-specific optimal actions only from expert state trajectories. Our proposed method starts with a model-free reinforcement learning algorithm with a heuristic reward signal to sample environment dynamics, which is then used to train the state-transition probability. Subsequently, we learn the optimal actions from expert state trajectories by supervised learning, while back-propagating the error gradients through the modeled environment dynamics. Experimental evaluations show that our proposed method successfully achieves performance similar to (state, action) trajectory-based traditional imitation learning methods even in the absence of action information, with much fewer iterations compared to conventional model-free reinforcement learning methods. We also demonstrate that our method can learn to act from only video demonstrations of expert agent for simple games and can learn to achieve desired performance in less number of iterations.
Neural Task Graph Execution
Authors: Sungryull Sohn, Junhyuk Oh, Honglak Lee
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Reinforcement Learning, Reinforcement Learning (RL)

In order to develop a scalable multi-task reinforcement learning (RL) agent that is able to execute many complex tasks, this paper introduces a new RL problem where the agent is required to execute a given task graph which describes a set of subtasks and dependencies among them. Unlike existing approaches which explicitly describe what the agent should do, our problem only describes properties of subtasks and relationships between them, which requires the agent to perform a complex reasoning to find the optimal subtask to execute. To solve this problem, we propose a neural task graph solver (NTS) which encodes the task graph using a recursive neural network. To overcome the difficulty of training, we propose a novel non-parametric gradient-based policy that performs back-propagation over a differentiable form of the task graph to compute the influence of each subtask on the other subtasks. Our NTS is pre-trained to approximate the proposed gradient-based policy and fine-tuned through actor-critic method. The experimental results on a 2D visual domain show that our method to pre-train from the gradient-based policy significantly improves the performance of NTS. We also demonstrate that our agent can perform a complex reasoning to find the optimal way of executing the task graph and generalize well to unseen task graphs. In addition, we compare our agent with a Monte-Carlo Tree Search (MCTS) method showing that our method is much more efficient than MCTS, and the performance of our agent can be further improved by combining with MCTS. The demo video is available at https://youtu.be/e_ZXVS5VutM.
UNSUPERVISED SENTENCE EMBEDDING USING DOCUMENT STRUCTURE-BASED CONTEXT
Authors: Taesung Lee, Youngja Park
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: coreference-resolution, Coreference Resolution, Sentence, Sentence Embedding, Sentence-Embedding, Sentence Embeddings

We present a new unsupervised method for learning general-purpose sentence embeddings.
Unlike existing methods which rely on local contexts, such as words
inside the sentence or immediately neighboring sentences, our method selects, for
each target sentence, influential sentences in the entire document based on a document
structure. We identify a dependency structure of sentences using metadata
or text styles. Furthermore, we propose a novel out-of-vocabulary word handling
technique to model many domain-specific terms, which were mostly discarded by
existing sentence embedding methods. We validate our model on several tasks
showing 30% precision improvement in coreference resolution in a technical domain,
and 7.5% accuracy increase in paraphrase detection compared to baselines.
Directing Generative Networks with Weighted Maximum Mean Discrepancy
Authors: Maurice Diesendruck, Guy W. Cole, Sinead Williamson
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

The maximum mean discrepancy (MMD) between two probability measures P
and Q is a metric that is zero if and only if all moments of the two measures
are equal, making it an appealing statistic for two-sample tests. Given i.i.d. samples
from P and Q, Gretton et al. (2012) show that we can construct an unbiased
estimator for the square of the MMD between the two distributions. If P is a
distribution of interest and Q is the distribution implied by a generative neural
network with stochastic inputs, we can use this estimator to train our neural network.
However, in practice we do not always have i.i.d. samples from our target
of interest. Data sets often exhibit biases—for example, under-representation of
certain demographics—and if we ignore this fact our machine learning algorithms
will propagate these biases. Alternatively, it may be useful to assume our data has
been gathered via a biased sample selection mechanism in order to manipulate
properties of the estimating distribution Q.
In this paper, we construct an estimator for the MMD between P and Q when we
only have access to P via some biased sample selection mechanism, and suggest
methods for estimating this sample selection mechanism when it is not already
known. We show that this estimator can be used to train generative neural networks
on a biased data sample, to give a simulator that reverses the effect of that
bias.
Soft Value Iteration Networks for Planetary Rover Path Planning
Authors: Max Pflueger, Ali Agha, Gaurav S. Sukhatme
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Motion Planning

Value iteration networks are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable. In this work, we study these networks in the context of robot motion planning, with a focus on applications to planetary rovers. The key challenging task in learning-based motion planning is to learn a transformation from terrain observations to a suitable navigation reward function. In order to deal with complex terrain observations and policy learning, we propose a value iteration recurrence, referred to as the soft value iteration network (SVIN). SVIN is designed to produce more effective training gradients through the value iteration network. It relies on a soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action. We demonstrate the effectiveness of the proposed method in robot motion planning scenarios. In particular, we study the application of SVIN to very challenging problems in planetary rover navigation and present early training results on data gathered by the  Curiosity rover that is currently operating on Mars.
A Classification-Based Perspective on GAN Distributions
Authors: Shibani Santurkar, Ludwig Schmidt, Aleksander Madry
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Classification, Diversity, General Classification

A fundamental, and still largely unanswered, question in the context of Generative Adversarial Networks (GANs) is whether GANs are actually able to capture the key characteristics of the datasets they are trained on. The current approaches to examining this issue require significant human supervision, such as visual inspection of sampled images, and often offer only fairly limited scalability. In this paper, we propose new techniques that employ classification-based perspective to evaluate synthetic GAN distributions and their capability to accurately reflect the essential properties of the training data. These techniques require only minimal human supervision and can easily be scaled and adapted to evaluate a variety of state-of-the-art GANs on large, popular datasets. They also indicate that GANs have significant problems in reproducing the more distributional properties of the training dataset. In particular, the diversity of such synthetic data is orders of magnitude smaller than that of the original data.
Achieving Strong Regularization for Deep Neural Networks
Authors: Dae Hoon Park, Chiu Man Ho, Yi Chang
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: L2 Regularization

L1 and L2 regularizers are critical tools in machine learning due to their ability to simplify solutions. However, imposing strong L1 or L2 regularization with gradient descent method easily fails, and this limits the generalization ability of the underlying neural networks. To understand this phenomenon, we investigate how and why training fails for strong regularization. Specifically, we examine how gradients change over time for different regularization strengths and provide an analysis why the gradients diminish so fast. We find that there exists a tolerance level of regularization strength, where the learning completely fails if the regularization strength goes beyond it. We propose a simple but novel method, Delayed Strong Regularization, in order to moderate the tolerance level. Experiment results show that our proposed approach indeed achieves strong regularization for both L1 and L2 regularizers and improves both accuracy and sparsity on public data sets. Our source code is published.
A Spectral Approach to Generalization and Optimization in Neural Networks
Authors: Farzan Farnia, Jesse Zhang, David Tse
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

The recent success of deep neural networks stems from their ability to generalize well on real data; however, Zhang et al. have observed that neural networks can easily overfit random labels. This observation demonstrates that with the existing theory, we cannot adequately explain why gradient methods can find generalizable solutions for neural networks. In this work, we use a Fourier-based approach to study the generalization properties of gradient-based methods over 2-layer neural networks with sinusoidal activation functions. We prove that if the underlying distribution of data has nice spectral properties such as bandlimitedness, then the gradient descent method will converge to generalizable local minima. We also establish a Fourier-based generalization bound for bandlimited spaces, which generalizes to other activation functions. Our generalization bound motivates a grouped version of path norms for measuring the complexity of 2-layer neural networks with ReLU activation functions. We demonstrate numerically that regularization of this group path norm results in neural network solutions that can fit true labels without losing test accuracy while not overfitting random labels.
Small Coresets to Represent Large Training Data for Support Vector Machines
Authors: Cenk Baykal, Murad Tukan, Dan Feldman, Daniela Rus
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Support Vector Machines (SVMs) are one of the most popular algorithms for classification and regression analysis. Despite their popularity, even efficient implementations have proven to be computationally expensive to train at a large-scale, especially in streaming settings. In this paper, we propose a novel coreset construction algorithm for efficiently generating compact representations of massive data sets to speed up SVM training. A coreset is a weighted subset of the original data points such that SVMs trained on the coreset are provably competitive with those trained on the original (massive) data set. We provide both lower and upper bounds on the number of samples required to obtain accurate approximations to the SVM problem as a function of the complexity of the input data. Our analysis also establishes sufficient conditions on the existence of sufficiently compact and representative coresets for the SVM problem. We empirically evaluate the practical effectiveness of our algorithm against synthetic and real-world data sets.
LSH Softmax: Sub-Linear Learning and Inference of the Softmax Layer in Deep Architectures
Authors: Daniel Levy, Danlu Chan, Stefano Ermon
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Deep Learning, Language Modeling, Language Modelling

Log-linear models models are widely used in machine learning, and in particular are ubiquitous in deep learning architectures in the form of the softmax. While exact inference and learning of these requires linear time, it can be done approximately in sub-linear time with strong concentrations guarantees. In this work, we present LSH Softmax, a method to perform sub-linear learning and inference of the softmax layer in the deep learning setting. Our method relies on the popular Locality-Sensitive Hashing to build a well-concentrated gradient estimator, using nearest neighbors and uniform samples. We also present an inference scheme in sub-linear time for LSH Softmax using the Gumbel distribution. On language modeling, we show that Recurrent Neural Networks trained with LSH Softmax perform on-par with computing the exact softmax while requiring sub-linear computations.
Recurrent Relational Networks for complex relational reasoning
Authors: Rasmus Berg Palm, Ulrich Paquet, Ole Winther
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Relational Reasoning

Humans possess an ability to abstractly reason about objects and their interactions, an ability not shared with state-of-the-art deep learning models. Relational networks, introduced by Santoro et al. (2017), add the capacity for relational reasoning to deep neural networks, but are limited in the complexity of the reasoning tasks they can address. We introduce recurrent relational networks which increase the suite of solvable tasks to those that require an order of magnitude more steps of relational reasoning. We use recurrent relational networks to solve Sudoku puzzles and achieve state-of-the-art results by solving 96.6% of the hardest Sudoku puzzles, where relational networks fail to solve any. We also apply our model to the BaBi textual QA dataset solving 19/20 tasks which is competitive with state-of-the-art sparse differentiable neural computers. The recurrent relational network is a general purpose module that can augment any neural network model with the capacity to do many-step relational reasoning.