pwc-archive/papers-with-abstracts huggingface dataset:
Neighbor-encoder
Authors: Chin-Chia Michael Yeh, Yan Zhu, Evangelos E. Papalexakis, Abdullah Mueen, Eamonn Keogh
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Decoder, Representation Learning, Time Series, Time Series Analysis

We propose a novel unsupervised representation learning framework called neighbor-encoder in which domain knowledge can be trivially incorporated into the learning process without modifying the general encoder-decoder architecture. In contrast to autoencoder, which reconstructs the input data, neighbor-encoder reconstructs the input data's neighbors. The proposed neighbor-encoder can be considered as a generalization of autoencoder as the input data can be treated as the nearest neighbor of itself with zero distance. By reformulating the representation learning problem as a neighbor reconstruction problem, domain knowledge can be easily incorporated with appropriate definition of similarity or distance between objects. As such, any existing similarity search algorithms can be easily integrated into our framework. Applications of other algorithms (e.g., association rule mining) in our framework is also possible since the concept of ``neighbor" is an abstraction which can be appropriately defined differently in different contexts. We have demonstrated the effectiveness of our framework in various domains, including images, time series, music, etc., with various neighbor definitions. Experimental results show that neighbor-encoder outperforms autoencoder in most scenarios we considered.
Benefits of Depth for Long-Term Memory of Recurrent Networks
Authors: Yoav Levine, Or Sharir, Amnon Shashua
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Attribute, Time Series Analysis

The key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ever-improving ability to model intricate long-term temporal dependencies. However, a well established measure of RNNs' long-term memory capacity is lacking, and thus formal understanding of their ability to correlate data throughout time is limited. Though depth efficiency in convolutional networks is well established by now, it does not suffice in order to account for the success of deep RNNs on inputs of varying lengths, and the need to address their 'time-series expressive power' arises. In this paper, we analyze the effect of depth on the ability of recurrent networks to express correlations ranging over long time-scales. To meet the above need, we introduce a measure of the information flow across time that can be supported by the network, referred to as the Start-End separation rank. Essentially, this measure reflects the distance of the function realized by the recurrent network from a function that models no interaction whatsoever between the beginning and end of the input sequence. We prove that deep recurrent networks support Start-End separation ranks which are exponentially higher than those supported by their shallow counterparts. Moreover, we show that the ability of deep recurrent networks to correlate different parts of the input sequence increases exponentially as the input sequence extends, while that of vanilla shallow recurrent networks does not adapt to the sequence length at all. Thus, we establish that depth brings forth an overwhelming advantage in the ability of recurrent networks to model long-term dependencies, and provide an exemplar of quantifying this key attribute which may be readily extended to other RNN architectures of interest, e.g. variants of LSTM networks. We obtain our results by considering a class of recurrent networks referred to as Recurrent Arithmetic Circuits (RACs), which merge the hidden state with the input via the Multiplicative Integration operation.
A Neural Method for Goal-Oriented Dialog Systems to interact with Named Entities
Authors: Janarthanan Rajendran, Jatin Ganhotra, Xiaoxiao Guo, Mo Yu, Satinder Singh
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Goal-Oriented Dialog, Question Answering

Many goal-oriented dialog tasks, especially ones in which the dialog system has to interact with external knowledge sources such as databases, have to handle a large number of Named Entities (NEs). There are at least two challenges in handling NEs using neural methods in such settings: individual NEs may occur only rarely making it hard to learn good representations of them, and many of the Out Of Vocabulary words that occur during test time may be NEs. Thus, the need to interact well with these NEs has emerged as a serious challenge to building neural methods for goal-oriented dialog tasks. In this paper, we propose a new neural method for this problem, and present empirical evaluations on a structured Question answering task and three related goal-oriented dialog tasks that show that our proposed method can be effective in interacting with NEs in these settings.
Comparison of Paragram and GloVe Results for Similarity Benchmarks
Authors: Jakub Dutkiewicz, Czesław Jędrzejek
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Semantic Similarity, Semantic Textual Similarity, Word Embeddings

Distributional Semantics Models(DSM) derive word space from linguistic items
in context. Meaning is obtained by defining a distance measure between vectors
corresponding to lexical entities. Such vectors present several problems. This
work concentrates on quality of word embeddings, improvement of word embedding
vectors, applicability of a novel similarity metric used ‘on top’ of the
word embeddings. In this paper we provide comparison between two methods
for post process improvements to the baseline DSM vectors. The counter-fitting
method which enforces antonymy and synonymy constraints into the Paragram
vector space representations recently showed improvement in the vectors’ capability
for judging semantic similarity. The second method is our novel RESM
method applied to GloVe baseline vectors. By applying the hubness reduction
method, implementing relational knowledge into the model by retrofitting synonyms
and providing a new ranking similarity definition RESM that gives maximum
weight to the top vector component values we equal the results for the ESL
and TOEFL sets in comparison with our calculations using the Paragram and Paragram
+ Counter-fitting methods. For SIMLEX-999 gold standard since we cannot
use the RESM the results using GloVe and PPDB are significantly worse compared
to Paragram. Apparently, counter-fitting corrects hubness. The Paragram
or our cosine retrofitting method are state-of-the-art results for the SIMLEX-999
gold standard. They are 0.2 better for SIMLEX-999 than word2vec with sense
de-conflation (that was announced to be state-of the-art method for less reliable
gold standards). Apparently relational knowledge and counter-fitting is more important
for judging semantic similarity than sense determination for words. It is to
be mentioned, though that Paragram hyperparameters are fitted to SIMLEX-999
results. The lesson is that many corrections to word embeddings are necessary
and methods with more parameters and hyperparameters perform better.
Domain Adaptation for Deep Reinforcement Learning in Visually Distinct Games
Authors: Dino S. Ratcliffe, Luca Citi, Sam Devlin, Udo Kruschwitz
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Deep Reinforcement Learning, Domain Adaptation, Multi-Task Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Many deep reinforcement learning approaches use graphical state representations,
this means visually distinct games that share the same underlying structure cannot
effectively share knowledge. This paper outlines a new approach for learning
underlying game state embeddings irrespective of the visual rendering of the game
state. We utilise approaches from multi-task learning and domain adaption in
order to place visually distinct game states on a shared embedding manifold. We
present our results in the context of deep reinforcement learning agents.
Ego-CNN: An Ego Network-based Representation of Graphs Detecting Critical Structures
Authors: Ruo-Chun Tzeng, Shan-Hung Wu
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Graph Embedding

While existing graph embedding models can generate useful embedding vectors that perform well on graph-related tasks, what valuable information can be jointly learned by a graph embedding model is less discussed. In this paper, we consider the possibility of detecting critical structures by a graph embedding model. We propose Ego-CNN to embed graph, which works in a local-to-global manner to take advantages of CNNs that gradually expanding the detectable local regions on the graph as the network depth increases. Critical structures can be detected if Ego-CNN is combined with a supervised task model. We show that Ego-CNN is (1) competitive to state-of-the-art graph embeddings models, (2) can nicely work with CNNs visualization techniques to show the detected structures, and (3) is efficient and can incorporate with scale-free priors, which commonly occurs in social network datasets, to further improve the training efficiency.
Lifelong Learning by Adjusting Priors
Authors: Ron Amit, Ron Meir
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Generalization Bounds, Lifelong learning

In representational lifelong learning an agent aims to continually learn to solve novel tasks while updating its representation in light of previous tasks. Under the assumption that future tasks are related to previous tasks, representations should be learned in such a way that they capture the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of a new task. We develop a framework for lifelong learning in deep neural networks that is based on generalization bounds, developed within the PAC-Bayes framework. Learning takes place through the construction of a distribution over networks based on the tasks seen so far, and its utilization for learning a new task. Thus, prior knowledge is incorporated through setting a history-dependent prior for novel tasks. We develop a gradient-based algorithm implementing these ideas, based on minimizing an objective function motivated by generalization bounds, and demonstrate its effectiveness through numerical examples.
A Simple Fully Connected Network for Composing Word Embeddings from Characters
Authors: Michael Traynor, Thomas Trappenberg
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Language Modeling, Language Modelling, Position, Word Embeddings

This work introduces a simple network for producing character aware word embeddings. Position agnostic and position aware character embeddings are combined to produce an embedding vector for each word. The learned word representations are shown to be very sparse and facilitate improved results on language modeling tasks, despite using markedly fewer parameters, and without the need to apply dropout. A final experiment suggests that weight sharing contributes to sparsity, increases performance, and prevents overfitting.
Censoring Representations with Multiple-Adversaries over Random Subspaces
Authors: Yusuke Iwasawa, Kotaro Nakayama, Yutaka Matsuo
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Adversarial feature learning (AFL) is one of the promising ways for explicitly constrains neural networks to learn desired representations; for example, AFL could help to learn anonymized representations so as to avoid privacy issues. AFL learn such a representations by training the networks to deceive the adversary that predict the sensitive information from the network, and therefore, the success of the AFL heavily relies on the choice of the adversary. This paper proposes a novel design of the adversary, {\em multiple adversaries over random subspaces} (MARS) that instantiate the concept of the {\em volunerableness}. The proposed method is motivated by an assumption that deceiving an adversary could fail to give meaningful information if the adversary is easily fooled, and adversary rely on single classifier suffer from this issues. 
In contrast, the proposed method is designed to be less vulnerable, by utilizing the ensemble of independent classifiers where each classifier tries to predict sensitive variables from a different {\em subset} of the representations. 
The empirical validations on three user-anonymization tasks show that our proposed method achieves state-of-the-art performances in all three datasets without significantly harming the utility of data. 
This is significant because it gives new implications about designing the adversary, which is important to improve the performance of AFL.
Evolutionary Expectation Maximization for Generative Models with Binary Latents
Authors: Enrico Guiraud, Jakob Drefs, Joerg Luecke
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Evolutionary Algorithms

We establish a theoretical link between evolutionary algorithms and variational parameter optimization of probabilistic generative models with binary hidden variables.
While the novel approach is independent of the actual generative model, here we use two such models to investigate its applicability and scalability: a noisy-OR Bayes Net (as a standard example of binary data) and Binary Sparse Coding (as a model for continuous data).

Learning of probabilistic generative models is first formulated as approximate maximum likelihood optimization using variational expectation maximization (EM).
We choose truncated posteriors as variational distributions in which discrete latent states serve as variational parameters. In the variational E-step,
the latent states are then  
optimized according to a tractable free-energy objective. Given a data point, we can show that evolutionary algorithms can be used for the variational optimization loop by (A)~considering the bit-vectors of the latent states as genomes of individuals, and by (B)~defining the fitness of the
individuals as the (log) joint probabilities given by the used generative model.

As a proof of concept, we apply the novel evolutionary EM approach to the optimization of the parameters of noisy-OR Bayes nets and binary sparse coding on artificial and real data (natural image patches). Using point mutations and single-point cross-over for the evolutionary algorithm, we find that scalable variational EM algorithms are obtained which efficiently improve the data likelihood. In general we believe that, with the link established here, standard as well as recent results in the field of evolutionary optimization can be leveraged to address the difficult problem of parameter optimization in generative models.
Learnability of Learned Neural Networks
Authors: Rahul Anand Sharma, Navin Goyal, Monojit Choudhury, Praneeth Netrapalli
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

This paper explores the simplicity of learned neural networks under various settings: learned on real vs random data, varying size/architecture and using large minibatch size vs small minibatch size. The notion of simplicity used here is that of learnability i.e., how accurately can the prediction function of a neural network be learned from labeled samples from it. While learnability is different from (in fact often higher than) test accuracy, the results herein suggest that there is a strong correlation between small generalization errors and high learnability.
This work also shows that there exist significant qualitative differences in shallow networks as compared to popular deep networks. More broadly, this paper extends in a new direction, previous work on understanding the properties of learned neural networks. Our hope is that such an empirical study of understanding learned neural networks might shed light on the right assumptions that can be made for a theoretical study of deep learning.
Online Hyper-Parameter Optimization
Authors: Damien Vincent, Sylvain Gelly, Nicolas Le Roux, Olivier Bousquet
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Hyperparameter Optimization

We propose an efficient online hyperparameter optimization method which uses a joint dynamical system to evaluate the gradient with respect to the hyperparameters. While similar methods are usually limited to hyperparameters with a smooth impact on the model, we show how to apply it to the probability of dropout in neural networks. Finally, we show its effectiveness on two distinct tasks.
Word2net: Deep Representations of Language
Authors: Maja Rudolph, Francisco Ruiz, David Blei
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Articles, Word Embeddings

Word embeddings extract semantic features of words from large datasets of text.
Most embedding methods rely on a log-bilinear model to predict the occurrence
of a word in a context of other words. Here we propose word2net, a method that
replaces their linear parametrization with neural networks. For each term in the
vocabulary, word2net posits a neural network that takes the context as input and
outputs a probability of occurrence. Further, word2net can use the hierarchical
organization of its word networks to incorporate additional meta-data, such as
syntactic features, into the embedding model. For example, we show how to share
parameters across word networks to develop an embedding model that includes
part-of-speech information. We study word2net with two datasets, a collection
of Wikipedia articles and a corpus of U.S. Senate speeches. Quantitatively, we
found that word2net outperforms popular embedding methods on predicting held-
out words and that sharing parameters based on part of speech further boosts
performance. Qualitatively, word2net learns interpretable semantic representations
and, compared to vector-based methods, better incorporates syntactic information.
The loss surface and expressivity of deep convolutional neural networks
Authors: Quynh Nguyen, Matthias Hein
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

We analyze the expressiveness and loss surface of practical deep convolutional
neural networks (CNNs) with shared weights and max pooling layers. We show
that such CNNs produce linearly independent features at a “wide” layer which
has more neurons than the number of training samples. This condition holds e.g.
for the VGG network. Furthermore, we provide for such wide CNNs necessary
and sufficient conditions for global minima with zero training error. For the case
where the wide layer is followed by a fully connected layer we show that almost
every critical point of the empirical loss is a global minimum with zero training
error. Our analysis suggests that both depth and width are very important in deep
learning. While depth brings more representational power and allows the network
to learn high level features, width smoothes the optimization landscape of the
loss function in the sense that a sufficiently wide network has a well-behaved loss
surface with almost no bad local minima.
Combination of Supervised and Reinforcement Learning For Vision-Based Autonomous Control
Authors: Dmitry Kangin, Nicolas Pugeault
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: MuJoCo, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

 Reinforcement learning methods have recently achieved impressive results on a wide range of control problems. However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution. This limitation largely prohibits  their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control. Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account reward-based control policies and is not capable to provide independent control policies.  In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments. We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality. Additionally, we demonstrate the applicability of the method to MuJoCo control problems.
Gating out sensory noise in a spike-based Long Short-Term Memory network
Authors: Davide Zambrano, Isabella Pozzi, Roeland Nusselder, Sander Bohte
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Reinforcement Learning

Spiking neural networks are being investigated both as biologically plausible models of neural computation and also as a potentially more efficient type of neural network. While convolutional spiking neural networks have been demonstrated to achieve near state-of-the-art performance, only one solution has been proposed to convert gated recurrent neural networks, so far.
Recurrent neural networks in the form of networks of gating memory cells have been central in state-of-the-art solutions in problem domains that involve sequence recognition or generation. Here, we design an analog gated LSTM cell where its neurons can be substituted for efficient stochastic spiking neurons. These adaptive spiking neurons implement an adaptive form of sigma-delta coding to convert internally computed analog activation values to spike-trains. For such neurons, we approximate the effective activation function, which resembles a sigmoid. We show how analog neurons with such activation functions can be used to create an analog LSTM cell; networks of these cells can then be trained with standard backpropagation. We train these LSTM networks on a noisy and noiseless version of the original sequence prediction task from Hochreiter & Schmidhuber (1997), and also on a noisy and noiseless version of a classical working memory reinforcement learning task, the T-Maze. Substituting the analog neurons for corresponding adaptive spiking neurons, we then show that almost all resulting spiking neural network equivalents correctly compute the original tasks.
Convolutional Sequence Modeling Revisited
Authors: Shaojie Bai, J. Zico Kolter, Vladlen Koltun
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Language Modelling, Time Series Analysis

This paper revisits the problem of sequence modeling using convolutional 
architectures.  Although both convolutional and recurrent architectures have a
long history in sequence prediction, the current "default" mindset in much of
the deep learning community is that generic sequence modeling is best handled
using recurrent networks.  The goal of this paper is to question this assumption. 
Specifically, we consider a simple generic temporal convolution network (TCN),
which adopts features from modern ConvNet architectures such as a dilations and 
residual connections.  We show that on a variety of sequence modeling tasks,
including many frequently used as benchmarks for evaluating recurrent networks,
the TCN outperforms baseline RNN methods (LSTMs, GRUs, and vanilla RNNs) and
sometimes even highly specialized approaches.  We further show that the
potential "infinite memory" advantage that RNNs have over TCNs is largely
absent in practice: TCNs indeed exhibit longer effective history sizes than their 
recurrent counterparts.   As a whole, we argue that it may be time to (re)consider 
ConvNets as the default "go to" architecture for sequence modeling.