pwc-archive/papers-with-abstracts huggingface dataset:
Discrete-Valued Neural Networks Using Variational Inference
Authors: Wolfgang Roth, Franz Pernkopf
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Quantization, Variational Inference

The increasing demand for neural networks (NNs) being employed on embedded devices has led to plenty of research investigating methods for training low precision NNs. While most methods involve a quantization step, we propose a principled Bayesian approach where we first infer a distribution over a discrete weight space from which we subsequently derive hardware-friendly low precision NNs. To this end, we introduce a probabilistic forward pass to approximate the intractable variational objective that allows us to optimize over discrete-valued weight distributions for NNs with sign activation functions. In our experiments, we show that our model achieves state of the art performance on several real world data sets. In addition, the resulting models exhibit a substantial amount of sparsity that can be utilized to further reduce the computational costs for inference.
Feat2Vec: Dense Vector Representation for Data with Arbitrary Features
Authors: Luis Armona, José P. González-Brenes, Ralph Edezhath
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Methods that calculate dense vector representations for features in unstructured data—such as words in a document—have proven to be very successful for knowledge representation. We study how to estimate dense representations when multiple feature types exist within a dataset for supervised learning where explicit labels are available, as well as for unsupervised learning where there are no labels. Feat2Vec calculates embeddings for data with multiple feature types enforcing that all different feature types exist in a common space. In the supervised case, we show that our method has advantages over recently proposed methods; such as enabling higher prediction accuracy, and providing a way to avoid the cold-start
problem. In the unsupervised case, our experiments suggest that Feat2Vec significantly outperforms existing algorithms that do not leverage the structure of the data. We believe that we are the first to propose a method for learning unsuper vised embeddings that leverage the structure of multiple feature types.
Tensor Contraction & Regression Networks
Authors: Jean Kossaifi, Zack Chase Lipton, Aran Khanna, Tommaso Furlanello, Anima Anandkumar
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: regression

Convolution neural networks typically consist of many convolutional layers followed by several fully-connected layers.  While convolutional layers map between high-order activation tensors, the fully-connected layers operate on flattened activation vectors.  Despite its success, this approach has notable drawbacks. Flattening discards the multi-dimensional structure of the activations, and the fully-connected layers require a large number of parameters. 
We present two new techniques to address these problems.  First, we introduce tensor contraction layers which can replace the ordinary fully-connected layers in a neural network. Second, we introduce tensor regression layers, which express the output of a neural network as a low-rank multi-linear mapping from a high-order activation tensor to the softmax layer.  Both the contraction and regression weights are learned end-to-end by backpropagation. By imposing low rank on both, we use significantly fewer parameters.  Experiments on the ImageNet dataset show that applied to the popular VGG and ResNet architectures, our methods significantly reduce the number of parameters in the fully connected layers (about 65% space savings) while negligibly impacting accuracy.
Fixing Weight Decay Regularization in Adam
Authors: Ilya Loshchilov, Frank Hutter
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: image-classification, Image Classification

We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. 
We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) 
decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).
We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence. Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. 
Our source code will become available after the review process.
Code Synthesis with Priority Queue Training
Authors: Daniel A. Abolafia, Quoc V. Le, Mohammad Norouzi
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Program Synthesis, Reinforcement Learning

We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards. We introduce a novel iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far. Then, we synthesize new programs and add them to the priority queue by sampling from the RNN. We benchmark our algorithm called priority queue training (PQT) against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF. Our experimental results show that our deceptively simple PQT algorithm significantly outperforms the baselines. By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.
Learning what to learn in a neural program
Authors: Richard Shin, Dawn Song
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Learning programs with neural networks is a challenging task, addressed by a long line of existing work. It is difficult to learn neural networks which will generalize to problem instances that are much larger than those used during training. Furthermore, even when the learned neural program empirically works on all test inputs, we cannot verify that it will work on every possible input. Recent work has shown that it is possible to address these issues by using recursion in the Neural Programmer-Interpreter, but this technique requires a verification set which is difficult to construct without knowledge of the internals of the oracle used to generate training data. In this work, we show how to automatically build such a verification set, which can also be directly used for training. By interactively querying an oracle, we can construct this set with minimal additional knowledge about the oracle. We empirically demonstrate that our method allows automated learning and verification of a recursive NPI program with provably perfect generalization.
Generative Models for Alignment and Data Efficiency in Language
Authors: Dustin Tran, Yura Burda, Ilya Sutskever
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Decipherment, Machine Translation, Translation, Unsupervised Machine Translation

We examine how learning from unaligned data can improve both the data efficiency of supervised tasks as well as enable alignments without any supervision. For example, consider unsupervised machine translation: the input is two corpora of English and French, and the task is to translate from one language to the other but without any pairs of English and French sentences. To address this, we develop feature-matching autoencoders (FMAEs). FMAEs ensure that the marginal distribution of feature layers are preserved across forward and inverse mappings between domains. We show that FMAEs achieve state of the art for data efficiency and alignment across three tasks: text decipherment, sentiment transfer, and neural machine translation for English-to-German and English-to-French. Most compellingly, FMAEs achieve state of the art for neural translation with limited supervision, with significant BLEU score differences of up to 5.7 and 6.3 over traditional supervised models. Furthermore, on English-to-German, they outperform last year's best fully supervised models such as ByteNet (Kalchbrenner et al., 2016) while using only half as many supervised examples.
Stochastic Training of Graph Convolutional Networks
Authors: Jianfei Chen, Jun Zhu
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Graph convolutional networks (GCNs) are powerful deep neural networks for graph-structured data. However, GCN computes nodes' representation recursively from their neighbors, making the receptive field size grow exponentially with the number of layers.  Previous attempts on reducing the receptive field size by subsampling neighbors do not have any convergence guarantee, and their receptive field size per node is still in the order of hundreds. In this paper, we develop a preprocessing strategy and two control variate based algorithms to further reduce the receptive field size. Our algorithms are guaranteed to converge to GCN's local optimum regardless of the neighbor sampling size. Empirical results show that our algorithms have a similar convergence speed per epoch with the exact algorithm even using only two neighbors per node. The time consumption of our algorithm on the Reddit dataset is only one fifth of previous neighbor sampling algorithms.
Nearest Neighbour Radial Basis Function Solvers for Deep Neural Networks
Authors: Benjamin J. Meyer, Ben Harwood, Tom Drummond
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Classification, General Classification, Metric Learning

We present a radial basis function solver for convolutional neural networks that can be directly applied to both distance metric learning and classification problems. Our method treats all training features from a deep neural network as radial basis function centres and computes loss by summing the influence of a feature's nearby centres in the embedding space. Having a radial basis function centred on each training feature is made scalable by treating it as an approximate nearest neighbour search problem. End-to-end learning of the network and solver is carried out, mapping high dimensional features into clusters of the same class. This results in a well formed embedding space, where semantically related instances are likely to be located near one another, regardless of whether or not the network was trained on those classes. The same loss function is used for both the metric learning and classification problems. We show that our radial basis function solver outperforms state-of-the-art embedding approaches on the Stanford Cars196 and CUB-200-2011 datasets. Additionally, we show that when used as a classifier, our method outperforms a conventional softmax classifier on the CUB-200-2011, Stanford Cars196, Oxford 102 Flowers and Leafsnap fine-grained classification datasets.
Deep Learning Inferences with Hybrid Homomorphic Encryption
Authors: Anthony Meehan, Ryan K. L. Ko, Geoff Holmes
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Deep Learning, Privacy Preserving, Representation Learning

When deep learning is applied to sensitive data sets, many privacy-related implementation issues arise. These issues are especially evident in the healthcare, finance, law and government industries. Homomorphic encryption could allow a server to make inferences on inputs encrypted by a client, but to our best knowledge, there has been no complete implementation of common deep learning operations, for arbitrary model depths, using homomorphic encryption. This paper demonstrates a novel approach, efficiently implementing many deep learning functions with bootstrapped homomorphic encryption. As part of our implementation, we demonstrate Single and Multi-Layer Neural Networks, for the Wisconsin Breast Cancer dataset, as well as a Convolutional Neural Network for MNIST. Our results give promising directions for privacy-preserving representation learning, and the return of data control to users.
Adversarial reading networks for machine comprehension
Authors: Quentin Grail, julien perez
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Machine Reading Comprehension, Question Answering, Reading Comprehension

Machine reading has recently shown remarkable progress thanks to differentiable
reasoning models. In this context, End-to-End trainable Memory Networks
(MemN2N) have demonstrated promising performance on simple natural language
based reasoning tasks such as factual reasoning and basic deduction. However,
the task of machine comprehension is currently bounded to a supervised setting
and available question answering dataset. In this paper we explore the paradigm
of adversarial learning and self-play for the task of machine reading comprehension.
Inspired by the successful propositions in the domain of game learning, we
present a novel approach of training for this task that is based on the definition
of a coupled attention-based memory model. On one hand, a reader network is
in charge of finding answers regarding a passage of text and a question. On the
other hand, a narrator network is in charge of obfuscating spans of text in order
to minimize the probability of success of the reader. We experimented the model
on several question-answering corpora. The proposed learning paradigm and associated
models present encouraging results.
Contextual memory bandit for pro-active dialog engagement
Authors: julien perez, Tomi Silander
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Multi-Armed Bandits

An objective of pro-activity in dialog systems is to enhance the usability of conversational
agents by enabling them to initiate conversation on their own. While
dialog systems have become increasingly popular during the last couple of years,
current task oriented dialog systems are still mainly reactive and users tend to
initiate conversations. In this paper, we propose to introduce the paradigm of contextual
bandits as framework for pro-active dialog systems. Contextual bandits
have been the model of choice for the problem of reward maximization with partial
feedback since they fit well to the task description. As a second contribution,
we introduce and explore the notion of memory into this paradigm. We propose
two differentiable memory models that act as parts of the parametric reward estimation
function. The first one, Convolutional Selective Memory Networks, uses
a selection of past interactions as part of the decision support. The second model,
called Contextual Attentive Memory Network, implements a differentiable attention
mechanism over the past interactions of the agent. The goal is to generalize
the classic model of contextual bandits to settings where temporal information
needs to be incorporated and leveraged in a learnable manner. Finally, we illustrate
the usability and performance of our model for building a pro-active mobile
assistant through an extensive set of experiments.
Sequence Transfer Learning for Neural Decoding
Authors: Venkatesh Elango*, Aashish N Patel*, Kai J Miller, Vikash Gilja
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Transfer Learning

A fundamental challenge in designing brain-computer interfaces (BCIs) is decoding behavior from time-varying neural oscillations. In typical applications, decoders are constructed for individual subjects and with limited data leading to restrictions on the types of models that can be utilized. Currently, the best performing decoders are typically linear models capable of utilizing rigid timing constraints with limited training data. Here we demonstrate the use of Long Short-Term Memory (LSTM) networks to take advantage of the temporal information present in sequential neural data collected from subjects implanted with electrocorticographic (ECoG) electrode arrays performing a finger flexion task. Our constructed models are capable of achieving accuracies that are comparable to existing techniques while also being robust to variation in sample data size. Moreover, we utilize the LSTM networks and an affine transformation layer to construct a novel architecture for transfer learning. We demonstrate that in scenarios where only the affine transform is learned for a new subject, it is possible to achieve results comparable to existing state-of-the-art techniques. The notable advantage is the increased stability of the model during training on novel subjects. Relaxing the constraint of only training the affine transformation, we establish our model as capable of exceeding performance of current models across all training data sizes. Overall, this work demonstrates that LSTMs are a versatile model that can accurately capture temporal patterns in neural data and can provide a foundation for transfer learning in neural decoding.
Character Level Based Detection of DGA Domain Names
Authors: Bin Yu, Jie Pan, Jiaming Hu, Anderson Nascimento, Martine De Cock
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: General Classification, text-classification, Text Classification

Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classification. Little studies are available that compare the effectiveness of these approaches for character based text classification with each other. In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm). Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overfitting.
Relational Multi-Instance Learning for Concept Annotation from Medical Time Series
Authors: Sanjay Purushotham, Zhengping Che, Bo Jiang, Tanachat Nilanon, Yan Liu
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Time Series, Time Series Analysis

Recent advances in computing technology and sensor design have made it easier to collect longitudinal or time series data from patients, resulting in a gigantic amount of available medical data. Most of the medical time series lack annotations or even when the annotations are available they could be subjective and prone to human errors. Earlier works have developed natural language processing techniques to extract concept annotations and/or clinical narratives from doctor notes. However, these approaches are slow and do not use the accompanying medical time series data. To address this issue, we introduce the problem of concept annotation for the medical time series data, i.e., the task of predicting and localizing medical concepts by using the time series data as input. We propose Relational Multi-Instance Learning (RMIL) - a deep Multi Instance Learning framework based on recurrent neural networks, which uses pooling functions and attention mechanisms for the concept annotation tasks. Empirical results on medical datasets show that our proposed models outperform various multi-instance learning models.
VOCABULARY-INFORMED VISUAL FEATURE AUGMENTATION FOR ONE-SHOT LEARNING
Authors: jianqi ma, Hangyu Lin, yinda zhang, Yanwei Fu, xiangyang xue
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Classification, Data Augmentation, Decoder, General Classification, One-Shot Learning

A natural solution for one-shot learning is to augment training data to handle the data deficiency problem. However, directly augmenting in the image domain may not necessarily generate training data that sufficiently explore the intra-class space for one-shot classification. Inspired by the recent vocabulary-informed learning, we propose to generate synthetic training data with the guide of the semantic word space. Essentially, we train an auto-encoder as a bridge to enable the transformation between the image feature space and the semantic space. Besides directly augmenting image features, we transform the image features to semantic space using the encoder and perform the data augmentation. The decoder then synthesizes the image features for the augmented instances from the semantic space. Experiments on three datasets show that our data augmentation method effectively improves the performance of one-shot classification. An extensive study shows that data augmented from semantic space are complementary with those from the image space, and thus boost the classification accuracy dramatically. Source code and dataset will be available.
Understanding Local Minima in Neural Networks by Loss Surface Decomposition
Authors: Hanock Kwak, Byoung-Tak Zhang
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

To provide principled ways of designing proper Deep Neural Network (DNN) models, it is essential to understand the loss surface of DNNs under realistic assumptions. We introduce interesting aspects for understanding the local minima and overall structure of the loss surface. The parameter domain of the loss surface can be decomposed into regions in which activation values (zero or one for rectified linear units) are consistent. We found that, in each region, the loss surface have properties similar to that of linear neural networks where every local minimum is a global minimum. This means that every differentiable local minimum is the global minimum of the corresponding region. We prove that for a neural network with one hidden layer using rectified linear units under realistic assumptions. There are poor regions that lead to poor local minima, and we explain why such regions exist even in the overparameterized DNNs.
Viterbi-based Pruning for Sparse Matrix with Fixed and High Index Compression Ratio
Authors: Dongsoo Lee, Daehyun Ahn, Taesu Kim, Pierce I. Chuang, Jae-Joon Kim
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Weight pruning has proven to be an effective method in reducing the model size and computation cost while not sacrificing the model accuracy. Conventional sparse matrix formats, however, involve irregular index structures with large storage requirement and sequential reconstruction process, resulting in inefficient use of highly parallel computing resources. Hence, pruning is usually restricted to inference with a batch size of one, for which an efficient parallel matrix-vector multiplication method exists. In this paper, a new class of sparse matrix representation utilizing Viterbi algorithm that has a high, and more importantly, fixed index compression ratio regardless of the pruning rate, is proposed. In this approach, numerous sparse matrix candidates are first generated by the Viterbi encoder, and then the one that aims to minimize the model accuracy degradation is selected by the Viterbi algorithm. The model pruning process based on the proposed Viterbi encoder and Viterbi algorithm is highly parallelizable, and can be implemented efficiently in hardware to achieve low-energy, high-performance index decoding process. Compared with the existing magnitude-based pruning methods, index data storage requirement can be further compressed by 85.2% in MNIST and 83.9% in AlexNet while achieving similar pruning rate. Even compared with the relative index compression technique, our method can still reduce the index storage requirement by 52.7% in MNIST and 35.5% in AlexNet.