pwc-archive/papers-with-abstracts huggingface dataset:
Espresso: Efficient Forward Propagation for Binary Deep Neural Networks
Authors: Fabrizio Pedersoli, George Tzanetakis, Andrea Tagliasacchi
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: CPU, GPU

  There are many applications scenarios for which the computational
  performance and memory footprint of the prediction phase of Deep
  Neural Networks (DNNs) need to be optimized. Binary Deep Neural
  Networks (BDNNs) have been shown to be an effective way of achieving
  this objective. In this paper, we show how Convolutional Neural
  Networks (CNNs) can be implemented using binary
  representations. Espresso is a compact, yet powerful
  library written in C/CUDA that features all the functionalities
  required for the forward propagation of CNNs, in a binary file less
  than 400KB, without any external dependencies. Although it is mainly
  designed to take advantage of massive GPU parallelism, Espresso also
  provides an equivalent CPU implementation for CNNs. Espresso
  provides special convolutional and dense layers for BCNNs,
  leveraging bit-packing and bit-wise computations
  for efficient execution. These techniques provide a speed-up of
  matrix-multiplication routines, and at the same time, reduce memory
  usage when storing parameters and activations. We experimentally
  show that Espresso is significantly faster than existing
  implementations of optimized binary neural networks (~ 2
  orders of magnitude). Espresso is released under the Apache 2.0
  license and is available at http://github.com/organization/project.
LatentPoison -- Adversarial Attacks On The Latent Space
Authors: Antonia Creswell, Biswa Sengupta, Anil A. Bharath
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Decoder, General Classification, Reinforcement Learning, Reinforcement Learning (RL)

Robustness and security of machine learning (ML) systems are intertwined, wherein a non-robust ML system (classifiers, regressors, etc.) can be subject to attacks using a wide variety of exploits. With the advent of scalable deep learning methodologies, a lot of emphasis has been put on the robustness of supervised, unsupervised and reinforcement learning algorithms. Here, we study the robustness of the latent space of a deep variational autoencoder (dVAE), an unsupervised generative framework, to show that it is indeed possible to perturb the latent space, flip the class predictions and keep the classification probability approximately equal before and after an attack. This means that an agent that looks at the outputs of a decoder would remain oblivious to an attack.
Understanding Grounded Language Learning Agents
Authors: Felix Hill, Karl Moritz Hermann, Phil Blunsom, Stephen Clark
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Grounded language learning, Policy Gradient Methods

Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and even execute symbolic instructions as first-person actors in partially-observable worlds. To achieve this so-called grounded language learning, models must overcome certain well-studied learning challenges that are also fundamental to infants learning their first words. While it is notable that models with no meaningful prior knowledge overcome these learning obstacles, AI researchers and practitioners currently lack a clear understanding of exactly how they do so. Here we address this question as a way of achieving a clearer general understanding of grounded language learning, both to inform future research and to improve confidence in model predictions. For maximum control and generality, we focus on a simple neural network-based language learning agent trained via policy-gradient methods to interpret synthetic linguistic instructions in a simulated 3D world. We apply experimental paradigms from developmental psychology to this agent, exploring the conditions under which established human biases and learning effects emerge. We further propose a novel way to visualise and analyse semantic representation in grounded language learning agents that yields a plausible computational account of the observed effects.
APPLICATION OF DEEP CONVOLUTIONAL NEURAL NETWORK TO PREVENT ATM FRAUD BY FACIAL DISGUISE IDENTIFICATION
Authors: Suraj Nandkishor Kothawade, Sumit Baburao Tamgale
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Face Identification

The paper proposes and demonstrates a Deep Convolutional Neural Network (DCNN) architecture to identify users with disguised face attempting a fraudulent ATM transaction. The recent introduction of Disguised Face Identification (DFI) framework proves the applicability of deep neural networks for this very problem. All the ATMs nowadays incorporate a hidden camera in them and capture the footage of their users. However, it is impossible for the police to track down the impersonators with disguised faces from the ATM footage. The proposed deep convolutional neural network is trained to identify, in real time, whether the user in the captured image is trying to cloak his identity or not. The output of the DCNN is then reported to the ATM to take appropriate steps and prevent the swindler from completing the transaction. The network is trained using a dataset of images captured in similar situations as of an ATM. The comparatively low background clutter in the images enables the network to demonstrate high accuracy in feature extraction and classification for all the different disguises.
An Ensemble of Retrieval-Based and Generation-Based Human-Computer Conversation Systems.
Authors: Yiping Song, Rui Yan, Cheng-Te Li, Jian-Yun Nie, Ming Zhang, Dongyan Zhao
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Re-Ranking, Retrieval

Human-computer conversation systems have attracted much attention in Natural Language Processing. Conversation systems can be roughly divided into two categories: retrieval-based and generation-based systems. Retrieval systems search a user-issued utterance (namely a query) in a large conversational repository and return a reply that best matches the query. Generative approaches synthesize new replies. Both ways have certain advantages but suffer from their own disadvantages. We propose a novel ensemble of retrieval-based and generation-based conversation system. The retrieved candidates, in addition to the original query, are fed to a reply generator via a neural network, so that the model is aware of more information. The generated reply together with the retrieved ones then participates in a re-ranking process to find the final reply to output. Experimental results show that such an ensemble system outperforms each single module by a large margin.
UNSUPERVISED METRIC LEARNING VIA NONLINEAR FEATURE SPACE TRANSFORMATIONS
Authors: Pin Zhang, Bibo Shi, JundongLiu
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Clustering, Metric Learning

In this paper, we propose a nonlinear unsupervised metric learning framework to boost of the performance of clustering algorithms. Under our framework, nonlinear distance metric learning and manifold embedding are integrated and conducted simultaneously to increase the natural separations among data samples. The metric learning component is implemented through feature space transformations, regulated by a nonlinear deformable model called Coherent Point Drifting (CPD). Driven by CPD, data points can get to a higher level of linear separability, which is subsequently picked up by the manifold embedding component to generate well-separable sample projections for clustering. Experimental results on synthetic and benchmark datasets show the effectiveness of our proposed approach over the state-of-the-art solutions in unsupervised metric learning.
Learning Deep Generative Models With Discrete Latent Variables
Authors: Hengyuan Hu, Ruslan Salakhutdinov
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Density Estimation

There have been numerous recent advancements on learning deep generative models with latent variables thanks to the reparameterization trick that allows to train deep directed models effectively.  However, since reparameterization trick only works on continuous variables, deep generative models with discrete latent variables still remain hard to train and perform considerably worse than their continuous counterparts. In this paper, we attempt to shrink this gap by introducing a new architecture and its learning procedure.  We develop a hybrid generative model with binary latent variables that consists of an undirected graphical model and a deep neural network. We propose an efficient two-stage pretraining and training procedure that is crucial for learning these models. Experiments on binarized digits and images of natural scenes demonstrate that our model achieves close to the state-of-the-art performance in terms of density estimation and is capable of generating coherent images of natural scenes.
A Self-Organizing Memory Network
Authors: Callie Federer, Joel Zylberberg
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Working memory requires information about external stimuli to be represented in the brain even after those stimuli go away. This information is encoded in the activities of neurons, and neural activities change over timescales of tens of milliseconds. Information in working memory, however, is retained for tens of seconds, suggesting the question of how time-varying neural activities maintain stable representations. Prior work shows that, if the neural dynamics are in the `  null space' of the representation - so that changes to neural activity do not affect the downstream read-out of stimulus information - then information can be retained for periods much longer than the time-scale of individual-neuronal activities. The prior work, however, requires precisely constructed synaptic connectivity matrices, without explaining how this would arise in a biological neural network. To identify mechanisms through which biological networks can self-organize to learn  memory function, we derived biologically plausible synaptic plasticity rules that dynamically modify the connectivity matrix to enable information storing. Networks implementing this plasticity rule can successfully learn to form memory representations even if only 10% of the synapses are plastic, they are robust to synaptic noise, and they can represent information about multiple stimuli.
Simple Fast Convolutional Feature Learning
Authors: David Macêdo, Cleber Zanchettin, Teresa Ludermir
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

The quality of the features used in visual recognition is of fundamental importance for the overall system. For a long time, low-level hand-designed feature algorithms as SIFT and HOG have obtained the best results on image recognition. Visual features have recently been extracted from trained convolutional neural networks. Despite the high-quality results, one of the main drawbacks of this approach, when compared with hand-designed features, is the training time required during the learning process. In this paper, we propose a simple and fast way to train supervised convolutional models to feature extraction while still maintaining its high-quality. This methodology is evaluated on different datasets and compared with state-of-the-art approaches.
Enhancing Batch Normalized Convolutional Networks using Displaced Rectifier Linear Units: A Systematic Comparative Study
Authors: David Macêdo, Cleber Zanchettin, Adriano L. I. Oliveira, Teresa Ludermir
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is a virtually mandatory technique to train deep networks currently. We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance. Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.
Deterministic Policy Imitation Gradient Algorithm
Authors: Fumihiro Sasaki, Atsuo Kawaguchi
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: continuous-control, Continuous Control, Imitation Learning

The goal of imitation learning (IL) is to enable a learner to imitate an expert’s behavior given the expert’s demonstrations. Recently, generative adversarial imitation learning (GAIL) has successfully achieved it even on complex continuous control tasks. However, GAIL requires a huge number of interactions with environment during training. We believe that IL algorithm could be more applicable to the real-world environments if the number of interactions could be reduced. To this end, we propose a model free, off-policy IL algorithm for continuous control. The keys of our algorithm are two folds: 1) adopting deterministic policy that allows us to derive a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG), 2) introducing a function which we call state screening function (SSF) to avoid noisy policy updates with states that are not typical of those appeared on the expert’s demonstrations. Experimental results show that our algorithm can achieve the goal of IL with at least tens of times less interactions than GAIL on a variety of continuous control tasks.
Neural Clustering By Predicting And Copying Noise
Authors: Sam Coope, Andrej Zukov-Gregoric, Yoram Bachrach
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Clustering

We propose a neural clustering model that jointly learns both latent features and how they cluster. Unlike similar methods our model does not require a predefined number of clusters. Using a supervised approach, we agglomerate latent features towards randomly sampled targets within the same space whilst progressively removing the targets until we are left with only targets which represent cluster centroids. To show the behavior of our model across different modalities we apply our model on both text and image data and very competitive results on MNIST. Finally, we also provide results against baseline models for fashion-MNIST, the 20 newsgroups dataset, and a Twitter dataset we ourselves create.
Network Iterative Learning for Dynamic Deep Neural Networks via Morphism
Authors: Tao Wei, Changhu Wang, Chang Wen Chen
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

In this research, we present a novel learning scheme called network iterative learning for deep neural networks. Different from traditional optimization algorithms that usually optimize directly on a static objective function, we propose in this work to optimize a dynamic objective function in an iterative fashion capable of adapting its function form when being optimized. The optimization is implemented as a series of intermediate neural net functions that is able to dynamically grow into the targeted neural net objective function. This is done via network morphism so that the network knowledge is fully preserved with each network growth. Experimental results demonstrate that the proposed network iterative learning scheme is able to significantly alleviate the degradation problem. Its effectiveness is verified on diverse benchmark datasets.
Generalized Graph Embedding Models
Authors: Qiao Liu, Xiao-Hui Yang, Rui Wan, Shouzhong Tu, Zufeng Wu
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: General Classification, Graph Embedding, Knowledge Base Completion, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

Many types of relations in physical, biological, social and information systems can be modeled as homogeneous or heterogeneous concept graphs. Hence, learning from and with graph embeddings has drawn a great deal of research interest recently, but only ad hoc solutions have been obtained this far. In this paper, we conjecture that the one-shot supervised learning mechanism is a bottleneck in improving the performance of the graph embedding learning algorithms, and propose to extend this by introducing a multi-shot unsupervised learning framework. Empirical results on several real-world data set show that the proposed model consistently and significantly outperforms existing state-of-the-art approaches on knowledge base completion and graph based multi-label classification tasks.
Link Weight Prediction with Node Embeddings
Authors: Yuchen Hou, Lawrence B. Holder
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Deep Learning, Graph Mining, Prediction, speech-recognition, Speech Recognition

Application of deep learning has been successful in various domains such as im-
age recognition, speech recognition and natural language processing. However,
the research on its application in graph mining is still in an early stage. Here we
present the first generic deep learning approach to the graph link weight prediction
problem based on node embeddings. We evaluate this approach with three differ-
ent node embedding techniques experimentally and compare its performance with
two state-of-the-art non deep learning baseline approaches. Our experiment re-
sults suggest that this deep learning approach outperforms the baselines by up to
70% depending on the dataset and embedding technique applied. This approach
shows that deep learning can be successfully applied to link weight prediction to
improve prediction accuracy.
A Bayesian Perspective on Generalization and Stochastic Gradient Descent
Authors: Samuel L. Smith and Quoc V. Le
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

We consider two questions at the heart of machine learning; how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? Our work responds to \citet{zhang2016understanding}, who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. We show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy. We propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the ``noise scale" $g = \epsilon (\frac{N}{B} - 1) \approx \epsilon N/B$, where $\epsilon$ is the learning rate, $N$ the training set size and $B$ the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, $B_{opt} \propto \epsilon N$. We verify these predictions empirically.
Multiscale Hidden Markov Models For Covariance Prediction
Authors: João Sedoc, Jordan Rodu, Dean Foster, Lyle Ungar
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Prediction

This paper presents a novel variant of hierarchical hidden Markov models (HMMs), the multiscale hidden Markov model (MSHMM), and an associated spectral estimation and prediction scheme that is consistent, finds global optima, and is computationally efficient. Our MSHMM is a generative model of multiple HMMs evolving at different rates where the observation is a result of the additive emissions of the HMMs. While estimation is relatively straightforward, prediction for the MSHMM poses a unique challenge, which we address in this paper.  Further, we show that spectral estimation of the MSHMM outperforms standard  methods of predicting the asset covariance of stock prices, a widely addressed problem that is multiscale, non-stationary, and requires processing huge amounts of data.
Recurrent Auto-Encoder Model for Multidimensional Time Series Representation
Authors: Timothy Wong, Zhiyuan Luo
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Clustering, Decoder, Time Series, Time Series Analysis

Recurrent auto-encoder model can summarise sequential data through an encoder structure into a fixed-length vector and then reconstruct into its original sequential form through the decoder structure. The summarised information can be used to represent time series features. In this paper, we propose relaxing the dimensionality of the decoder output so that it performs partial reconstruction. The fixed-length vector can therefore represent features only in the selected dimensions. In addition, we propose using rolling fixed window approach to generate samples. The change of time series features over time can be summarised as a smooth trajectory path. The fixed-length vectors are further analysed through additional visualisation and unsupervised clustering techniques. 

This proposed method can be applied in large-scale industrial processes for sensors signal analysis purpose where clusters of the vector representations can be used to reflect the operating states of selected aspects of the industrial system.
Sparse-Complementary Convolution for Efficient Model Utilization on CNNs
Authors: Chun-Fu (Richard) Chen, Jinwook Oh, Quanfu Fan, Marco Pistoia, Gwo Giun (Chris) Lee
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

We introduce an efficient way to increase the accuracy of convolution neural networks (CNNs) based on high model utilization without increasing any computational complexity.
The proposed sparse-complementary convolution replaces regular convolution with sparse and complementary shapes of kernels, covering the same receptive field. 
By the nature of deep learning, high model utilization of a CNN can be achieved with more simpler kernels rather than fewer complex kernels.
This simple but insightful model reuses of recent network architectures, ResNet and DenseNet, can provide better accuracy for most classification tasks (CIFAR-10/100 and ImageNet) compared to their baseline models. By simply replacing the convolution of a CNN with our sparse-complementary convolution, at the same FLOPs and parameters, we can improve top-1 accuracy on ImageNet by 0.33% and 0.18% for ResNet-101 and ResNet-152, respectively. A similar accuracy improvement could be gained by increasing the number of layers in those networks by ~1.5x.
Achieving morphological agreement with Concorde
Authors: Daniil Polykovskiy, Dmitry Soloviev
Date: 2018-01-01 00:00:00
Proceeding: ICLR 2018 1

Neural conversational models are widely used in applications like personal assistants and chat bots. These models seem to give better performance when operating on word level. However, for fusion languages like French, Russian and Polish vocabulary size sometimes become infeasible since most of the words have lots of word forms. We propose a neural network architecture for transforming normalized text into a grammatically correct one. Our model efficiently employs correspondence between normalized and target words and significantly outperforms character-level models while being 2x faster in training and 20\% faster at evaluation. We also propose a new pipeline for building conversational models: first generate a normalized answer and then transform it into a grammatically correct one using our network. The proposed pipeline gives better performance than character-level conversational models according to assessor testing.