pwc-archive/papers-with-abstracts huggingface dataset:
Learning Representations of Categorical Feature Combinations via Self-Attention
Authors: Chen Xu, Chengzhen Fu, Peng Jiang, Wenwu Ou
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Click-Through Rate Prediction

Self-attention has been widely used to model the sequential data and achieved remarkable results in many applications. Although it can be used to model dependencies without regard to positions of sequences, self-attention is seldom applied to non-sequential data. In this work, we propose to learn representations of multi-field categorical data in prediction tasks via self-attention mechanism, where features are orderless but have intrinsic relations over different fields. In most current DNN based models, feature embeddings are simply concatenated for further processing by networks. Instead, by applying self-attention to transform the embeddings, we are able to relate features in different fields and automatically learn representations of their combinations, which are known as the factors of many prevailing linear models. To further improve the effect of feature combination mining, we modify the original self-attention structure by restricting the similarity weight to have at most k non-zero values, which additionally regularizes the model. We experimentally evaluate the effectiveness of our self-attention model on non-sequential data. Across two click through rate prediction benchmark datasets, i.e., Cretio and Avazu, our model with top-k restricted self-attention achieves the state-of-the-art performance. Compared with the vanilla MLP, the gain by adding self-attention is significantly larger than that by modifying the network structures, which most current works focus on.
Total Style Transfer with a Single Feed-Forward Network
Authors: Minseong Kim, Hyun-Chul Choi
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Decoder, Style Transfer

Recent image style transferring methods achieved arbitrary stylization with input content and style images. To transfer the style of an arbitrary image to a content image, these methods used a feed-forward network with a lowest-scaled feature transformer or a cascade of the networks with a feature transformer of a corresponding scale. However, their approaches did not consider either multi-scaled style in their single-scale feature transformer or dependency between the transformed feature statistics across the cascade networks. This shortcoming resulted in generating partially and inexactly transferred style in the generated images.
To overcome this limitation of partial style transfer, we propose a total style transferring method which transfers multi-scaled feature statistics through a single feed-forward process. First, our method transforms multi-scaled feature maps of a content image into those of a target style image by considering both inter-channel correlations in each single scaled feature map and inter-scale correlations between multi-scaled feature maps. Second, each transformed feature map is inserted into the decoder layer of the corresponding scale using skip-connection. Finally, the skip-connected multi-scaled feature maps are decoded into a stylized image through our trained decoder network.
Multi-Scale Stacked Hourglass Network for Human Pose Estimation
Authors: Chunsheng Guo, Wenlong Du, Na Ying
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Pose Estimation

Stacked hourglass network has become an important model for Human pose estimation. The estimation of human body posture depends on the global information of the keypoints type and the local information of the keypoints location. The consistent processing of inputs and constraints makes it difficult to form differentiated and determined collaboration mechanisms for each stacked hourglass network. In this paper, we propose a Multi-Scale Stacked Hourglass (MSSH) network to high-light the differentiation capabilities of each Hourglass network for human pose estimation.  The pre-processing network forms feature maps of different scales,and dispatch them to various locations of the stack hourglass network, where the small-scale features reach the front of stacked hourglass network, and large-scale features reach the rear of stacked hourglass network.   And a new loss function is proposed for multi-scale stacked hourglass network.  Different keypoints have different weight coefficients of loss function at different scales, and the keypoints weight coefficients are dynamically adjusted from the top-level hourglass network to the bottom-level hourglass network.  Experimental results show that the pro-posed method is competitive with respect to the comparison algorithm on MPII and LSP datasets.
A quantifiable testing of global translational invariance in Convolutional and Capsule Networks
Authors: Weikai Qi
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Data Augmentation, Translation

 We design simple and quantifiable testing of global translation-invariance in deep learning models trained on the MNIST dataset. Experiments on convolutional and capsules neural networks show that both models have poor performance in dealing with global translation-invariance; however, the performance improved by using data augmentation. Although the capsule network is better on the MNIST testing dataset, the convolutional neural network generally has better performance on the translation-invariance.
Multiple Encoder-Decoders Net for Lane Detection
Authors: Yuetong Du, Xiaodong Gu, Junqin Liu, Liwen He
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Decoder, Image Segmentation, Lane Detection, Semantic Segmentation

For semantic image segmentation and lane detection, nets with a single spatial pyramid structure or encoder-decoder structure are usually exploited. Convolutional neural networks (CNNs) show great results on both high-level and low-level features representations, however, the capability has not been fully embodied for lane detection task. In especial, it's still a challenge for model-based lane detection to combine the multi-scale context with a pixel-level accuracy because of the weak visual appearance and strong prior information. In this paper, we we propose an novel network for lane detection, the three main contributions are as follows. First, we employ multiple encoder-decoders module in end-to-end ways and show the promising results for lane detection. Second, we analysis different configurations of multiple encoder-decoders nets. Third, we make our attempts to rethink the evaluation methods of lane detection for the limitation of the popular methods based on IoU.
Generative model based on minimizing exact empirical Wasserstein distance
Authors: Akihiro Iohara, Takahito Ogawa, Toshiyuki Tanaka
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Generative Adversarial Networks (GANs) are a very powerful framework for generative modeling. However, they are often hard to train, and learning of GANs often becomes unstable. Wasserstein GAN (WGAN) is a promising framework to deal with the instability problem as it has a good convergence property. One drawback of the WGAN is that it evaluates the Wasserstein distance in the dual domain, which requires some approximation, so that it may fail to optimize the true Wasserstein distance. In this paper, we propose evaluating the exact empirical optimal transport cost efficiently in the primal domain and performing gradient descent with respect to its derivative to train the generator network. Experiments on the MNIST dataset show that our method is significantly stable to converge, and achieves the lowest Wasserstein distance among the WGAN variants at the cost of some sharpness of generated images. Experiments on the 8-Gaussian toy dataset show that better gradients for the generator are obtained in our method. In addition, the proposed method enables more flexible generative modeling than WGAN.
Deli-Fisher GAN: Stable and Efficient Image Generation With Structured Latent Generative Space
Authors: Boli Fang, Chuck Jia, Miao Jiang, Dhawal Chaturvedi
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Image Generation

Generative Adversarial Networks (GANs) are powerful tools for realistic image generation. However, a major drawback of GANs is that they are especially hard to train, often requiring large amounts of data and long training time. In this paper we propose the Deli-Fisher GAN, a GAN that generates photo-realistic images by enforcing structure on the latent generative space using similar approaches in \cite{deligan}. The structure of the latent space we consider in this paper is modeled as a mixture of Gaussians, whose parameters are learned in the training process. Furthermore, to improve stability and efficiency, we use the Fisher Integral Probability Metric as the divergence measure in our GAN model, instead of the Jensen-Shannon divergence. We show by experiments that the Deli-Fisher GAN performs better than DCGAN, WGAN, and the Fisher GAN as measured by inception score.
Stacking for Transfer Learning
Authors: Peng Yuankai
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Domain Adaptation, Ensemble Learning, Transfer Learning

In machine learning tasks, overtting frequently crops up when the number of samples of target domain is insufﬁcient, for the generalization ability of the classiﬁer is poor in this circumstance. To solve this problem, transfer learning utilizes the knowledge of similar domains to improve the robustness of the learner. The main idea of existing transfer learning algorithms is to reduce the dierence between domains by sample selection or domain adaptation. However, no matter what transfer learning algorithm we use, the difference always exists and the hybrid training of source and target data leads to reducing ﬁtting capability of the learner on target domain. Moreover, when the relatedness between domains is too low, negative transfer is more likely to occur. To tackle the problem, we proposed a two-phase transfer learning architecture based on ensemble learning, which uses the existing transfer learning algorithms to train the weak learners in the ﬁrst stage, and uses the predictions of target data to train the ﬁnal learner in the second stage. Under this architecture, the ﬁtting capability and generalization capability can be guaranteed at the same time. We evaluated the proposed method on public datasets, which demonstrates the effectiveness and robustness of our proposed method.
Graph Spectral Regularization For Neural Network Interpretability
Authors: Alexander Tong, David van Dijk, Jay Stanley, Guy Wolf, Smita Krishnaswamy
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Deep neural networks can learn meaningful representations of data. However, these representations are hard to interpret. For example, visualizing a latent layer is generally only possible for at most three dimensions. Neural networks are able to learn and benefit from much higher dimensional representations but these are not visually interpretable because nodes have arbitrary ordering within a layer. Here, we utilize the ability of the human observer to identify patterns in structured representations to visualize higher dimensions. To do so, we propose a class of regularizations we call \textit{Graph Spectral Regularizations} that impose graph-structure on latent layers. This is achieved by treating activations as signals on a predefined graph and constraining those activations using graph filters, such as low pass and wavelet-like filters. This framework allows for any kind of graph as well as filter to achieve a wide range of structured regularizations depending on the inference needs of the data. First, we show a synthetic example that the graph-structured layer can reveal topological features of the data. Next, we show that a smoothing regularization can impose semantically consistent ordering of nodes when applied to capsule nets. Further, we show that the graph-structured layer, using wavelet-like spatially localized filters, can form localized receptive fields for improved image and biomedical data interpretation. In other words, the mapping between latent layer, neurons and the output space becomes clear due to the localization of the activations. Finally, we show that when structured as a grid, the representations create coherent images that allow for image-processing techniques such as convolutions.
RETHINKING SELF-DRIVING : MULTI -TASK KNOWLEDGE FOR BETTER GENERALIZATION AND ACCIDENT EXPLANATION ABILITY
Authors: Zhihao LI, Toshiyuki MOTOYOSHI, Kazuma Sasaki, Tetsuya OGATA, Shigeki SUGANO
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Diversity

Current end-to-end deep learning driving models have two problems: (1) Poor
generalization ability of unobserved driving environment when diversity of train-
ing driving dataset is limited (2) Lack of accident explanation ability when driving
models don’t work as expected. To tackle these two problems, rooted on the be-
lieve that knowledge of associated easy task is benificial for addressing difficult
task, we proposed a new driving model which is composed of perception module
for see and think and driving module for behave, and trained it with multi-task
perception-related basic knowledge and driving knowledge stepwisely.  Specifi-
cally segmentation map and depth map (pixel level understanding of images) were
considered as what & where and how far knowledge for tackling easier driving-
related perception problems before generating final control commands for difficult
driving task. The results of experiments demonstrated the effectiveness of multi-
task perception knowledge for better generalization and accident explanation abil-
ity. With our method the average sucess rate of finishing most difficult navigation
tasks in untrained city of CoRL test surpassed current benchmark method for 15
percent in trained weather and 20 percent in untrained weathers.
Normalization Gradients are Least-squares Residuals
Authors: Yi Liu
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks. Discussions of why this normalization works so well remain unsettled.  We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN. We recast the back-propagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations. This view, which we term {\em gradient-least-squares}, is an extensible and arithmetically accurate description of BN. To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN.
Task-GAN for Improved GAN based Image Restoration
Authors: Jiahong Ouyang, Guanhua Wang, Enhao Gong, Kevin Chen, John Pauly and Greg Zaharchuk
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Diagnostic, Generative Adversarial Network, Hallucination, Image Enhancement, Image Restoration, Medical Image Enhancement

Deep Learning (DL) algorithms based on Generative Adversarial Network (GAN) have demonstrated great potentials in computer vision tasks such as image restoration. Despite the rapid development of image restoration algorithms using DL and GANs, image restoration for specific scenarios, such as medical image enhancement and super-resolved identity recognition, are still facing challenges. How to ensure visually realistic restoration while avoiding hallucination or mode- collapse? How to make sure the visually plausible results do not contain hallucinated features jeopardizing downstream tasks such as pathology identification and subject identification?
Here we propose to resolve these challenges by coupling the GAN based image restoration framework with another task-specific network. With medical imaging restoration as an example, the proposed model conducts additional pathology recognition/classification task to ensure the preservation of detailed structures that are important to this task. Validated on multiple medical datasets, we demonstrate the proposed method leads to improved deep learning based image restoration while preserving the detailed structure and diagnostic features. Additionally, the trained task network show potentials to achieve super-human level performance in identifying pathology and diagnosis.
Further validation on super-resolved identity recognition tasks also show that the proposed method can be generalized for diverse image restoration tasks.
Characterizing the Accuracy/Complexity Landscape of Explanations of Deep Networks through Knowledge Extraction
Authors: Simon Odense, Artur d'Avila Garcez
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Knowledge extraction techniques are used to convert neural networks into symbolic descriptions with the objective of producing more comprehensible learning models. The central challenge is to find an explanation which is more comprehensible than the original model while still representing that model faithfully. The distributed nature of deep networks has led many to believe that the hidden features of a neural network cannot be explained by logical descriptions simple enough to be understood by humans, and that decompositional knowledge extraction should be abandoned in favour of other methods. In this paper we examine this question systematically by proposing a knowledge extraction method using \textit{M-of-N} rules which allows us to map the complexity/accuracy landscape of rules describing hidden features in a Convolutional Neural Network (CNN). Experiments reported in this paper show that the shape of this landscape reveals an optimal trade off between comprehensibility and accuracy, showing that each latent variable has an optimal \textit{M-of-N} rule to describe its behaviour. We find that the rules with optimal tradeoff in the first and final layer have a high degree of explainability whereas the rules with the optimal tradeoff in the second and third layer are less explainable. The results shed light on the feasibility of rule extraction from deep networks, and point to the value of decompositional knowledge extraction as a method of explainability.
A NON-LINEAR THEORY FOR SENTENCE EMBEDDING
Authors: Hichem Mezaoui, Isar Nejadgholi
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Sentence, Sentence Embedding, Sentence-Embedding

This paper revisits the Random Walk model for sentence embedding in the context of non-extensive statistics. We propose a non-extensive algebra to compute the discourse vector. We argue that by doing so we are taking into account high non-linearity in the semantic space. Furthermore, we show that by considering a non-extensive algebra, the compounding effect of the vector length is mitigated. Overall, we show that the proposed model leads to good sentence embedding. We evaluate the embedding method on textual similarity tasks.
Neural Variational Inference For Embedding Knowledge Graphs
Authors: Alexander I. Cowen-Rivers, Pasquale Minervini
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Knowledge Graphs, Variational Inference

Recent advances in Neural Variational Inference allowed for a renaissance in latent variable models in a variety of domains involving high-dimensional data. In this paper, we introduce two generic Variational Inference frameworks for generative models of Knowledge Graphs; Latent Fact Model and Latent Information Model.  While traditional variational methods derive an analytical approximation for the intractable distribution over the latent variables, here we construct an inference network conditioned on the symbolic representation of entities and relation types in the Knowledge Graph, to provide the variational distributions. The new framework can create models able to discover underlying probabilistic semantics for the symbolic representation by utilising parameterisable distributions which permit training by back-propagation in the context of neural variational inference, resulting in a highly-scalable method. Under a Bernoulli sampling framework, we provide an alternative justification for commonly used techniques in large-scale stochastic variational inference, which drastically reduces training time at a cost of an additional approximation to the variational lower bound.  The generative frameworks are flexible enough to allow training under any prior distribution that permits a re-parametrisation trick, as well as under any scoring function that permits maximum likelihood estimation of the parameters. Experiment results display the potential and efficiency of this framework by improving upon multiple benchmarks with Gaussian prior representations. Code publicly available on Github.
The GAN Landscape: Losses, Architectures, Regularization, and Normalization
Authors: Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, Sylvain Gelly
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of ``tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.
Amortized Bayesian Meta-Learning
Authors: Sachin Ravi, Alex Beatson
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Few-Shot Image Classification, Few-Shot Learning, Meta-Learning, Reinforcement Learning, Variational Inference

Meta-learning, or learning-to-learn, has proven to be a successful strategy in attacking problems in supervised learning and reinforcement learning that involve small amounts of data. State-of-the-art solutions involve learning an initialization and/or learning algorithm using a set of training episodes so that the meta learner can generalize to an evaluation episode quickly. These methods perform well but often lack good quantification of uncertainty, which can be vital to real-world applications when data is lacking. We propose a meta-learning method which efficiently amortizes hierarchical variational inference across tasks, learning a prior distribution over neural network weights so that a few steps of Bayes by Backprop will produce a good task-specific approximate posterior. We show that our method produces good uncertainty estimates on contextual bandit and few-shot learning benchmarks.