pwc-archive/papers-with-abstracts huggingface dataset:
Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions
Authors: Anna Sergeevna Bosman, Andries Engelbrecht, Mardé Helbig
Date: 2019-01-08 00:00:00

Quantification of the stationary points and the associated basins of
attraction of neural network loss surfaces is an important step towards a
better understanding of neural network loss surfaces at large. This work
proposes a novel method to visualise basins of attraction together with the
associated stationary points via gradient-based random sampling. The proposed
technique is used to perform an empirical study of the loss surfaces generated
by two different error metrics: quadratic loss and entropic loss. The empirical
observations confirm the theoretical hypothesis regarding the nature of neural
network attraction basins. Entropic loss is shown to exhibit stronger gradients
and fewer stationary points than quadratic loss, indicating that entropic loss
has a more searchable landscape. Quadratic loss is shown to be more resilient
to overfitting than entropic loss. Both losses are shown to exhibit local
minima, but the number of local minima is shown to decrease with an increase in
dimensionality. Thus, the proposed visualisation technique successfully
captures the local minima properties exhibited by the neural network loss
surfaces, and can be used for the purpose of fitness landscape analysis of
neural networks.
Spectral Clustering via Ensemble Deep Autoencoder Learning (SC-EDAE)
Authors: Severine Affeldt, Lazhar Labiod, Mohamed Nadif
Date: 2019-01-08 00:00:00
Tasks: Clustering, Deep Clustering, Ensemble Learning

Recently, a number of works have studied clustering strategies that combine classical clustering algorithms and deep learning methods. These approaches follow either a sequential way, where a deep representation is learned using a deep autoencoder before obtaining clusters with k-means, or a simultaneous way, where deep representation and clusters are learned jointly by optimizing a single objective function. Both strategies improve clustering performance, however the robustness of these approaches is impeded by several deep autoencoder setting issues, among which the weights initialization, the width and number of layers or the number of epochs. To alleviate the impact of such hyperparameters setting on the clustering performance, we propose a new model which combines the spectral clustering and deep autoencoder strengths in an ensemble learning framework. Extensive experiments on various benchmark datasets demonstrate the potential and robustness of our approach compared to state-of-the-art deep clustering methods.
Online learning using multiple times weight updating
Authors: Charanjeet, Anuj Sharma
Date: 2018-10-26 00:00:00

Online learning makes sequence of decisions with partial data arrival where
next movement of data is unknown. In this paper, we have presented a new
technique as multiple times weight updating that update the weight iteratively
forsame instance. The proposed technique analyzed with popular state-of-art
algorithms from literature and experimented using established tool. The results
indicates that mistake rate reduces to zero or close to zero for various
datasets and algorithms. The overhead running cost is not too expensive and
achieving mistake rate close to zero further strengthen the proposed technique.
The present work include bound nature of weight updating for single instance
and achieve optimal weight value. This proposed work could be extended to big
datasets problems to reduce mistake rate in online learning environment. Also,
the proposed technique could be helpful to meet real life challenges.
Unpaired Pose Guided Human Image Generation
Authors: Xu Chen, Jie Song, Otmar Hilliges
Date: 2019-01-08 00:00:00
Tasks: Image Generation, Image-to-Image Translation, Translation

This paper studies the task of full generative modelling of realistic images of humans, guided only by coarse sketch of the pose, while providing control over the specific instance or type of outfit worn by the user. This is a difficult problem because input and output domain are very different and direct image-to-image translation becomes infeasible. We propose an end-to-end trainable network under the generative adversarial framework, that provides detailed control over the final appearance while not requiring paired training data and hence allows us to forgo the challenging problem of fitting 3D poses to 2D images. The model allows to generate novel samples conditioned on either an image taken from the target domain or a class label indicating the style of clothing (e.g., t-shirt). We thoroughly evaluate the architecture and the contributions of the individual components experimentally. Finally, we show in a large scale perceptual study that our approach can generate realistic looking images and that participants struggle in detecting fake images versus real samples, especially if faces are blurred.
Cost Sensitive Learning in the Presence of Symmetric Label Noise
Authors: Sandhya Tripathi, N. Hemachandra
Date: 2019-01-08 00:00:00
Tasks: Binary Classification

In binary classification framework, we are interested in making cost sensitive label predictions in the presence of uniform/symmetric label noise. We first observe that $0$-$1$ Bayes classifiers are not (uniform) noise robust in cost sensitive setting. To circumvent this impossibility result, we present two schemes; unlike the existing methods, our schemes do not require noise rate. The first one uses $\alpha$-weighted $\gamma$-uneven margin squared loss function, $l_{\alpha, usq}$, which can handle cost sensitivity arising due to domain requirement (using user given $\alpha$) or class imbalance (by tuning $\gamma$) or both. However, we observe that $l_{\alpha, usq}$ Bayes classifiers are also not cost sensitive and noise robust. We show that regularized ERM of this loss function over the class of linear classifiers yields a cost sensitive uniform noise robust classifier as a solution of a system of linear equations. We also provide a performance bound for this classifier. The second scheme that we propose is a re-sampling based scheme that exploits the special structure of the uniform noise models and uses in-class probability $\eta$ estimates. Our computational experiments on some UCI datasets with class imbalance show that classifiers of our two schemes are on par with the existing methods and in fact better in some cases w.r.t. Accuracy and Arithmetic Mean, without using/tuning noise rate. We also consider other cost sensitive performance measures viz., F measure and Weighted Cost for evaluation. As our re-sampling scheme requires estimates of $\eta$, we provide a detailed comparative study of various $\eta$ estimation methods on synthetic datasets, w.r.t. half a dozen evaluation criterion. Also, we provide understanding on the interpretation of cost parameters $\alpha$ and $\gamma$ using different synthetic data experiments.
Multi-style Generative Reading Comprehension
Authors: Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazutoshi Shinoda, Atsushi Otsuka, Hisako Asano, Junji Tomita
Date: 2019-01-08 00:00:00
Proceeding: ACL 2019 7
Tasks: Abstractive Text Summarization, Question Answering, Reading Comprehension, Text Generation

This study tackles generative reading comprehension (RC), which consists of answering questions based on textual evidence and natural language generation (NLG). We propose a multi-style abstractive summarization model for question answering, called Masque. The proposed model has two key characteristics. First, unlike most studies on RC that have focused on extracting an answer span from the provided passages, our model instead focuses on generating a summary from the question and multiple passages. This serves to cover various answer styles required for real-world applications. Second, whereas previous studies built a specific model for each answer style because of the difficulty of acquiring one general model, our approach learns multi-style answers within a model to improve the NLG capability for all styles involved. This also enables our model to give an answer in the target style. Experiments show that our model achieves state-of-the-art performance on the Q&A task and the Q&A + NLG task of MS MARCO 2.1 and the summary task of NarrativeQA. We observe that the transfer of the style-independent NLG capability to the target style is the key to its success.
Multi-Perspective Fusion Network for Commonsense Reading Comprehension
Authors: Chunhua Liu, Yan Zhao, Qingyi Si, Haiou Zhang, Bohan Li, Dong Yu
Date: 2019-01-08 00:00:00
Tasks: Reading Comprehension

Commonsense Reading Comprehension (CRC) is a significantly challenging task,
aiming at choosing the right answer for the question referring to a narrative
passage, which may require commonsense knowledge inference. Most of the
existing approaches only fuse the interaction information of choice, passage,
and question in a simple combination manner from a \emph{union} perspective,
which lacks the comparison information on a deeper level. Instead, we propose a
Multi-Perspective Fusion Network (MPFN), extending the single fusion method
with multiple perspectives by introducing the \emph{difference} and
\emph{similarity} fusion\deleted{along with the \emph{union}}. More
comprehensive and accurate information can be captured through the three types
of fusion. We design several groups of experiments on MCScript dataset
\cite{Ostermann:LREC18:MCScript} to evaluate the effectiveness of the three
types of fusion respectively. From the experimental results, we can conclude
that the difference fusion is comparable with union fusion, and the similarity
fusion needs to be activated by the union fusion. The experimental result also
shows that our MPFN model achieves the state-of-the-art with an accuracy of
83.52\% on the official test set.
Artificial Intelligence and Machine Learning to Predict and Improve Efficiency in Manufacturing Industry
Authors: Ibtissam El Hassani, Choumicha El Mazgualdi, Tawfik Masrour
Date: 2019-01-08 00:00:00
Tasks: BIG-bench Machine Learning, Deep Learning

The overall equipment effectiveness (OEE) is a performance measurement metric
widely used. Its calculation provides to the managers the possibility to
identify the main losses that reduce the machine effectiveness and then take
the necessary decisions in order to improve the situation. However, this
calculation is done a-posterior which is often too late. In the present
research, we implemented different Machine Learning algorithms namely; Support
vector machine, Optimized Support vector Machine (using Genetic Algorithm),
Random Forest, XGBoost and Deep Learning to predict the estimate OEE value. The
data used to train our models was provided by an automotive cable production
industry. The results show that the Deep Learning and Random Forest are more
accurate and present better performance for the prediction of the overall
equipment effectiveness in our case study.
DEMN: Distilled-Exposition Enhanced Matching Network for Story Comprehension
Authors: Chunhua Liu, Haiou Zhang, Shan Jiang, Dong Yu
Date: 2019-01-08 00:00:00
Proceeding: PACLIC 2018 12
Tasks: Cloze Test

This paper proposes a Distilled-Exposition Enhanced Matching Network (DEMN)
for story-cloze test, which is still a challenging task in story comprehension.
We divide a complete story into three narrative segments: an
\textit{exposition}, a \textit{climax}, and an \textit{ending}. The model
consists of three modules: input module, matching module, and distillation
module. The input module provides semantic representations for the three
segments and then feeds them into the other two modules. The matching module
collects interaction features between the ending and the climax. The
distillation module distills the crucial semantic information in the exposition
and infuses it into the matching module in two different ways. We evaluate our
single and ensemble model on ROCStories Corpus \cite{Mostafazadeh2016ACA},
achieving an accuracy of 80.1\% and 81.2\% on the test set respectively. The
experimental results demonstrate that our DEMN model achieves a
state-of-the-art performance.
CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
Authors: Alexandros Koliousis, Pijika Watcharapichat, Matthias Weidlich, Luo Mai, Paolo Costa, Peter Pietzuch
Date: 2019-01-08 00:00:00
Tasks: Deep Learning, GPU

Deep learning models are trained on servers with many GPUs, and training must
scale with the number of GPUs. Systems such as TensorFlow and Caffe2 train
models with parallel synchronous stochastic gradient descent: they process a
batch of training data at a time, partitioned across GPUs, and average the
resulting partial gradients to obtain an updated global model. To fully utilise
all GPUs, systems must increase the batch size, which hinders statistical
efficiency. Users tune hyper-parameters such as the learning rate to compensate
for this, which is complex and model-specific.
  We describe CROSSBOW, a new single-server multi-GPU system for training deep
learning models that enables users to freely choose their preferred batch size
- however small - while scaling to multiple GPUs. CROSSBOW uses many parallel
model replicas and avoids reduced statistical efficiency through a new
synchronous training method. We introduce SMA, a synchronous variant of model
averaging in which replicas independently explore the solution space with
gradient descent, but adjust their search synchronously based on the trajectory
of a globally-consistent average model. CROSSBOW achieves high hardware
efficiency with small batch sizes by potentially training multiple model
replicas per GPU, automatically tuning the number of replicas to maximise
throughput. Our experiments show that CROSSBOW improves the training time of
deep learning models on an 8-GPU server by 1.3-4x compared to TensorFlow.
3D Object Detection Using Scale Invariant and Feature Reweighting Networks
Authors: Xin Zhao, Zhe Liu, Ruolan Hu, Kaiqi Huang
Date: 2019-01-08 00:00:00
Tasks: 3D Object Detection, object-detection, Object Detection

3D object detection plays an important role in a large number of real-world
applications. It requires us to estimate the localizations and the orientations
of 3D objects in real scenes. In this paper, we present a new network
architecture which focuses on utilizing the front view images and frustum point
clouds to generate 3D detection results. On the one hand, a PointSIFT module is
utilized to improve the performance of 3D segmentation. It can capture the
information from different orientations in space and the robustness to
different scale shapes. On the other hand, our network obtains the useful
features and suppresses the features with less information by a SENet module.
This module reweights channel features and estimates the 3D bounding boxes more
effectively. Our method is evaluated on both KITTI dataset for outdoor scenes
and SUN-RGBD dataset for indoor scenes. The experimental results illustrate
that our method achieves better performance than the state-of-the-art methods
especially when point clouds are highly sparse.
Soft-Bayes: Prod for Mixtures of Experts with Log-Loss
Authors: Laurent Orseau, Tor Lattimore, Shane Legg
Date: 2019-01-08 00:00:00

We consider prediction with expert advice under the log-loss with the goal of
deriving efficient and robust algorithms. We argue that existing algorithms
such as exponentiated gradient, online gradient descent and online Newton step
do not adequately satisfy both requirements. Our main contribution is an
analysis of the Prod algorithm that is robust to any data sequence and runs in
linear time relative to the number of experts in each round. Despite the
unbounded nature of the log-loss, we derive a bound that is independent of the
largest loss and of the largest gradient, and depends only on the number of
experts and the time horizon. Furthermore we give a Bayesian interpretation of
Prod and adapt the algorithm to derive a tracking regret.
Interpretable BoW Networks for Adversarial Example Detection
Authors: Krishna Kanth Nakka, Mathieu Salzmann
Date: 2019-01-08 00:00:00
Tasks: Generative Adversarial Network

The standard approach to providing interpretability to deep convolutional
neural networks (CNNs) consists of visualizing either their feature maps, or
the image regions that contribute the most to the prediction. In this paper, we
introduce an alternative strategy to interpret the results of a CNN. To this
end, we leverage a Bag of visual Word representation within the network and
associate a visual and semantic meaning to the corresponding codebook elements
via the use of a generative adversarial network. The reason behind the
prediction for a new sample can then be interpreted by looking at the visual
representation of the most highly activated codeword. We then propose to
exploit our interpretable BoW networks for adversarial example detection. To
this end, we build upon the intuition that, while adversarial samples look very
similar to real images, to produce incorrect predictions, they should activate
codewords with a significantly different visual representation. We therefore
cast the adversarial example detection problem as that of comparing the input
image with the most highly activated visual codeword. As evidenced by our
experiments, this allows us to outperform the state-of-the-art adversarial
example detection methods on standard benchmarks, independently of the attack
strategy.
Multi-turn Inference Matching Network for Natural Language Inference
Authors: Chunhua Liu, Shan Jiang, Hainan Yu, Dong Yu
Date: 2019-01-08 00:00:00
Tasks: Natural Language Inference

Natural Language Inference (NLI) is a fundamental and challenging task in
Natural Language Processing (NLP). Most existing methods only apply one-pass
inference process on a mixed matching feature, which is a concatenation of
different matching features between a premise and a hypothesis. In this paper,
we propose a new model called Multi-turn Inference Matching Network (MIMN) to
perform multi-turn inference on different matching features. In each turn, the
model focuses on one particular matching feature instead of the mixed matching
feature. To enhance the interaction between different matching features, a
memory component is employed to store the history inference information. The
inference of each turn is performed on the current matching feature and the
memory. We conduct experiments on three different NLI datasets. The
experimental results show that our model outperforms or achieves the
state-of-the-art performance on all the three datasets.
Deep Neural Network Approximation Theory
Authors: Dennis Elbrächter, Dmytro Perekrestenko, Philipp Grohs, Helmut Bölcskei
Date: 2019-01-08 00:00:00
Tasks: Handwritten Digit Recognition, Image Classification, Speech Recognition

This paper develops fundamental limits of deep neural network learning by characterizing what is possible if no constraints are imposed on the learning algorithm and on the amount of training data. Concretely, we consider Kolmogorov-optimal approximation through deep neural networks with the guiding theme being a relation between the complexity of the function (class) to be approximated and the complexity of the approximating network in terms of connectivity and memory requirements for storing the network topology and the associated quantized weights. The theory we develop establishes that deep networks are Kolmogorov-optimal approximants for markedly different function classes, such as unit balls in Besov spaces and modulation spaces. In addition, deep networks provide exponential approximation accuracy - i.e., the approximation error decays exponentially in the number of nonzero weights in the network - of the multiplication operation, polynomials, sinusoidal functions, and certain smooth functions. Moreover, this holds true even for one-dimensional oscillatory textures and the Weierstrass function - a fractal function, neither of which has previously known methods achieving exponential approximation accuracy. We also show that in the approximation of sufficiently smooth functions finite-width deep networks require strictly smaller connectivity than finite-depth wide networks.
Uncertainty-Based Out-of-Distribution Detection in Deep Reinforcement Learning
Authors: Andreas Sedlmeier, Thomas Gabor, Thomy Phan, Lenz Belzner, Claudia Linnhoff-Popien
Date: 2019-01-08 00:00:00
Tasks: Bayesian Inference, Deep Reinforcement Learning, Open-Ended Question Answering, Out-of-Distribution Detection, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Variational Inference

We consider the problem of detecting out-of-distribution (OOD) samples in
deep reinforcement learning. In a value based reinforcement learning setting,
we propose to use uncertainty estimation techniques directly on the agent's
value estimating neural network to detect OOD samples. The focus of our work
lies in analyzing the suitability of approximate Bayesian inference methods and
related ensembling techniques that generate uncertainty estimates. Although
prior work has shown that dropout-based variational inference techniques and
bootstrap-based approaches can be used to model epistemic uncertainty, the
suitability for detecting OOD samples in deep reinforcement learning remains an
open question. Our results show that uncertainty estimation can be used to
differentiate in- from out-of-distribution samples. Over the complete training
process of the reinforcement learning agents, bootstrap-based approaches tend
to produce more reliable epistemic uncertainty estimates, when compared to
dropout-based approaches.
Tree Tensor Networks for Generative Modeling
Authors: Song Cheng, Lei Wang, Tao Xiang, Pan Zhang
Date: 2019-01-08 00:00:00
Tasks: BIG-bench Machine Learning, GPU, Tensor Networks

Matrix product states (MPS), a tensor network designed for one-dimensional
quantum systems, has been recently proposed for generative modeling of natural
data (such as images) in terms of `Born machine'. However, the exponential
decay of correlation in MPS restricts its representation power heavily for
modeling complex data such as natural images. In this work, we push forward the
effort of applying tensor networks to machine learning by employing the Tree
Tensor Network (TTN) which exhibits balanced performance in expressibility and
efficient training and sampling. We design the tree tensor network to utilize
the 2-dimensional prior of the natural images and develop sweeping learning and
sampling algorithms which can be efficiently implemented utilizing Graphical
Processing Units (GPU). We apply our model to random binary patterns and the
binary MNIST datasets of handwritten digits. We show that TTN is superior to
MPS for generative modeling in keeping correlation of pixels in natural images,
as well as giving better log-likelihood scores in standard datasets of
handwritten digits. We also compare its performance with state-of-the-art
generative models such as the Variational AutoEncoders, Restricted Boltzmann
machines, and PixelCNN. Finally, we discuss the future development of Tensor
Network States in machine learning problems.