pwc-archive/papers-with-abstracts huggingface dataset:
Unsupervised Learning of Interpretable Dialog Models
Authors: Dhiraj Madan, Dinesh Raghu, Gaurav Pandey, Sachindra Joshi
Date: 2018-11-02 00:00:00

Recently several deep learning based models have been proposed for end-to-end
learning of dialogs. While these models can be trained from data without the
need for any additional annotations, it is hard to interpret them. On the other
hand, there exist traditional state based dialog systems, where the states of
the dialog are discrete and hence easy to interpret. However these states need
to be handcrafted and annotated in the data. To achieve the best of both
worlds, we propose Latent State Tracking Network (LSTN) using which we learn an
interpretable model in unsupervised manner. The model defines a discrete latent
variable at each turn of the conversation which can take a finite set of
values. Since these discrete variables are not present in the training data, we
use EM algorithm to train our model in unsupervised manner. In the experiments,
we show that LSTN can help achieve interpretability in dialog models without
much decrease in performance compared to end-to-end approaches.
On Evaluating the Generalization of LSTM Models in Formal Languages
Authors: Mirac Suzgun, Yonatan Belinkov, Stuart M. Shieber
Date: 2018-11-02 00:00:00
Proceeding: WS 2019 1
Tasks: Inductive Learning

Recurrent Neural Networks (RNNs) are theoretically Turing-complete and
established themselves as a dominant model for language processing. Yet, there
still remains an uncertainty regarding their language learning capabilities. In
this paper, we empirically evaluate the inductive learning capabilities of Long
Short-Term Memory networks, a popular extension of simple RNNs, to learn simple
formal languages, in particular $a^nb^n$, $a^nb^nc^n$, and $a^nb^nc^nd^n$. We
investigate the influence of various aspects of learning, such as training data
regimes and model capacity, on the generalization to unobserved samples. We
find striking differences in model performances under different training
settings and highlight the need for careful analysis and assessment when making
claims about the learning capabilities of neural network models.
Analysing Dropout and Compounding Errors in Neural Language Models
Authors: James O' Neill, Danushka Bollegala
Date: 2018-11-02 00:00:00
Tasks: Decoder, Language Modelling

This paper carries out an empirical analysis of various dropout techniques
for language modelling, such as Bernoulli dropout, Gaussian dropout, Curriculum
Dropout, Variational Dropout and Concrete Dropout. Moreover, we propose an
extension of variational dropout to concrete dropout and curriculum dropout
with varying schedules. We find these extensions to perform well when compared
to standard dropout approaches, particularly variational curriculum dropout
with a linear schedule. Largest performance increases are made when applying
dropout on the decoder layer. Lastly, we analyze where most of the errors occur
at test time as a post-analysis step to determine if the well-known problem of
compounding errors is apparent and to what end do the proposed methods mitigate
this issue for each dataset. We report results on a 2-hidden layer LSTM, GRU
and Highway network with embedding dropout, dropout on the gated hidden layers
and the output projection layer for each model. We report our results on
Penn-TreeBank and WikiText-2 word-level language modelling datasets, where the
former reduces the long-tail distribution through preprocessing and one which
preserves rare words in the training and test set.
Invertible Residual Networks
Authors: Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, Jörn-Henrik Jacobsen
Date: 2018-11-02 00:00:00
Tasks: Density Estimation, General Classification, Image Generation

We show that standard ResNet architectures can be made invertible, allowing the same model to be used for classification, density estimation, and generation. Typically, enforcing invertibility requires partitioning dimensions or restricting network architectures. In contrast, our approach only requires adding a simple normalization step during training, already available in standard frameworks. Invertible ResNets define a generative model which can be trained by maximum likelihood on unlabeled data. To compute likelihoods, we introduce a tractable approximation to the Jacobian log-determinant of a residual block. Our empirical evaluation shows that invertible ResNets perform competitively with both state-of-the-art image classifiers and flow-based generative models, something that has not been previously achieved with a single architecture.
CMI: An Online Multi-objective Genetic Autoscaler for Scientific and Engineering Workflows in Cloud Infrastructures with Unreliable Virtual Machines
Authors: David A. Monge, Elina Pacini, Cristian Mateos, Enrique Alba, Carlos García Garino
Date: 2018-11-02 00:00:00
Tasks: Cloud Computing

Cloud Computing is becoming the leading paradigm for executing scientific and
engineering workflows. The large-scale nature of the experiments they model and
their variable workloads make clouds the ideal execution environment due to
prompt and elastic access to huge amounts of computing resources. Autoscalers
are middleware-level software components that allow scaling up and down the
computing platform by acquiring or terminating virtual machines (VM) at the
time that workflow's tasks are being scheduled. In this work we propose a novel
online multi-objective autoscaler for workflows denominated Cloud
Multi-objective Intelligence (CMI), that aims at the minimization of makespan,
monetary cost and the potential impact of errors derived from unreliable VMs.
In addition, this problem is subject to monetary budget constraints. CMI is
responsible for periodically solving the autoscaling problems encountered along
the execution of a workflow. Simulation experiments on four well-known
workflows exhibit that CMI significantly outperforms a state-of-the-art
autoscaler of similar characteristics called Spot Instances Aware Autoscaling
(SIAA). These results convey a solid base for deepening in the study of other
meta-heuristic methods for autoscaling workflow applications using cheap but
unreliable infrastructures.
Anomaly Detection for imbalanced datasets with Deep Generative Models
Authors: Nazly Rocio Santos Buitrago, Loek Tonnaer, Vlado Menkovski, Dimitrios Mavroeidis
Date: 2018-11-02 00:00:00
Tasks: Anomaly Detection, Medical Image Analysis

Many important data analysis applications present with severely imbalanced
datasets with respect to the target variable. A typical example is medical
image analysis, where positive samples are scarce, while performance is
commonly estimated against the correct detection of these positive examples. We
approach this challenge by formulating the problem as anomaly detection with
generative models. We train a generative model without supervision on the
`negative' (common) datapoints and use this model to estimate the likelihood of
unseen data. A successful model allows us to detect the `positive' case as low
likelihood datapoints.
  In this position paper, we present the use of state-of-the-art deep
generative models (GAN and VAE) for the estimation of a likelihood of the data.
Our results show that on the one hand both GANs and VAEs are able to separate
the `positive' and `negative' samples in the MNIST case. On the other hand, for
the NLST case, neither GANs nor VAEs were able to capture the complexity of the
data and discriminate anomalies at the level that this task requires. These
results show that even though there are a number of successes presented in the
literature for using generative models in similar applications, there remain
further challenges for broad successful implementation.
The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
Authors: Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, Vittorio Ferrari
Date: 2018-11-02 00:00:00
Tasks: General Classification, image-classification, Image Classification, Object, object-detection, Object Detection, Relationship Detection, Visual Relationship Detection

We present Open Images V4, a dataset of 9.2M images with unified annotations for image classification, object detection and visual relationship detection. The images have a Creative Commons Attribution license that allows to share and adapt the material, and they have been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias. Open Images V4 offers large scale across several dimensions: 30.1M image-level labels for 19.8k concepts, 15.4M bounding boxes for 600 object classes, and 375k visual relationship annotations involving 57 classes. For object detection in particular, we provide 15x more bounding boxes than the next largest datasets (15.4M boxes on 1.9M images). The images often show complex scenes with several objects (8 annotated objects per image on average). We annotated visual relationships between them, which support visual relationship detection, an emerging task that requires structured reasoning. We provide in-depth comprehensive statistics about the dataset, we validate the quality of the annotations, we study how the performance of several modern models evolves with increasing amounts of training data, and we demonstrate two applications made possible by having unified annotations of multiple types coexisting in the same images. We hope that the scale, quality, and variety of Open Images V4 will foster further research and innovation even beyond the areas of image classification, object detection, and visual relationship detection.
Proximal Gradient Method for Nonsmooth Optimization over the Stiefel Manifold
Authors: Shixiang Chen, Shiqian Ma, Anthony Man-Cho So, Tong Zhang
Date: 2018-11-02 00:00:00

We consider optimization problems over the Stiefel manifold whose objective function is the summation of a smooth function and a nonsmooth function. Existing methods for solving this kind of problems can be classified into three classes. Algorithms in the first class rely on information of the subgradients of the objective function and thus tend to converge slowly in practice. Algorithms in the second class are proximal point algorithms, which involve subproblems that can be as difficult as the original problem. Algorithms in the third class are based on operator-splitting techniques, but they usually lack rigorous convergence guarantees. In this paper, we propose a retraction-based proximal gradient method for solving this class of problems. We prove that the proposed method globally converges to a stationary point. Iteration complexity for obtaining an $\epsilon$-stationary solution is also analyzed. Numerical results on solving sparse PCA and compressed modes problems are reported to demonstrate the advantages of the proposed method.
Neural Likelihoods via Cumulative Distribution Functions
Authors: Pawel Chilinski, Ricardo Silva
Date: 2018-11-02 00:00:00
Tasks: Density Estimation

We leverage neural networks as universal approximators of monotonic functions to build a parameterization of conditional cumulative distribution functions (CDFs). By the application of automatic differentiation with respect to response variables and then to parameters of this CDF representation, we are able to build black box CDF and density estimators. A suite of families is introduced as alternative constructions for the multivariate case. At one extreme, the simplest construction is a competitive density estimator against state-of-the-art deep learning methods, although it does not provide an easily computable representation of multivariate CDFs. At the other extreme, we have a flexible construction from which multivariate CDF evaluations and marginalizations can be obtained by a simple forward pass in a deep neural net, but where the computation of the likelihood scales exponentially with dimensionality. Alternatives in between the extremes are discussed. We evaluate the different representations empirically on a variety of tasks involving tail area probabilities, tail dependence and (partial) density estimation.
Clustering and Learning from Imbalanced Data
Authors: Naman D. Singh, Abhinav Dhall
Date: 2018-11-02 00:00:00
Tasks: Clustering

A learning classifier must outperform a trivial solution, in case of
imbalanced data, this condition usually does not hold true. To overcome this
problem, we propose a novel data level resampling method - Clustering Based
Oversampling for improved learning from class imbalanced datasets. The
essential idea behind the proposed method is to use the distance between a
minority class sample and its respective cluster centroid to infer the number
of new sample points to be generated for that minority class sample. The
proposed algorithm has very less dependence on the technique used for finding
cluster centroids and does not effect the majority class learning in any way.
It also improves learning from imbalanced data by incorporating the
distribution structure of minority class samples in generation of new data
samples. The newly generated minority class data is handled in a way as to
prevent outlier production and overfitting. Implementation analysis on
different datasets using deep neural networks as the learning classifier shows
the effectiveness of this method as compared to other synthetic data resampling
techniques across several evaluation metrics.
One-Bit OFDM Receivers via Deep Learning
Authors: Eren Balevi, Jeffrey G. Andrews
Date: 2018-11-02 00:00:00
Tasks: Decoder, Deep Learning, Quantization

This paper develops novel deep learning-based architectures and design methodologies for an orthogonal frequency division multiplexing (OFDM) receiver under the constraint of one-bit complex quantization. Single bit quantization greatly reduces complexity and power consumption, but makes accurate channel estimation and data detection difficult. This is particularly true for multicarrier waveforms, which have high peak-to-average ratio in the time domain and fragile subcarrier orthogonality in the frequency domain. The severe distortion for one-bit quantization typically results in an error floor even at moderately low signal-to-noise-ratio (SNR) such as 5 dB. For channel estimation (using pilots), we design a novel generative supervised deep neural network (DNN) that can be trained with a reasonable number of pilots. After channel estimation, a neural network-based receiver -- specifically, an autoencoder -- jointly learns a precoder and decoder for data symbol detection. Since quantization prevents end-to-end training, we propose a two-step sequential training policy for this model. With synthetic data, our deep learning-based channel estimation can outperform least squares (LS) channel estimation for unquantized (full-resolution) OFDM at average SNRs up to 14 dB. For data detection, our proposed design achieves lower bit error rate (BER) in fading than unquantized OFDM at average SNRs up to 10 dB.
Neural Response Ranking for Social Conversation: A Data-Efficient Approach
Authors: Igor Shalyminov, Ondřej Dušek, Oliver Lemon
Date: 2018-11-02 00:00:00
Proceeding: WS 2018 10

The overall objective of 'social' dialogue systems is to support engaging,
entertaining, and lengthy conversations on a wide variety of topics, including
social chit-chat. Apart from raw dialogue data, user-provided ratings are the
most common signal used to train such systems to produce engaging responses. In
this paper we show that social dialogue systems can be trained effectively from
raw unannotated data. Using a dataset of real conversations collected in the
2017 Alexa Prize challenge, we developed a neural ranker for selecting 'good'
system responses to user utterances, i.e. responses which are likely to lead to
long and engaging conversations. We show that (1) our neural ranker
consistently outperforms several strong baselines when trained to optimise for
user ratings; (2) when trained on larger amounts of data and only using
conversation length as the objective, the ranker performs better than the one
trained using ratings -- ultimately reaching a Precision@1 of 0.87. This
advance will make data collection for social conversational agents simpler and
less expensive in the future.
Discovering conservation laws from data for control
Authors: Eurika Kaiser, J. Nathan Kutz, Steven L. Brunton
Date: 2018-11-02 00:00:00

Conserved quantities, i.e. constants of motion, are critical for
characterizing many dynamical systems in science and engineering. These
quantities are related to underlying symmetries and they provide fundamental
knowledge about physical laws, describe the evolution of the system, and enable
system reduction. In this work, we formulate a data-driven architecture for
discovering conserved quantities based on Koopman theory. The Koopman operator
has emerged as a principled linear embedding of nonlinear dynamics, and its
eigenfunctions establish intrinsic coordinates along which the dynamics behave
linearly. Interestingly, eigenfunctions of the Koopman operator associated with
vanishing eigenvalues correspond to conserved quantities of the underlying
system. In this paper, we show that these invariants may be identified with
data-driven regression and power series expansions, based on the infinitesimal
generator of the Koopman operator. We further establish a connection between
the Koopman framework, conserved quantities, and the Lie-Poisson bracket. This
data-driven method for discovering conserved quantities is demonstrated on the
three-dimensional rigid body equations, where we simultaneously discover the
total energy and angular momentum and use these intrinsic coordinates to
develop a model predictive controller to track a given reference value.
Improving the Coverage and the Generalization Ability of Neural Word Sense Disambiguation through Hypernymy and Hyponymy Relationships
Authors: Loïc Vial, Benjamin Lecouteux, Didier Schwab
Date: 2018-11-02 00:00:00
Tasks: Word Sense Disambiguation

In Word Sense Disambiguation (WSD), the predominant approach generally
involves a supervised system trained on sense annotated corpora. The limited
quantity of such corpora however restricts the coverage and the performance of
these systems. In this article, we propose a new method that solves these
issues by taking advantage of the knowledge present in WordNet, and especially
the hypernymy and hyponymy relationships between synsets, in order to reduce
the number of different sense tags that are necessary to disambiguate all words
of the lexical database. Our method leads to state of the art results on most
WSD evaluation tasks, while improving the coverage of supervised systems,
reducing the training time and the size of the models, without additional
training data. In addition, we exhibit results that significantly outperform
the state of the art when our method is combined with an ensembling technique
and the addition of the WordNet Gloss Tagged as training corpus.
Dantzig Selector with an Approximately Optimal Denoising Matrix and its Application to Reinforcement Learning
Authors: Bo Liu, Luwan Zhang, Ji Liu
Date: 2018-11-02 00:00:00
Tasks: compressed sensing, Denoising, feature selection, Reinforcement Learning, Reinforcement Learning (RL), Sparse Learning

Dantzig Selector (DS) is widely used in compressed sensing and sparse
learning for feature selection and sparse signal recovery. Since the DS
formulation is essentially a linear programming optimization, many existing
linear programming solvers can be simply applied for scaling up. The DS
formulation can be explained as a basis pursuit denoising problem, wherein the
data matrix (or measurement matrix) is employed as the denoising matrix to
eliminate the observation noise. However, we notice that the data matrix may
not be the optimal denoising matrix, as shown by a simple counter-example. This
motivates us to pursue a better denoising matrix for defining a general DS
formulation. We first define the optimal denoising matrix through a minimax
optimization, which turns out to be an NPhard problem. To make the problem
computationally tractable, we propose a novel algorithm, termed as Optimal
Denoising Dantzig Selector (ODDS), to approximately estimate the optimal
denoising matrix. Empirical experiments validate the proposed method. Finally,
a novel sparse reinforcement learning algorithm is formulated by extending the
proposed ODDS algorithm to temporal difference learning, and empirical
experimental results demonstrate to outperform the conventional vanilla DS-TD
algorithm.
A Fast Algorithm for Clustering High Dimensional Feature Vectors
Authors: Shahina Rahman, Valen E. Johnson
Date: 2018-11-02 00:00:00
Tasks: Clustering, Vocal Bursts Intensity Prediction

We propose an algorithm for clustering high dimensional data. If $P$ features
for $N$ objects are represented in an $N\times P$ matrix ${\bf X}$, where $N\ll
P$, the method is based on exploiting the cluster-dependent structure of the
$N\times N$ matrix ${\bf XX}^T$. Computational burden thus depends primarily on
$N$, the number of objects to be clustered, rather than $P$, the number of
features that are measured. This makes the method particularly useful in high
dimensional settings, where it is substantially faster than a number of other
popular clustering algorithms. Aside from an upper bound on the number of
potential clusters, the method is independent of tuning parameters. When
compared to $16$ other clustering algorithms on $32$ genomic datasets with gold
standards, we show that it provides the most accurate cluster configuration
more than twice as often than its closest competitors. We illustrate the method
on data taken from highly cited genomic studies.
Image Chat: Engaging Grounded Conversations
Authors: Kurt Shuster, Samuel Humeau, Antoine Bordes, Jason Weston
Date: 2018-11-02 00:00:00
Tasks: Text Retrieval

To achieve the long-term goal of machines being able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art image and text representations, considering various ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emotional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and human evaluations of engagingness show the efficacy of our approach; in particular, we obtain state-of-the-art performance on the existing IGC task, and our best performing model is almost on par with humans on the Image-Chat test set (preferred 47.7% of the time).