pwc-archive/papers-with-abstracts huggingface dataset:
Efficient online algorithms for fast-rate regret bounds under sparsity
Authors: Pierre Gaillard, Olivier Wintenberger
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12

We consider the online convex optimization problem. In the setting of
arbitrary sequences and finite set of parameters, we establish a new fast-rate
quantile regret bound. Then we investigate the optimization into the L1-ball by
discretizing the parameter space. Our algorithm is projection free and we
propose an efficient solution by restarting the algorithm on adaptive
discretization grids. In the adversarial setting, we develop an algorithm that
achieves several rates of convergence with different dependencies on the
sparsity of the objective. In the i.i.d. setting, we establish new risk bounds
that are adaptive to the sparsity of the problem and to the regularity of the
risk (ranging from a rate 1 / $\sqrt T$ for general convex risk to 1 /T for
strongly convex risk). These results generalize previous works on sparse online
learning. They are obtained under a weak assumption on the risk
({\L}ojasiewicz's assumption) that allows multiple optima which is crucial when
dealing with degenerate situations.
Matrix Co-completion for Multi-label Classification with Missing Features and Labels
Authors: Miao Xu, Gang Niu, Bo Han, Ivor W. Tsang, Zhi-Hua Zhou, Masashi Sugiyama
Date: 2018-05-23 00:00:00
Tasks: General Classification, Matrix Completion, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

We consider a challenging multi-label classification problem where both
feature matrix $\X$ and label matrix $\Y$ have missing entries. An existing
method concatenated $\X$ and $\Y$ as $[\X; \Y]$ and applied a matrix completion
(MC) method to fill the missing entries, under the assumption that $[\X; \Y]$
is of low-rank. However, since entries of $\Y$ take binary values in the
multi-label setting, it is unlikely that $\Y$ is of low-rank. Moreover, such
assumption implies a linear relationship between $\X$ and $\Y$ which may not
hold in practice. In this paper, we consider a latent matrix $\Z$ that produces
the probability $\sigma(Z_{ij})$ of generating label $Y_{ij}$, where
$\sigma(\cdot)$ is nonlinear. Considering label correlation, we assume $[\X;
\Z]$ is of low-rank, and propose an MC algorithm based on subgradient descent
named co-completion (COCO) motivated by elastic net and one-bit MC. We give a
theoretical bound on the recovery effect of COCO and demonstrate its practical
usefulness through experiments.
RDF2Vec-based Classification of Ontology Alignment Changes
Authors: Matthias Jurisch, Bodo Igler
Date: 2018-05-23 00:00:00
Tasks: Classification, General Classification

When ontologies cover overlapping topics, the overlap can be represented
using ontology alignments. These alignments need to be continuously adapted to
changing ontologies. Especially for large ontologies this is a costly task
often consisting of manual work. Finding changes that do not lead to an
adaption of the alignment can potentially make this process significantly
easier. This work presents an approach to finding these changes based on RDF
embeddings and common classification techniques. To examine the feasibility of
this approach, an evaluation on a real-world dataset is presented. In this
evaluation, the best classifiers reached a precision of 0.8.
Dataflow Matrix Machines and V-values: a Bridge between Programs and Neural Nets
Authors: Michael Bukatin, Jon Anthony
Date: 2017-12-20 00:00:00

1) Dataflow matrix machines (DMMs) generalize neural nets by replacing
streams of numbers with linear streams (streams supporting linear
combinations), allowing arbitrary input and output arities for activation
functions, countable-sized networks with finite dynamically changeable active
part capable of unbounded growth, and a very expressive self-referential
mechanism.
  2) DMMs are suitable for general-purpose programming, while retaining the key
property of recurrent neural networks: programs are expressed via matrices of
real numbers, and continuous changes to those matrices produce arbitrarily
small variations in the associated programs.
  3) Spaces of V-values (vector-like elements based on nested maps) are
particularly useful, enabling DMMs with variadic activation functions and
conveniently representing conventional data structures.
Probabilistic Riemannian submanifold learning with wrapped Gaussian process latent variable models
Authors: Anton Mallasto, Søren Hauberg, Aasa Feragen
Date: 2018-05-23 00:00:00
Tasks: Uncertainty Quantification

Latent variable models (LVMs) learn probabilistic models of data manifolds
lying in an \emph{ambient} Euclidean space. In a number of applications, a
priori known spatial constraints can shrink the ambient space into a
considerably smaller manifold. Additionally, in these applications the
Euclidean geometry might induce a suboptimal similarity measure, which could be
improved by choosing a different metric. Euclidean models ignore such
information and assign probability mass to data points that can never appear as
data, and vastly different likelihoods to points that are similar under the
desired metric. We propose the wrapped Gaussian process latent variable model
(WGPLVM), that extends Gaussian process latent variable models to take values
strictly on a given ambient Riemannian manifold, making the model blind to
impossible data points. This allows non-linear, probabilistic inference of
low-dimensional Riemannian submanifolds from data. Our evaluation on diverse
datasets show that we improve performance on several tasks, including encoding,
visualization and uncertainty quantification.
Selecting Machine-Translated Data for Quick Bootstrapping of a Natural Language Understanding System
Authors: Judith Gaspers, Penny Karanasou, Rajen Chatterjee
Date: 2018-05-23 00:00:00
Proceeding: NAACL 2018 6
Tasks: Machine Translation, Natural Language Understanding, Translation

This paper investigates the use of Machine Translation (MT) to bootstrap a
Natural Language Understanding (NLU) system for a new language for the use case
of a large-scale voice-controlled device. The goal is to decrease the cost and
time needed to get an annotated corpus for the new language, while still having
a large enough coverage of user requests. Different methods of filtering MT
data in order to keep utterances that improve NLU performance and
language-specific post-processing methods are investigated. These methods are
tested in a large-scale NLU task with translating around 10 millions training
utterances from English to German. The results show a large improvement for
using MT data over a grammar-based and over an in-house data collection
baseline, while reducing the manual effort greatly. Both filtering and
post-processing approaches improve results further.
Optimal Transport for structured data with application on graphs
Authors: Titouan Vayer, Laetitia Chapel, Rémi Flamary, Romain Tavenard, Nicolas Courty
Date: 2018-05-23 00:00:00
Tasks: Clustering, Graph Classification, Graph Clustering, Time Series Analysis

This work considers the problem of computing distances between structured objects such as undirected graphs, seen as probability distributions in a specific metric space. We consider a new transportation distance (i.e. that minimizes a total cost of transporting probability masses) that unveils the geometric nature of the structured objects space. Unlike Wasserstein or Gromov-Wasserstein metrics that focus solely and respectively on features (by considering a metric in the feature space) or structure (by seeing structure as a metric space), our new distance exploits jointly both information, and is consequently called Fused Gromov-Wasserstein (FGW). After discussing its properties and computational aspects, we show results on a graph classification task, where our method outperforms both graph kernels and deep graph convolutional networks. Exploiting further on the metric properties of FGW, interesting geometric objects such as Fr\'echet means or barycenters of graphs are illustrated and discussed in a clustering context.
Hyperbolic Neural Networks
Authors: Octavian-Eugen Ganea, Gary Bécigneul, Thomas Hofmann
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Graph Representation Learning, Natural Language Inference, Sentence, Sentence Embeddings

Hyperbolic spaces have recently gained momentum in the context of machine
learning due to their high capacity and tree-likeliness properties. However,
the representational power of hyperbolic geometry is not yet on par with
Euclidean geometry, mostly because of the absence of corresponding hyperbolic
neural network layers. This makes it hard to use hyperbolic embeddings in
downstream tasks. Here, we bridge this gap in a principled manner by combining
the formalism of M\"obius gyrovector spaces with the Riemannian geometry of the
Poincar\'e model of hyperbolic spaces. As a result, we derive hyperbolic
versions of important deep learning tools: multinomial logistic regression,
feed-forward and recurrent neural networks such as gated recurrent units. This
allows to embed sequential data and perform classification in the hyperbolic
space. Empirically, we show that, even if hyperbolic optimization tools are
limited, hyperbolic sentence embeddings either outperform or are on par with
their Euclidean variants on textual entailment and noisy-prefix recognition
tasks.
Bayesian Alignments of Warped Multi-Output Gaussian Processes
Authors: Markus Kaiser, Clemens Otte, Thomas Runkler, Carl Henrik Ek
Date: 2017-10-08 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Gaussian Processes, Time Series, Time Series Analysis

We propose a novel Bayesian approach to modelling nonlinear alignments of
time series based on latent shared information. We apply the method to the
real-world problem of finding common structure in the sensor data of wind
turbines introduced by the underlying latent and turbulent wind field. The
proposed model allows for both arbitrary alignments of the inputs and
non-parametric output warpings to transform the observations. This gives rise
to multiple deep Gaussian process models connected via latent generating
processes. We present an efficient variational approximation based on nested
variational compression and show how the model can be used to extract shared
information between dependent time series, recovering an interpretable
functional decomposition of the learning problem. We show results for an
artificial data set and real-world data of two wind turbines.
Image Restoration by Estimating Frequency Distribution of Local Patches
Authors: Jaeyoung Yoo, Sang-ho Lee, Nojun Kwak
Date: 2018-05-23 00:00:00
Proceeding: CVPR 2018 6
Tasks: General Classification, Image Compression, Image Restoration

In this paper, we propose a method to solve the image restoration problem,
which tries to restore the details of a corrupted image, especially due to the
loss caused by JPEG compression. We have treated an image in the frequency
domain to explicitly restore the frequency components lost during image
compression. In doing so, the distribution in the frequency domain is learned
using the cross entropy loss. Unlike recent approaches, we have reconstructed
the details of an image without using the scheme of adversarial training.
Rather, the image restoration problem is treated as a classification problem to
determine the frequency coefficient for each frequency band in an image patch.
In this paper, we show that the proposed method effectively restores a
JPEG-compressed image with more detailed high frequency components, making the
restored image more vivid.
Excitation Dropout: Encouraging Plasticity in Deep Neural Networks
Authors: Andrea Zunino, Sarah Adel Bargal, Pietro Morerio, Jianming Zhang, Stan Sclaroff, Vittorio Murino
Date: 2018-05-23 00:00:00
Tasks: Decision Making, Video Recognition

We propose a guided dropout regularizer for deep networks based on the evidence of a network prediction defined as the firing of neurons in specific paths. In this work, we utilize the evidence at each neuron to determine the probability of dropout, rather than dropping out neurons uniformly at random as in standard dropout. In essence, we dropout with higher probability those neurons which contribute more to decision making at training time. This approach penalizes high saliency neurons that are most relevant for model prediction, i.e. those having stronger evidence. By dropping such high-saliency neurons, the network is forced to learn alternative paths in order to maintain loss minimization, resulting in a plasticity-like behavior, a characteristic of human brains too. We demonstrate better generalization ability, an increased utilization of network neurons, and a higher resilience to network compression using several metrics over four image/video recognition benchmarks.
Neural networks for post-processing ensemble weather forecasts
Authors: Stephan Rasp, Sebastian Lerch
Date: 2018-05-23 00:00:00

Ensemble weather predictions require statistical post-processing of
systematic errors to obtain reliable and accurate probabilistic forecasts.
Traditionally, this is accomplished with distributional regression models in
which the parameters of a predictive distribution are estimated from a training
period. We propose a flexible alternative based on neural networks that can
incorporate nonlinear relationships between arbitrary predictor variables and
forecast distribution parameters that are automatically learned in a
data-driven way rather than requiring pre-specified link functions. In a case
study of 2-meter temperature forecasts at surface stations in Germany, the
neural network approach significantly outperforms benchmark post-processing
methods while being computationally more affordable. Key components to this
improvement are the use of auxiliary predictor variables and station-specific
information with the help of embeddings. Furthermore, the trained neural
network can be used to gain insight into the importance of meteorological
variables thereby challenging the notion of neural networks as uninterpretable
black boxes. Our approach can easily be extended to other statistical
post-processing and forecasting problems. We anticipate that recent advances in
deep learning combined with the ever-increasing amounts of model and
observation data will transform the post-processing of numerical weather
forecasts in the coming decade.
A Parameter Estimation of Fractional Order Grey Model Based on Adaptive Dynamic Cat Swarm Algorithm
Authors: Binyan Lin, Fei Gao, Meng Wang, Yuyao Xiong, Ansheng Li
Date: 2018-05-22 00:00:00
Tasks: parameter estimation, Prediction

In this paper, we utilize ADCSO (Adaptive Dynamic Cat Swarm Optimization) to
estimate the parameters of Fractional Order Grey Model. The parameters of
Fractional Order Grey Model affect the prediction accuracy of the model. In
order to solve the problem that general swarm intelligence algorithms easily
fall into the local optimum and optimize the accuracy of the model, ADCSO is
utilized to reduce the error of the model. Experimental results for the data of
container throughput of Wuhan Port and marine capture productions of Zhejiang
Province show that the different parameter values affect the prediction
results. The parameters estimated by ADCSO make the prediction error of the
model smaller and the convergence speed higher, and it is not easy to fall into
the local convergence compared with PSO (Particle Swarm Optimization) and LSM
(Least Square Method). The feasibility and advantage of ADCSO for the parameter
estimation of Fractional Order Grey Model are verified.
Constrained Graph Variational Autoencoders for Molecule Design
Authors: Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, Alexander L. Gaunt
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Decoder

Graphs are ubiquitous data structures for representing interactions between
entities. With an emphasis on the use of graphs to represent chemical
molecules, we explore the task of learning to generate graphs that conform to a
distribution observed in training data. We propose a variational autoencoder
model in which both encoder and decoder are graph-structured. Our decoder
assumes a sequential ordering of graph extension steps and we discuss and
analyze design choices that mitigate the potential downsides of this
linearization. Experiments compare our approach with a wide range of baselines
on the molecule generation task and show that our method is more successful at
matching the statistics of the original dataset on semantically important
metrics. Furthermore, we show that by using appropriate shaping of the latent
space, our model allows us to design molecules that are (locally) optimal in
desired properties.
Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application
Authors: Yujing Hu, Qing Da, An-Xiang Zeng, Yang Yu, Yinghui Xu
Date: 2018-03-02 00:00:00
Tasks: Decision Making, Learning-To-Rank, Reinforcement Learning, Reinforcement Learning (RL)

In e-commerce platforms such as Amazon and TaoBao, ranking items in a search
session is a typical multi-step decision-making problem. Learning to rank (LTR)
methods have been widely applied to ranking problems. However, such methods
often consider different ranking steps in a session to be independent, which
conversely may be highly correlated to each other. For better utilizing the
correlation between different ranking steps, in this paper, we propose to use
reinforcement learning (RL) to learn an optimal ranking policy which maximizes
the expected accumulative rewards in a search session. Firstly, we formally
define the concept of search session Markov decision process (SSMDP) to
formulate the multi-step ranking problem. Secondly, we analyze the property of
SSMDP and theoretically prove the necessity of maximizing accumulative rewards.
Lastly, we propose a novel policy gradient algorithm for learning an optimal
ranking policy, which is able to deal with the problem of high reward variance
and unbalanced reward distribution of an SSMDP. Experiments are conducted in
simulation and TaoBao search engine. The results demonstrate that our algorithm
performs much better than online LTR methods, with more than 40% and 30% growth
of total transaction amount in the simulation and the real application,
respectively.
Grounding the Semantics of Part-of-Day Nouns Worldwide using Twitter
Authors: David Vilares, Carlos Gómez-Rodríguez
Date: 2018-05-23 00:00:00
Proceeding: WS 2018 6

The usage of part-of-day nouns, such as 'night', and their time-specific
greetings ('good night'), varies across languages and cultures. We show the
possibilities that Twitter offers for studying the semantics of these terms and
its variability between countries. We mine a worldwide sample of multilingual
tweets with temporal greetings, and study how their frequencies vary in
relation with local time. The results provide insights into the semantics of
these temporal expressions and the cultural and sociological factors
influencing their usage.
Deep learning generalizes because the parameter-function map is biased towards simple functions
Authors: Guillermo Valle-Pérez, Chico Q. Camargo, Ard A. Louis
Date: 2018-05-22 00:00:00
Proceeding: ICLR 2019 5
Tasks: Gaussian Processes, Learning Theory

Deep neural networks (DNNs) generalize remarkably well without explicit
regularization even in the strongly over-parametrized regime where classical
learning theory would instead predict that they would severely overfit. While
many proposals for some kind of implicit regularization have been made to
rationalise this success, there is no consensus for the fundamental reason why
DNNs do not strongly overfit. In this paper, we provide a new explanation. By
applying a very general probability-complexity bound recently derived from
algorithmic information theory (AIT), we argue that the parameter-function map
of many DNNs should be exponentially biased towards simple functions. We then
provide clear evidence for this strong simplicity bias in a model DNN for
Boolean functions, as well as in much larger fully connected and convolutional
networks applied to CIFAR10 and MNIST. As the target functions in many real
problems are expected to be highly structured, this intrinsic simplicity bias
helps explain why deep networks generalize well on real world problems. This
picture also facilitates a novel PAC-Bayes approach where the prior is taken
over the DNN input-output function space, rather than the more conventional
prior over parameter space. If we assume that the training algorithm samples
parameters close to uniformly within the zero-error region then the PAC-Bayes
theorem can be used to guarantee good expected generalization for target
functions producing high-likelihood training sets. By exploiting recently
discovered connections between DNNs and Gaussian processes to estimate the
marginal likelihood, we produce relatively tight generalization PAC-Bayes error
bounds which correlate well with the true error on realistic datasets such as
MNIST and CIFAR10 and for architectures including convolutional and fully
connected networks.
Representation Balancing MDPs for Off-Policy Policy Evaluation
Authors: Yao Liu, Omer Gottesman, Aniruddh Raghu, Matthieu Komorowski, Aldo Faisal, Finale Doshi-Velez, Emma Brunskill
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12

We study the problem of off-policy policy evaluation (OPPE) in RL. In
contrast to prior work, we consider how to estimate both the individual policy
value and average policy value accurately. We draw inspiration from recent work
in causal reasoning, and propose a new finite sample generalization error bound
for value estimates from MDP models. Using this upper bound as an objective, we
develop a learning algorithm of an MDP model with a balanced representation,
and show that our approach can yield substantially lower MSE in common
synthetic benchmarks and a HIV treatment simulation domain.
Generalisation of structural knowledge in the hippocampal-entorhinal system
Authors: James C. R. Whittington, Timothy H. Muller, Shirley Mark, Caswell Barry, Timothy E. J. Behrens
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12

A central problem to understanding intelligence is the concept of
generalisation. This allows previously learnt structure to be exploited to
solve tasks in novel situations differing in their particularities. We take
inspiration from neuroscience, specifically the hippocampal-entorhinal system
known to be important for generalisation. We propose that to generalise
structural knowledge, the representations of the structure of the world, i.e.
how entities in the world relate to each other, need to be separated from
representations of the entities themselves. We show, under these principles,
artificial neural networks embedded with hierarchy and fast Hebbian memory, can
learn the statistics of memories and generalise structural knowledge. Spatial
neuronal representations mirroring those found in the brain emerge, suggesting
spatial cognition is an instance of more general organising principles. We
further unify many entorhinal cell types as basis functions for constructing
transition graphs, and show these representations effectively utilise memories.
We experimentally support model assumptions, showing a preserved relationship
between entorhinal grid and hippocampal place cells across environments.