pwc-archive/papers-with-abstracts huggingface dataset:
Simulation assisted machine learning
Authors: Timo M. Deist, Andrew Patti, Zhaoqi Wang, David Krane, Taylor Sorenson, David Craft
Date: 2018-02-15 00:00:00
Tasks: BIG-bench Machine Learning

Motivation: In a predictive modeling setting, if sufficient details of the system behavior are known, one can build and use a simulation for making predictions. When sufficient system details are not known, one typically turns to machine learning, which builds a black-box model of the system using a large dataset of input sample features and outputs. We consider a setting which is between these two extremes: some details of the system mechanics are known but not enough for creating simulations that can be used to make high quality predictions. In this context we propose using approximate simulations to build a kernel for use in kernelized machine learning methods, such as support vector machines. The results of multiple simulations (under various uncertainty scenarios) are used to compute similarity measures between every pair of samples: sample pairs are given a high similarity score if they behave similarly under a wide range of simulation parameters. These similarity values, rather than the original high dimensional feature data, are used to build the kernel. Results: We demonstrate and explore the simulation based kernel (SimKern) concept using four synthetic complex systems--three biologically inspired models and one network flow optimization model. We show that, when the number of training samples is small compared to the number of features, the SimKern approach dominates over no-prior-knowledge methods. This approach should be applicable in all disciplines where predictive models are sought and informative yet approximate simulations are available. Availability: The Python SimKern software, the demonstration models (in MATLAB, R), and the datasets are available at https://github.com/davidcraft/SimKern.
Towards Deep Representation Learning with Genetic Programming
Authors: Lino Rodriguez-Coayahuitl, Alicia Morales-Reyes, Hugo Jair Escalante
Date: 2018-02-20 00:00:00
Tasks: BIG-bench Machine Learning, Representation Learning

Genetic Programming (GP) is an evolutionary algorithm commonly used for
machine learning tasks. In this paper we present a method that allows GP to
transform the representation of a large-scale machine learning dataset into a
more compact representation, by means of processing features from the original
representation at individual level. We develop as a proof of concept of this
method an autoencoder. We tested a preliminary version of our approach in a
variety of well-known machine learning image datasets. We speculate that this
method, used in an iterative manner, can produce results competitive with
state-of-art deep neural networks.
Rover Descent: Learning to optimize by learning to navigate on prototypical loss surfaces
Authors: Louis Faury, Flavian vasile
Date: 2018-01-22 00:00:00
Tasks: Navigate

Learning to optimize - the idea that we can learn from data algorithms that
optimize a numerical criterion - has recently been at the heart of a growing
number of research efforts. One of the most challenging issues within this
approach is to learn a policy that is able to optimize over classes of
functions that are fairly different from the ones that it was trained on. We
propose a novel way of framing learning to optimize as a problem of learning a
good navigation policy on a partially observable loss surface. To this end, we
develop Rover Descent, a solution that allows us to learn a fairly broad
optimization policy from training on a small set of prototypical
two-dimensional surfaces that encompasses the classically hard cases such as
valleys, plateaus, cliffs and saddles and by using strictly zero-order
information. We show that, without having access to gradient or curvature
information, we achieve state-of-the-art convergence speed on optimization
problems not presented at training time such as the Rosenbrock function and
other hard cases in two dimensions. We extend our framework to optimize over
high dimensional landscapes, while still handling only two-dimensional local
landscape information and show good preliminary results.
Improved TDNNs using Deep Kernels and Frequency Dependent Grid-RNNs
Authors: Florian Kreyssig, Chao Zhang, Philip Woodland
Date: 2018-02-18 00:00:00
Tasks: speech-recognition, Speech Recognition

Time delay neural networks (TDNNs) are an effective acoustic model for large
vocabulary speech recognition. The strength of the model can be attributed to
its ability to effectively model long temporal contexts. However, current TDNN
models are relatively shallow, which limits the modelling capability. This
paper proposes a method of increasing the network depth by deepening the kernel
used in the TDNN temporal convolutions. The best performing kernel consists of
three fully connected layers with a residual (ResNet) connection from the
output of the first to the output of the third. The addition of
spectro-temporal processing as the input to the TDNN in the form of a
convolutional neural network (CNN) and a newly designed Grid-RNN was
investigated. The Grid-RNN strongly outperforms a CNN if different sets of
parameters for different frequency bands are used and can be further enhanced
by using a bi-directional Grid-RNN. Experiments using the multi-genre broadcast
(MGB3) English data (275h) show that deep kernel TDNNs reduces the word error
rate (WER) by 6% relative and when combined with the frequency dependent
Grid-RNN gives a relative WER reduction of 9%.
Combining Textual Content and Structure to Improve Dialog Similarity
Authors: Ana Paula Appel, Paulo Rodrigo Cavalin, Marisa Affonso Vasconcelos, Claudio Santos Pinhanez
Date: 2018-02-20 00:00:00
Tasks: Chatbot

Chatbots, taking advantage of the success of the messaging apps and recent
advances in Artificial Intelligence, have become very popular, from helping
business to improve customer services to chatting to users for the sake of
conversation and engagement (celebrity or personal bots). However, developing
and improving a chatbot requires understanding their data generated by its
users. Dialog data has a different nature of a simple question and answering
interaction, in which context and temporal properties (turn order) creates a
different understanding of such data. In this paper, we propose a novelty
metric to compute dialogs' similarity based not only on the text content but
also on the information related to the dialog structure. Our experimental
results performed over the Switchboard dataset show that using evidence from
both textual content and the dialog structure leads to more accurate results
than using each measure in isolation.
Learning of Optimal Forecast Aggregation in Partial Evidence Environments
Authors: Yakov Babichenko, Dan Garber
Date: 2018-02-20 00:00:00

We consider the forecast aggregation problem in repeated settings, where the
forecasts are done on a binary event. At each period multiple experts provide
forecasts about an event. The goal of the aggregator is to aggregate those
forecasts into a subjective accurate forecast. We assume that experts are
Bayesian; namely they share a common prior, each expert is exposed to some
evidence, and each expert applies Bayes rule to deduce his forecast. The
aggregator is ignorant with respect to the information structure (i.e.,
distribution over evidence) according to which experts make their prediction.
The aggregator observes the experts' forecasts only. At the end of each period
the actual state is realized. We focus on the question whether the aggregator
can learn to aggregate optimally the forecasts of the experts, where the
optimal aggregation is the Bayesian aggregation that takes into account all the
information (evidence) in the system.
  We consider the class of partial evidence information structures, where each
expert is exposed to a different subset of conditionally independent signals.
Our main results are positive; We show that optimal aggregation can be learned
in polynomial time in a quite wide range of instances of the partial evidence
environments. We provide a tight characterization of the instances where
learning is possible and impossible.
Do Less, Get More: Streaming Submodular Maximization with Subsampling
Authors: Moran Feldman, Amin Karbasi, Ehsan Kazemi
Date: 2018-02-20 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Video Summarization

In this paper, we develop the first one-pass streaming algorithm for
submodular maximization that does not evaluate the entire stream even once. By
carefully subsampling each element of data stream, our algorithm enjoys the
tightest approximation guarantees in various settings while having the smallest
memory footprint and requiring the lowest number of function evaluations. More
specifically, for a monotone submodular function and a $p$-matchoid constraint,
our randomized algorithm achieves a $4p$ approximation ratio (in expectation)
with $O(k)$ memory and $O(km/p)$ queries per element ($k$ is the size of the
largest feasible solution and $m$ is the number of matroids used to define the
constraint). For the non-monotone case, our approximation ratio increases only
slightly to $4p+2-o(1)$. To the best or our knowledge, our algorithm is the
first that combines the benefits of streaming and subsampling in a novel way in
order to truly scale submodular maximization to massive machine learning
problems. To showcase its practicality, we empirically evaluated the
performance of our algorithm on a video summarization application and observed
that it outperforms the state-of-the-art algorithm by up to fifty fold, while
maintaining practically the same utility.
Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow
Authors: Eddy Ilg, Özgün Çiçek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas Brox
Date: 2018-02-20 00:00:00
Proceeding: ECCV 2018 9
Tasks: Optical Flow Estimation

Optical flow estimation can be formulated as an end-to-end supervised
learning problem, which yields estimates with a superior accuracy-runtime
tradeoff compared to alternative methodology. In this paper, we make such
networks estimate their local uncertainty about the correctness of their
prediction, which is vital information when building decisions on top of the
estimations. For the first time we compare several strategies and techniques to
estimate uncertainty in a large-scale computer vision task like optical flow
estimation. Moreover, we introduce a new network architecture utilizing the
Winner-Takes-All loss and show that this can provide complementary hypotheses
and uncertainty estimates efficiently with a single forward pass and without
the need for sampling or ensembles. Finally, we demonstrate the quality of the
different uncertainty estimates, which is clearly above previous confidence
measures on optical flow and allows for interactive frame rates.
Camera-based vehicle velocity estimation from monocular video
Authors: Moritz Kampelmühler, Michael G. Müller, Christoph Feichtenhofer
Date: 2018-02-20 00:00:00
Tasks: Autonomous Driving, CPU, Optical Flow Estimation

This paper documents the winning entry at the CVPR2017 vehicle velocity
estimation challenge. Velocity estimation is an emerging task in autonomous
driving which has not yet been thoroughly explored. The goal is to estimate the
relative velocity of a specific vehicle from a sequence of images. In this
paper, we present a light-weight approach for directly regressing vehicle
velocities from their trajectories using a multilayer perceptron. Another
contribution is an explorative study of features for monocular vehicle velocity
estimation. We find that light-weight trajectory based features outperform
depth and motion cues extracted from deep ConvNets, especially for far-distance
predictions where current disparity and optical flow estimators are challenged
significantly. Our light-weight approach is real-time capable on a single CPU
and outperforms all competing entries in the velocity estimation challenge. On
the test set, we report an average error of 1.12 m/s which is comparable to a
(ground-truth) system that combines LiDAR and radar techniques to achieve an
error of around 0.71 m/s.
An Efficient Semismooth Newton Based Algorithm for Convex Clustering
Authors: Yancheng Yuan, Defeng Sun, Kim-Chuan Toh
Date: 2018-02-20 00:00:00
Proceeding: ICML 2018 7
Tasks: Clustering

Clustering may be the most fundamental problem in unsupervised learning which
is still active in machine learning research because its importance in many
applications. Popular methods like K-means, may suffer from instability as they
are prone to get stuck in its local minima. Recently, the sum-of-norms (SON)
model (also known as clustering path), which is a convex relaxation of
hierarchical clustering model, has been proposed in [7] and [5] Although
numerical algorithms like ADMM and AMA are proposed to solve convex clustering
model [2], it is known to be very challenging to solve large-scale problems. In
this paper, we propose a semi-smooth Newton based augmented Lagrangian method
for large-scale convex clustering problems. Extensive numerical experiments on
both simulated and real data demonstrate that our algorithm is highly efficient
and robust for solving large-scale problems. Moreover, the numerical results
also show the superior performance and scalability of our algorithm compared to
existing first-order methods.
Attentive Tensor Product Learning
Authors: Qiuyuan Huang, Li Deng, Dapeng Wu, Chang Liu, Xiaodong He
Date: 2018-02-20 00:00:00
Tasks: Constituency Parsing, Deep Learning, Image Captioning, Part-Of-Speech Tagging, POS, POS Tagging, Sentence

This paper proposes a new architecture - Attentive Tensor Product Learning
(ATPL) - to represent grammatical structures in deep learning models. ATPL is a
new architecture to bridge this gap by exploiting Tensor Product
Representations (TPR), a structured neural-symbolic model developed in
cognitive science, aiming to integrate deep learning with explicit language
structures and rules. The key ideas of ATPL are: 1) unsupervised learning of
role-unbinding vectors of words via TPR-based deep neural network; 2) employing
attention modules to compute TPR; and 3) integration of TPR with typical deep
learning architectures including Long Short-Term Memory (LSTM) and Feedforward
Neural Network (FFNN). The novelty of our approach lies in its ability to
extract the grammatical structure of a sentence by using role-unbinding
vectors, which are obtained in an unsupervised manner. This ATPL approach is
applied to 1) image captioning, 2) part of speech (POS) tagging, and 3)
constituency parsing of a sentence. Experimental results demonstrate the
effectiveness of the proposed approach.
i-RevNet: Deep Invertible Networks
Authors: Jörn-Henrik Jacobsen, Arnold Smeulders, Edouard Oyallon
Date: 2018-02-20 00:00:00
Proceeding: ICLR 2018 1

It is widely believed that the success of deep convolutional networks is
based on progressively discarding uninformative variability about the input
with respect to the problem at hand. This is supported empirically by the
difficulty of recovering images from their hidden representations, in most
commonly used network architectures. In this paper we show via a one-to-one
mapping that this loss of information is not a necessary condition to learn
representations that generalize well on complicated problems, such as ImageNet.
Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can
be fully inverted up to the final projection onto the classes, i.e. no
information is discarded. Building an invertible architecture is difficult, for
one, because the local inversion is ill-conditioned, we overcome this by
providing an explicit inverse. An analysis of i-RevNets learned representations
suggests an alternative explanation for the success of deep networks by a
progressive contraction and linear separation with depth. To shed light on the
nature of the model learned by the i-RevNet we reconstruct linear
interpolations between natural image representations.
Composite Optimization by Nonconvex Majorization-Minimization
Authors: Jonas Geiping, Michael Moeller
Date: 2018-02-20 00:00:00
Tasks: Super-Resolution

The minimization of a nonconvex composite function can model a variety of
imaging tasks. A popular class of algorithms for solving such problems are
majorization-minimization techniques which iteratively approximate the
composite nonconvex function by a majorizing function that is easy to minimize.
Most techniques, e.g. gradient descent, utilize convex majorizers in order to
guarantee that the majorizer is easy to minimize. In our work we consider a
natural class of nonconvex majorizers for these functions, and show that these
majorizers are still sufficient for a globally convergent optimization scheme.
Numerical results illustrate that by applying this scheme, one can often obtain
superior local optima compared to previous majorization-minimization methods,
when the nonconvex majorizers are solved to global optimality. Finally, we
illustrate the behavior of our algorithm for depth super-resolution from raw
time-of-flight data.
Large Scale Novel Object Discovery in 3D
Authors: Siddharth Srivastava, Gaurav Sharma, Brejesh lall
Date: 2017-01-22 00:00:00
Tasks: Clustering, Object, Object Discovery

We present a method for discovering never-seen-before objects in 3D point
clouds obtained from sensors like Microsoft Kinect. We generate supervoxels
directly from the point cloud data and use them with a Siamese network, built
on a recently proposed 3D convolutional neural network architecture. We use
known objects to train a non-linear embedding of supervoxels, by optimizing the
criteria that supervoxels which fall on the same object should be closer than
those which fall on different objects, in the embedding space. We test on
unknown objects, which were not seen during training, and perform clustering in
the learned embedding space of supervoxels to effectively perform novel object
discovery. We validate the method with extensive experiments, quantitatively
showing that it can discover numerous unseen objects while being trained on
only a few dense 3D models. We also show very good qualitative results of
object discovery in point cloud data when the test objects, either specific
instances or even categories, were never seen during training.
Novel View Synthesis for Large-scale Scene using Adversarial Loss
Authors: Xiaochuan Yin, Henglai Wei, Penghong Lin, Xiangwei Wang, Qijun Chen
Date: 2018-02-20 00:00:00
Tasks: Novel View Synthesis

Novel view synthesis aims to synthesize new images from different viewpoints
of given images. Most of previous works focus on generating novel views of
certain objects with a fixed background. However, for some applications, such
as virtual reality or robotic manipulations, large changes in background may
occur due to the egomotion of the camera. Generated images of a large-scale
environment from novel views may be distorted if the structure of the
environment is not considered. In this work, we propose a novel fully
convolutional network, that can take advantage of the structural information
explicitly by incorporating the inverse depth features. The inverse depth
features are obtained from CNNs trained with sparse labeled depth values. This
framework can easily fuse multiple images from different viewpoints. To fill
the missing textures in the generated image, adversarial loss is applied, which
can also improve the overall image quality. Our method is evaluated on the
KITTI dataset. The results show that our method can generate novel views of
large-scale scene without distortion. The effectiveness of our approach is
demonstrated through qualitative and quantitative evaluation.
Soft-DTW: a Differentiable Loss Function for Time-Series
Authors: Marco Cuturi, Mathieu Blondel
Date: 2017-03-05 00:00:00
Proceeding: ICML 2017 8
Tasks: Dynamic Time Warping, Time Series, Time Series Analysis

We propose in this paper a differentiable learning loss between time series,
building upon the celebrated dynamic time warping (DTW) discrepancy. Unlike the
Euclidean distance, DTW can compare time series of variable size and is robust
to shifts or dilatations across the time dimension. To compute DTW, one
typically solves a minimal-cost alignment problem between two time series using
dynamic programming. Our work takes advantage of a smoothed formulation of DTW,
called soft-DTW, that computes the soft-minimum of all alignment costs. We show
in this paper that soft-DTW is a differentiable loss function, and that both
its value and gradient can be computed with quadratic time/space complexity
(DTW has quadratic time but linear space complexity). We show that this
regularization is particularly well suited to average and cluster time series
under the DTW geometry, a task for which our proposal significantly outperforms
existing baselines. Next, we propose to tune the parameters of a machine that
outputs time series by minimizing its fit with ground-truth labels in a
soft-DTW sense.
On Learning Causal Structures from Non-Experimental Data without Any Faithfulness Assumption
Authors: Hanti Lin, Jiji Zhang
Date: 2018-02-20 00:00:00

Consider the problem of learning, from non-experimental data, the causal (Markov equivalence) structure of the true, unknown causal Bayesian network (CBN) on a given, fixed set of (categorical) variables. This learning problem is known to be so hard that there is no learning algorithm that converges to the truth for all possible CBNs (on the given set of variables). So the convergence property has to be sacrificed for some CBNs---but for which? In response, the standard practice has been to design and employ learning algorithms that secure the convergence property for at least all the CBNs that satisfy the famous faithfulness condition, which implies sacrificing the convergence property for some CBNs that violate the faithfulness condition (Spirtes et al. 2000). This standard design practice can be justified by assuming---that is, accepting on faith---that the true, unknown CBN satisfies the faithfulness condition. But the real question is this: Is it possible to explain, without assuming the faithfulness condition or any of its weaker variants, why it is mandatory rather than optional to follow the standard design practice? This paper aims to answer the above question in the affirmative. We first define an array of modes of convergence to the truth as desiderata that might or might not be achieved by a causal learning algorithm. Those modes of convergence concern (i) how pervasive the domain of convergence is on the space of all possible CBNs and (ii) how uniformly the convergence happens. Then we prove a result to the following effect: for any learning algorithm that tackles the causal learning problem in question, if it achieves the best achievable mode of convergence (considered in this paper), then it must follow the standard design practice of converging to the truth for at least all CBNs that satisfy the faithfulness condition---it is a requirement, not an option.