pwc-archive/papers-with-abstracts huggingface dataset:
Deep Recurrent Gaussian Process with Variational Sparse Spectrum Approximation
Authors: Roman Föll, Bernard Haasdonk, Markus Hanselmann, Holger Ulmer
Date: 2017-11-02 00:00:00
Tasks: Autonomous Driving, Variational Inference, Weather Forecasting

Modeling sequential data has become more and more important in practice. Some
applications are autonomous driving, virtual sensors and weather forecasting.
To model such systems so called recurrent models are used. In this article we
introduce two new Deep Recurrent Gaussian Process (DRGP) models based on the
Sparse Spectrum Gaussian Process (SSGP) and the improved variational version
called Variational Sparse Spectrum Gaussian Process (VSSGP). We follow the
recurrent structure given by an existing DRGP based on a specific sparse
Nystr\"om approximation. Therefore, we also variationally integrate out the
input-space and hence can propagate uncertainty through the layers. We can show
that for the resulting lower bound an optimal variational distribution exists.
Training is realized through optimizing the variational lower bound. Using
Distributed Variational Inference (DVI), we can reduce the computational
complexity. We improve over current state of the art methods in prediction
accuracy for experimental data-sets used for their evaluation and introduce a
new data-set for engine control, named Emission. Furthermore, our method can
easily be adapted for unsupervised learning, e.g. the latent variable model and
its deep version.
Aggregated Wasserstein Metric and State Registration for Hidden Markov Models
Authors: Yukun Chen, Jianbo Ye, Jia Li
Date: 2017-11-12 00:00:00
Tasks: Retrieval, Time Series Analysis

We propose a framework, named Aggregated Wasserstein, for computing a
dissimilarity measure or distance between two Hidden Markov Models with state
conditional distributions being Gaussian. For such HMMs, the marginal
distribution at any time position follows a Gaussian mixture distribution, a
fact exploited to softly match, aka register, the states in two HMMs. We refer
to such HMMs as Gaussian mixture model-HMM (GMM-HMM). The registration of
states is inspired by the intrinsic relationship of optimal transport and the
Wasserstein metric between distributions. Specifically, the components of the
marginal GMMs are matched by solving an optimal transport problem where the
cost between components is the Wasserstein metric for Gaussian distributions.
The solution of the optimization problem is a fast approximation to the
Wasserstein metric between two GMMs. The new Aggregated Wasserstein distance is
a semi-metric and can be computed without generating Monte Carlo samples. It is
invariant to relabeling or permutation of states. The distance is defined
meaningfully even for two HMMs that are estimated from data of different
dimensionality, a situation that can arise due to missing variables. This
distance quantifies the dissimilarity of GMM-HMMs by measuring both the
difference between the two marginal GMMs and that between the two transition
matrices. Our new distance is tested on tasks of retrieval, classification, and
t-SNE visualization of time series. Experiments on both synthetic and real data
have demonstrated its advantages in terms of accuracy as well as efficiency in
comparison with existing distances based on the Kullback-Leibler divergence.
The destiny of constant structure discrete time closed semantic systems
Authors: Evgeny Ivanko
Date: 2017-11-19 00:00:00

Constant structure closed semantic systems are the systems each element of
which receives its definition through the correspondent unchangeable set of
other elements of the system. Discrete time means here that the definitions of
the elements change iteratively and simultaneously based on the "neighbor
portraits" from the previous iteration. I prove that the iterative redefinition
process in such class of systems will quickly degenerate into a series of
pairwise isomorphic states and discuss some directions of further research.
Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
Authors: Liwei Wang, Alexander G. Schwing, Svetlana Lazebnik
Date: 2017-11-19 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Caption Generation, Image Description

This paper explores image caption generation using conditional variational
auto-encoders (CVAEs). Standard CVAEs with a fixed Gaussian prior yield
descriptions with too little variability. Instead, we propose two models that
explicitly structure the latent space around $K$ components corresponding to
different types of image content, and combine components to create priors for
images that contain multiple types of content simultaneously (e.g., several
kinds of objects). Our first model uses a Gaussian Mixture model (GMM) prior,
while the second one defines a novel Additive Gaussian (AG) prior that linearly
combines component means. We show that both models produce captions that are
more diverse and more accurate than a strong LSTM baseline or a "vanilla" CVAE
with a fixed Gaussian prior, with AG-CVAE showing particular promise.
Prior-aware Dual Decomposition: Document-specific Topic Inference for Spectral Topic Models
Authors: Moontae Lee, David Bindel, David Mimno
Date: 2017-11-19 00:00:00
Tasks: Topic Models

Spectral topic modeling algorithms operate on matrices/tensors of word
co-occurrence statistics to learn topic-specific word distributions. This
approach removes the dependence on the original documents and produces
substantial gains in efficiency and provable topic inference, but at a cost:
the model can no longer provide information about the topic composition of
individual documents. Recently Thresholded Linear Inverse (TLI) is proposed to
map the observed words of each document back to its topic composition. However,
its linear characteristics limit the inference quality without considering the
important prior information over topics. In this paper, we evaluate Simple
Probabilistic Inverse (SPI) method and novel Prior-aware Dual Decomposition
(PADD) that is capable of learning document-specific topic compositions in
parallel. Experiments show that PADD successfully leverages topic correlations
as a prior, notably outperforming TLI and learning quality topic compositions
comparable to Gibbs sampling on various data.
A Zero-Positive Learning Approach for Diagnosing Software Performance Regressions
Authors: Mejbah Alam, Justin Gottschlich, Nesime Tatbul, Javier Turek, Timothy Mattson, Abdullah Muzahid
Date: 2017-09-21 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: regression

The field of machine programming (MP), the automation of the development of software, is making notable research advances. This is, in part, due to the emergence of a wide range of novel techniques in machine learning. In this paper, we apply MP to the automation of software performance regression testing. A performance regression is a software performance degradation caused by a code change. We present AutoPerf - a novel approach to automate regression testing that utilizes three core techniques: (i) zero-positive learning, (ii) autoencoders, and (iii) hardware telemetry. We demonstrate AutoPerf's generality and efficacy against 3 types of performance regressions across 10 real performance bugs in 7 benchmark and open-source programs. On average, AutoPerf exhibits 4% profiling overhead and accurately diagnoses more performance bugs than prior state-of-the-art approaches. Thus far, AutoPerf has produced no false negatives.
A Transfer-Learning Approach for Accelerated MRI using Deep Neural Networks
Authors: Salman Ul Hassan Dar, Muzaffer Özbey, Ahmet Burak Çatlı, Tolga Çukur
Date: 2017-10-07 00:00:00
Tasks: 4k, MRI Reconstruction, Transfer Learning

Purpose: Neural networks have received recent interest for reconstruction of undersampled MR acquisitions. Ideally network performance should be optimized by drawing the training and testing data from the same domain. In practice, however, large datasets comprising hundreds of subjects scanned under a common protocol are rare. The goal of this study is to introduce a transfer-learning approach to address the problem of data scarcity in training deep networks for accelerated MRI. Methods: Neural networks were trained on thousands of samples from public datasets of either natural images or brain MR images. The networks were then fine-tuned using only few tens of brain MR images in a distinct testing domain. Domain-transferred networks were compared to networks trained directly in the testing domain. Network performance was evaluated for varying acceleration factors (2-10), number of training samples (0.5-4k) and number of fine-tuning samples (0-100). Results: The proposed approach achieves successful domain transfer between MR images acquired with different contrasts (T1- and T2-weighted images), and between natural and MR images (ImageNet and T1- or T2-weighted images). Networks obtained via transfer-learning using only tens of images in the testing domain achieve nearly identical performance to networks trained directly in the testing domain using thousands of images. Conclusion: The proposed approach might facilitate the use of neural networks for MRI reconstruction without the need for collection of extensive imaging datasets.
Linear Maximum Margin Classifier for Learning from Uncertain Data
Authors: Christos Tzelepis, Vasileios Mezaris, Ioannis Patras
Date: 2015-04-15 00:00:00

In this paper, we propose a maximum margin classifier that deals with
uncertainty in data input. More specifically, we reformulate the SVM framework
such that each training example can be modeled by a multi-dimensional Gaussian
distribution described by its mean vector and its covariance matrix -- the
latter modeling the uncertainty. We address the classification problem and
define a cost function that is the expected value of the classical SVM cost
when data samples are drawn from the multi-dimensional Gaussian distributions
that form the set of the training examples. Our formulation approximates the
classical SVM formulation when the training examples are isotropic Gaussians
with variance tending to zero. We arrive at a convex optimization problem,
which we solve efficiently in the primal form using a stochastic gradient
descent approach. The resulting classifier, which we name SVM with Gaussian
Sample Uncertainty (SVM-GSU), is tested on synthetic data and five publicly
available and popular datasets; namely, the MNIST, WDBC, DEAP, TV News Channel
Commercial Detection, and TRECVID MED datasets. Experimental results verify the
effectiveness of the proposed method.
Deep learning for inferring cause of data anomalies
Authors: V. Azzolini, M. Borisyak, G. Cerminara, D. Derkach, G. Franzoni, F. De Guio, O. Koval, M. Pierini, A. Pol, F. Ratnikov, F. Siroky, A. Ustyuzhanin, J-R. Vlimant
Date: 2017-11-19 00:00:00
Tasks: Deep Learning

Daily operation of a large-scale experiment is a resource consuming task,
particularly from perspectives of routine data quality monitoring. Typically,
data comes from different sub-detectors and the global quality of data depends
on the combinatorial performance of each of them. In this paper, the problem of
identifying channels in which anomalies occurred is considered. We introduce a
generic deep learning model and prove that, under reasonable assumptions, the
model learns to identify 'channels' which are affected by an anomaly. Such
model could be used for data quality manager cross-check and assistance and
identifying good channels in anomalous data samples. The main novelty of the
method is that the model does not require ground truth labels for each channel,
only global flag is used. This effectively distinguishes the model from
classical classification methods. Being applied to CMS data collected in the
year 2010, this approach proves its ability to decompose anomaly by separate
channels.
A Classifying Variational Autoencoder with Application to Polyphonic Music Generation
Authors: Jay A. Hennig, Akash Umakantha, Ryan C. Williamson
Date: 2017-11-19 00:00:00
Tasks: Music Generation

The variational autoencoder (VAE) is a popular probabilistic generative
model. However, one shortcoming of VAEs is that the latent variables cannot be
discrete, which makes it difficult to generate data from different modes of a
distribution. Here, we propose an extension of the VAE framework that
incorporates a classifier to infer the discrete class of the modeled data. To
model sequential data, we can combine our Classifying VAE with a recurrent
neural network such as an LSTM. We apply this model to algorithmic music
generation, where our model learns to generate musical sequences in different
keys. Most previous work in this area avoids modeling key by transposing data
into only one or two keys, as opposed to the 10+ different keys in the original
music. We show that our Classifying VAE and Classifying VAE+LSTM models
outperform the corresponding non-classifying models in generating musical
samples that stay in key. This benefit is especially apparent when trained on
untransposed music data in the original keys.
An Improved Oscillating-Error Classifier with Branching
Authors: Kieran Greer
Date: 2017-11-19 00:00:00

This paper extends the earlier work on an oscillating error correction technique. Specifically, it extends the design to include further corrections, by adding new layers to the classifier through a branching method. This technique is still consistent with earlier work and also neural networks in general. With this extended design, the classifier can now achieve the high levels of accuracy reported previously.
Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees
Authors: Francesco Locatello, Michael Tschannen, Gunnar Rätsch, Martin Jaggi
Date: 2017-05-31 00:00:00
Proceeding: NeurIPS 2017 12

Greedy optimization methods such as Matching Pursuit (MP) and Frank-Wolfe
(FW) algorithms regained popularity in recent years due to their simplicity,
effectiveness and theoretical guarantees. MP and FW address optimization over
the linear span and the convex hull of a set of atoms, respectively. In this
paper, we consider the intermediate case of optimization over the convex cone,
parametrized as the conic hull of a generic atom set, leading to the first
principled definitions of non-negative MP algorithms for which we give explicit
convergence rates and demonstrate excellent empirical performance. In
particular, we derive sublinear ($\mathcal{O}(1/t)$) convergence on general
smooth and convex objectives, and linear convergence ($\mathcal{O}(e^{-t})$) on
strongly convex objectives, in both cases for general sets of atoms.
Furthermore, we establish a clear correspondence of our algorithms to known
algorithms from the MP and FW literature. Our novel algorithms and analyses
target general atom sets and general objective functions, and hence are
directly applicable to a large variety of learning settings.
Understanding Kernel Size in Blind Deconvolution
Authors: Li Si-Yao, Dongwei Ren, Qian Yin
Date: 2017-06-06 00:00:00
Tasks: Deblurring

Most blind deconvolution methods usually pre-define a large kernel size to
guarantee the support domain. Blur kernel estimation error is likely to be
introduced, yielding severe artifacts in deblurring results. In this paper, we
first theoretically and experimentally analyze the mechanism to estimation
error in oversized kernel, and show that it holds even on blurry images without
noises. Then to suppress this adverse effect, we propose a low rank-based
regularization on blur kernel to exploit the structural information in degraded
kernels, by which larger-kernel effect can be effectively suppressed. And we
propose an efficient optimization algorithm to solve it. Experimental results
on benchmark datasets show that the proposed method is comparable with the
state-of-the-arts by accordingly setting proper kernel size, and performs much
better in handling larger-size kernels quantitatively and qualitatively. The
deblurring results on real-world blurry images further validate the
effectiveness of the proposed method.
Vision Recognition using Discriminant Sparse Optimization Learning
Authors: Qingxiang Feng, Yicong Zhou
Date: 2017-11-19 00:00:00

To better select the correct training sample and obtain the robust
representation of the query sample, this paper proposes a discriminant-based
sparse optimization learning model. This learning model integrates discriminant
and sparsity together. Based on this model, we then propose a classifier called
locality-based discriminant sparse representation (LDSR). Because discriminant
can help to increase the difference of samples in different classes and to
decrease the difference of samples within the same class, LDSR can obtain
better sparse coefficients and constitute a better sparse representation for
classification. In order to take advantages of kernel techniques, discriminant
and sparsity, we further propose a nonlinear classifier called kernel
locality-based discriminant sparse representation (KLDSR). Experiments on
several well-known databases prove that the performance of LDSR and KLDSR is
better than that of several state-of-the-art methods including deep learning
based methods.
MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Face Images
Authors: İlke Çuğu, Eren Şener, Emre Akbaş
Date: 2017-11-19 00:00:00
Tasks: CPU, Facial Expression Recognition, Facial Expression Recognition (FER), Knowledge Distillation, Model Compression

This paper is aimed at creating extremely small and fast convolutional neural networks (CNN) for the problem of facial expression recognition (FER) from frontal face images. To this end, we employed the popular knowledge distillation (KD) method and identified two major shortcomings with its use: 1) a fine-grained grid search is needed for tuning the temperature hyperparameter and 2) to find the optimal size-accuracy balance, one needs to search for the final network size (or the compression rate). On the other hand, KD is proved to be useful for model compression for the FER problem, and we discovered that its effects gets more and more significant with the decreasing model size. In addition, we hypothesized that translation invariance achieved using max-pooling layers would not be useful for the FER problem as the expressions are sensitive to small, pixel-wise changes around the eye and the mouth. However, we have found an intriguing improvement on generalization when max-pooling is used. We conducted experiments on two widely-used FER datasets, CK+ and Oulu-CASIA. Our smallest model (MicroExpNet), obtained using knowledge distillation, is less than 1MB in size and works at 1851 frames per second on an Intel i7 CPU. Despite being less accurate than the state-of-the-art, MicroExpNet still provides significant insights for designing a microarchitecture for the FER problem.
Convergence Analysis of the Dynamics of a Special Kind of Two-Layered Neural Networks with $\ell_1$ and $\ell_2$ Regularization
Authors: Zhifeng Kong
Date: 2017-11-19 00:00:00

In this paper, we made an extension to the convergence analysis of the
dynamics of two-layered bias-free networks with one $ReLU$ output. We took into
consideration two popular regularization terms: the $\ell_1$ and $\ell_2$ norm
of the parameter vector $w$, and added it to the square loss function with
coefficient $\lambda/2$. We proved that when $\lambda$ is small, the weight
vector $w$ converges to the optimal solution $\hat{w}$ (with respect to the new
loss function) with probability $\geq (1-\varepsilon)(1-A_d)/2$ under random
initiations in a sphere centered at the origin, where $\varepsilon$ is a small
value and $A_d$ is a constant. Numerical experiments including phase diagrams
and repeated simulations verified our theory.
Kill Two Birds with One Stone: Weakly-Supervised Neural Network for Image Annotation and Tag Refinement
Authors: Jun-Jie Zhang, Qi Wu, Jian Zhang, Chunhua Shen, Jianfeng Lu
Date: 2017-11-19 00:00:00
Tasks: Retrieval, TAG

The number of social images has exploded by the wide adoption of social
networks, and people like to share their comments about them. These comments
can be a description of the image, or some objects, attributes, scenes in it,
which are normally used as the user-provided tags. However, it is well-known
that user-provided tags are incomplete and imprecise to some extent. Directly
using them can damage the performance of related applications, such as the
image annotation and retrieval. In this paper, we propose to learn an image
annotation model and refine the user-provided tags simultaneously in a
weakly-supervised manner. The deep neural network is utilized as the image
feature learning and backbone annotation model, while visual consistency,
semantic dependency, and user-error sparsity are introduced as the constraints
at the batch level to alleviate the tag noise. Therefore, our model is highly
flexible and stable to handle large-scale image sets. Experimental results on
two benchmark datasets indicate that our proposed model achieves the best
performance compared to the state-of-the-art methods.
Sequential Randomized Matrix Factorization for Gaussian Processes: Efficient Predictions and Hyper-parameter Optimization
Authors: Shaunak D. Bopardikar, George S. Eskander Ekladious
Date: 2017-11-19 00:00:00
Tasks: Computational Efficiency, Gaussian Processes

This paper presents a sequential randomized lowrank matrix factorization
approach for incrementally predicting values of an unknown function at test
points using the Gaussian Processes framework. It is well-known that in the
Gaussian processes framework, the computational bottlenecks are the inversion
of the (regularized) kernel matrix and the computation of the hyper-parameters
defining the kernel. The main contributions of this paper are two-fold. First,
we formalize an approach to compute the inverse of the kernel matrix using
randomized matrix factorization algorithms in a streaming scenario, i.e., data
is generated incrementally over time. The metrics of accuracy and computational
efficiency of the proposed method are compared against a batch approach based
on use of randomized matrix factorization and an existing streaming approach
based on approximating the Gaussian process by a finite set of basis vectors.
Second, we extend the sequential factorization approach to a class of kernel
functions for which the hyperparameters can be efficiently optimized. All
results are demonstrated on two publicly available datasets.