pwc-archive/papers-with-abstracts huggingface dataset:
Compatible extensions and consistent closures: a fuzzy approach
Authors: Irina Georgescu
Date: 2017-05-22 00:00:00
Tasks: Relation

In this paper $\ast$--compatible extensions of fuzzy relations are studied,
generalizing some results obtained by Duggan in case of crisp relations. From
this general result are obtained as particular cases fuzzy versions of some
important extension theorems for crisp relations (Szpilrajn, Hansson,
Suzumura). Two notions of consistent closure of a fuzzy relation are
introduced.
Large Scale Empirical Risk Minimization via Truncated Adaptive Newton Method
Authors: Mark Eisen, Aryan Mokhtari, Alejandro Ribeiro
Date: 2017-05-22 00:00:00
Tasks: Second-order methods

We consider large scale empirical risk minimization (ERM) problems, where
both the problem dimension and variable size is large. In these cases, most
second order methods are infeasible due to the high cost in both computing the
Hessian over all samples and computing its inverse in high dimensions. In this
paper, we propose a novel adaptive sample size second-order method, which
reduces the cost of computing the Hessian by solving a sequence of ERM problems
corresponding to a subset of samples and lowers the cost of computing the
Hessian inverse using a truncated eigenvalue decomposition. We show that while
we geometrically increase the size of the training set at each stage, a single
iteration of the truncated Newton method is sufficient to solve the new ERM
within its statistical accuracy. Moreover, for a large number of samples we are
allowed to double the size of the training set at each stage, and the proposed
method subsequently reaches the statistical accuracy of the full training set
approximately after two effective passes. In addition to this theoretical
result, we show empirically on a number of well known data sets that the
proposed truncated adaptive sample size algorithm outperforms stochastic
alternatives for solving ERM problems.
The Physical Systems Behind Optimization Algorithms
Authors: Lin F. Yang, R. Arora, V. Braverman, Tuo Zhao
Date: 2016-12-08 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: BIG-bench Machine Learning

We use differential equations based approaches to provide some {\it
\textbf{physics}} insights into analyzing the dynamics of popular optimization
algorithms in machine learning. In particular, we study gradient descent,
proximal gradient descent, coordinate gradient descent, proximal coordinate
gradient, and Newton's methods as well as their Nesterov's accelerated variants
in a unified framework motivated by a natural connection of optimization
algorithms to physical systems. Our analysis is applicable to more general
algorithms and optimization problems {\it \textbf{beyond}} convexity and strong
convexity, e.g. Polyak-\L ojasiewicz and error bound conditions (possibly
nonconvex).
Facial Affect Estimation in the Wild Using Deep Residual and Convolutional Networks
Authors: Behzad Hasani, Mohammad H. Mahoor
Date: 2017-05-22 00:00:00

Automated affective computing in the wild is a challenging task in the field
of computer vision. This paper presents three neural network-based methods
proposed for the task of facial affect estimation submitted to the First
Affect-in-the-Wild challenge. These methods are based on Inception-ResNet
modules redesigned specifically for the task of facial affect estimation. These
methods are: Shallow Inception-ResNet, Deep Inception-ResNet, and
Inception-ResNet with LSTMs. These networks extract facial features in
different scales and simultaneously estimate both the valence and arousal in
each frame. Root Mean Square Error (RMSE) rates of 0.4 and 0.3 are achieved for
the valence and arousal respectively with corresponding Concordance Correlation
Coefficient (CCC) rates of 0.04 and 0.29 using Deep Inception-ResNet method.
Reducing Reparameterization Gradient Variance
Authors: Andrew C. Miller, Nicholas J. Foti, Alexander D'Amour, Ryan P. Adams
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Variational Inference

Optimization with noisy gradients has become ubiquitous in statistics and
machine learning. Reparameterization gradients, or gradient estimates computed
via the "reparameterization trick," represent a class of noisy gradients often
used in Monte Carlo variational inference (MCVI). However, when these gradient
estimators are too noisy, the optimization procedure can be slow or fail to
converge. One way to reduce noise is to use more samples for the gradient
estimate, but this can be computationally expensive. Instead, we view the noisy
gradient as a random variable, and form an inexpensive approximation of the
generating procedure for the gradient sample. This approximation has high
correlation with the noisy gradient by construction, making it a useful control
variate for variance reduction. We demonstrate our approach on non-conjugate
multi-level hierarchical models and a Bayesian neural net where we observed
gradient variance reductions of multiple orders of magnitude (20-2,000x).
Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks
Authors: Behzad Hasani, Mohammad H. Mahoor
Date: 2017-05-22 00:00:00
Tasks: Facial Expression Recognition, Facial Expression Recognition (FER)

Deep Neural Networks (DNNs) have shown to outperform traditional methods in
various visual recognition tasks including Facial Expression Recognition (FER).
In spite of efforts made to improve the accuracy of FER systems using DNN,
existing methods still are not generalizable enough in practical applications.
This paper proposes a 3D Convolutional Neural Network method for FER in videos.
This new network architecture consists of 3D Inception-ResNet layers followed
by an LSTM unit that together extracts the spatial relations within facial
images as well as the temporal relations between different frames in the video.
Facial landmark points are also used as inputs to our network which emphasize
on the importance of facial components rather than the facial regions that may
not contribute significantly to generating facial expressions. Our proposed
method is evaluated using four publicly available databases in
subject-independent and cross-database tasks and outperforms state-of-the-art
methods.
Sequential Optimization for Efficient High-Quality Object Proposal Generation
Authors: Ziming Zhang, Yun Liu, Xi Chen, Yanjun Zhu, Ming-Ming Cheng, Venkatesh Saligrama, Philip H. S. Torr
Date: 2015-11-14 00:00:00
Tasks: Computational Efficiency, CPU, Object, object-detection, Object Detection, Object Proposal Generation, Vocal Bursts Intensity Prediction

We are motivated by the need for a generic object proposal generation
algorithm which achieves good balance between object detection recall, proposal
localization quality and computational efficiency. We propose a novel object
proposal algorithm, BING++, which inherits the virtue of good computational
efficiency of BING but significantly improves its proposal localization
quality. At high level we formulate the problem of object proposal generation
from a novel probabilistic perspective, based on which our BING++ manages to
improve the localization quality by employing edges and segments to estimate
object boundaries and update the proposals sequentially. We propose learning
the parameters efficiently by searching for approximate solutions in a
quantized parameter space for complexity reduction. We demonstrate the
generalization of BING++ with the same fixed parameters across different object
classes and datasets. Empirically our BING++ can run at half speed of BING on
CPU, but significantly improve the localization quality by 18.5% and 16.7% on
both VOC2007 and Microhsoft COCO datasets, respectively. Compared with other
state-of-the-art approaches, BING++ can achieve comparable performance, but run
significantly faster.
SmartPaste: Learning to Adapt Source Code
Authors: Miltiadis Allamanis, Marc Brockschmidt
Date: 2017-05-22 00:00:00
Tasks: Machine Translation, Program Repair, Text Summarization, Translation

Deep Neural Networks have been shown to succeed at a range of natural
language tasks such as machine translation and text summarization. While tasks
on source code (ie, formal languages) have been considered recently, most work
in this area does not attempt to capitalize on the unique opportunities offered
by its known syntax and structure. In this work, we introduce SmartPaste, a
first task that requires to use such information. The task is a variant of the
program repair problem that requires to adapt a given (pasted) snippet of code
to surrounding, existing source code. As first solutions, we design a set of
deep neural models that learn to represent the context of each variable
location and variable usage in a data flow-sensitive way. Our evaluation
suggests that our models can learn to solve the SmartPaste task in many cases,
achieving 58.6% accuracy, while learning meaningful representation of variable
usages.
On-the-fly Operation Batching in Dynamic Computation Graphs
Authors: Graham Neubig, Yoav Goldberg, Chris Dyer
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2017 12

Dynamic neural network toolkits such as PyTorch, DyNet, and Chainer offer
more flexibility for implementing models that cope with data of varying
dimensions and structure, relative to toolkits that operate on statically
declared computations (e.g., TensorFlow, CNTK, and Theano). However, existing
toolkits - both static and dynamic - require that the developer organize the
computations into the batches necessary for exploiting high-performance
algorithms and hardware. This batching task is generally difficult, but it
becomes a major hurdle as architectures become complex. In this paper, we
present an algorithm, and its implementation in the DyNet toolkit, for
automatically batching operations. Developers simply write minibatch
computations as aggregations of single instance computations, and the batching
algorithm seamlessly executes them, on the fly, using computationally efficient
batched operations. On a variety of tasks, we obtain throughput similar to that
obtained with manual batches, as well as comparable speedups over
single-instance learning on architectures that are impractical to batch
manually.
Real Time Image Saliency for Black Box Classifiers
Authors: Piotr Dabkowski, Yarin Gal
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Saliency Detection

In this work we develop a fast saliency detection method that can be applied
to any differentiable image classifier. We train a masking model to manipulate
the scores of the classifier by masking salient parts of the input image. Our
model generalises well to unseen images and requires a single forward pass to
perform saliency detection, therefore suitable for use in real-time systems. We
test our approach on CIFAR-10 and ImageNet datasets and show that the produced
saliency maps are easily interpretable, sharp, and free of artifacts. We
suggest a new metric for saliency and test our method on the ImageNet object
localisation task. We achieve results outperforming other weakly supervised
methods.
Concrete Dropout
Authors: Yarin Gal, Jiri Hron, Alex Kendall
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Reinforcement Learning, Reinforcement Learning (RL)

Dropout is used as a practical tool to obtain uncertainty estimates in large
vision models and reinforcement learning (RL) tasks. But to obtain
well-calibrated uncertainty estimates, a grid-search over the dropout
probabilities is necessary - a prohibitive operation with large models, and an
impossible one with RL. We propose a new dropout variant which gives improved
performance and better calibrated uncertainties. Relying on recent developments
in Bayesian deep learning, we use a continuous relaxation of dropout's discrete
masks. Together with a principled optimisation objective, this allows for
automatic tuning of the dropout probability in large models, and as a result
faster experimentation cycles. In RL this allows the agent to adapt its
uncertainty dynamically as more data is observed. We analyse the proposed
variant extensively on a range of tasks, and give insights into common practice
in the field where larger dropout probabilities are often used in deeper model
layers.
Stabilizing GAN Training with Multiple Random Projections
Authors: Behnam Neyshabur, Srinadh Bhojanapalli, Ayan Chakrabarti
Date: 2017-05-22 00:00:00
Proceeding: ICLR 2018 1

Training generative adversarial networks is unstable in high-dimensions as
the true data distribution tends to be concentrated in a small fraction of the
ambient space. The discriminator is then quickly able to classify nearly all
generated samples as fake, leaving the generator without meaningful gradients
and causing it to deteriorate after a point in training. In this work, we
propose training a single generator simultaneously against an array of
discriminators, each of which looks at a different random low-dimensional
projection of the data. Individual discriminators, now provided with restricted
views of the input, are unable to reject generated samples perfectly and
continue to provide meaningful gradients to the generator throughout training.
Meanwhile, the generator learns to produce samples consistent with the full
data distribution to satisfy all discriminators simultaneously. We demonstrate
the practical utility of this approach experimentally, and show that it is able
to produce image samples with higher quality than traditional training with a
single discriminator.
TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation
Authors: Li Ding, Chenliang Xu
Date: 2017-05-22 00:00:00
Tasks: Action Segmentation, Decoder, Segmentation

Action segmentation as a milestone towards building automatic systems to
understand untrimmed videos has received considerable attention in the recent
years. It is typically being modeled as a sequence labeling problem but
contains intrinsic and sufficient differences than text parsing or speech
processing. In this paper, we introduce a novel hybrid temporal convolutional
and recurrent network (TricorNet), which has an encoder-decoder architecture:
the encoder consists of a hierarchy of temporal convolutional kernels that
capture the local motion changes of different actions; the decoder is a
hierarchy of recurrent neural networks that are able to learn and memorize
long-term action dependencies after the encoding stage. Our model is simple but
extremely effective in terms of video sequence labeling. The experimental
results on three public action segmentation datasets have shown that the
proposed model achieves superior performance over the state of the art.
Minimax Statistical Learning with Wasserstein Distances
Authors: Jaeho Lee, Maxim Raginsky
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Domain Adaptation, Generalization Bounds

As opposed to standard empirical risk minimization (ERM), distributionally
robust optimization aims to minimize the worst-case risk over a larger
ambiguity set containing the original empirical distribution of the training
data. In this work, we describe a minimax framework for statistical learning
with ambiguity sets given by balls in Wasserstein space. In particular, we
prove generalization bounds that involve the covering number properties of the
original ERM problem. As an illustrative example, we provide generalization
guarantees for transport-based domain adaptation problems where the Wasserstein
distance between the source and target domain distributions can be reliably
estimated from unlabeled samples.
Optimal Multi-Object Segmentation with Novel Gradient Vector Flow Based Shape Priors
Authors: Junjie Bai, Abhay Shah, Xiaodong Wu
Date: 2017-05-22 00:00:00
Tasks: Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation

Shape priors have been widely utilized in medical image segmentation to
improve segmentation accuracy and robustness. A major way to encode such a
prior shape model is to use a mesh representation, which is prone to causing
self-intersection or mesh folding. Those problems require complex and expensive
algorithms to mitigate. In this paper, we propose a novel shape prior directly
embedded in the voxel grid space, based on gradient vector flows of a
pre-segmentation. The flexible and powerful prior shape representation is ready
to be extended to simultaneously segmenting multiple interacting objects with
minimum separation distance constraint. The problem is formulated as a Markov
random field problem whose exact solution can be efficiently computed with a
single minimum s-t cut in an appropriately constructed graph. The proposed
algorithm is validated on two multi-object segmentation applications: the brain
tissue segmentation in MRI images, and the bladder/prostate segmentation in CT
images. Both sets of experiments show superior or competitive performance of
the proposed method to other state-of-the-art methods.
Time Series Structure Discovery via Probabilistic Program Synthesis
Authors: Ulrich Schaechtle, Feras Saad, Alexey Radul, Vikash Mansinghka
Date: 2016-11-21 00:00:00
Tasks: Clustering, Program Synthesis, Time Series, Time Series Analysis, Time Series Clustering

There is a widespread need for techniques that can discover structure from
time series data. Recently introduced techniques such as Automatic Bayesian
Covariance Discovery (ABCD) provide a way to find structure within a single
time series by searching through a space of covariance kernels that is
generated using a simple grammar. While ABCD can identify a broad class of
temporal patterns, it is difficult to extend and can be brittle in practice.
This paper shows how to extend ABCD by formulating it in terms of probabilistic
program synthesis. The key technical ideas are to (i) represent models using
abstract syntax trees for a domain-specific probabilistic language, and (ii)
represent the time series model prior, likelihood, and search strategy using
probabilistic programs in a sufficiently expressive language. The final
probabilistic program is written in under 70 lines of probabilistic code in
Venture. The paper demonstrates an application to time series clustering that
involves a non-parametric extension to ABCD, experiments for interpolation and
extrapolation on real-world econometric data, and improvements in accuracy over
both non-parametric and standard regression baselines.
Solving ill-posed inverse problems using iterative deep neural networks
Authors: Jonas Adler, Ozan Öktem
Date: 2017-04-13 00:00:00
Tasks: GPU

We propose a partially learned approach for the solution of ill posed inverse
problems with not necessarily linear forward operators. The method builds on
ideas from classical regularization theory and recent advances in deep learning
to perform learning while making use of prior information about the inverse
problem encoded in the forward operator, noise model and a regularizing
functional. The method results in a gradient-like iterative scheme, where the
"gradient" component is learned using a convolutional network that includes the
gradients of the data discrepancy and regularizer as input in each iteration.
We present results of such a partially learned gradient scheme on a non-linear
tomographic inversion problem with simulated data from both the Sheep-Logan
phantom as well as a head CT. The outcome is compared against FBP and TV
reconstruction and the proposed method provides a 5.4 dB PSNR improvement over
the TV reconstruction while being significantly faster, giving reconstructions
of 512 x 512 volumes in about 0.4 seconds using a single GPU.
A unified view of entropy-regularized Markov decision processes
Authors: Gergely Neu, Anders Jonsson, Vicenç Gómez
Date: 2017-05-22 00:00:00
Tasks: Policy Gradient Methods, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We propose a general framework for entropy-regularized average-reward
reinforcement learning in Markov decision processes (MDPs). Our approach is
based on extending the linear-programming formulation of policy optimization in
MDPs to accommodate convex regularization functions. Our key result is showing
that using the conditional entropy of the joint state-action distributions as
regularization yields a dual optimization problem closely resembling the
Bellman optimality equations. This result enables us to formalize a number of
state-of-the-art entropy-regularized reinforcement learning algorithms as
approximate variants of Mirror Descent or Dual Averaging, and thus to argue
about the convergence properties of these methods. In particular, we show that
the exact version of the TRPO algorithm of Schulman et al. (2015) actually
converges to the optimal policy, while the entropy-regularized policy gradient
methods of Mnih et al. (2016) may fail to converge to a fixed point. Finally,
we illustrate empirically the effects of using various regularization
techniques on learning performance in a simple reinforcement learning setup.
Accelerated Stochastic Quasi-Newton Optimization on Riemann Manifolds
Authors: Anirban Roychowdhury
Date: 2017-04-06 00:00:00

We propose an L-BFGS optimization algorithm on Riemannian manifolds using
minibatched stochastic variance reduction techniques for fast convergence with
constant step sizes, without resorting to linesearch methods designed to
satisfy Wolfe conditions. We provide a new convergence proof for strongly
convex functions without using curvature conditions on the manifold, as well as
a convergence discussion for nonconvex functions. We discuss a couple of ways
to obtain the correction pairs used to calculate the product of the gradient
with the inverse Hessian, and empirically demonstrate their use in synthetic
experiments on computation of Karcher means for symmetric positive definite
matrices and leading eigenvalues of large scale data matrices. We compare our
method to VR-PCA for the latter experiment, along with Riemannian SVRG for both
cases, and show strong convergence results for a range of datasets.
Robust Localized Multi-view Subspace Clustering
Authors: Yanbo Fan, Jian Liang, Ran He, Bao-Gang Hu, Siwei Lyu
Date: 2017-05-22 00:00:00
Tasks: Clustering, Multi-view Subspace Clustering

In multi-view clustering, different views may have different confidence
levels when learning a consensus representation. Existing methods usually
address this by assigning distinctive weights to different views. However, due
to noisy nature of real-world applications, the confidence levels of samples in
the same view may also vary. Thus considering a unified weight for a view may
lead to suboptimal solutions. In this paper, we propose a novel localized
multi-view subspace clustering model that considers the confidence levels of
both views and samples. By assigning weight to each sample under each view
properly, we can obtain a robust consensus representation via fusing the
noiseless structures among views and samples. We further develop a regularizer
on weight parameters based on the convex conjugacy theory, and samples weights
are determined in an adaptive manner. An efficient iterative algorithm is
developed with a convergence guarantee. Experimental results on four benchmarks
demonstrate the correctness and effectiveness of the proposed model.
Convolutional Networks with MuxOut Layers as Multi-rate Systems for Image Upscaling
Authors: Pablo Navarrete Michelini, Hanwen Liu
Date: 2017-05-22 00:00:00

We interpret convolutional networks as adaptive filters and combine them with
so-called MuxOut layers to efficiently upscale low resolution images. We
formalize this interpretation by deriving a linear and space-variant structure
of a convolutional network when its activations are fixed. We introduce general
purpose algorithms to analyze a network and show its overall filter effect for
each given location. We use this analysis to evaluate two types of image
upscalers: deterministic upscalers that target the recovery of details from
original content; and second, a new generation of upscalers that can sample the
distribution of upscale aliases (images that share the same downscale version)
that look like real content.