pwc-archive/papers-with-abstracts huggingface dataset:
Variational Dropout and the Local Reparameterization Trick
Authors: Diederik P. Kingma, Tim Salimans, Max Welling
Date: 2015-06-08 00:00:00
Proceeding: NeurIPS 2015 12
Tasks: Bayesian Inference

We investigate a local reparameterizaton technique for greatly reducing the
variance of stochastic gradients for variational Bayesian inference (SGVB) of a
posterior over model parameters, while retaining parallelizability. This local
reparameterization translates uncertainty about global parameters into local
noise that is independent across datapoints in the minibatch. Such
parameterizations can be trivially parallelized and have variance that is
inversely proportional to the minibatch size, generally leading to much faster
convergence. Additionally, we explore a connection with dropout: Gaussian
dropout objectives correspond to SGVB with local reparameterization, a
scale-invariant prior and proportionally fixed posterior variance. Our method
allows inference of more flexibly parameterized posteriors; specifically, we
propose variational dropout, a generalization of Gaussian dropout where the
dropout rates are learned, often leading to better models. The method is
demonstrated through several experiments.
Revisiting Differentially Private Regression: Lessons From Learning Theory and their Consequences
Authors: Xi Wu, Matthew Fredrikson, Wentao Wu, Somesh Jha, Jeffrey F. Naughton
Date: 2015-12-20 00:00:00
Tasks: Learning Theory, regression

Private regression has received attention from both database and security
communities. Recent work by Fredrikson et al. (USENIX Security 2014) analyzed
the functional mechanism (Zhang et al. VLDB 2012) for training linear
regression models over medical data. Unfortunately, they found that model
accuracy is already unacceptable with differential privacy when $\varepsilon =
5$. We address this issue, presenting an explicit connection between
differential privacy and stable learning theory through which a substantially
better privacy/utility tradeoff can be obtained. Perhaps more importantly, our
theory reveals that the most basic mechanism in differential privacy, output
perturbation, can be used to obtain a better tradeoff for all
convex-Lipschitz-bounded learning tasks. Since output perturbation is simple to
implement, it means that our approach is potentially widely applicable in
practice. We go on to apply it on the same medical data as used by Fredrikson
et al. Encouragingly, we achieve accurate models even for $\varepsilon = 0.1$.
In the last part of this paper, we study the impact of our improved
differentially private mechanisms on model inversion attacks, a privacy attack
introduced by Fredrikson et al. We observe that the improved tradeoff makes the
resulting differentially private model more susceptible to inversion attacks.
We analyze this phenomenon formally.
Tensorizing Neural Networks
Authors: Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, Dmitry Vetrov
Date: 2015-09-22 00:00:00
Proceeding: NeurIPS 2015 12
Tasks: Image Classification

Deep neural networks currently demonstrate state-of-the-art performance in
several domains. At the same time, models of this class are very demanding in
terms of computational resources. In particular, a large amount of memory is
required by commonly used fully-connected layers, making it hard to use the
models on low-end devices and stopping the further increase of the model size.
In this paper we convert the dense weight matrices of the fully-connected
layers to the Tensor Train format such that the number of parameters is reduced
by a huge factor and at the same time the expressive power of the layer is
preserved. In particular, for the Very Deep VGG networks we report the
compression factor of the dense weight matrix of a fully-connected layer up to
200000 times leading to the compression factor of the whole network up to 7
times.
Kernel principal component analysis network for image classification
Authors: Dan Wu, Jiasong Wu, Rui Zeng, Longyu Jiang, Lotfi Senhadji, Huazhong Shu
Date: 2015-12-20 00:00:00
Tasks: Classification, Face Recognition, General Classification, image-classification, Image Classification, Object Recognition

In order to classify the nonlinear feature with linear classifier and improve
the classification accuracy, a deep learning network named kernel principal
component analysis network (KPCANet) is proposed. First, mapping the data into
higher space with kernel principal component analysis to make the data linearly
separable. Then building a two-layer KPCANet to obtain the principal components
of image. Finally, classifying the principal components with linearly
classifier. Experimental results show that the proposed KPCANet is effective in
face recognition, object recognition and hand-writing digits recognition, it
also outperforms principal component analysis network (PCANet) generally as
well. Besides, KPCANet is invariant to illumination and stable to occlusion and
slight deformation.
Detecting the large entries of a sparse covariance matrix in sub-quadratic time
Authors: Ofer Shwartz, Boaz Nadler
Date: 2015-05-12 00:00:00

The covariance matrix of a $p$-dimensional random variable is a fundamental
quantity in data analysis. Given $n$ i.i.d. observations, it is typically
estimated by the sample covariance matrix, at a computational cost of
$O(np^{2})$ operations. When $n,p$ are large, this computation may be
prohibitively slow. Moreover, in several contemporary applications, the
population matrix is approximately sparse, and only its few large entries are
of interest. This raises the following question, at the focus of our work:
Assuming approximate sparsity of the covariance matrix, can its large entries
be detected much faster, say in sub-quadratic time, without explicitly
computing all its $p^{2}$ entries? In this paper, we present and theoretically
analyze two randomized algorithms that detect the large entries of an
approximately sparse sample covariance matrix using only $O(np\text{ poly log }
p)$ operations. Furthermore, assuming sparsity of the population matrix, we
derive sufficient conditions on the underlying random variable and on the
number of samples $n$, for the sample covariance matrix to satisfy our
approximate sparsity requirements. Finally, we illustrate the performance of
our algorithms via several simulations.
Inference and Mixture Modeling with the Elliptical Gamma Distribution
Authors: Reshad Hosseini, Suvrit Sra, Lucas Theis, Matthias Bethge
Date: 2014-10-17 00:00:00
Tasks: Density Estimation

We study modeling and inference with the Elliptical Gamma Distribution (EGD).
We consider maximum likelihood (ML) estimation for EGD scatter matrices, a task
for which we develop new fixed-point algorithms. Our algorithms are efficient
and converge to global optima despite nonconvexity. Moreover, they turn out to
be much faster than both a well-known iterative algorithm of Kent & Tyler
(1991) and sophisticated manifold optimization algorithms. Subsequently, we
invoke our ML algorithms as subroutines for estimating parameters of a mixture
of EGDs. We illustrate our methods by applying them to model natural image
statistics---the proposed EGD mixture model yields the most parsimonious model
among several competing approaches.
On non-iterative training of a neural classifier
Authors: K. Eswaran, K. Damodhar Rao
Date: 2015-12-14 00:00:00
Tasks: General Classification

Recently an algorithm, was discovered, which separates points in n-dimension
by planes in such a manner that no two points are left un-separated by at least
one plane{[}1-3{]}. By using this new algorithm we show that there are two ways
of classification by a neural network, for a large dimension feature space,
both of which are non-iterative and deterministic. To demonstrate the power of
both these methods we apply them exhaustively to the classical pattern
recognition problem: The Fisher-Anderson's, IRIS flower data set and present
the results.
  It is expected these methods will now be widely used for the training of
neural networks for Deep Learning not only because of their non-iterative and
deterministic nature but also because of their efficiency and speed and will
supersede other classification methods which are iterative in nature and rely
on error minimization.
A dense subgraph based algorithm for compact salient image region detection
Authors: Souradeep Chakraborty, Pabitra Mitra
Date: 2015-11-20 00:00:00
Tasks: Few-Shot Semantic Segmentation

We present an algorithm for graph based saliency computation that utilizes
the underlying dense subgraphs in finding visually salient regions in an image.
To compute the salient regions, the model first obtains a saliency map using
random walks on a Markov chain. Next, k-dense subgraphs are detected to further
enhance the salient regions in the image. Dense subgraphs convey more
information about local graph structure than simple centrality measures. To
generate the Markov chain, intensity and color features of an image in addition
to region compactness is used. For evaluating the proposed model, we do
extensive experiments on benchmark image data sets. The proposed method
performs comparable to well-known algorithms in salient region detection.
Using machine learning for medium frequency derivative portfolio trading
Authors: Abhijit Sharang, Chetan Rao
Date: 2015-12-19 00:00:00
Tasks: BIG-bench Machine Learning, General Classification

We use machine learning for designing a medium frequency trading strategy for
a portfolio of 5 year and 10 year US Treasury note futures. We formulate this
as a classification problem where we predict the weekly direction of movement
of the portfolio using features extracted from a deep belief network trained on
technical indicators of the portfolio constituents. The experimentation shows
that the resulting pipeline is effective in making a profitable trade.
A Robust Transformation-Based Learning Approach Using Ripple Down Rules for Part-of-Speech Tagging
Authors: Dat Quoc Nguyen, Dai Quoc Nguyen, Dang Duc Pham, Son Bao Pham
Date: 2014-12-12 00:00:00
Tasks: Part-Of-Speech Tagging, POS, POS Tagging

In this paper, we propose a new approach to construct a system of
transformation rules for the Part-of-Speech (POS) tagging task. Our approach is
based on an incremental knowledge acquisition method where rules are stored in
an exception structure and new rules are only added to correct the errors of
existing rules; thus allowing systematic control of the interaction between the
rules. Experimental results on 13 languages show that our approach is fast in
terms of training time and tagging speed. Furthermore, our approach obtains
very competitive accuracy in comparison to state-of-the-art POS and
morphological taggers.
Combining patch-based strategies and non-rigid registration-based label fusion methods
Authors: Carlos Platero, M. Carmen Tobar
Date: 2015-12-19 00:00:00

The objective of this study is to develop a patch-based labeling method that
cooperates with a label fusion using non-rigid registrations. We present a
novel patch-based label fusion method, whose selected patches and their weights
are calculated from a combination of similarity measures between patches using
intensity-based distances and labeling-based distances, where a previous
labeling of the target image is inferred through a label fusion method using
non-rigid registrations. These combined similarity measures result in better
selection of the patches, and their weights are more robust, which improves the
segmentation results compared to other label fusion methods, including the
conventional patch-based labeling method. To evaluate the performance and the
robustness of the proposed label fusion method, we employ two available
databases of T1-weighted (T1W) magnetic resonance imaging (MRI) of human
brains. We compare our approach with other label fusion methods in the
automatic hippocampal segmentation from T1W-MRI.
  Our label fusion method yields mean Dice coefficients of 0.847 and 0.798 for
the two databases used with mean times of approximately 180 and 320 seconds,
respectively. The collaboration between the patch-based labeling method and the
label fusion using non-rigid registrations is given in the several levels: (a)
The pre-selection of the patches in the atlases are improved, (b) The weights
of our selected patches are also more robust, (c) our approach imposes
geometrical restrictions, such as shape priors, and (d) the work-flow is very
efficient. We show that the proposed approach is very competitive with respect
to recently reported methods.
A new robust adaptive algorithm for underwater acoustic channel equalization
Authors: Dariush Kari, Muhammed Omer Sayin, Suleyman Serdar Kozat
Date: 2015-12-19 00:00:00

We introduce a novel family of adaptive robust equalizers for highly
challenging underwater acoustic (UWA) channel equalization. Since the
underwater environment is highly non-stationary and subjected to impulsive
noise, we use adaptive filtering techniques based on a relative logarithmic
cost function inspired by the competitive methods from the online learning
literature. To improve the convergence performance of the conventional linear
equalization methods, while mitigating the stability issues, we intrinsically
combine different norms of the error in the cost function, using logarithmic
functions. Hence, we achieve a comparable convergence performance to least mean
fourth (LMF) equalizer, while significantly enhancing the stability performance
in such an adverse communication medium. We demonstrate the performance of our
algorithms through highly realistic experiments performed on accurately
simulated underwater acoustic channels.
Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
Authors: Hao Zhang, Zhiting Hu, Jinliang Wei, Pengtao Xie, Gunhee Kim, Qirong Ho, Eric Xing
Date: 2015-12-19 00:00:00
Tasks: CPU, GPU, Object Recognition

Deep learning (DL) has achieved notable successes in many machine learning
tasks. A number of frameworks have been developed to expedite the process of
designing and training deep neural networks (DNNs), such as Caffe, Torch and
Theano. Currently they can harness multiple GPUs on a single machine, but are
unable to use GPUs that are distributed across multiple machines; as even
average-sized DNNs can take days to train on a single GPU with 100s of GBs to
TBs of data, distributed GPUs present a prime opportunity for scaling up DL.
However, the limited bandwidth available on commodity Ethernet networks
presents a bottleneck to distributed GPU training, and prevents its trivial
realization.
  To investigate how to adapt existing frameworks to efficiently support
distributed GPUs, we propose Poseidon, a scalable system architecture for
distributed inter-machine communication in existing DL frameworks. We integrate
Poseidon with Caffe and evaluate its performance at training DNNs for object
recognition. Poseidon features three key contributions that accelerate DNN
training on clusters: (1) a three-level hybrid architecture that allows
Poseidon to support both CPU-only and GPU-equipped clusters, (2) a distributed
wait-free backpropagation (DWBP) algorithm to improve GPU utilization and to
balance communication, and (3) a structure-aware communication protocol (SACP)
to minimize communication overheads. We empirically show that Poseidon
converges to same objectives as a single machine, and achieves state-of-art
training speedup across multiple models and well-established datasets using a
commodity GPU cluster of 8 nodes (e.g. 4.5x speedup on AlexNet, 4x on
GoogLeNet, 4x on CIFAR-10). On the much larger ImageNet22K dataset, Poseidon
with 8 nodes achieves better speedup and competitive accuracy to recent
CPU-based distributed systems such as Adam and Le et al., which use 10s to
1000s of nodes.
Test-Driven Development of ontologies (extended version)
Authors: C. Maria Keet, Agnieszka Lawrynowicz
Date: 2015-12-19 00:00:00
Tasks: test driven development

Emerging ontology authoring methods to add knowledge to an ontology focus on
ameliorating the validation bottleneck. The verification of the newly added
axiom is still one of trying and seeing what the reasoner says, because a
systematic testbed for ontology authoring is missing. We sought to address this
by introducing the approach of test-driven development for ontology authoring.
We specify 36 generic tests, as TBox queries and TBox axioms tested through
individuals, and structure their inner workings in an `open box'-way, which
cover the OWL 2 DL language features. This is implemented as a Protege plugin
so that one can perform a TDD test as a black box test. We evaluated the two
test approaches on their performance. The TBox queries were faster, and that
effect is more pronounced the larger the ontology is. We provide a general
sequence of a TDD process for ontology engineering as a foundation for a TDD
methodology.
Discriminative Subnetworks with Regularized Spectral Learning for Global-state Network Data
Authors: Xuan Hong Dang, Ambuj K. Singh, Petko Bogdanov, Hongyuan You, Bayyuan Hsu
Date: 2015-12-19 00:00:00
Tasks: Graph Learning

Data mining practitioners are facing challenges from data with network
structure. In this paper, we address a specific class of global-state networks
which comprises of a set of network instances sharing a similar structure yet
having different values at local nodes. Each instance is associated with a
global state which indicates the occurrence of an event. The objective is to
uncover a small set of discriminative subnetworks that can optimally classify
global network values. Unlike most existing studies which explore an
exponential subnetwork space, we address this difficult problem by adopting a
space transformation approach. Specifically, we present an algorithm that
optimizes a constrained dual-objective function to learn a low-dimensional
subspace that is capable of discriminating networks labelled by different
global states, while reconciling with common network topology sharing across
instances. Our algorithm takes an appealing approach from spectral graph
learning and we show that the globally optimum solution can be achieved via
matrix eigen-decomposition.
What is India speaking: The "Hinglish" invasion
Authors: Rana D. Parshad, Vineeta Chand, Neha Sinha, Nitu Kumari
Date: 2014-06-12 00:00:00

While language competition models of diachronic language shift are
increasingly sophisticated, drawing on sociolinguistic components like variable
language prestige, distance from language centers and intermediate bilingual
transitionary populations, in one significant way they fall short. They fail to
consider contact-based outcomes resulting in mixed language practices, e.g.
outcome scenarios such as creoles or unmarked code switching as an emergent
communicative norm. On these lines something very interesting is uncovered in
India, where traditionally there have been monolingual Hindi speakers and
Hindi/English bilinguals, but virtually no monolingual English speakers. While
the Indian census data reports a sharp increase in the proportion of
Hindi/English bilinguals, we argue that the number of Hindi/English bilinguals
in India is inaccurate, given a new class of urban individuals speaking a mixed
lect of Hindi and English, popularly known as "Hinglish". Based on
predator-prey, sociolinguistic theories, salient local ecological factors and
the rural-urban divide in India, we propose a new mathematical model of
interacting monolingual Hindi speakers, Hindi/English bilinguals and Hinglish
speakers. The model yields globally asymptotic stable states of coexistence, as
well as bilingual extinction. To validate our model, sociolinguistic data from
different Indian classes are contrasted with census reports: We see that
purported urban Hindi/English bilinguals are unable to maintain fluent Hindi
speech and instead produce Hinglish, whereas rural speakers evidence
monolingual Hindi. Thus we present evidence for the first time where an
unrecognized mixed lect involving English but not "English", has possibly taken
over a sizeable faction of a large global population.
Lossy Compression via Sparse Linear Regression: Performance under Minimum-distance Encoding
Authors: Ramji Venkataramanan, Antony Joseph, Sekhar Tatikonda
Date: 2012-02-03 00:00:00
Tasks: regression

We study a new class of codes for lossy compression with the squared-error
distortion criterion, designed using the statistical framework of
high-dimensional linear regression. Codewords are linear combinations of
subsets of columns of a design matrix. Called a Sparse Superposition or Sparse
Regression codebook, this structure is motivated by an analogous construction
proposed recently by Barron and Joseph for communication over an AWGN channel.
For i.i.d Gaussian sources and minimum-distance encoding, we show that such a
code can attain the Shannon rate-distortion function with the optimal error
exponent, for all distortions below a specified value. It is also shown that
sparse regression codes are robust in the following sense: a codebook designed
to compress an i.i.d Gaussian source of variance $\sigma^2$ with
(squared-error) distortion $D$ can compress any ergodic source of variance less
than $\sigma^2$ to within distortion $D$. Thus the sparse regression ensemble
retains many of the good covering properties of the i.i.d random Gaussian
ensemble, while having having a compact representation in terms of a matrix
whose size is a low-order polynomial in the block-length.
Bayesian anti-sparse coding
Authors: Clément Elvira, Pierre Chainais, Nicolas Dobigeon
Date: 2015-12-18 00:00:00

Sparse representations have proven their efficiency in solving a wide class
of inverse problems encountered in signal and image processing. Conversely,
enforcing the information to be spread uniformly over representation
coefficients exhibits relevant properties in various applications such as
digital communications. Anti-sparse regularization can be naturally expressed
through an $\ell_{\infty}$-norm penalty. This paper derives a probabilistic
formulation of such representations. A new probability distribution, referred
to as the democratic prior, is first introduced. Its main properties as well as
three random variate generators for this distribution are derived. Then this
probability distribution is used as a prior to promote anti-sparsity in a
Gaussian linear inverse problem, yielding a fully Bayesian formulation of
anti-sparse coding. Two Markov chain Monte Carlo (MCMC) algorithms are proposed
to generate samples according to the posterior distribution. The first one is a
standard Gibbs sampler. The second one uses Metropolis-Hastings moves that
exploit the proximity mapping of the log-posterior distribution. These samples
are used to approximate maximum a posteriori and minimum mean square error
estimators of both parameters and hyperparameters. Simulations on synthetic
data illustrate the performances of the two proposed samplers, for both
complete and over-complete dictionaries. All results are compared to the recent
deterministic variational FITRA algorithm.
Modeling Colors of Single Attribute Variations with Application to Food Appearance
Authors: Yaser Yacoob
Date: 2015-12-18 00:00:00
Tasks: Attribute, Object, object-detection, Object Detection

This paper considers the intra-image color-space of an object or a scene when
these are subject to a dominant single-source of variation. The source of
variation can be intrinsic or extrinsic (i.e., imaging conditions) to the
object. We observe that the quantized colors for such objects typically lie on
a planar subspace of RGB, and in some cases linear or polynomial curves on this
plane are effective in capturing these color variations. We also observe that
the inter-image color sub-spaces are robust as long as drastic illumination
change is not involved.
  We illustrate the use of this analysis for: discriminating between
shading-change and reflectance-change for patches, and object detection,
segmentation and recognition based on a single exemplar. We focus on images of
food items to illustrate the effectiveness of the proposed approach.