pwc-archive/papers-with-abstracts huggingface dataset:
Learning Mixtures of Gaussians in High Dimensions
Authors: Rong Ge, Qingqing Huang, Sham M. Kakade
Date: 2015-03-02 00:00:00
Tasks: Learning Theory, Vocal Bursts Intensity Prediction

Efficiently learning mixture of Gaussians is a fundamental problem in
statistics and learning theory. Given samples coming from a random one out of k
Gaussian distributions in Rn, the learning problem asks to estimate the means
and the covariance matrices of these Gaussians. This learning problem arises in
many areas ranging from the natural sciences to the social sciences, and has
also found many machine learning applications. Unfortunately, learning mixture
of Gaussians is an information theoretically hard problem: in order to learn
the parameters up to a reasonable accuracy, the number of samples required is
exponential in the number of Gaussian components in the worst case. In this
work, we show that provided we are in high enough dimensions, the class of
Gaussian mixtures is learnable in its most general form under a smoothed
analysis framework, where the parameters are randomly perturbed from an
adversarial starting point. In particular, given samples from a mixture of
Gaussians with randomly perturbed parameters, when n > {\Omega}(k^2), we give
an algorithm that learns the parameters with polynomial running time and using
polynomial number of samples. The central algorithmic ideas consist of new ways
to decompose the moment tensor of the Gaussian mixture by exploiting its
structural properties. The symmetries of this tensor are derived from the
combinatorial structure of higher order moments of Gaussian distributions
(sometimes referred to as Isserlis' theorem or Wick's theorem). We also develop
new tools for bounding smallest singular values of structured random matrices,
which could be useful in other smoothed analysis settings.
Large-Scale Distributed Bayesian Matrix Factorization using Stochastic Gradient MCMC
Authors: Sungjin Ahn, Anoop Korattikara, Nathan Liu, Suju Rajan, Max Welling
Date: 2015-03-05 00:00:00
Tasks: Prediction

Despite having various attractive qualities such as high prediction accuracy
and the ability to quantify uncertainty and avoid over-fitting, Bayesian Matrix
Factorization has not been widely adopted because of the prohibitive cost of
inference. In this paper, we propose a scalable distributed Bayesian matrix
factorization algorithm using stochastic gradient MCMC. Our algorithm, based on
Distributed Stochastic Gradient Langevin Dynamics, can not only match the
prediction accuracy of standard MCMC methods like Gibbs sampling, but at the
same time is as fast and simple as stochastic gradient descent. In our
experiments, we show that our algorithm can achieve the same level of
prediction accuracy as Gibbs sampling an order of magnitude faster. We also
show that our method reduces the prediction error as fast as distributed
stochastic gradient descent, achieving a 4.1% improvement in RMSE for the
Netflix dataset and an 1.8% for the Yahoo music dataset.
On the Intrinsic Limits to Representationally-Adaptive Machine-Learning
Authors: David Windridge
Date: 2015-03-09 00:00:00
Tasks: BIG-bench Machine Learning, Transfer Learning

Online learning is a familiar problem setting within Machine-Learning in
which data is presented serially in time to a learning agent, requiring it to
progressively adapt within the constraints of the learning algorithm. More
sophisticated variants may involve concepts such as transfer-learning which
increase this adaptive capability, enhancing the learner's cognitive capacities
in a manner that can begin to imitate the open-ended learning capabilities of
human beings.
  We shall argue in this paper, however, that a full realization of this notion
requires that, in addition to the capacity to adapt to novel data, autonomous
online learning must ultimately incorporate the capacity to update its own
representational capabilities in relation to the data. We therefore enquire
about the philosophical limits of this process, and argue that only fully
embodied learners exhibiting an a priori perception-action link in order to
ground representational adaptations are capable of exhibiting the full range of
human cognitive capability.
Sublinear-Time Approximate MCMC Transitions for Probabilistic Programs
Authors: Yutian Chen, Vikash Mansinghka, Zoubin Ghahramani
Date: 2014-11-06 00:00:00
Tasks: parameter estimation, Probabilistic Programming, State Estimation, State Space Models

Probabilistic programming languages can simplify the development of machine
learning techniques, but only if inference is sufficiently scalable.
Unfortunately, Bayesian parameter estimation for highly coupled models such as
regressions and state-space models still scales poorly; each MCMC transition
takes linear time in the number of observations. This paper describes a
sublinear-time algorithm for making Metropolis-Hastings (MH) updates to latent
variables in probabilistic programs. The approach generalizes recently
introduced approximate MH techniques: instead of subsampling data items assumed
to be independent, it subsamples edges in a dynamically constructed graphical
model. It thus applies to a broader class of problems and interoperates with
other general-purpose inference techniques. Empirical results, including
confirmation of sublinear per-transition scaling, are presented for Bayesian
logistic regression, nonlinear classification via joint Dirichlet process
mixtures, and parameter estimation for stochastic volatility models (with state
estimation via particle MCMC). All three applications use the same
implementation, and each requires under 20 lines of probabilistic code.
Distilling the Knowledge in a Neural Network
Authors: Geoffrey Hinton, Oriol Vinyals, Jeff Dean
Date: 2015-03-09 00:00:00
Tasks: Knowledge Distillation, Mixture-of-Experts

A very simple way to improve the performance of almost any machine learning
algorithm is to train many different models on the same data and then to
average their predictions. Unfortunately, making predictions using a whole
ensemble of models is cumbersome and may be too computationally expensive to
allow deployment to a large number of users, especially if the individual
models are large neural nets. Caruana and his collaborators have shown that it
is possible to compress the knowledge in an ensemble into a single model which
is much easier to deploy and we develop this approach further using a different
compression technique. We achieve some surprising results on MNIST and we show
that we can significantly improve the acoustic model of a heavily used
commercial system by distilling the knowledge in an ensemble of models into a
single model. We also introduce a new type of ensemble composed of one or more
full models and many specialist models which learn to distinguish fine-grained
classes that the full models confuse. Unlike a mixture of experts, these
specialist models can be trained rapidly and in parallel.
Brain Tumor Segmentation: A Comparative Analysis
Authors: Muhammad Ali Qadar, Yan Zhaowen
Date: 2015-03-09 00:00:00
Tasks: Brain Tumor Segmentation, Image Segmentation, Segmentation, Semantic Segmentation, Tumor Segmentation

Five different threshold segmentation based approaches have been reviewed and
compared over here to extract the tumor from set of brain images. This research
focuses on the analysis of image segmentation methods, a comparison of five
semi-automated methods have been undertaken for evaluating their relative
performance in the segmentation of tumor. Consequently, results are compared on
the basis of quantitative and qualitative analysis of respective methods. The
purpose of this study was to analytically identify the methods, most suitable
for application for a particular genre of problems. The results show that of
the region growing segmentation performed better than rest in most cases.
Structured Prediction of Sequences and Trees using Infinite Contexts
Authors: Ehsan Shareghi, Gholamreza Haffari, Trevor Cohn, Ann Nicholson
Date: 2015-03-09 00:00:00
Tasks: Part-Of-Speech Tagging, Structured Prediction

Linguistic structures exhibit a rich array of global phenomena, however
commonly used Markov models are unable to adequately describe these phenomena
due to their strong locality assumptions. We propose a novel hierarchical model
for structured prediction over sequences and trees which exploits global
context by conditioning each generation decision on an unbounded context of
prior decisions. This builds on the success of Markov models but without
imposing a fixed bound in order to better represent global phenomena. To
facilitate learning of this large and unbounded model, we use a hierarchical
Pitman-Yor process prior which provides a recursive form of smoothing. We
propose prediction algorithms based on A* and Markov Chain Monte Carlo
sampling. Empirical results demonstrate the potential of our model compared to
baseline finite-context Markov models on part-of-speech tagging and syntactic
parsing.
Deep Learning and the Information Bottleneck Principle
Authors: Naftali Tishby, Noga Zaslavsky
Date: 2015-03-09 00:00:00
Tasks: Deep Learning, Generalization Bounds

Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the
information bottleneck (IB) principle. We first show that any DNN can be
quantified by the mutual information between the layers and the input and
output variables. Using this representation we can calculate the optimal
information theoretic limits of the DNN and obtain finite sample generalization
bounds. The advantage of getting closer to the theoretical limit is
quantifiable both by the generalization bound and by the network's simplicity.
We argue that both the optimal architecture, number of layers and
features/connections at each layer, are related to the bifurcation points of
the information bottleneck tradeoff, namely, relevant compression of the input
layer with respect to the output layer. The hierarchical representations at the
layered network naturally correspond to the structural phase transitions along
the information curve. We believe that this new insight can lead to new
optimality bounds and deep learning algorithms.
Belief-Rule-Based Expert Systems for Evaluation of E- Government: A Case Study
Authors: Shahadat Hossein, Par-Ola Zander, Md. Kamal, Linkon Chowdhury
Date: 2014-03-22 00:00:00
Tasks: Decision Making

Little knowledge exists on the impact and results associated with
e-government projects in many specific use domains. Therefore it is necessary
to evaluate the efficiency and effectiveness of e-government systems. Since the
development of e-government is a continuous process of improvement, it requires
continuous evaluation of the overall e-government system as well as evaluation
of its various dimensions such as determinants, characteristics and results.
E-government development is often complex with multiple stakeholders, large
user bases and complex goals. Consequently, even experts have difficulties in
evaluating these systems, especially in an integrated and comprehensive way as
well as on an aggregate level. Expert systems are a candidate solution to
evaluate such complex e-government systems. However, it is difficult for expert
systems to cope with uncertain evaluation data that are vague, inconsistent,
highly subjective or in other ways challenging to formalize. This paper
presents an approach that can handle uncertainty in e-government evaluation:
The combination of Belief Rule Base (BRB) knowledge representation and
Evidential Reasoning (ES). This approach is illustrated with a concrete
prototype, known as Belief Rule Based Expert System (BRBES) and put to use in
the local e-government of Bangladesh. The results have been compared with a
recently developed method of evaluating e-Government, and it is shown that the
results of BRBES are more accurate and reliable. BRBES can be used to identify
the factors that need to be improved to achieve the overall aim of an
e-government project. In addition, various "what if" scenarios can be generated
and developers and managers can get a forecast of the outcomes. In this way,
the system can be used to facilitate decision making processes under
uncertainty.
Deep Human Parsing with Active Template Regression
Authors: Xiaodan Liang, Si Liu, Xiaohui Shen, Jianchao Yang, Luoqi Liu, Jian Dong, Liang Lin, Shuicheng Yan
Date: 2015-03-09 00:00:00
Tasks: Human Parsing, Position, regression

In this work, the human parsing task, namely decomposing a human image into
semantic fashion/body regions, is formulated as an Active Template Regression
(ATR) problem, where the normalized mask of each fashion/body item is expressed
as the linear combination of the learned mask templates, and then morphed to a
more precise mask with the active shape parameters, including position, scale
and visibility of each semantic region. The mask template coefficients and the
active shape parameters together can generate the human parsing results, and
are thus called the structure outputs for human parsing. The deep Convolutional
Neural Network (CNN) is utilized to build the end-to-end relation between the
input human image and the structure outputs for human parsing. More
specifically, the structure outputs are predicted by two separate networks. The
first CNN network is with max-pooling, and designed to predict the template
coefficients for each label mask, while the second CNN network is without
max-pooling to preserve sensitivity to label mask position and accurately
predict the active shape parameters. For a new image, the structure outputs of
the two networks are fused to generate the probability of each label for each
pixel, and super-pixel smoothing is finally used to refine the human parsing
result. Comprehensive evaluations on a large dataset well demonstrate the
significant superiority of the ATR framework over other state-of-the-arts for
human parsing. In particular, the F1-score reaches $64.38\%$ by our ATR
framework, significantly higher than $44.76\%$ based on the state-of-the-art
algorithm.
Persistent Homology in Sparse Regression and Its Application to Brain Morphometry
Authors: Moo. K. Chung, Jamie L. Hanson, Jieping Ye, Richard J. Davidson, Seth D. Pollak
Date: 2014-08-31 00:00:00
Tasks: Brain Morphometry, regression

Sparse systems are usually parameterized by a tuning parameter that
determines the sparsity of the system. How to choose the right tuning parameter
is a fundamental and difficult problem in learning the sparse system. In this
paper, by treating the the tuning parameter as an additional dimension,
persistent homological structures over the parameter space is introduced and
explored. The structures are then further exploited in speeding up the
computation using the proposed soft-thresholding technique. The topological
structures are further used as multivariate features in the tensor-based
morphometry (TBM) in characterizing white matter alterations in children who
have experienced severe early life stress and maltreatment. These analyses
reveal that stress-exposed children exhibit more diffuse anatomical
organization across the whole white matter region.
Mathematical understanding of detailed balance condition violation and its application to Langevin dynamics
Authors: M. Ohzeki, A. Ichiki
Date: 2015-03-09 00:00:00

We develop an efficient sampling method by simulating Langevin dynamics with
an artificial force rather than a natural force by using the gradient of the
potential energy. The standard technique for sampling following the
predetermined distribution such as the Gibbs-Boltzmann one is performed under
the detailed balance condition. In the present study, we propose a modified
Langevin dynamics violating the detailed balance condition on the
transition-probability formulation. We confirm that the numerical
implementation of the proposed method actually demonstrates two major
beneficial improvements: acceleration of the relaxation to the predetermined
distribution and reduction of the correlation time between two different
realizations in the steady state.
Fully Connected Deep Structured Networks
Authors: Alexander G. Schwing, Raquel Urtasun
Date: 2015-03-09 00:00:00
Tasks: General Classification, image-classification, Image Classification, Image Segmentation, Lesion Segmentation, object-detection, Object Detection, Segmentation, Semantic Segmentation

Convolutional neural networks with many layers have recently been shown to
achieve excellent results on many high-level tasks such as image
classification, object detection and more recently also semantic segmentation.
Particularly for semantic segmentation, a two-stage procedure is often
employed. Hereby, convolutional networks are trained to provide good local
pixel-wise features for the second step being traditionally a more global
graphical model. In this work we unify this two-stage process into a single
joint training algorithm. We demonstrate our method on the semantic image
segmentation task and show encouraging results on the challenging PASCAL VOC
2012 dataset.
An Unsupervised Method for Uncovering Morphological Chains
Authors: Karthik Narasimhan, Regina Barzilay, Tommi Jaakkola
Date: 2015-03-08 00:00:00
Proceeding: TACL 2015 1
Tasks: Morphological Analysis

Most state-of-the-art systems today produce morphological analysis based only
on orthographic patterns. In contrast, we propose a model for unsupervised
morphological analysis that integrates orthographic and semantic views of
words. We model word formation in terms of morphological chains, from base
words to the observed words, breaking the chains into parent-child relations.
We use log-linear models with morpheme and word-level features to predict
possible parents, including their modifications, for each word. The limited set
of candidate parents for each word render contrastive estimation feasible. Our
model consistently matches or outperforms five state-of-the-art systems on
Arabic, English and Turkish.
Fully Convolutional Networks for Semantic Segmentation
Authors: Jonathan Long, Evan Shelhamer, Trevor Darrell
Date: 2014-11-14 00:00:00
Proceeding: CVPR 2015 6
Tasks: Crack Segmentation, Multispectral Object Detection, Multi-tissue Nucleus Segmentation, Segmentation, Semantic Segmentation, Thermal Image Segmentation

Convolutional networks are powerful visual models that yield hierarchies of
features. We show that convolutional networks by themselves, trained
end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic
segmentation. Our key insight is to build "fully convolutional" networks that
take input of arbitrary size and produce correspondingly-sized output with
efficient inference and learning. We define and detail the space of fully
convolutional networks, explain their application to spatially dense prediction
tasks, and draw connections to prior models. We adapt contemporary
classification networks (AlexNet, the VGG net, and GoogLeNet) into fully
convolutional networks and transfer their learned representations by
fine-tuning to the segmentation task. We then define a novel architecture that
combines semantic information from a deep, coarse layer with appearance
information from a shallow, fine layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves state-of-the-art
segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012),
NYUDv2, and SIFT Flow, while inference takes one third of a second for a
typical image.
TuPAQ: An Efficient Planner for Large-scale Predictive Analytic Queries
Authors: Evan R. Sparks, Ameet Talwalkar, Michael J. Franklin, Michael. I. Jordan, Tim Kraska
Date: 2015-01-31 00:00:00

The proliferation of massive datasets combined with the development of
sophisticated analytical techniques have enabled a wide variety of novel
applications such as improved product recommendations, automatic image tagging,
and improved speech-driven interfaces. These and many other applications can be
supported by Predictive Analytic Queries (PAQs). A major obstacle to supporting
PAQs is the challenging and expensive process of identifying and training an
appropriate predictive model. Recent efforts aiming to automate this process
have focused on single node implementations and have assumed that model
training itself is a black box, thus limiting the effectiveness of such
approaches on large-scale problems. In this work, we build upon these recent
efforts and propose an integrated PAQ planning architecture that combines
advanced model search techniques, bandit resource allocation via runtime
algorithm introspection, and physical optimization via batching. The result is
TuPAQ, a component of the MLbase system, which solves the PAQ planning problem
with comparable quality to exhaustive strategies but an order of magnitude more
efficiently than the standard baseline approach, and can scale to models
trained on terabytes of data across hundreds of machines.
Fitting 3D Morphable Models using Local Features
Authors: Patrik Huber, Zhen-Hua Feng, William Christmas, Josef Kittler, Matthias Rätsch
Date: 2015-03-08 00:00:00
Tasks: regression

In this paper, we propose a novel fitting method that uses local image
features to fit a 3D Morphable Model to 2D images. To overcome the obstacle of
optimising a cost function that contains a non-differentiable feature
extraction operator, we use a learning-based cascaded regression method that
learns the gradient direction from data. The method allows to simultaneously
solve for shape and pose parameters. Our method is thoroughly evaluated on
Morphable Model generated data and first results on real data are presented.
Compared to traditional fitting methods, which use simple raw features like
pixel colour or edge maps, local features have been shown to be much more
robust against variations in imaging conditions. Our approach is unique in that
we are the first to use local features to fit a Morphable Model.
  Because of the speed of our method, it is applicable for realtime
applications. Our cascaded regression framework is available as an open source
library (https://github.com/patrikhuber).
Financial Market Prediction
Authors: Mike Wu
Date: 2015-03-08 00:00:00
Tasks: Prediction

Given financial data from popular sites like Yahoo and the London Exchange,
the presented paper attempts to model and predict stocks that can be considered
"good investments". Stocks are characterized by 125 features ranging from gross
domestic product to EDIBTA, and are labeled by discrepancies between stock and
market price returns. An artificial neural network (Self-Organizing Map) is
fitted to train on more than a million data points to predict "good
investments" given testing stocks from 2013 and after.
DESAT: an SSW tool for SDO/AIA image de-saturation
Authors: Richard A Schwartz, Gabriele Torre, Anna Maria Massone, Michele Piana
Date: 2015-03-08 00:00:00

Saturation affects a significant rate of images recorded by the Atmospheric
Imaging Assembly on the Solar Dynamics Observatory. This paper describes a
computational method and a technological pipeline for the de-saturation of such
images, based on several mathematical ingredients like Expectation
Maximization, image correlation and interpolation. An analysis of the
computational properties and demands of the pipeline, together with an
assessment of its reliability are performed against a set of data recorded from
the Feburary 25 2014 flaring event.