pwc-archive/papers-with-abstracts huggingface dataset:
Exploring the Imposition of Synaptic Precision Restrictions For Evolutionary Synthesis of Deep Neural Networks
Authors: Mohammad Javad Shafiee, Francis Li, Alexander Wong
Date: 2017-07-01 00:00:00

A key contributing factor to incredible success of deep neural networks has
been the significant rise on massively parallel computing devices allowing
researchers to greatly increase the size and depth of deep neural networks,
leading to significant improvements in modeling accuracy. Although deeper,
larger, or complex deep neural networks have shown considerable promise, the
computational complexity of such networks is a major barrier to utilization in
resource-starved scenarios. We explore the synaptogenesis of deep neural
networks in the formation of efficient deep neural network architectures within
an evolutionary deep intelligence framework, where a probabilistic generative
modeling strategy is introduced to stochastically synthesize increasingly
efficient yet effective offspring deep neural networks over generations,
mimicking evolutionary processes such as heredity, random mutation, and natural
selection in a probabilistic manner. In this study, we primarily explore the
imposition of synaptic precision restrictions and its impact on the
evolutionary synthesis of deep neural networks to synthesize more efficient
network architectures tailored for resource-starved scenarios. Experimental
results show significant improvements in synaptic efficiency (~10X decrease for
GoogLeNet-based DetectNet) and inference speed (>5X increase for
GoogLeNet-based DetectNet) while preserving modeling accuracy.
Do GANs actually learn the distribution? An empirical study
Authors: Sanjeev Arora, Yi Zhang
Date: 2017-06-26 00:00:00

Do GANS (Generative Adversarial Nets) actually learn the target distribution?
The foundational paper of (Goodfellow et al 2014) suggested they do, if they
were given sufficiently large deep nets, sample size, and computation time. A
recent theoretical analysis in Arora et al (to appear at ICML 2017) raised
doubts whether the same holds when discriminator has finite size. It showed
that the training objective can approach its optimum value even if the
generated distribution has very low support ---in other words, the training
objective is unable to prevent mode collapse. The current note reports
experiments suggesting that such problems are not merely theoretical. It
presents empirical evidence that well-known GANs approaches do learn
distributions of fairly low support, and thus presumably are not learning the
target distribution. The main technical contribution is a new proposed test,
based upon the famous birthday paradox, for estimating the support size of the
generated distribution.
Synthesizing Deep Neural Network Architectures using Biological Synaptic Strength Distributions
Authors: A. H. Karimi, M. J. Shafiee, A. Ghodsi, A. Wong
Date: 2017-07-01 00:00:00
Tasks: Small Data Image Classification

In this work, we perform an exploratory study on synthesizing deep neural
networks using biological synaptic strength distributions, and the potential
influence of different distributions on modelling performance particularly for
the scenario associated with small data sets. Surprisingly, a CNN with
convolutional layer synaptic strengths drawn from biologically-inspired
distributions such as log-normal or correlated center-surround distributions
performed relatively well suggesting a possibility for designing deep neural
network architectures that do not require many data samples to learn, and can
sidestep current training procedures while maintaining or boosting modelling
performance.
Adaptive Candidate Generation for Scalable Edge-discovery Tasks on Data Graphs
Authors: Mayank Kejriwal
Date: 2016-05-02 00:00:00
Tasks: Blocking, Entity Resolution, Link Prediction

Several `edge-discovery' applications over graph-based data models are known
to have worst-case quadratic time complexity in the nodes, even if the
discovered edges are sparse. One example is the generic link discovery problem
between two graphs, which has invited research interest in several communities.
Specific versions of this problem include link prediction in social networks,
ontology alignment between metadata-rich RDF data, approximate joins, and
entity resolution between instance-rich data. As large datasets continue to
proliferate, reducing quadratic complexity to make the task practical is an
important research problem. Within the entity resolution community, the problem
is commonly referred to as blocking. A particular class of learnable blocking
schemes is known as Disjunctive Normal Form (DNF) blocking schemes, and has
emerged as state-of-the art for homogeneous (i.e. same-schema) tabular data.
Despite the promise of these schemes, a formalism or learning framework has not
been developed for them when input data instances are generic, attributed
graphs possessing both node and edge heterogeneity. With such a development,
the complexity-reducing scope of DNF schemes becomes applicable to a variety of
problems, including entity resolution and type alignment between heterogeneous
graphs, and link prediction in networks represented as attributed graphs. This
paper presents a graph-theoretic formalism for DNF schemes, and investigates
their learnability in an optimization framework. We also briefly describe an
empirical case study encapsulating some of the principles in this paper.
Better than Real: Complex-valued Neural Nets for MRI Fingerprinting
Authors: Patrick Virtue, Stella X. Yu, Michael Lustig
Date: 2017-07-01 00:00:00

The task of MRI fingerprinting is to identify tissue parameters from
complex-valued MRI signals. The prevalent approach is dictionary based, where a
test MRI signal is compared to stored MRI signals with known tissue parameters
and the most similar signals and tissue parameters retrieved. Such an approach
does not scale with the number of parameters and is rather slow when the tissue
parameter space is large.
  Our first novel contribution is to use deep learning as an efficient
nonlinear inverse mapping approach. We generate synthetic (tissue, MRI) data
from an MRI simulator, and use them to train a deep net to map the MRI signal
to the tissue parameters directly.
  Our second novel contribution is to develop a complex-valued neural network
with new cardioid activation functions. Our results demonstrate that
complex-valued neural nets could be much more accurate than real-valued neural
nets at complex-valued MRI fingerprinting.
Adversarial Image Alignment and Interpolation
Authors: Viren Jain
Date: 2017-06-30 00:00:00
Tasks: Patch Matching, Super-Resolution

Volumetric (3d) images are acquired for many scientific and biomedical
purposes using imaging methods such as serial section microscopy, CT scans, and
MRI. A frequent step in the analysis and reconstruction of such data is the
alignment and registration of images that were acquired in succession along a
spatial or temporal dimension. For example, in serial section electron
microscopy, individual 2d sections are imaged via electron microscopy and then
must be aligned to one another in order to produce a coherent 3d volume. State
of the art approaches find image correspondences derived from patch matching
and invariant feature detectors, and then solve optimization problems that
rigidly or elastically deform series of images into an aligned volume. Here we
show how fully convolutional neural networks trained with an adversarial loss
function can be used for two tasks: (1) synthesis of missing or damaged image
data from adjacent sections, and (2) fine-scale alignment of block-face
electron microscopy data. Finally, we show how these two capabilities can be
combined in order to produce artificial isotropic volumes from anisotropic
image volumes using a super-resolution adversarial alignment and interpolation
approach.
Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English
Authors: Su Lin Blodgett, Brendan O'Connor
Date: 2017-06-30 00:00:00
Tasks: Fairness, Language Identification

We highlight an important frontier in algorithmic fairness: disparity in the
quality of natural language processing algorithms when applied to language from
authors of different social groups. For example, current systems sometimes
analyze the language of females and minorities more poorly than they do of
whites and males. We conduct an empirical analysis of racial disparity in
language identification for tweets written in African-American English, and
discuss implications of disparity in NLP.
Multiple VLAD encoding of CNNs for image classification
Authors: Qing Li, Qiang Peng, Chuan Yan
Date: 2017-06-30 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification

Despite the effectiveness of convolutional neural networks (CNNs) especially
in image classification tasks, the effect of convolution features on learned
representations is still limited. It mostly focuses on the salient object of
the images, but ignores the variation information on clutter and local. In this
paper, we propose a special framework, which is the multiple VLAD encoding
method with the CNNs features for image classification. Furthermore, in order
to improve the performance of the VLAD coding method, we explore the
multiplicity of VLAD encoding with the extension of three kinds of encoding
algorithms, which are the VLAD-SA method, the VLAD-LSA and the VLAD-LLC method.
Finally, we equip the spatial pyramid patch (SPM) on VLAD encoding to add the
spatial information of CNNs feature. In particular, the power of SPM leads our
framework to yield better performance compared to the existing method.
Fairer and more accurate, but for whom?
Authors: Alexandra Chouldechova, Max G'Sell
Date: 2017-06-30 00:00:00
Tasks: Decision Making, Fairness

Complex statistical machine learning models are increasingly being used or
considered for use in high-stakes decision-making pipelines in domains such as
financial services, health care, criminal justice and human services. These
models are often investigated as possible improvements over more classical
tools such as regression models or human judgement. While the modeling approach
may be new, the practice of using some form of risk assessment to inform
decisions is not. When determining whether a new model should be adopted, it is
therefore essential to be able to compare the proposed model to the existing
approach across a range of task-relevant accuracy and fairness metrics. Looking
at overall performance metrics, however, may be misleading. Even when two
models have comparable overall performance, they may nevertheless disagree in
their classifications on a considerable fraction of cases. In this paper we
introduce a model comparison framework for automatically identifying subgroups
in which the differences between models are most pronounced. Our primary focus
is on identifying subgroups where the models differ in terms of
fairness-related quantities such as racial or gender disparities. We present
experimental results from a recidivism prediction task and a hypothetical
lending example.
Understanding Traffic Density from Large-Scale Web Camera Data
Authors: Shanghang Zhang, Guanhang Wu, João P. Costeira, José M. F. Moura
Date: 2017-03-17 00:00:00
Proceeding: CVPR 2017 7
Tasks: regression, vehicle detection

Understanding traffic density from large-scale web camera (webcam) videos is
a challenging problem because such videos have low spatial and temporal
resolution, high occlusion and large perspective. To deeply understand traffic
density, we explore both deep learning based and optimization based methods. To
avoid individual vehicle detection and tracking, both methods map the image
into vehicle density map, one based on rank constrained regression and the
other one based on fully convolution networks (FCN). The regression based
method learns different weights for different blocks in the image to increase
freedom degrees of weights and embed perspective information. The FCN based
method jointly estimates vehicle density map and vehicle count with a residual
learning framework to perform end-to-end dense prediction, allowing arbitrary
image resolution, and adapting to different vehicle scales and perspectives. We
analyze and compare both methods, and get insights from optimization based
method to improve deep model. Since existing datasets do not cover all the
challenges in our work, we collected and labelled a large-scale traffic video
dataset, containing 60 million frames from 212 webcams. Both methods are
extensively evaluated and compared on different counting tasks and datasets.
FCN based method significantly reduces the mean absolute error from 10.99 to
5.31 on the public dataset TRANCOS compared with the state-of-the-art baseline.
Automatically Redundant Features Removal for Unsupervised Feature Selection via Sparse Feature Graph
Authors: Shuchu Han, Hao Huang, Hong Qin
Date: 2017-05-13 00:00:00
Tasks: feature selection, Sparse Learning

The redundant features existing in high dimensional datasets always affect
the performance of learning and mining algorithms. How to detect and remove
them is an important research topic in machine learning and data mining
research. In this paper, we propose a graph based approach to find and remove
those redundant features automatically for high dimensional data. Based on the
sparse learning based unsupervised feature selection framework, Sparse Feature
Graph (SFG) is introduced not only to model the redundancy between two
features, but also to disclose the group redundancy between two groups of
features. With SFG, we can divide the whole features into different groups, and
improve the intrinsic structure of data by removing detected redundant
features. With accurate data structure, quality indicator vectors can be
obtained to improve the learning performance of existing unsupervised feature
selection algorithms such as multi-cluster feature selection (MCFS). Our
experimental results on benchmark datasets show that the proposed SFG and
feature redundancy remove algorithm can improve the performance of unsupervised
feature selection algorithms consistently.
Likelihood Inflating Sampling Algorithm
Authors: Reihaneh Entezari, Radu V. Craiu, Jeffrey S. Rosenthal
Date: 2016-05-06 00:00:00

Markov Chain Monte Carlo (MCMC) sampling from a posterior distribution
corresponding to a massive data set can be computationally prohibitive since
producing one sample requires a number of operations that is linear in the data
size. In this paper, we introduce a new communication-free parallel method, the
Likelihood Inflating Sampling Algorithm (LISA), that significantly reduces
computational costs by randomly splitting the dataset into smaller subsets and
running MCMC methods independently in parallel on each subset using different
processors. Each processor will be used to run an MCMC chain that samples
sub-posterior distributions which are defined using an "inflated" likelihood
function. We develop a strategy for combining the draws from different
sub-posteriors to study the full posterior of the Bayesian Additive Regression
Trees (BART) model. The performance of the method is tested using both
simulated and real data.
An artificial neural network to find correlation patterns in an arbitrary number of variables
Authors: Alessandro Fontana
Date: 2016-06-21 00:00:00

Methods to find correlation among variables are of interest to many
disciplines, including statistics, machine learning, (big) data mining and
neurosciences. Parameters that measure correlation between two variables are of
limited utility when used with multiple variables. In this work, I propose a
simple criterion to measure correlation among an arbitrary number of variables,
based on a data set. The central idea is to i) design a function of the
variables that can take different forms depending on a set of parameters, ii)
calculate the difference between a statistics associated to the function
computed on the data set and the same statistics computed on a randomised
version of the data set, called "scrambled" data set, and iii) optimise the
parameters to maximise this difference. Many such functions can be organised in
layers, which can in turn be stacked one on top of the other, forming a neural
network. The function parameters are searched with an enhanced genetic
algortihm called POET and the resulting method is tested on a cancer gene data
set. The method may have potential implications for some issues that affect the
field of neural networks, such as overfitting, the need to process huge amounts
of data for training and the presence of "adversarial examples".
Bolt: Accelerated Data Mining with Fast Vector Compression
Authors: Davis W. Blalock, John V. Guttag
Date: 2017-06-30 00:00:00
Tasks: Quantization

Vectors of data are at the heart of machine learning and data mining.
Recently, vector quantization methods have shown great promise in reducing both
the time and space costs of operating on vectors. We introduce a vector
quantization algorithm that can compress vectors over 12x faster than existing
techniques while also accelerating approximate vector operations such as
distance and dot product computations by up to 10x. Because it can encode over
2GB of vectors per second, it makes vector quantization cheap enough to employ
in many more circumstances. For example, using our technique to compute
approximate dot products in a nested loop can multiply matrices faster than a
state-of-the-art BLAS implementation, even when our algorithm must first
compress the matrices.
  In addition to showing the above speedups, we demonstrate that our approach
can accelerate nearest neighbor search and maximum inner product search by over
100x compared to floating point operations and up to 10x compared to other
vector quantization methods. Our approximate Euclidean distance and dot product
computations are not only faster than those of related algorithms with slower
encodings, but also faster than Hamming distance computations, which have
direct hardware support on the tested platforms. We also assess the errors of
our algorithm's approximate distances and dot products, and find that it is
competitive with existing, slower vector quantization algorithms.
Nuclear penalized multinomial regression with an application to predicting at bat outcomes in baseball
Authors: Scott Powers, Trevor Hastie, Robert Tibshirani
Date: 2017-06-30 00:00:00
Tasks: regression

We propose the nuclear norm penalty as an alternative to the ridge penalty
for regularized multinomial regression. This convex relaxation of reduced-rank
multinomial regression has the advantage of leveraging underlying structure
among the response categories to make better predictions. We apply our method,
nuclear penalized multinomial regression (NPMR), to Major League Baseball
play-by-play data to predict outcome probabilities based on batter-pitcher
matchups. The interpretation of the results meshes well with subject-area
expertise and also suggests a novel understanding of what differentiates
players.
Lifelong Learning in Costly Feature Spaces
Authors: Maria-Florina Balcan, Avrim Blum, Vaishnavh Nagarajan
Date: 2017-06-30 00:00:00
Tasks: Lifelong learning

An important long-term goal in machine learning systems is to build learning
agents that, like humans, can learn many tasks over their lifetime, and
moreover use information from these tasks to improve their ability to do so
efficiently. In this work, our goal is to provide new theoretical insights into
the potential of this paradigm. In particular, we propose a lifelong learning
framework that adheres to a novel notion of resource efficiency that is
critical in many real-world domains where feature evaluations are costly. That
is, our learner aims to reuse information from previously learned related tasks
to learn future tasks in a feature-efficient manner. Furthermore, we consider
novel combinatorial ways in which learning tasks can relate. Specifically, we
design lifelong learning algorithms for two structurally different and widely
used families of target functions: decision trees/lists and
monomials/polynomials. We also provide strong feature-efficiency guarantees for
these algorithms; in fact, we show that in order to learn future targets, we
need only slightly more feature evaluations per training example than what is
needed to predict on an arbitrary example using those targets. We also provide
algorithms with guarantees in an agnostic model where not all the targets are
related to each other. Finally, we also provide lower bounds on the performance
of a lifelong learner in these models, which are in fact tight under some
conditions.
SafetyNets: Verifiable Execution of Deep Neural Networks on an Untrusted Cloud
Authors: Zahra Ghodsi, Tianyu Gu, Siddharth Garg
Date: 2017-06-30 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: speech-recognition, Speech Recognition

Inference using deep neural networks is often outsourced to the cloud since
it is a computationally demanding task. However, this raises a fundamental
issue of trust. How can a client be sure that the cloud has performed inference
correctly? A lazy cloud provider might use a simpler but less accurate model to
reduce its own computational load, or worse, maliciously modify the inference
results sent to the client. We propose SafetyNets, a framework that enables an
untrusted server (the cloud) to provide a client with a short mathematical
proof of the correctness of inference tasks that they perform on behalf of the
client. Specifically, SafetyNets develops and implements a specialized
interactive proof (IP) protocol for verifiable execution of a class of deep
neural networks, i.e., those that can be represented as arithmetic circuits.
Our empirical results on three- and four-layer deep neural networks demonstrate
the run-time costs of SafetyNets for both the client and server are low.
SafetyNets detects any incorrect computations of the neural network by the
untrusted server with high probability, while achieving state-of-the-art
accuracy on the MNIST digit recognition (99.4%) and TIMIT speech recognition
tasks (75.22%).
Probabilistic Line Searches for Stochastic Optimization
Authors: Maren Mahsereci, Philipp Hennig
Date: 2017-03-29 00:00:00
Proceeding: NeurIPS 2015
Tasks: Bayesian Optimization, Stochastic Optimization

In deterministic optimization, line searches are a standard tool ensuring
stability and efficiency. Where only stochastic gradients are available, no
direct equivalent has so far been formulated, because uncertain gradients do
not allow for a strict sequence of decisions collapsing the search space. We
construct a probabilistic line search by combining the structure of existing
deterministic methods with notions from Bayesian optimization. Our method
retains a Gaussian process surrogate of the univariate optimization objective,
and uses a probabilistic belief over the Wolfe conditions to monitor the
descent. The algorithm has very low computational cost, and no user-controlled
parameters. Experiments show that it effectively removes the need to define a
learning rate for stochastic gradient descent.
Foundations for a Probabilistic Event Calculus
Authors: Fabio Aurelio D'Asaro, Antonis Bikakis, Luke Dickens, Rob Miller
Date: 2017-03-20 00:00:00

We present PEC, an Event Calculus (EC) style action language for reasoning
about probabilistic causal and narrative information. It has an action language
style syntax similar to that of the EC variant Modular-E. Its semantics is
given in terms of possible worlds which constitute possible evolutions of the
domain, and builds on that of EFEC, an epistemic extension of EC. We also
describe an ASP implementation of PEC and show the sense in which this is sound
and complete.