pwc-archive/papers-with-abstracts huggingface dataset:
VisualBackProp: efficient visualization of CNNs
Authors: Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, Karol Zieba
Date: 2016-11-16 00:00:00
Tasks: Self-Driving Cars

This paper proposes a new method, that we call VisualBackProp, for
visualizing which sets of pixels of the input image contribute most to the
predictions made by the convolutional neural network (CNN). The method heavily
hinges on exploring the intuition that the feature maps contain less and less
irrelevant information to the prediction decision when moving deeper into the
network. The technique we propose was developed as a debugging tool for
CNN-based systems for steering self-driving cars and is therefore required to
run in real-time, i.e. it was designed to require less computations than a
forward propagation. This makes the presented visualization method a valuable
debugging tool which can be easily used during both training and inference. We
furthermore justify our approach with theoretical arguments and theoretically
confirm that the proposed method identifies sets of input pixels, rather than
individual pixels, that collaboratively contribute to the prediction. Our
theoretical findings stand in agreement with the experimental results. The
empirical evaluation shows the plausibility of the proposed approach on the
road video data as well as in other applications and reveals that it compares
favorably to the layer-wise relevance propagation approach, i.e. it obtains
similar visualization results and simultaneously achieves order of magnitude
speed-ups.
A Deep Compositional Framework for Human-like Language Acquisition in Virtual Environment
Authors: Haonan Yu, Haichao Zhang, Wei Xu
Date: 2017-03-28 00:00:00
Tasks: Language Acquisition, Navigate, Zero-Shot Learning

We tackle a task where an agent learns to navigate in a 2D maze-like
environment called XWORLD. In each session, the agent perceives a sequence of
raw-pixel frames, a natural language command issued by a teacher, and a set of
rewards. The agent learns the teacher's language from scratch in a grounded and
compositional manner, such that after training it is able to correctly execute
zero-shot commands: 1) the combination of words in the command never appeared
before, and/or 2) the command contains new object concepts that are learned
from another task but never learned from navigation. Our deep framework for the
agent is trained end to end: it learns simultaneously the visual
representations of the environment, the syntax and semantics of the language,
and the action module that outputs actions. The zero-shot learning capability
of our framework results from its compositionality and modularity with
parameter tying. We visualize the intermediate outputs of the framework,
demonstrating that the agent truly understands how to solve the problem. We
believe that our results provide some preliminary insights on how to train an
agent with similar abilities in a 3D environment.
Real-Time Machine Learning: The Missing Pieces
Authors: Robert Nishihara, Philipp Moritz, Stephanie Wang, Alexey Tumanov, William Paul, Johann Schleier-Smith, Richard Liaw, Mehrdad Niknami, Michael. I. Jordan, Ion Stoica
Date: 2017-03-11 00:00:00
Tasks: BIG-bench Machine Learning, Decision Making

Machine learning applications are increasingly deployed not only to serve
predictions using static models, but also as tightly-integrated components of
feedback loops involving dynamic, real-time decision making. These applications
pose a new set of requirements, none of which are difficult to achieve in
isolation, but the combination of which creates a challenge for existing
distributed execution frameworks: computation with millisecond latency at high
throughput, adaptive construction of arbitrary task graphs, and execution of
heterogeneous kernels over diverse sets of resources. We assert that a new
distributed execution framework is needed for such ML applications and propose
a candidate approach with a proof-of-concept architecture that achieves a 63x
performance improvement over a state-of-the-art execution framework for a
representative application.
Two-temperature logistic regression based on the Tsallis divergence
Authors: Ehsan Amid, Manfred K. Warmuth, Sriram Srinivasan
Date: 2017-05-19 00:00:00
Tasks: regression, Vocal Bursts Valence Prediction

We develop a variant of multiclass logistic regression that is significantly more robust to noise. The algorithm has one weight vector per class and the surrogate loss is a function of the linear activations (one per class). The surrogate loss of an example with linear activation vector $\mathbf{a}$ and class $c$ has the form $-\log_{t_1} \exp_{t_2} (a_c - G_{t_2}(\mathbf{a}))$ where the two temperatures $t_1$ and $t_2$ ''temper'' the $\log$ and $\exp$, respectively, and $G_{t_2}(\mathbf{a})$ is a scalar value that generalizes the log-partition function. We motivate this loss using the Tsallis divergence. Our method allows transitioning between non-convex and convex losses by the choice of the temperature parameters. As the temperature $t_1$ of the logarithm becomes smaller than the temperature $t_2$ of the exponential, the surrogate loss becomes ''quasi convex''. Various tunings of the temperatures recover previous methods and tuning the degree of non-convexity is crucial in the experiments. In particular, quasi-convexity and boundedness of the loss provide significant robustness to the outliers. We explain this by showing that $t_1 < 1$ caps the surrogate loss and $t_2 >1$ makes the predictive distribution have a heavy tail. We show that the surrogate loss is Bayes-consistent, even in the non-convex case. Additionally, we provide efficient iterative algorithms for calculating the log-partition value only in a few number of iterations. Our compelling experimental results on large real-world datasets show the advantage of using the two-temperature variant in the noisy as well as the noise free case.
Multi-Stage Variational Auto-Encoders for Coarse-to-Fine Image Generation
Authors: Lei Cai, Hongyang Gao, Shuiwang Ji
Date: 2017-05-19 00:00:00
Tasks: Decoder, Image Generation

Variational auto-encoder (VAE) is a powerful unsupervised learning framework
for image generation. One drawback of VAE is that it generates blurry images
due to its Gaussianity assumption and thus L2 loss. To allow the generation of
high quality images by VAE, we increase the capacity of decoder network by
employing residual blocks and skip connections, which also enable efficient
optimization. To overcome the limitation of L2 loss, we propose to generate
images in a multi-stage manner from coarse to fine. In the simplest case, the
proposed multi-stage VAE divides the decoder into two components in which the
second component generates refined images based on the course images generated
by the first component. Since the second component is independent of the VAE
model, it can employ other loss functions beyond the L2 loss and different
model architectures. The proposed framework can be easily generalized to
contain more than two components. Experiment results on the MNIST and CelebA
datasets demonstrate that the proposed multi-stage VAE can generate sharper
images as compared to those from the original VAE.
The High-Dimensional Geometry of Binary Neural Networks
Authors: Alexander G. Anderson, Cory P. Berg
Date: 2017-05-19 00:00:00
Proceeding: ICLR 2018 1
Tasks: Vocal Bursts Intensity Prediction

Recent research has shown that one can train a neural network with binary
weights and activations at train time by augmenting the weights with a
high-precision continuous latent variable that accumulates small changes from
stochastic gradient descent. However, there is a dearth of theoretical analysis
to explain why we can effectively capture the features in our data with binary
weights and activations. Our main result is that the neural networks with
binary weights and activations trained using the method of Courbariaux, Hubara
et al. (2016) work because of the high-dimensional geometry of binary vectors.
In particular, the ideal continuous vectors that extract out features in the
intermediate representations of these BNNs are well-approximated by binary
vectors in the sense that dot products are approximately preserved. Compared to
previous research that demonstrated the viability of such BNNs, our work
explains why these BNNs work in terms of the HD geometry. Our theory serves as
a foundation for understanding not only BNNs but a variety of methods that seek
to compress traditional neural networks. Furthermore, a better understanding of
multilayer binary neural networks serves as a starting point for generalizing
BNNs to other neural network architectures such as recurrent neural networks.
Distributed and parallel time series feature extraction for industrial big data applications
Authors: Maximilian Christ, Andreas W. Kempa-Liehr, Michael Feindt
Date: 2016-10-25 00:00:00
Tasks: Binary Classification, Classification, Feature Importance, feature selection, General Classification, regression, Time Series, Time Series Analysis, Time Series Classification

The all-relevant problem of feature selection is the identification of all
strongly and weakly relevant attributes. This problem is especially hard to
solve for time series classification and regression in industrial applications
such as predictive maintenance or production line optimization, for which each
label or regression target is associated with several time series and
meta-information simultaneously. Here, we are proposing an efficient, scalable
feature extraction algorithm for time series, which filters the available
features in an early stage of the machine learning pipeline with respect to
their significance for the classification or regression task, while controlling
the expected percentage of selected but irrelevant features. The proposed
algorithm combines established feature extraction methods with a feature
importance filter. It has a low computational complexity, allows to start on a
problem with only limited domain knowledge available, can be trivially
parallelized, is highly scalable and based on well studied non-parametric
hypothesis tests. We benchmark our proposed algorithm on all binary
classification problems of the UCR time series classification archive as well
as time series from a production line optimization project and simulated
stochastic processes with underlying qualitative change of dynamics.
Noise-Tolerant Interactive Learning from Pairwise Comparisons
Authors: Yichong Xu, Hongyang Zhang, Aarti Singh, Kyle Miller, Artur Dubrawski
Date: 2017-04-19 00:00:00

We study the problem of interactively learning a binary classifier using
noisy labeling and pairwise comparison oracles, where the comparison oracle
answers which one in the given two instances is more likely to be positive.
Learning from such oracles has multiple applications where obtaining direct
labels is harder but pairwise comparisons are easier, and the algorithm can
leverage both types of oracles. In this paper, we attempt to characterize how
the access to an easier comparison oracle helps in improving the label and
total query complexity. We show that the comparison oracle reduces the learning
problem to that of learning a threshold function. We then present an algorithm
that interactively queries the label and comparison oracles and we characterize
its query complexity under Tsybakov and adversarial noise conditions for the
comparison and labeling oracles. Our lower bounds show that our label and total
query complexity is almost optimal.
Classification of Alzheimer's Disease Structural MRI Data by Deep Learning Convolutional Neural Networks
Authors: Saman Sarraf, Ghassem Tofighi
Date: 2016-07-22 00:00:00
Tasks: BIG-bench Machine Learning, General Classification

Recently, machine learning techniques especially predictive modeling and
pattern recognition in biomedical sciences from drug delivery system to medical
imaging has become one of the important methods which are assisting researchers
to have deeper understanding of entire issue and to solve complex medical
problems. Deep learning is a powerful machine learning algorithm in
classification while extracting low to high-level features. In this paper, we
used convolutional neural network to classify Alzheimer's brain from normal
healthy brain. The importance of classifying this kind of medical data is to
potentially develop a predict model or system in order to recognize the type
disease from normal subjects or to estimate the stage of the disease.
Classification of clinical data such as Alzheimer's disease has been always
challenging and most problematic part has been always selecting the most
discriminative features. Using Convolutional Neural Network (CNN) and the
famous architecture LeNet-5, we successfully classified structural MRI data of
Alzheimer's subjects from normal controls where the accuracy of test data on
trained data reached 98.84%. This experiment suggests us the shift and scale
invariant features extracted by CNN followed by deep learning classification is
most powerful method to distinguish clinical data from healthy data in fMRI.
This approach also enables us to expand our methodology to predict more
complicated systems.
Doubly Robust Data-Driven Distributionally Robust Optimization
Authors: Jose Blanchet, Yang Kang, Fan Zhang, Fei He, Zhangyi Hu
Date: 2017-05-19 00:00:00

Data-driven Distributionally Robust Optimization (DD-DRO) via optimal
transport has been shown to encompass a wide range of popular machine learning
algorithms. The distributional uncertainty size is often shown to correspond to
the regularization parameter. The type of regularization (e.g. the norm used to
regularize) corresponds to the shape of the distributional uncertainty. We
propose a data-driven robust optimization methodology to inform the
transportation cost underlying the definition of the distributional
uncertainty. We show empirically that this additional layer of robustification,
which produces a method we called doubly robust data-driven distributionally
robust optimization (DD-R-DRO), allows to enhance the generalization properties
of regularized estimators while reducing testing error relative to
state-of-the-art classifiers in a wide range of data sets.
Scaffolding Networks: Incremental Learning and Teaching Through Questioning
Authors: Asli Celikyilmaz, Li Deng, Lihong Li, Chong Wang
Date: 2017-02-28 00:00:00
Tasks: Incremental Learning, Reinforcement Learning, Sentence

We introduce a new paradigm of learning for reasoning, understanding, and
prediction, as well as the scaffolding network to implement this paradigm. The
scaffolding network embodies an incremental learning approach that is
formulated as a teacher-student network architecture to teach machines how to
understand text and do reasoning. The key to our computational scaffolding
approach is the interactions between the teacher and the student through
sequential questioning. The student observes each sentence in the text
incrementally, and it uses an attention-based neural net to discover and
register the key information in relation to its current memory. Meanwhile, the
teacher asks questions about the observed text, and the student network gets
rewarded by correctly answering these questions. The entire network is updated
continually using reinforcement learning. Our experimental results on synthetic
and real datasets show that the scaffolding network not only outperforms
state-of-the-art methods but also learns to do reasoning in a scalable way even
with little human generated input.
Clustering under Local Stability: Bridging the Gap between Worst-Case and Beyond Worst-Case Analysis
Authors: Maria-Florina Balcan, Colin White
Date: 2017-05-19 00:00:00
Tasks: Clustering

Recently, there has been substantial interest in clustering research that
takes a beyond worst-case approach to the analysis of algorithms. The typical
idea is to design a clustering algorithm that outputs a near-optimal solution,
provided the data satisfy a natural stability notion. For example, Bilu and
Linial (2010) and Awasthi et al. (2012) presented algorithms that output
near-optimal solutions, assuming the optimal solution is preserved under small
perturbations to the input distances. A drawback to this approach is that the
algorithms are often explicitly built according to the stability assumption and
give no guarantees in the worst case; indeed, several recent algorithms output
arbitrarily bad solutions even when just a small section of the data does not
satisfy the given stability notion.
  In this work, we address this concern in two ways. First, we provide
algorithms that inherit the worst-case guarantees of clustering approximation
algorithms, while simultaneously guaranteeing near-optimal solutions when the
data is stable. Our algorithms are natural modifications to existing
state-of-the-art approximation algorithms. Second, we initiate the study of
local stability, which is a property of a single optimal cluster rather than an
entire optimal solution. We show our algorithms output all optimal clusters
which satisfy stability locally. Specifically, we achieve strong positive
results in our local framework under recent stability notions including metric
perturbation resilience (Angelidakis et al. 2017) and robust perturbation
resilience (Balcan and Liang 2012) for the $k$-median, $k$-means, and
symmetric/asymmetric $k$-center objectives.
Data-driven Optimal Cost Selection for Distributionally Robust Optimization
Authors: Jose Blanchet, Yang Kang, Fan Zhang, Karthyek Murthy
Date: 2017-05-19 00:00:00
Tasks: BIG-bench Machine Learning, regression

Recently, (Blanchet, Kang, and Murhy 2016, and Blanchet, and Kang 2017)
showed that several machine learning algorithms, such as square-root Lasso,
Support Vector Machines, and regularized logistic regression, among many
others, can be represented exactly as distributionally robust optimization
(DRO) problems. The distributional uncertainty is defined as a neighborhood
centered at the empirical distribution. We propose a methodology which learns
such neighborhood in a natural data-driven way. We show rigorously that our
framework encompasses adaptive regularization as a particular case. Moreover,
we demonstrate empirically that our proposed methodology is able to improve
upon a wide range of popular machine learning estimators.
A New 3D Segmentation Technique for QCT Scans of the Lumbar Spine to Determine BMD and Vertebral Geometry
Authors: Andre Mastmeyer, Klaus Engelke, Christina Fuchs, Willi Kalender
Date: 2017-05-19 00:00:00
Tasks: Anatomy, Segmentation

Quantitative computed tomography (QCT) is a standard method to determine bone
mineral density (BMD) in the spine. Traditionally single 8 - 10 mm thick slices
have been analyzed only. Current spiral CT scanners provide true 3D acquisition
schemes allowing for a more differential BMD analysis and an assessment of
geometric parameters, which may improve fracture prediction. We developed a
novel 3D segmentation approach that combines deformable balloons, multi seeded
volume growing, and dedicated morphological operations to extract the vertebral
bodies. An anatomy-oriented coordinate system attached automatically to each
vertebra is used to define volumes of interest. We analyzed intra-operator
precision of the segmentation procedure using abdominal scans from 10 patients
(60 mAs, 120 kV, slice thickness 1mm, B40s, Siemens Sensation 16). Our new
segmentation method shows excellent precision errors in the order of < 1 % for
BMD and < 2 % for volume.
Kernel Bandwidth Selection for SVDD: Peak Criterion Approach for Large Data
Authors: Sergiy Peredriy, Deovrat Kakde, Arin Chaudhuri
Date: 2016-10-31 00:00:00
Tasks: Outlier Detection

Support Vector Data Description (SVDD) provides a useful approach to
construct a description of multivariate data for single-class classification
and outlier detection with various practical applications. Gaussian kernel used
in SVDD formulation allows flexible data description defined by observations
designated as support vectors. The data boundary of such description is
non-spherical and conforms to the geometric features of the data. By varying
the Gaussian kernel bandwidth parameter, the SVDD-generated boundary can be
made either smoother (more spherical) or tighter/jagged. The former case may
lead to under-fitting, whereas the latter may result in overfitting. Peak
criterion has been proposed to select an optimal value of the kernel bandwidth
to strike the balance between the data boundary smoothness and its ability to
capture the general geometric shape of the data. Peak criterion involves
training SVDD at various values of the kernel bandwidth parameter. When
training datasets are large, the time required to obtain the optimal value of
the Gaussian kernel bandwidth parameter according to Peak method can become
prohibitively large. This paper proposes an extension of Peak method for the
case of large data. The proposed method gives good results when applied to
several datasets. Two existing alternative methods of computing the Gaussian
kernel bandwidth parameter (Coefficient of Variation and Distance to the
Farthest Neighbor) were modified to allow comparison with the proposed method
on convergence. Empirical comparison demonstrates the advantage of the proposed
method.
Local Information with Feedback Perturbation Suffices for Dictionary Learning in Neural Circuits
Authors: Tsung-Han Lin
Date: 2017-05-19 00:00:00
Tasks: Dictionary Learning

While the sparse coding principle can successfully model information
processing in sensory neural systems, it remains unclear how learning can be
accomplished under neural architectural constraints. Feasible learning rules
must rely solely on synaptically local information in order to be implemented
on spatially distributed neurons. We describe a neural network with spiking
neurons that can address the aforementioned fundamental challenge and solve the
L1-minimizing dictionary learning problem, representing the first model able to
do so. Our major innovation is to introduce feedback synapses to create a
pathway to turn the seemingly non-local information into local ones. The
resulting network encodes the error signal needed for learning as the change of
network steady states caused by feedback, and operates akin to the classical
stochastic gradient descent method.
A New 3D Method to Segment the Lumbar Vertebral Bodies and to Determine Bone Mineral Density and Geometry
Authors: Andre Mastmeyer, Klaus Engelke, Sebastian Meller, Willi Kalender
Date: 2017-05-19 00:00:00
Tasks: Segmentation

In this paper we present a new 3D segmentation approach for the vertebrae of the lower thoracic and the lumbar spine in spiral computed tomography datasets. We implemented a multi-step procedure. Its main components are deformable models, volume growing, and morphological operations. The performance analysis that included an evaluation of accuracy using the European Spine Phantom, and of intra-operator precision using clinical CT datasets from 10 patients highlight the potential for clinical use. The intra-operator precision of the segmentation procedure was better than 1% for Bone Mineral Density (BMD) and better than 1.8% for volume. The long-term goal of this work is to enable better fracture prediction and improved patient monitoring in the field of osteoporosis. A true 3D segmentation also enables an accurate measurement of geometrical parameters that can augment the classical measurement of BMD.