pwc-archive/papers-with-abstracts huggingface dataset:
Eigenvectors of Orthogonally Decomposable Functions
Authors: Mikhail Belkin, Luis Rademacher, James Voss
Date: 2014-11-05 00:00:00
Tasks: Clustering, Topic Models

The Eigendecomposition of quadratic forms (symmetric matrices) guaranteed by
the spectral theorem is a foundational result in applied mathematics. Motivated
by a shared structure found in inferential problems of recent interest---namely
orthogonal tensor decompositions, Independent Component Analysis (ICA), topic
models, spectral clustering, and Gaussian mixture learning---we generalize the
eigendecomposition from quadratic forms to a broad class of "orthogonally
decomposable" functions. We identify a key role of convexity in our extension,
and we generalize two traditional characterizations of eigenvectors: First, the
eigenvectors of a quadratic form arise from the optima structure of the
quadratic form on the sphere. Second, the eigenvectors are the fixed points of
the power iteration.
  In our setting, we consider a simple first order generalization of the power
method which we call gradient iteration. It leads to efficient and easily
implementable methods for basis recovery. It includes influential Machine
Learning methods such as cumulant-based FastICA and the tensor power iteration
for orthogonally decomposable tensors as special cases.
  We provide a complete theoretical analysis of gradient iteration using the
structure theory of discrete dynamical systems to show almost sure convergence
and fast (super-linear) convergence rates. The analysis also extends to the
case when the observed function is only approximately orthogonally
decomposable, with bounds that are polynomial in dimension and other relevant
parameters, such as perturbation size. Our perturbation results can be
considered as a non-linear version of the classical Davis-Kahan theorem for
perturbations of eigenvectors of symmetric matrices.
Approximation Algorithms for D-optimal Design
Authors: Mohit Singh, Weijun Xie
Date: 2018-02-23 00:00:00
Tasks: Experimental Design

Experimental design is a classical statistics problem and its aim is to estimate an unknown $m$-dimensional vector $\beta$ from linear measurements where a Gaussian noise is introduced in each measurement. For the combinatorial experimental design problem, the goal is to pick $k$ out of the given $n$ experiments so as to make the most accurate estimate of the unknown parameters, denoted as $\hat{\beta}$. In this paper, we will study one of the most robust measures of error estimation - $D$-optimality criterion, which corresponds to minimizing the volume of the confidence ellipsoid for the estimation error $\beta-\hat{\beta}$. The problem gives rise to two natural variants depending on whether repetitions of experiments are allowed or not. We first propose an approximation algorithm with a $\frac1e$-approximation for the $D$-optimal design problem with and without repetitions, giving the first constant factor approximation for the problem. We then analyze another sampling approximation algorithm and prove that it is $(1-\epsilon)$-approximation if $k\geq \frac{4m}{\epsilon}+\frac{12}{\epsilon^2}\log(\frac{1}{\epsilon})$ for any $\epsilon \in (0,1)$. Finally, for $D$-optimal design with repetitions, we study a different algorithm proposed by literature and show that it can improve this asymptotic approximation ratio.
Do WaveNets Dream of Acoustic Waves?
Authors: Kanru Hua
Date: 2018-02-23 00:00:00

Various sources have reported the WaveNet deep learning architecture being
able to generate high-quality speech, but to our knowledge there haven't been
studies on the interpretation or visualization of trained WaveNets. This study
investigates the possibility that WaveNet understands speech by unsupervisedly
learning an acoustically meaningful latent representation of the speech signals
in its receptive field; we also attempt to interpret the mechanism by which the
feature extraction is performed. Suggested by singular value decomposition and
linear regression analysis on the activations and known acoustic features (e.g.
F0), the key findings are (1) activations in the higher layers are highly
correlated with spectral features; (2) WaveNet explicitly performs pitch
extraction despite being trained to directly predict the next audio sample and
(3) for the said feature analysis to take place, the latent signal
representation is converted back and forth between baseband and wideband
components.
Missing Data Reconstruction in Remote Sensing image with a Unified Spatial-Temporal-Spectral Deep Convolutional Neural Network
Authors: Qiang Zhang, Qiangqiang Yuan, Chao Zeng, Xinghua Li, Yancong Wei
Date: 2018-02-23 00:00:00
Tasks: Cloud Removal, STS

Because of the internal malfunction of satellite sensors and poor atmospheric
conditions such as thick cloud, the acquired remote sensing data often suffer
from missing information, i.e., the data usability is greatly reduced. In this
paper, a novel method of missing information reconstruction in remote sensing
images is proposed. The unified spatial-temporal-spectral framework based on a
deep convolutional neural network (STS-CNN) employs a unified deep
convolutional neural network combined with spatial-temporal-spectral
supplementary information. In addition, to address the fact that most methods
can only deal with a single missing information reconstruction task, the
proposed approach can solve three typical missing information reconstruction
tasks: 1) dead lines in Aqua MODIS band 6; 2) the Landsat ETM+ Scan Line
Corrector (SLC)-off problem; and 3) thick cloud removal. It should be noted
that the proposed model can use multi-source data (spatial, spectral, and
temporal) as the input of the unified framework. The results of both simulated
and real-data experiments demonstrate that the proposed model exhibits high
effectiveness in the three missing information reconstruction tasks listed
above.
An efficient $k$-means-type algorithm for clustering datasets with incomplete records
Authors: Andrew Lithio, Ranjan Maitra
Date: 2018-02-23 00:00:00
Tasks: Clustering, Missing Values, Nonparametric Clustering

The $k$-means algorithm is arguably the most popular nonparametric clustering
method but cannot generally be applied to datasets with incomplete records. The
usual practice then is to either impute missing values under an assumed
missing-completely-at-random mechanism or to ignore the incomplete records, and
apply the algorithm on the resulting dataset. We develop an efficient version
of the $k$-means algorithm that allows for clustering in the presence of
incomplete records. Our extension is called $k_m$-means and reduces to the
$k$-means algorithm when all records are complete. We also provide
initialization strategies for our algorithm and methods to estimate the number
of groups in the dataset. Illustrations and simulations demonstrate the
efficacy of our approach in a variety of settings and patterns of missing data.
Our methods are also applied to the analysis of activation images obtained from
a functional Magnetic Resonance Imaging experiment.
Real-Time End-to-End Action Detection with Two-Stream Networks
Authors: Alaaeldin El-Nouby, Graham W. Taylor
Date: 2018-02-23 00:00:00
Tasks: Action Detection, Action Recognition, GPU, Optical Flow Estimation, Temporal Action Localization, Vocal Bursts Valence Prediction

Two-stream networks have been very successful for solving the problem of
action detection. However, prior work using two-stream networks train both
streams separately, which prevents the network from exploiting regularities
between the two streams. Moreover, unlike the visual stream, the dominant forms
of optical flow computation typically do not maximally exploit GPU parallelism.
We present a real-time end-to-end trainable two-stream network for action
detection. First, we integrate the optical flow computation in our framework by
using Flownet2. Second, we apply early fusion for the two streams and train the
whole pipeline jointly end-to-end. Finally, for better network initialization,
we transfer from the task of action recognition to action detection by
pre-training our framework using the recently released large-scale Kinetics
dataset. Our experimental results show that training the pipeline jointly
end-to-end with fine-tuning the optical flow for the objective of action
detection improves detection performance significantly. Additionally, we
observe an improvement when initializing with parameters pre-trained using
Kinetics. Last, we show that by integrating the optical flow computation, our
framework is more efficient, running at real-time speeds (up to 31 fps).
Fisher-Schultz Lecture: Generic Machine Learning Inference on Heterogenous Treatment Effects in Randomized Experiments, with an Application to Immunization in India
Authors: Victor Chernozhukov, Mert Demirer, Esther Duflo, Iván Fernández-Val
Date: 2017-12-13 00:00:00
Tasks: BIG-bench Machine Learning, Variational Inference

We propose strategies to estimate and make inference on key features of heterogeneous effects in randomized experiments. These key features include best linear predictors of the effects using machine learning proxies, average effects sorted by impact groups, and average characteristics of most and least impacted units. The approach is valid in high dimensional settings, where the effects are proxied (but not necessarily consistently estimated) by predictive and causal machine learning methods. We post-process these proxies into estimates of the key features. Our approach is generic, it can be used in conjunction with penalized methods, neural networks, random forests, boosted trees, and ensemble methods, both predictive and causal. Estimation and inference are based on repeated data splitting to avoid overfitting and achieve validity. We use quantile aggregation of the results across many potential splits, in particular taking medians of p-values and medians and other quantiles of confidence intervals. We show that quantile aggregation lowers estimation risks over a single split procedure, and establish its principal inferential properties. Finally, our analysis reveals ways to build provably better machine learning proxies through causal learning: we can use the objective functions that we develop to construct the best linear predictors of the effects, to obtain better machine learning proxies in the initial step. We illustrate the use of both inferential tools and causal learners with a randomized field experiment that evaluates a combination of nudges to stimulate demand for immunization in India.
Monotonic Chunkwise Attention
Authors: Chung-Cheng Chiu, Colin Raffel
Date: 2017-12-14 00:00:00
Tasks: Document Summarization, speech-recognition, Speech Recognition

Sequence-to-sequence models with soft attention have been successfully
applied to a wide variety of problems, but their decoding process incurs a
quadratic time and space cost and is inapplicable to real-time sequence
transduction. To address these issues, we propose Monotonic Chunkwise Attention
(MoChA), which adaptively splits the input sequence into small chunks over
which soft attention is computed. We show that models utilizing MoChA can be
trained efficiently with standard backpropagation while allowing online and
linear-time decoding at test time. When applied to online speech recognition,
we obtain state-of-the-art results and match the performance of a model using
an offline soft attention mechanism. In document summarization experiments
where we do not expect monotonic alignments, we show significantly improved
performance compared to a baseline monotonic attention-based model.
BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite
Authors: Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Xu Wen, Rui Ren, Chen Zheng, Xiwen He, Hainan Ye, Haoning Tang, Zheng Cao, Shujie Zhang, Jiahui Dai
Date: 2018-02-23 00:00:00
Tasks: Benchmarking, CPU, Diversity, Management

Several fundamental changes in technology indicate domain-specific hardware
and software co-design is the only path left. In this context, architecture,
system, data management, and machine learning communities pay greater attention
to innovative big data and AI algorithms, architecture, and systems.
Unfortunately, complexity, diversity, frequently-changed workloads, and rapid
evolution of big data and AI systems raise great challenges. First, the
traditional benchmarking methodology that creates a new benchmark or proxy for
every possible workload is not scalable, or even impossible for Big Data and AI
benchmarking. Second, it is prohibitively expensive to tailor the architecture
to characteristics of one or more application or even a domain of applications.
We consider each big data and AI workload as a pipeline of one or more classes
of units of computation performed on different initial or intermediate data
inputs, each class of which we call a data motif. On the basis of our previous
work that identifies eight data motifs taking up most of the run time of a wide
variety of big data and AI workloads, we propose a scalable benchmarking
methodology that uses the combination of one or more data motifs---to represent
diversity of big data and AI workloads. Following this methodology, we present
a unified big data and AI benchmark suite---BigDataBench 4.0, publicly
available from~\url{http://prof.ict.ac.cn/BigDataBench}. This unified benchmark
suite sheds new light on domain-specific hardware and software co-design:
tailoring the system and architecture to characteristics of the unified eight
data motifs other than one or more application case by case. Also, for the
first time, we comprehensively characterize the CPU pipeline efficiency using
the benchmarks of seven workload types in BigDataBench 4.0.
Learning to Make Predictions on Graphs with Autoencoders
Authors: Phi Vu Tran
Date: 2018-02-23 00:00:00
Tasks: Classification, General Classification, Graph Representation Learning, Link Prediction, Multi-Task Learning, Node Classification, Prediction, Representation Learning

We examine two fundamental tasks associated with graph representation
learning: link prediction and semi-supervised node classification. We present a
novel autoencoder architecture capable of learning a joint representation of
both local graph structure and available node features for the multi-task
learning of link prediction and node classification. Our autoencoder
architecture is efficiently trained end-to-end in a single learning stage to
simultaneously perform link prediction and node classification, whereas
previous related methods require multiple training steps that are difficult to
optimize. We provide a comprehensive empirical evaluation of our models on nine
benchmark graph-structured datasets and demonstrate significant improvement
over related methods for graph representation learning. Reference code and data
are available at https://github.com/vuptran/graph-representation-learning
Generalization in Deep Learning
Authors: Kenji Kawaguchi, Leslie Pack Kaelbling, Yoshua Bengio
Date: 2017-10-16 00:00:00
Tasks: Deep Learning, Open-Ended Question Answering

This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuous generalization guarantees for deep learning. Based on theoretical observations, we propose new open problems and discuss the limitations of our results.
FigureQA: An Annotated Figure Dataset for Visual Reasoning
Authors: Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, Yoshua Bengio
Date: 2017-10-19 00:00:00
Proceeding: ICLR 2018 1
Tasks: BIG-bench Machine Learning, Chart Question Answering, Relation Network, Visual Question Answering (VQA), Visual Reasoning

We introduce FigureQA, a visual reasoning corpus of over one million
question-answer pairs grounded in over 100,000 images. The images are
synthetic, scientific-style figures from five classes: line plots, dot-line
plots, vertical and horizontal bar graphs, and pie charts. We formulate our
reasoning task by generating questions from 15 templates; questions concern
various relationships between plot elements and examine characteristics like
the maximum, the minimum, area-under-the-curve, smoothness, and intersection.
To resolve, such questions often require reference to multiple plot elements
and synthesis of information distributed spatially throughout a figure. To
facilitate the training of machine learning systems, the corpus also includes
side data that can be used to formulate auxiliary objectives. In particular, we
provide the numerical data used to generate each figure as well as bounding-box
annotations for all plot elements. We study the proposed visual reasoning task
by training several models, including the recently proposed Relation Network as
a strong baseline. Preliminary results indicate that the task poses a
significant machine learning challenge. We envision FigureQA as a first step
towards developing models that can intuitively recognize patterns from visual
representations of data.
Deep Multimodal Learning for Emotion Recognition in Spoken Language
Authors: Yue Gu, Shuhong Chen, Ivan Marsic
Date: 2018-02-22 00:00:00
Tasks: Emotion Recognition, Sentence

In this paper, we present a novel deep multimodal framework to predict human
emotions based on sentence-level spoken language. Our architecture has two
distinctive characteristics. First, it extracts the high-level features from
both text and audio via a hybrid deep multimodal structure, which considers the
spatial information from text, temporal information from audio, and high-level
associations from low-level handcrafted features. Second, we fuse all features
by using a three-layer deep neural network to learn the correlations across
modalities and train the feature extraction and fusion modules together,
allowing optimal global fine-tuning of the entire structure. We evaluated the
proposed framework on the IEMOCAP dataset. Our result shows promising
performance, achieving 60.4% in weighted accuracy for five emotion categories.
Diverse Exploration for Fast and Safe Policy Improvement
Authors: Andrew Cohen, Lei Yu, Robert Wright
Date: 2018-02-22 00:00:00
Tasks: Diversity, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We study an important yet under-addressed problem of quickly and safely
improving policies in online reinforcement learning domains. As its solution,
we propose a novel exploration strategy - diverse exploration (DE), which
learns and deploys a diverse set of safe policies to explore the environment.
We provide DE theory explaining why diversity in behavior policies enables
effective exploration without sacrificing exploitation. Our empirical study
shows that an online policy improvement algorithm framework implementing the DE
strategy can achieve both fast policy improvement and safe online performance.
Hierarchical Representations for Efficient Architecture Search
Authors: Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu
Date: 2017-11-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: General Classification, image-classification, Image Classification, Neural Architecture Search

We explore efficient neural architecture search methods and show that a
simple yet powerful evolutionary algorithm can discover new architectures with
excellent performance. Our approach combines a novel hierarchical genetic
representation scheme that imitates the modularized design pattern commonly
adopted by human experts, and an expressive search space that supports complex
topologies. Our algorithm efficiently discovers architectures that outperform a
large number of manually designed models for image classification, obtaining
top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which
is competitive with the best existing neural architecture search approaches. We
also present results using random search, achieving 0.3% less top-1 accuracy on
CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36
hours down to 1 hour.
On Looking for Local Expansion Invariants in Argumentation Semantics: a Preliminary Report
Authors: Stefano Bistarelli, Francesco Santini, Carlo Taticchi
Date: 2018-02-22 00:00:00
Tasks: Abstract Argumentation

We study invariant local expansion operators for conflict-free and admissible
sets in Abstract Argumentation Frameworks (AFs). Such operators are directly
applied on AFs, and are invariant with respect to a chosen "semantics" (that is
w.r.t. each of the conflict free/admissible set of arguments). Accordingly, we
derive a definition of robustness for AFs in terms of the number of times such
operators can be applied without producing any change in the chosen semantics.
Deep learning algorithm for data-driven simulation of noisy dynamical system
Authors: Kyongmin Yeo, Igor Melnyk
Date: 2018-02-22 00:00:00
Tasks: Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Prediction, Time Series, Time Series Analysis

We present a deep learning model, DE-LSTM, for the simulation of a stochastic
process with an underlying nonlinear dynamics. The deep learning model aims to
approximate the probability density function of a stochastic process via
numerical discretization and the underlying nonlinear dynamics is modeled by
the Long Short-Term Memory (LSTM) network. It is shown that, when the numerical
discretization is used, the function estimation problem can be solved by a
multi-label classification problem. A penalized maximum log likelihood method
is proposed to impose a smoothness condition in the prediction of the
probability distribution. We show that the time evolution of the probability
distribution can be computed by a high-dimensional integration of the
transition probability of the LSTM internal states. A Monte Carlo algorithm to
approximate the high-dimensional integration is outlined. The behavior of
DE-LSTM is thoroughly investigated by using the Ornstein-Uhlenbeck process and
noisy observations of nonlinear dynamical systems; Mackey-Glass time series and
forced Van der Pol oscillator. It is shown that DE-LSTM makes a good prediction
of the probability distribution without assuming any distributional properties
of the stochastic process. For a multiple-step forecast of the Mackey-Glass
time series, the prediction uncertainty, denoted by the 95\% confidence
interval, first grows, then dynamically adjusts following the evolution of the
system, while in the simulation of the forced Van der Pol oscillator, the
prediction uncertainty does not grow in time even for a 3,000-step forecast.
High Order Recurrent Neural Networks for Acoustic Modelling
Authors: Chao Zhang, Philip Woodland
Date: 2018-02-22 00:00:00
Tasks: Acoustic Modelling, speech-recognition, Speech Recognition, Vocal Bursts Intensity Prediction

Vanishing long-term gradients are a major issue in training standard
recurrent neural networks (RNNs), which can be alleviated by long short-term
memory (LSTM) models with memory cells. However, the extra parameters
associated with the memory cells mean an LSTM layer has four times as many
parameters as an RNN with the same hidden vector size. This paper addresses the
vanishing gradient problem using a high order RNN (HORNN) which has additional
connections from multiple previous time steps. Speech recognition experiments
using British English multi-genre broadcast (MGB3) data showed that the
proposed HORNN architectures for rectified linear unit and sigmoid activation
functions reduced word error rates (WER) by 4.2% and 6.3% over the
corresponding RNNs, and gave similar WERs to a (projected) LSTM while using
only 20%--50% of the recurrent layer parameters and computation.