pwc-archive/papers-with-abstracts huggingface dataset:
Gaussian Quadrature for Kernel Features
Authors: Tri Dao, Christopher De Sa, Christopher Ré
Date: 2017-09-08 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: speech-recognition, Speech Recognition

Kernel methods have recently attracted resurgent interest, showing
performance competitive with deep neural networks in tasks such as speech
recognition. The random Fourier features map is a technique commonly used to
scale up kernel machines, but employing the randomized feature map means that
$O(\epsilon^{-2})$ samples are required to achieve an approximation error of at
most $\epsilon$. We investigate some alternative schemes for constructing
feature maps that are deterministic, rather than random, by approximating the
kernel in the frequency domain using Gaussian quadrature. We show that
deterministic feature maps can be constructed, for any $\gamma > 0$, to achieve
error $\epsilon$ with $O(e^{e^\gamma} + \epsilon^{-1/\gamma})$ samples as
$\epsilon$ goes to 0. Our method works particularly well with sparse ANOVA
kernels, which are inspired by the convolutional layer of CNNs. We validate our
methods on datasets in different domains, such as MNIST and TIMIT, showing that
deterministic features are faster to generate and achieve accuracy comparable
to the state-of-the-art kernel methods based on random Fourier features.
Nested LSTMs
Authors: Joel Ruben Antony Moniz, David Krueger
Date: 2018-01-31 00:00:00
Tasks: Language Modeling, Language Modelling

We propose Nested LSTMs (NLSTM), a novel RNN architecture with multiple
levels of memory. Nested LSTMs add depth to LSTMs via nesting as opposed to
stacking. The value of a memory cell in an NLSTM is computed by an LSTM cell,
which has its own inner memory cell. Specifically, instead of computing the
value of the (outer) memory cell as $c^{outer}_t = f_t \odot c_{t-1} + i_t
\odot g_t$, NLSTM memory cells use the concatenation $(f_t \odot c_{t-1}, i_t
\odot g_t)$ as input to an inner LSTM (or NLSTM) memory cell, and set
$c^{outer}_t$ = $h^{inner}_t$. Nested LSTMs outperform both stacked and
single-layer LSTMs with similar numbers of parameters in our experiments on
various character-level language modeling tasks, and the inner memories of an
LSTM learn longer term dependencies compared with the higher-level units of a
stacked LSTM.
A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent
Authors: Ben London
Date: 2017-09-19 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Generalization Bounds

We study the generalization error of randomized learning algorithms -- focusing on stochastic gradient descent (SGD) -- using a novel combination of PAC-Bayes and algorithmic stability. Importantly, our generalization bounds hold for all posterior distributions on an algorithm's random hyperparameters, including distributions that depend on the training data. This inspires an adaptive sampling algorithm for SGD that optimizes the posterior at runtime. We analyze this algorithm in the context of our generalization bounds and evaluate it on a benchmark dataset. Our experiments demonstrate that adaptive sampling can reduce empirical risk faster than uniform sampling while also improving out-of-sample accuracy.
Visual aesthetic analysis using deep neural network: model and techniques to increase accuracy without transfer learning
Authors: Muktabh Mayank Srivastava, Sonaal Kant
Date: 2017-12-09 00:00:00
Tasks: Transfer Learning

We train a deep Convolutional Neural Network (CNN) from scratch for visual
aesthetic analysis in images and discuss techniques we adopt to improve the
accuracy. We avoid the prevalent best transfer learning approaches of using
pretrained weights to perform the task and train a model from scratch to get
accuracy of 78.7% on AVA2 Dataset close to the best models available (85.6%).
We further show that accuracy increases to 81.48% on increasing the training
set by incremental 10 percentile of entire AVA dataset showing our algorithm
gets better with more data.
Netizen-Style Commenting on Fashion Photos: Dataset and Diversity Measures
Authors: Wen Hua Lin, Kuan-Ting Chen, Hung Yueh Chiang, Winston Hsu
Date: 2018-01-31 00:00:00
Tasks: Cultural Vocal Bursts Intensity Prediction, Diversity, Image Captioning, Topic Models

Recently, deep neural network models have achieved promising results in image
captioning task. Yet, "vanilla" sentences, only describing shallow appearances
(e.g., types, colors), generated by current works are not satisfied netizen
style resulting in lacking engagements, contexts, and user intentions. To
tackle this problem, we propose Netizen Style Commenting (NSC), to
automatically generate characteristic comments to a user-contributed fashion
photo. We are devoted to modulating the comments in a vivid "netizen" style
which reflects the culture in a designated social community and hopes to
facilitate more engagement with users. In this work, we design a novel
framework that consists of three major components: (1) We construct a
large-scale clothing dataset named NetiLook, which contains 300K posts (photos)
with 5M comments to discover netizen-style comments. (2) We propose three
unique measures to estimate the diversity of comments. (3) We bring diversity
by marrying topic models with neural networks to make up the insufficiency of
conventional image captioning works. Experimenting over Flickr30k and our
NetiLook datasets, we demonstrate our proposed approaches benefit fashion photo
commenting and improve image captioning tasks both in accuracy and diversity.
Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
Authors: Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, Chengqi Zhang
Date: 2018-01-31 00:00:00
Tasks: Hard Attention, Natural Language Inference, Sentence

Many natural language processing tasks solely rely on sparse dependencies
between a few tokens in a sentence. Soft attention mechanisms show promising
performance in modeling local/global dependencies by soft probabilities between
every two tokens, but they are not effective and efficient when applied to long
sentences. By contrast, hard attention mechanisms directly select a subset of
tokens but are difficult and inefficient to train due to their combinatorial
nature. In this paper, we integrate both soft and hard attention into one
context fusion model, "reinforced self-attention (ReSA)", for the mutual
benefit of each other. In ReSA, a hard attention trims a sequence for a soft
self-attention to process, while the soft attention feeds reward signals back
to facilitate the training of the hard one. For this purpose, we develop a
novel hard attention called "reinforced sequence sampling (RSS)", selecting
tokens in parallel and trained via policy gradient. Using two RSS modules, ReSA
efficiently extracts the sparse dependencies between each pair of selected
tokens. We finally propose an RNN/CNN-free sentence-encoding model, "reinforced
self-attention network (ReSAN)", solely based on ReSA. It achieves
state-of-the-art performance on both Stanford Natural Language Inference (SNLI)
and Sentences Involving Compositional Knowledge (SICK) datasets.
Paraphrase-Supervised Models of Compositionality
Authors: Avneesh Saluja, Chris Dyer, Jean-David Ruvini
Date: 2018-01-31 00:00:00
Tasks: Machine Translation, Translation

Compositional vector space models of meaning promise new solutions to
stubborn language understanding problems. This paper makes two contributions
toward this end: (i) it uses automatically-extracted paraphrase examples as a
source of supervision for training compositional models, replacing previous
work which relied on manual annotations used for the same purpose, and (ii)
develops a context-aware model for scoring phrasal compositionality.
Experimental results indicate that these multiple sources of information can be
used to learn partial semantic supervision that matches previous techniques in
intrinsic evaluation tasks. Our approaches are also evaluated for their impact
on a machine translation system where we show improvements in translation
quality, demonstrating that compositionality in interpretation correlates with
compositionality in translation.
A Cross Entropy based Optimization Algorithm with Global Convergence Guarantees
Authors: Ajin George Joseph, Shalabh Bhatnagar
Date: 2018-01-31 00:00:00
Tasks: global-optimization

The cross entropy (CE) method is a model based search method to solve
optimization problems where the objective function has minimal structure. The
Monte-Carlo version of the CE method employs the naive sample averaging
technique which is inefficient, both computationally and space wise. We provide
a novel stochastic approximation version of the CE method, where the sample
averaging is replaced with incremental geometric averaging. This approach can
save considerable computational and storage costs. Our algorithm is incremental
in nature and possesses additional attractive features such as accuracy,
stability, robustness and convergence to the global optimum for a particular
class of objective functions. We evaluate the algorithm on a variety of global
optimization benchmark problems and the results obtained corroborate our
theoretical findings.
An Incremental Off-policy Search in a Model-free Markov Decision Process Using a Single Sample Path
Authors: Ajin George Joseph, Shalabh Bhatnagar
Date: 2018-01-31 00:00:00

In this paper, we consider a modified version of the control problem in a
model free Markov decision process (MDP) setting with large state and action
spaces. The control problem most commonly addressed in the contemporary
literature is to find an optimal policy which maximizes the value function,
i.e., the long run discounted reward of the MDP. The current settings also
assume access to a generative model of the MDP with the hidden premise that
observations of the system behaviour in the form of sample trajectories can be
obtained with ease from the model. In this paper, we consider a modified
version, where the cost function is the expectation of a non-convex function of
the value function without access to the generative model. Rather, we assume
that a sample trajectory generated using a priori chosen behaviour policy is
made available. In this restricted setting, we solve the modified control
problem in its true sense, i.e., to find the best possible policy given this
limited information. We propose a stochastic approximation algorithm based on
the well-known cross entropy method which is data (sample trajectory)
efficient, stable, robust as well as computationally and storage efficient. We
provide a proof of convergence of our algorithm to a policy which is globally
optimal relative to the behaviour policy. We also present experimental results
to corroborate our claims and we demonstrate the superiority of the solution
produced by our algorithm compared to the state-of-the-art algorithms under
appropriately chosen behaviour policy.
Stochastic Non-convex Ordinal Embedding with Stabilized Barzilai-Borwein Step Size
Authors: Ke Ma, Jinshan Zeng, Jiechao Xiong, Qianqian Xu, Xiaochun Cao, Wei Liu, Yuan YAO
Date: 2017-11-17 00:00:00

Learning representation from relative similarity comparisons, often called
ordinal embedding, gains rising attention in recent years. Most of the existing
methods are batch methods designed mainly based on the convex optimization,
say, the projected gradient descent method. However, they are generally
time-consuming due to that the singular value decomposition (SVD) is commonly
adopted during the update, especially when the data size is very large. To
overcome this challenge, we propose a stochastic algorithm called SVRG-SBB,
which has the following features: (a) SVD-free via dropping convexity, with
good scalability by the use of stochastic algorithm, i.e., stochastic variance
reduced gradient (SVRG), and (b) adaptive step size choice via introducing a
new stabilized Barzilai-Borwein (SBB) method as the original version for convex
problems might fail for the considered stochastic \textit{non-convex}
optimization problem. Moreover, we show that the proposed algorithm converges
to a stationary point at a rate $\mathcal{O}(\frac{1}{T})$ in our setting,
where $T$ is the number of total iterations. Numerous simulations and
real-world data experiments are conducted to show the effectiveness of the
proposed algorithm via comparing with the state-of-the-art methods,
particularly, much lower computational cost with good prediction performance.
Learning Video-Story Composition via Recurrent Neural Network
Authors: Guangyu Zhong, Yi-Hsuan Tsai, Sifei Liu, Zhixun Su, Ming-Hsuan Yang
Date: 2018-01-31 00:00:00

In this paper, we propose a learning-based method to compose a video-story
from a group of video clips that describe an activity or experience. We learn
the coherence between video clips from real videos via the Recurrent Neural
Network (RNN) that jointly incorporates the spatial-temporal semantics and
motion dynamics to generate smooth and relevant compositions. We further
rearrange the results generated by the RNN to make the overall video-story
compatible with the storyline structure via a submodular ranking optimization
process. Experimental results on the video-story dataset show that the proposed
algorithm outperforms the state-of-the-art approach.
Kernel Distillation for Fast Gaussian Processes Prediction
Authors: Congzheng Song, Yiming Sun
Date: 2018-01-31 00:00:00
Tasks: Gaussian Processes, Prediction

Gaussian processes (GPs) are flexible models that can capture complex
structure in large-scale dataset due to their non-parametric nature. However,
the usage of GPs in real-world application is limited due to their high
computational cost at inference time. In this paper, we introduce a new
framework, \textit{kernel distillation}, to approximate a fully trained teacher
GP model with kernel matrix of size $n\times n$ for $n$ training points. We
combine inducing points method with sparse low-rank approximation in the
distillation procedure. The distilled student GP model only costs $O(m^2)$
storage for $m$ inducing points where $m \ll n$ and improves the inference time
complexity. We demonstrate empirically that kernel distillation provides better
trade-off between the prediction time and the test performance compared to the
alternatives.
Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition
Authors: Jianshu Zhang, Jun Du, Li-Rong Dai
Date: 2018-01-05 00:00:00
Tasks: Decoder, Handwritten Mathmatical Expression Recognition, Math

Handwritten mathematical expression recognition is a challenging problem due
to the complicated two-dimensional structures, ambiguous handwriting input and
variant scales of handwritten math symbols. To settle this problem, we utilize
the attention based encoder-decoder model that recognizes mathematical
expression images from two-dimensional layouts to one-dimensional LaTeX
strings. We improve the encoder by employing densely connected convolutional
networks as they can strengthen feature extraction and facilitate gradient
propagation especially on a small training set. We also present a novel
multi-scale attention model which is employed to deal with the recognition of
math symbols in different scales and save the fine-grained details that will be
dropped by pooling operations. Validated on the CROHME competition task, the
proposed method significantly outperforms the state-of-the-art methods with an
expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME
2016, by only using the official training dataset.
Engineering fast multilevel support vector machines
Authors: E. Sadrfaridpour, T. Razzaghi, I. Safro
Date: 2017-07-24 00:00:00

The computational complexity of solving nonlinear support vector machine
(SVM) is prohibitive on large-scale data. In particular, this issue becomes
very sensitive when the data represents additional difficulties such as highly
imbalanced class sizes. Typically, nonlinear kernels produce significantly
higher classification quality to linear kernels but introduce extra kernel and
model parameters which requires computationally expensive fitting. This
increases the quality but also reduces the performance dramatically. We
introduce a generalized fast multilevel framework for regular and weighted SVM
and discuss several versions of its algorithmic components that lead to a good
trade-off between quality and time. Our framework is implemented using PETSc
which allows an easy integration with scientific computing tasks. The
experimental results demonstrate significant speed up compared to the
state-of-the-art nonlinear SVM libraries.
  Reproducibility: our source code, documentation and parameters are available
at https:// github.com/esadr/mlsvm.
DxNAT - Deep Neural Networks for Explaining Non-Recurring Traffic Congestion
Authors: Fangzhou sun, Abhishek Dubey, Jules White
Date: 2018-01-30 00:00:00
Tasks: Data Augmentation

Non-recurring traffic congestion is caused by temporary disruptions, such as
accidents, sports games, adverse weather, etc. We use data related to real-time
traffic speed, jam factors (a traffic congestion indicator), and events
collected over a year from Nashville, TN to train a multi-layered deep neural
network. The traffic dataset contains over 900 million data records. The
network is thereafter used to classify the real-time data and identify
anomalous operations. Compared with traditional approaches of using statistical
or machine learning techniques, our model reaches an accuracy of 98.73 percent
when identifying traffic congestion caused by football games. Our approach
first encodes the traffic across a region as a scaled image. After that the
image data from different timestamps is fused with event- and time-related
data. Then a crossover operator is used as a data augmentation method to
generate training datasets with more balanced classes. Finally, we use the
receiver operating characteristic (ROC) analysis to tune the sensitivity of the
classifier. We present the analysis of the training time and the inference time
separately.
FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
Authors: Jie Chen, Tengfei Ma, Cao Xiao
Date: 2018-01-30 00:00:00
Proceeding: ICLR 2018 1
Tasks: Node Classification

The graph convolutional networks (GCN) recently proposed by Kipf and Welling
are an effective graph model for semi-supervised learning. This model, however,
was originally designed to be learned with the presence of both training and
test data. Moreover, the recursive neighborhood expansion across layers poses
time and memory challenges for training with large, dense graphs. To relax the
requirement of simultaneous availability of test data, we interpret graph
convolutions as integral transforms of embedding functions under probability
measures. Such an interpretation allows for the use of Monte Carlo approaches
to consistently estimate the integrals, which in turn leads to a batched
training scheme as we propose in this work---FastGCN. Enhanced with importance
sampling, FastGCN not only is efficient for training but also generalizes well
for inference. We show a comprehensive set of experiments to demonstrate its
effectiveness compared with GCN and related models. In particular, training is
orders of magnitude more efficient while predictions remain comparably
accurate.
Low-Rank Bandit Methods for High-Dimensional Dynamic Pricing
Authors: Jonas Mueller, Vasilis Syrgkanis, Matt Taddy
Date: 2018-01-30 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Vocal Bursts Intensity Prediction

We consider dynamic pricing with many products under an evolving but low-dimensional demand model. Assuming the temporal variation in cross-elasticities exhibits low-rank structure based on fixed (latent) features of the products, we show that the revenue maximization problem reduces to an online bandit convex optimization with side information given by the observed demands. We design dynamic pricing algorithms whose revenue approaches that of the best fixed price vector in hindsight, at a rate that only depends on the intrinsic rank of the demand model and not the number of products. Our approach applies a bandit convex optimization algorithm in a projected low-dimensional space spanned by the latent product features, while simultaneously learning this span via online singular value decomposition of a carefully-crafted matrix containing the observed demands.
Czech Text Document Corpus v 2.0
Authors: Pavel Král, Ladislav Lenc
Date: 2017-10-06 00:00:00
Proceeding: LREC 2018 5
Tasks: Classification, Document Classification, General Classification

This paper introduces "Czech Text Document Corpus v 2.0", a collection of
text documents for automatic document classification in Czech language. It is
composed of the text documents provided by the Czech News Agency and is freely
available for research purposes at http://ctdc.kiv.zcu.cz/. This corpus was
created in order to facilitate a straightforward comparison of the document
classification approaches on Czech data. It is particularly dedicated to
evaluation of multi-label document classification approaches, because one
document is usually labelled with more than one label. Besides the information
about the document classes, the corpus is also annotated at the morphological
layer. This paper further shows the results of selected state-of-the-art
methods on this corpus to offer the possibility of an easy comparison with
these approaches.
Approximate Supermodularity Bounds for Experimental Design
Authors: Luiz. F. O. Chamon, Alejandro Ribeiro
Date: 2017-11-04 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Experimental Design

This work provides performance guarantees for the greedy solution of
experimental design problems. In particular, it focuses on A- and E-optimal
designs, for which typical guarantees do not apply since the mean-square error
and the maximum eigenvalue of the estimation error covariance matrix are not
supermodular. To do so, it leverages the concept of approximate supermodularity
to derive non-asymptotic worst-case suboptimality bounds for these greedy
solutions. These bounds reveal that as the SNR of the experiments decreases,
these cost functions behave increasingly as supermodular functions. As such,
greedy A- and E-optimal designs approach (1-1/e)-optimality. These results
reconcile the empirical success of greedy experimental design with the
non-supermodularity of the A- and E-optimality criteria.
Towards Robust Deep Neural Networks with BANG
Authors: Andras Rozsa, Manuel Gunther, Terrance E. Boult
Date: 2016-12-01 00:00:00
Tasks: BIG-bench Machine Learning, Data Augmentation, General Classification

Machine learning models, including state-of-the-art deep neural networks, are
vulnerable to small perturbations that cause unexpected classification errors.
This unexpected lack of robustness raises fundamental questions about their
generalization properties and poses a serious concern for practical
deployments. As such perturbations can remain imperceptible - the formed
adversarial examples demonstrate an inherent inconsistency between vulnerable
machine learning models and human perception - some prior work casts this
problem as a security issue. Despite the significance of the discovered
instabilities and ensuing research, their cause is not well understood and no
effective method has been developed to address the problem. In this paper, we
present a novel theory to explain why this unpleasant phenomenon exists in deep
neural networks. Based on that theory, we introduce a simple, efficient, and
effective training approach, Batch Adjusted Network Gradients (BANG), which
significantly improves the robustness of machine learning models. While the
BANG technique does not rely on any form of data augmentation or the
utilization of adversarial images for training, the resultant classifiers are
more resistant to adversarial perturbations while maintaining or even enhancing
the overall classification performance.