pwc-archive/papers-with-abstracts huggingface dataset:
Avoiding overfitting of multilayer perceptrons by training derivatives
Authors: V. I. Avrutskiy
Date: 2018-02-28 00:00:00

Resistance to overfitting is observed for neural networks trained with
extended backpropagation algorithm. In addition to target values, its cost
function uses derivatives of those up to the $4^{\mathrm{th}}$ order. For
common applications of neural networks, high order derivatives are not readily
available, so simpler cases are considered: training network to approximate
analytical function inside 2D and 5D domains and solving Poisson equation
inside a 2D circle. For function approximation, the cost is a sum of squared
differences between output and target as well as their derivatives with respect
to the input. Differential equations are usually solved by putting a multilayer
perceptron in place of unknown function and training its weights, so that
equation holds within some margin of error. Commonly used cost is the
equation's residual squared. Added terms are squared derivatives of said
residual with respect to the independent variables. To investigate overfitting,
the cost is minimized for points of regular grids with various spacing, and its
root mean is compared with its value on much denser test set. Fully connected
perceptrons with six hidden layers and $2\cdot10^{4}$, $1\cdot10^{6}$ and
$5\cdot10^{6}$ weights in total are trained with Rprop until cost changes by
less than 10% for last 1000 epochs, or when the $10000^{\mathrm{th}}$ epoch is
reached. Training the network with $5\cdot10^{6}$ weights to represent simple
2D function using 10 points with 8 extra derivatives in each produces cost test
to train ratio of $1.5$, whereas for classical backpropagation in comparable
conditions this ratio is $2\cdot10^{4}$.
Hardware-Efficient Guided Image Filtering For Multi-Label Problem
Authors: Longquan Dai, Mengke Yuan, Zechao Li, Xiaopeng Zhang, Jinhui Tang
Date: 2018-02-28 00:00:00
Proceeding: CVPR 2017 7

The Guided Filter (GF) is well-known for its linear complexity. However, when
filtering an image with an n-channel guidance, GF needs to invert an n x n
matrix for each pixel. To the best of our knowledge existing matrix inverse
algorithms are inefficient on current hardwares. This shortcoming limits
applications of multichannel guidance in computation intensive system such as
multi-label system. We need a new GF-like filter that can perform fast
multichannel image guided filtering. Since the optimal linear complexity of GF
cannot be minimized further, the only way thus is to bring all potentialities
of current parallel computing hardwares into full play. In this paper we
propose a hardware-efficient Guided Filter (HGF), which solves the efficiency
problem of multichannel guided image filtering and yields competent results
when applying it to multi-label problems with synthesized polynomial
multichannel guidance. Specifically, in order to boost the filtering
performance, HGF takes a new matrix inverse algorithm which only involves two
hardware-efficient operations: element-wise arithmetic calculations and box
filtering. In order to break the linear model restriction, HGF synthesizes a
polynomial multichannel guidance to introduce nonlinearity. Benefiting from our
polynomial guidance and hardware-efficient matrix inverse algorithm, HGF not
only is more sensitive to the underlying structure of guidance but also
achieves the fastest computing speed. Due to these merits, HGF obtains
state-of-the-art results in terms of accuracy and efficiency in the computation
intensive multi-label
Speeding Up the Bilateral Filter: A Joint Acceleration Way
Authors: Longquan Dai, Mengke Yuan, Xiaopeng Zhang
Date: 2018-02-28 00:00:00

Computational complexity of the brute-force implementation of the bilateral
filter (BF) depends on its filter kernel size. To achieve the constant-time BF
whose complexity is irrelevant to the kernel size, many techniques have been
proposed, such as 2D box filtering, dimension promotion, and shiftability
property. Although each of the above techniques suffers from accuracy and
efficiency problems, previous algorithm designers were used to take only one of
them to assemble fast implementations due to the hardness of combining them
together. Hence, no joint exploitation of these techniques has been proposed to
construct a new cutting edge implementation that solves these problems. Jointly
employing five techniques: kernel truncation, best N -term approximation as
well as previous 2D box filtering, dimension promotion, and shiftability
property, we propose a unified framework to transform BF with arbitrary spatial
and range kernels into a set of 3D box filters that can be computed in linear
time. To the best of our knowledge, our algorithm is the first method that can
integrate all these acceleration techniques and, therefore, can draw upon one
another's strong point to overcome deficiencies. The strength of our method has
been corroborated by several carefully designed experiments. In particular, the
filtering accuracy is significantly improved without sacrificing the efficiency
at running time.
Deep Rotation Equivariant Network
Authors: Junying Li, Zichen Yang, Haifeng Liu, Deng Cai
Date: 2017-05-24 00:00:00
Tasks: Rotated MNIST

Recently, learning equivariant representations has attracted considerable
research attention. Dieleman et al. introduce four operations which can be
inserted into convolutional neural network to learn deep representations
equivariant to rotation. However, feature maps should be copied and rotated
four times in each layer in their approach, which causes much running time and
memory overhead. In order to address this problem, we propose Deep Rotation
Equivariant Network consisting of cycle layers, isotonic layers and decycle
layers. Our proposed layers apply rotation transformation on filters rather
than feature maps, achieving a speed up of more than 2 times with even less
memory overhead. We evaluate DRENs on Rotated MNIST and CIFAR-10 datasets and
demonstrate that it can improve the performance of state-of-the-art
architectures.
Learning Graphs with Monotone Topology Properties and Multiple Connected Components
Authors: Eduardo Pavez, Hilmi E. Egilmez, Antonio Ortega
Date: 2017-05-31 00:00:00

Recent papers have formulated the problem of learning graphs from data as an
inverse covariance estimation with graph Laplacian constraints. While such
problems are convex, existing methods cannot guarantee that solutions will have
specific graph topology properties (e.g., being $k$-partite), which are
desirable for some applications. In fact, the problem of learning a graph with
given topology properties, e.g., finding the $k$-partite graph that best
matches the data, is in general non-convex. In this paper, we develop novel
theoretical results that provide performance guarantees for an approach to
solve these problems. Our solution decomposes this problem into two
sub-problems, for which efficient solutions are known. Specifically, a graph
topology inference (GTI) step is employed to select a feasible graph topology,
i.e., one having the desired property. Then, a graph weight estimation (GWE)
step is performed by solving a generalized graph Laplacian estimation problem,
where edges are constrained by the topology found in the GTI step. Our main
result is a bound on the error of the GWE step as a function of the error in
the GTI step. This error bound indicates that the GTI step should be solved
using an algorithm that approximates the similarity matrix by another matrix
whose entries have been thresholded to zero to have the desired type of graph
topology. The GTI stage can leverage existing methods (e.g., state of the art
approaches for graph coloring) which are typically based on minimizing the
total weight of removed edges. Since the GWE stage is formulated as an inverse
covariance estimation problem with linear constraints, it can be solved using
existing convex optimization methods. We demonstrate that our two step approach
can achieve good results for both synthetic and texture image data.
Multi-resolution Tensor Learning for Large-Scale Spatial Data
Authors: Stephan Zheng, Rose Yu, Yisong Yue
Date: 2018-02-19 00:00:00
Tasks: Meta-Learning

High-dimensional tensor models are notoriously computationally expensive to
train. We present a meta-learning algorithm, MMT, that can significantly speed
up the process for spatial tensor models. MMT leverages the property that
spatial data can be viewed at multiple resolutions, which are related by
coarsening and finegraining from one resolution to another. Using this
property, MMT learns a tensor model by starting from a coarse resolution and
iteratively increasing the model complexity. In order to not "over-train" on
coarse resolution models, we investigate an information-theoretic fine-graining
criterion to decide when to transition into higher-resolution models. We
provide both theoretical and empirical evidence for the advantages of this
approach. When applied to two real-world large-scale spatial datasets for
basketball player and animal behavior modeling, our approach demonstrate 3 key
benefits: 1) it efficiently captures higher-order interactions (i.e., tensor
latent factors), 2) it is orders of magnitude faster than fixed resolution
learning and scales to very fine-grained spatial resolutions, and 3) it
reliably yields accurate and interpretable models.
Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs
Authors: Xuhao Chen
Date: 2018-02-28 00:00:00

Deep neural networks have achieved remarkable accuracy in many artificial
intelligence applications, e.g. computer vision, at the cost of a large number
of parameters and high computational complexity. Weight pruning can compress
DNN models by removing redundant parameters in the networks, but it brings
sparsity in the weight matrix, and therefore makes the computation inefficient
on GPUs. Although pruning can remove more than 80% of the weights, it actually
hurts inference performance (speed) when running models on GPUs.
  Two major problems cause this unsatisfactory performance on GPUs. First,
lowering convolution onto matrix multiplication reduces data reuse
opportunities and wastes memory bandwidth. Second, the sparsity brought by
pruning makes the computation irregular, which leads to inefficiency when
running on massively parallel GPUs. To overcome these two limitations, we
propose Escort, an efficient sparse convolutional neural networks on GPUs.
Instead of using the lowering method, we choose to compute the sparse
convolutions directly. We then orchestrate the parallelism and locality for the
direct sparse convolution kernel, and apply customized optimization techniques
to further improve performance. Evaluation on NVIDIA GPUs show that Escort can
improve sparse convolution speed by 2.63x and 3.07x, and inference speed by
1.43x and 1.69x, compared to CUBLAS and CUSPARSE respectively.
Medical Exam Question Answering with Large-scale Reading Comprehension
Authors: Xiao Zhang, Ji Wu, ZhiYang He, Xien Liu, Ying Su
Date: 2018-02-28 00:00:00
Tasks: MedQA, Question Answering, Reading Comprehension

Reading and understanding text is one important component in computer aided
diagnosis in clinical medicine, also being a major research problem in the
field of NLP. In this work, we introduce a question-answering task called MedQA
to study answering questions in clinical medicine using knowledge in a
large-scale document collection. The aim of MedQA is to answer real-world
questions with large-scale reading comprehension. We propose our solution
SeaReader--a modular end-to-end reading comprehension model based on LSTM
networks and dual-path attention architecture. The novel dual-path attention
models information flow from two perspectives and has the ability to
simultaneously read individual documents and integrate information across
multiple documents. In experiments our SeaReader achieved a large increase in
accuracy on MedQA over competing models. Additionally, we develop a series of
novel techniques to demonstrate the interpretation of the question answering
process in SeaReader.
Solving for high dimensional committor functions using artificial neural networks
Authors: Yuehaw Khoo, Jianfeng Lu, Lexing Ying
Date: 2018-02-28 00:00:00
Tasks: Vocal Bursts Intensity Prediction

In this note we propose a method based on artificial neural network to study
the transition between states governed by stochastic processes. In particular,
we aim for numerical schemes for the committor function, the central object of
transition path theory, which satisfies a high-dimensional Fokker-Planck
equation. By working with the variational formulation of such partial
differential equation and parameterizing the committor function in terms of a
neural network, approximations can be obtained via optimizing the neural
network weights using stochastic algorithms. The numerical examples show that
moderate accuracy can be achieved for high-dimensional problems.
Selective Experience Replay for Lifelong Learning
Authors: David Isele, Akansel Cosgun
Date: 2018-02-28 00:00:00
Tasks: Deep Reinforcement Learning, Lifelong learning, Reinforcement Learning

Deep reinforcement learning has emerged as a powerful tool for a variety of
learning tasks, however deep nets typically exhibit forgetting when learning
multiple tasks in sequence. To mitigate forgetting, we propose an experience
replay process that augments the standard FIFO buffer and selectively stores
experiences in a long-term memory. We explore four strategies for selecting
which experiences will be stored: favoring surprise, favoring reward, matching
the global training distribution, and maximizing coverage of the state space.
We show that distribution matching successfully prevents catastrophic
forgetting, and is consistently the best approach on all domains tested. While
distribution matching has better and more consistent performance, we identify
one case in which coverage maximization is beneficial - when tasks that receive
less trained are more important. Overall, our results show that selective
experience replay, when suitable selection algorithms are employed, can prevent
catastrophic forgetting.
Defense against Universal Adversarial Perturbations
Authors: Naveed Akhtar, Jian Liu, Ajmal Mian
Date: 2017-11-16 00:00:00
Proceeding: CVPR 2018 6

Recent advances in Deep Learning show the existence of image-agnostic
quasi-imperceptible perturbations that when applied to `any' image can fool a
state-of-the-art network classifier to change its prediction about the image
label. These `Universal Adversarial Perturbations' pose a serious threat to the
success of Deep Learning in practice. We present the first dedicated framework
to effectively defend the networks against such perturbations. Our approach
learns a Perturbation Rectifying Network (PRN) as `pre-input' layers to a
targeted model, such that the targeted model needs no modification. The PRN is
learned from real and synthetic image-agnostic perturbations, where an
efficient method to compute the latter is also proposed. A perturbation
detector is separately trained on the Discrete Cosine Transform of the
input-output difference of the PRN. A query image is first passed through the
PRN and verified by the detector. If a perturbation is detected, the output of
the PRN is used for label prediction instead of the actual image. A rigorous
evaluation shows that our framework can defend the network classifiers against
unseen adversarial perturbations in the real-world scenarios with up to 97.5%
success rate. The PRN also generalizes well in the sense that training for one
targeted network defends another network with a comparable success rate.
Semi-Analytic Resampling in Lasso
Authors: Tomoyuki Obuchi, Yoshiyuki Kabashima
Date: 2018-02-28 00:00:00
Tasks: Variable Selection

An approximate method for conducting resampling in Lasso, the $\ell_1$
penalized linear regression, in a semi-analytic manner is developed, whereby
the average over the resampled datasets is directly computed without repeated
numerical sampling, thus enabling an inference free of the statistical
fluctuations due to sampling finiteness, as well as a significant reduction of
computational time. The proposed method is based on a message passing type
algorithm, and its fast convergence is guaranteed by the state evolution
analysis, when covariates are provided as zero-mean independently and
identically distributed Gaussian random variables. It is employed to implement
bootstrapped Lasso (Bolasso) and stability selection, both of which are
variable selection methods using resampling in conjunction with Lasso, and
resolves their disadvantage regarding computational cost. To examine
approximation accuracy and efficiency, numerical experiments were carried out
using simulated datasets. Moreover, an application to a real-world dataset, the
wine quality dataset, is presented. To process such real-world datasets, an
objective criterion for determining the relevance of selected variables is also
introduced by the addition of noise variables and resampling.
IM2HEIGHT: Height Estimation from Single Monocular Imagery via Fully Residual Convolutional-Deconvolutional Network
Authors: Lichao Mou, Xiao Xiang Zhu
Date: 2018-02-28 00:00:00
Tasks: Instance Segmentation, Semantic Segmentation

In this paper we tackle a very novel problem, namely height estimation from a
single monocular remote sensing image, which is inherently ambiguous, and a
technically ill-posed problem, with a large source of uncertainty coming from
the overall scale. We propose a fully convolutional-deconvolutional network
architecture being trained end-to-end, encompassing residual learning, to model
the ambiguous mapping between monocular remote sensing images and height maps.
Specifically, it is composed of two parts, i.e., convolutional sub-network and
deconvolutional sub-network. The former corresponds to feature extractor that
transforms the input remote sensing image to high-level multidimensional
feature representation, whereas the latter plays the role of a height generator
that produces height map from the feature extracted from the convolutional
sub-network. Moreover, to preserve fine edge details of estimated height maps,
we introduce a skip connection to the network, which is able to shuttle
low-level visual information, e.g., object boundaries and edges, directly
across the network. To demonstrate the usefulness of single-view height
prediction, we show a practical example of instance segmentation of buildings
using estimated height map. This paper, for the first time in the remote
sensing community, attempts to estimate height from monocular vision. The
proposed network is validated using a large-scale high resolution aerial image
data set covered an area of Berlin. Both visual and quantitative analysis of
the experimental results demonstrate the effectiveness of our approach.
Tractable and Scalable Schatten Quasi-Norm Approximations for Rank Minimization
Authors: Fanhua Shang, Yuanyuan Liu, James Cheng
Date: 2018-02-28 00:00:00

The Schatten quasi-norm was introduced to bridge the gap between the trace
norm and rank function. However, existing algorithms are too slow or even
impractical for large-scale problems. Motivated by the equivalence relation
between the trace norm and its bilinear spectral penalty, we define two
tractable Schatten norms, i.e.\ the bi-trace and tri-trace norms, and prove
that they are in essence the Schatten-$1/2$ and $1/3$ quasi-norms,
respectively. By applying the two defined Schatten quasi-norms to various rank
minimization problems such as MC and RPCA, we only need to solve much smaller
factor matrices. We design two efficient linearized alternating minimization
algorithms to solve our problems and establish that each bounded sequence
generated by our algorithms converges to a critical point. We also provide the
restricted strong convexity (RSC) based and MC error bounds for our algorithms.
Our experimental results verified both the efficiency and effectiveness of our
algorithms compared with the state-of-the-art methods.
Learners that Use Little Information
Authors: Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, Amir Yehudayoff
Date: 2017-10-14 00:00:00

We study learning algorithms that are restricted to using a small amount of
information from their input sample. We introduce a category of learning
algorithms we term $d$-bit information learners, which are algorithms whose
output conveys at most $d$ bits of information of their input. A central theme
in this work is that such algorithms generalize.
  We focus on the learning capacity of these algorithms, and prove sample
complexity bounds with tight dependencies on the confidence and error
parameters. We also observe connections with well studied notions such as
sample compression schemes, Occam's razor, PAC-Bayes and differential privacy.
  We discuss an approach that allows us to prove upper bounds on the amount of
information that algorithms reveal about their inputs, and also provide a lower
bound by showing a simple concept class for which every (possibly randomized)
empirical risk minimizer must reveal a lot of information. On the other hand,
we show that in the distribution-dependent setting every VC class has empirical
risk minimizers that do not reveal a lot of information.
Neural Aesthetic Image Reviewer
Authors: Wenshan Wang, Su Yang, Weishan Zhang, Jiulong Zhang
Date: 2018-02-28 00:00:00
Tasks: Multi-Task Learning

Recently, there is a rising interest in perceiving image aesthetics. The
existing works deal with image aesthetics as a classification or regression
problem. To extend the cognition from rating to reasoning, a deeper
understanding of aesthetics should be based on revealing why a high- or
low-aesthetic score should be assigned to an image. From such a point of view,
we propose a model referred to as Neural Aesthetic Image Reviewer, which can
not only give an aesthetic score for an image, but also generate a textual
description explaining why the image leads to a plausible rating score.
Specifically, we propose two multi-task architectures based on shared
aesthetically semantic layers and task-specific embedding layers at a high
level for performance improvement on different tasks. To facilitate researches
on this problem, we collect the AVA-Reviews dataset, which contains 52,118
images and 312,708 comments in total. Through multi-task learning, the proposed
models can rate aesthetic images as well as produce comments in an end-to-end
manner. It is confirmed that the proposed models outperform the baselines
according to the performance evaluation on the AVA-Reviews dataset. Moreover,
we demonstrate experimentally that our model can generate textual reviews
related to aesthetics, which are consistent with human perception.