pwc-archive/papers-with-abstracts huggingface dataset:
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Authors: Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington
Date: 2018-06-14 00:00:00
Proceeding: ICML 2018 7

In recent years, state-of-the-art methods in computer vision have utilized
increasingly deep convolutional neural network architectures (CNNs), with some
of the most successful models employing hundreds or even thousands of layers. A
variety of pathologies such as vanishing/exploding gradients make training such
deep networks challenging. While residual connections and batch normalization
do enable training at these depths, it has remained unclear whether such
specialized architecture designs are truly necessary to train deep CNNs. In
this work, we demonstrate that it is possible to train vanilla CNNs with ten
thousand layers or more simply by using an appropriate initialization scheme.
We derive this initialization scheme theoretically by developing a mean field
theory for signal propagation and by characterizing the conditions for
dynamical isometry, the equilibration of singular values of the input-output
Jacobian matrix. These conditions require that the convolution operator be an
orthogonal transformation in the sense that it is norm-preserving. We present
an algorithm for generating such random initial orthogonal convolution kernels
and demonstrate empirically that they enable efficient training of extremely
deep architectures.
How Do Source-side Monolingual Word Embeddings Impact Neural Machine Translation?
Authors: Shuoyang Ding, Kevin Duh
Date: 2018-06-05 00:00:00
Tasks: Machine Translation, NMT, Translation, Word Embeddings

Using pre-trained word embeddings as input layer is a common practice in many
natural language processing (NLP) tasks, but it is largely neglected for neural
machine translation (NMT). In this paper, we conducted a systematic analysis on
the effect of using pre-trained source-side monolingual word embedding in NMT.
We compared several strategies, such as fixing or updating the embeddings
during NMT training on varying amounts of data, and we also proposed a novel
strategy called dual-embedding that blends the fixing and updating strategies.
Our results suggest that pre-trained embeddings can be helpful if properly
incorporated into NMT, especially when parallel data is limited or additional
in-domain monolingual data is readily available.
Theory of Estimation-of-Distribution Algorithms
Authors: Martin S. Krejca, Carsten Witt
Date: 2018-06-14 00:00:00
Tasks: Evolutionary Algorithms

Estimation-of-distribution algorithms (EDAs) are general metaheuristics used
in optimization that represent a more recent alternative to classical
approaches like evolutionary algorithms. In a nutshell, EDAs typically do not
directly evolve populations of search points but build probabilistic models of
promising solutions by repeatedly sampling and selecting points from the
underlying search space. Recently, there has been made significant progress in
the theoretical understanding of EDAs. This article provides an up-to-date
overview of the most commonly analyzed EDAs and the most recent theoretical
results in this area. In particular, emphasis is put on the runtime analysis of
simple univariate EDAs, including a description of typical benchmark functions
and tools for the analysis. Along the way, open problems and directions for
future research are described.
Convex Coupled Matrix and Tensor Completion
Authors: Kishan Wimalawarne, Makoto Yamada, Hiroshi Mamitsuka
Date: 2017-05-15 00:00:00

We propose a set of convex low rank inducing norms for a coupled matrices and
tensors (hereafter coupled tensors), which shares information between matrices
and tensors through common modes. More specifically, we propose a mixture of
the overlapped trace norm and the latent norms with the matrix trace norm, and
then, we propose a new completion algorithm based on the proposed norms. A key
advantage of the proposed norms is that it is convex and can find a globally
optimal solution, while existing methods for coupled learning are non-convex.
Furthermore, we analyze the excess risk bounds of the completion model
regularized by our proposed norms which show that our proposed norms can
exploit the low rankness of coupled tensors leading to better bounds compared
to uncoupled norms. Through synthetic and real-world data experiments, we show
that the proposed completion algorithm compares favorably with existing
completion algorithms.
Parameter Learning and Change Detection Using a Particle Filter With Accelerated Adaptation
Authors: Karol Gellert, Erik Schlögl
Date: 2018-06-14 00:00:00
Tasks: Change Detection

This paper presents the construction of a particle filter, which incorporates
elements inspired by genetic algorithms, in order to achieve accelerated
adaptation of the estimated posterior distribution to changes in model
parameters. Specifically, the filter is designed for the situation where the
subsequent data in online sequential filtering does not match the model
posterior filtered based on data up to a current point in time. The examples
considered encompass parameter regime shifts and stochastic volatility. The
filter adapts to regime shifts extremely rapidly and delivers a clear heuristic
for distinguishing between regime shifts and stochastic volatility, even though
the model dynamics assumed by the filter exhibit neither of those features.
Joint Blind Motion Deblurring and Depth Estimation of Light Field
Authors: Dongwoo Lee, Haesol Park, In Kyu Park, Kyoung Mu Lee
Date: 2017-11-29 00:00:00
Proceeding: ECCV 2018 9
Tasks: Deblurring, Depth Estimation

Removing camera motion blur from a single light field is a challenging task
since it is highly ill-posed inverse problem. The problem becomes even worse
when blur kernel varies spatially due to scene depth variation and high-order
camera motion. In this paper, we propose a novel algorithm to estimate all blur
model variables jointly, including latent sub-aperture image, camera motion,
and scene depth from the blurred 4D light field. Exploiting multi-view nature
of a light field relieves the inverse property of the optimization by utilizing
strong depth cues and multi-view blur observation. The proposed joint
estimation achieves high quality light field deblurring and depth estimation
simultaneously under arbitrary 6-DOF camera motion and unconstrained scene
depth. Intensive experiment on real and synthetic blurred light field confirms
that the proposed algorithm outperforms the state-of-the-art light field
deblurring and depth estimation methods.
PCAS: Pruning Channels with Attention Statistics for Deep Network Compression
Authors: Kohei Yamamoto, Kurato Maeno
Date: 2018-06-14 00:00:00
Tasks: channel selection

Compression techniques for deep neural networks are important for implementing them on small embedded devices. In particular, channel-pruning is a useful technique for realizing compact networks. However, many conventional methods require manual setting of compression ratios in each layer. It is difficult to analyze the relationships between all layers, especially for deeper models. To address these issues, we propose a simple channel-pruning technique based on attention statistics that enables to evaluate the importance of channels. We improved the method by means of a criterion for automatic channel selection, using a single compression ratio for the entire model in place of per-layer model analysis. The proposed approach achieved superior performance over conventional methods with respect to accuracy and the computational costs for various models and datasets. We provide analysis results for behavior of the proposed criterion on different datasets to demonstrate its favorable properties for channel pruning.
Single Image Reflection Separation with Perceptual Losses
Authors: Xuaner Zhang, Ren Ng, Qifeng Chen
Date: 2018-06-14 00:00:00
Proceeding: CVPR 2018 6
Tasks: Image Enhancement, Reflection Removal, SSIM

We present an approach to separating reflection from a single image. The
approach uses a fully convolutional network trained end-to-end with losses that
exploit low-level and high-level image information. Our loss function includes
two perceptual losses: a feature loss from a visual perception network, and an
adversarial loss that encodes characteristics of images in the transmission
layers. We also propose a novel exclusion loss that enforces pixel-level layer
separation. We create a dataset of real-world images with reflection and
corresponding ground-truth transmission layers for quantitative evaluation and
model training. We validate our method through comprehensive quantitative
experiments and show that our approach outperforms state-of-the-art reflection
removal methods in PSNR, SSIM, and perceptual user study. We also extend our
method to two other image enhancement tasks to demonstrate the generality of
our approach.
Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition
Authors: Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding
Date: 2018-06-14 00:00:00
Proceeding: ECCV 2018 9
Tasks: Fine-Grained Image Recognition, Metric Learning

Attention-based learning for fine-grained image recognition remains a
challenging task, where most of the existing methods treat each object part in
isolation, while neglecting the correlations among them. In addition, the
multi-stage or multi-scale mechanisms involved make the existing methods less
efficient and hard to be trained end-to-end. In this paper, we propose a novel
attention-based convolutional neural network (CNN) which regulates multiple
object parts among different input images. Our method first learns multiple
attention region features of each input image through the one-squeeze
multi-excitation (OSME) module, and then apply the multi-attention multi-class
constraint (MAMC) in a metric learning framework. For each anchor feature, the
MAMC functions by pulling same-attention same-class features closer, while
pushing different-attention or different-class features away. Our method can be
easily trained end-to-end, and is highly efficient which requires only one
training stage. Moreover, we introduce Dogs-in-the-Wild, a comprehensive dog
species dataset that surpasses similar existing datasets by category coverage,
data volume and annotation quality. This dataset will be released upon
acceptance to facilitate the research of fine-grained image recognition.
Extensive experiments are conducted to show the substantial improvements of our
method on four benchmark datasets.
A Fast Proximal Point Method for Computing Exact Wasserstein Distance
Authors: Yujia Xie, Xiangfeng Wang, Ruijia Wang, Hongyuan Zha
Date: 2018-02-12 00:00:00
Tasks: BIG-bench Machine Learning

Wasserstein distance plays increasingly important roles in machine learning, stochastic programming and image processing. Major efforts have been under way to address its high computational complexity, some leading to approximate or regularized variations such as Sinkhorn distance. However, as we will demonstrate, regularized variations with large regularization parameter will degradate the performance in several important machine learning applications, and small regularization parameter will fail due to numerical stability issues with existing algorithms. We address this challenge by developing an Inexact Proximal point method for exact Optimal Transport problem (IPOT) with the proximal operator approximately evaluated at each iteration using projections to the probability simplex. The algorithm (a) converges to exact Wasserstein distance with theoretical guarantee and robust regularization parameter selection, (b) alleviates numerical stability issue, (c) has similar computational complexity to Sinkhorn, and (d) avoids the shrinking problem when apply to generative models. Furthermore, a new algorithm is proposed based on IPOT to obtain sharper Wasserstein barycenter.
Fire SSD: Wide Fire Modules based Single Shot Detector on Edge Device
Authors: Hengfui Liau, Nimmagadda Yamini, YengLiong Wong
Date: 2018-06-14 00:00:00
Tasks: CPU, Edge-computing, GPU, Object, object-detection, Object Detection

With the emergence of edge computing, there is an increasing need for running
convolutional neural network based object detection on small form factor edge
computing devices with limited compute and thermal budget for applications such
as video surveillance. To address this problem, efficient object detection
frameworks such as YOLO and SSD were proposed. However, SSD based object
detection that uses VGG16 as backend network is insufficient to achieve real
time speed on edge devices. To further improve the detection speed, the backend
network is replaced by more efficient networks such as SqueezeNet and
MobileNet. Although the speed is greatly improved, it comes with a price of
lower accuracy. In this paper, we propose an efficient SSD named Fire SSD. Fire
SSD achieves 70.7mAP on Pascal VOC 2007 test set. Fire SSD achieves the speed
of 30.6FPS on low power mainstream CPU and is about 6 times faster than SSD300
and has about 4 times smaller model size. Fire SSD also achieves 22.2FPS on
integrated GPU.
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
Authors: Rachel Ward, Xiaoxia Wu, Leon Bottou
Date: 2018-06-05 00:00:00
Tasks: Stochastic Optimization

Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule. Yet, the theoretical guarantees to date for AdaGrad are for online and convex optimization. We bridge this gap by providing theoretical guarantees for the convergence of AdaGrad for smooth, nonconvex functions. We show that the norm version of AdaGrad (AdaGrad-Norm) converges to a stationary point at the $\mathcal{O}(\log(N)/\sqrt{N})$ rate in the stochastic setting, and at the optimal $\mathcal{O}(1/N)$ rate in the batch (non-stochastic) setting -- in this sense, our convergence guarantees are 'sharp'. In particular, the convergence of AdaGrad-Norm is robust to the choice of all hyper-parameters of the algorithm, in contrast to stochastic gradient descent whose convergence depends crucially on tuning the step-size to the (generally unknown) Lipschitz smoothness constant and level of stochastic noise on the gradient. Extensive numerical experiments are provided to corroborate our theory; moreover, the experiments suggest that the robustness of AdaGrad-Norm extends to state-of-the-art models in deep learning, without sacrificing generalization.
View-volume Network for Semantic Scene Completion from a Single Depth Image
Authors: Yu-Xiao Guo, Xin Tong
Date: 2018-06-14 00:00:00
Tasks: 3D Semantic Scene Completion

We introduce a View-Volume convolutional neural network (VVNet) for inferring
the occupancy and semantic labels of a volumetric 3D scene from a single depth
image. The VVNet concatenates a 2D view CNN and a 3D volume CNN with a
differentiable projection layer. Given a single RGBD image, our method extracts
the detailed geometric features from the input depth image with a 2D view CNN
and then projects the features into a 3D volume according to the input depth
map via a projection layer. After that, we learn the 3D context information of
the scene with a 3D volume CNN for computing the result volumetric occupancy
and semantic labels. With combined 2D and 3D representations, the VVNet
efficiently reduces the computational cost, enables feature extraction from
multi-channel high resolution inputs, and thus significantly improves the
result accuracy. We validate our method and demonstrate its efficiency and
effectiveness on both synthetic SUNCG and real NYU dataset.
Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning
Authors: Dong Yin, Yudong Chen, Kannan Ramchandran, Peter Bartlett
Date: 2018-06-14 00:00:00

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior. In this setting, the Byzantine machines may create fake local minima near a saddle point that is far away from any true local minimum, even when robust gradient estimators are used. We develop ByzantinePGD, a robust first-order algorithm that can provably escape saddle points and fake local minima, and converge to an approximate true local minimizer with low iteration complexity. As a by-product, we give a simpler algorithm and analysis for escaping saddle points in the usual non-Byzantine setting. We further discuss three robust gradient estimators that can be used in ByzantinePGD, including median, trimmed mean, and iterative filtering. We characterize their performance in concrete statistical settings, and argue for their near-optimality in low and high dimensional regimes.
Deep Multi-Output Forecasting: Learning to Accurately Predict Blood Glucose Trajectories
Authors: Ian Fox, Lynn Ang, Mamta Jaiswal, Rodica Pop-Busui, Jenna Wiens
Date: 2018-06-14 00:00:00

In many forecasting applications, it is valuable to predict not only the
value of a signal at a certain time point in the future, but also the values
leading up to that point. This is especially true in clinical applications,
where the future state of the patient can be less important than the patient's
overall trajectory. This requires multi-step forecasting, a forecasting variant
where one aims to predict multiple values in the future simultaneously.
Standard methods to accomplish this can propagate error from prediction to
prediction, reducing quality over the long term. In light of these challenges,
we propose multi-output deep architectures for multi-step forecasting in which
we explicitly model the distribution of future values of the signal over a
prediction horizon. We apply these techniques to the challenging and clinically
relevant task of blood glucose forecasting. Through a series of experiments on
a real-world dataset consisting of 550K blood glucose measurements, we
demonstrate the effectiveness of our proposed approaches in capturing the
underlying signal dynamics. Compared to existing shallow and deep methods, we
find that our proposed approaches improve performance individually and capture
complementary information, leading to a large improvement over the baseline
when combined (4.87 vs. 5.31 absolute percentage error (APE)). Overall, the
results suggest the efficacy of our proposed approach in predicting blood
glucose level and multi-step forecasting more generally.
Finding GEMS: Multi-Scale Dictionaries for High-Dimensional Graph Signals
Authors: Yael Yankelevsky, Michael Elad
Date: 2018-06-14 00:00:00
Tasks: Dictionary Learning, Vocal Bursts Intensity Prediction

Modern data introduces new challenges to classic signal processing
approaches, leading to a growing interest in the field of graph signal
processing. A powerful and well established model for real world signals in
various domains is sparse representation over a dictionary, combined with the
ability to train the dictionary from signal examples. This model has been
successfully applied to graph signals as well by integrating the underlying
graph topology into the learned dictionary. Nonetheless, dictionary learning
methods for graph signals are typically restricted to small dimensions due to
the computational constraints that the dictionary learning problem entails, and
due to the direct use of the graph Laplacian matrix. In this paper, we propose
a dictionary learning algorithm that applies to a broader class of graph
signals, and is capable of handling much higher dimensional data. We
incorporate the underlying graph topology both implicitly, by forcing the
learned dictionary atoms to be sparse combinations of graph-wavelet functions,
and explicitly, by adding direct graph constraints to promote smoothness in
both the feature and manifold domains. The resulting atoms are thus adapted to
the data of interest while adhering to the underlying graph structure and
possessing a desired multi-scale property. Experimental results on several
datasets, representing both synthetic and real network data of different
nature, demonstrate the effectiveness of the proposed algorithm for graph
signal processing even in high dimensions.
Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
Authors: Yibo Yang, Nicholas Ruozzi, Vibhav Gogate
Date: 2018-06-14 00:00:00
Tasks: Clustering, Neural Network Compression, Quantization

We propose a simple and easy to implement neural network compression
algorithm that achieves results competitive with more complicated
state-of-the-art methods. The key idea is to modify the original optimization
problem by adding K independent Gaussian priors (corresponding to the k-means
objective) over the network parameters to achieve parameter quantization, as
well as an L1 penalty to achieve pruning. Unlike many existing
quantization-based methods, our method uses hard clustering assignments of
network parameters, which adds minimal change or overhead to standard network
training. We also demonstrate experimentally that tying neural network
parameters provides less gain in generalization performance than changing
network architecture and connectivity patterns entirely.
q-Neurons: Neuron Activations based on Stochastic Jackson's Derivative Operators
Authors: Frank Nielsen, Ke Sun
Date: 2018-06-01 00:00:00

We propose a new generic type of stochastic neurons, called $q$-neurons, that
considers activation functions based on Jackson's $q$-derivatives with
stochastic parameters $q$. Our generalization of neural network architectures
with $q$-neurons is shown to be both scalable and very easy to implement. We
demonstrate experimentally consistently improved performances over
state-of-the-art standard activation functions, both on training and testing
loss functions.
Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
Authors: Jianbo Chen, Le Song, Martin J. Wainwright, Michael. I. Jordan
Date: 2018-02-21 00:00:00
Proceeding: ICML 2018 7
Tasks: feature selection

We introduce instancewise feature selection as a methodology for model
interpretation. Our method is based on learning a function to extract a subset
of features that are most informative for each given example. This feature
selector is trained to maximize the mutual information between selected
features and the response variable, where the conditional distribution of the
response variable given the input is the model to be explained. We develop an
efficient variational approximation to the mutual information, and show the
effectiveness of our method on a variety of synthetic and real data sets using
both quantitative metrics and human evaluation.