pwc-archive/papers-with-abstracts huggingface dataset:
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks
Authors: Charles H. Martin, Michael W. Mahoney
Date: 2019-01-24 00:00:00

Given two or more Deep Neural Networks (DNNs) with the same or similar architectures, and trained on the same dataset, but trained with different solvers, parameters, hyper-parameters, regularization, etc., can we predict which DNN will have the best test accuracy, and can we do so without peeking at the test data? In this paper, we show how to use a new Theory of Heavy-Tailed Self-Regularization (HT-SR) to answer this. HT-SR suggests, among other things, that modern DNNs exhibit what we call Heavy-Tailed Mechanistic Universality (HT-MU), meaning that the correlations in the layer weight matrices can be fit to a power law (PL) with exponents that lie in common Universality classes from Heavy-Tailed Random Matrix Theory (HT-RMT). From this, we develop a Universal capacity control metric that is a weighted average of PL exponents. Rather than considering small toy NNs, we examine over 50 different, large-scale pre-trained DNNs, ranging over 15 different architectures, trained on ImagetNet, each of which has been reported to have different test accuracies. We show that this new capacity metric correlates very well with the reported test accuracies of these DNNs, looking across each architecture (VGG16/.../VGG19, ResNet10/.../ResNet152, etc.). We also show how to approximate the metric by the more familiar Product Norm capacity measure, as the average of the log Frobenius norm of the layer weight matrices. Our approach requires no changes to the underlying DNN or its loss function, it does not require us to train a model (although it could be used to monitor training), and it does not even require access to the ImageNet data.
Federated Deep Reinforcement Learning
Authors: Hankz Hankui Zhuo, Wenfeng Feng, Yufeng Lin, Qian Xu, Qiang Yang
Date: 2019-01-24 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Transfer Learning

In deep reinforcement learning, building policies of high-quality is challenging when the feature space of states is small and the training data is limited. Despite the success of previous transfer learning approaches in deep reinforcement learning, directly transferring data or models from an agent to another agent is often not allowed due to the privacy of data and/or models in many privacy-aware applications. In this paper, we propose a novel deep reinforcement learning framework to federatively build models of high-quality for agents with consideration of their privacies, namely Federated deep Reinforcement Learning (FedRL). To protect the privacy of data and models, we exploit Gausian differentials on the information shared with each other when updating their local models. In the experiment, we evaluate our FedRL framework in two diverse domains, Grid-world and Text2Action domains, by comparing to various baselines.
Traditional and Heavy-Tailed Self Regularization in Neural Network Models
Authors: Charles H. Martin, Michael W. Mahoney
Date: 2019-01-24 00:00:00

Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep
Neural Networks (DNNs), including both production quality, pre-trained models
such as AlexNet and Inception, and smaller models trained from scratch, such as
LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly
indicate that the empirical spectral density (ESD) of DNN layer matrices
displays signatures of traditionally-regularized statistical models, even in
the absence of exogenously specifying traditional forms of regularization, such
as Dropout or Weight Norm constraints. Building on recent results in RMT, most
notably its extension to Universality classes of Heavy-Tailed matrices, we
develop a theory to identify \emph{5+1 Phases of Training}, corresponding to
increasing amounts of \emph{Implicit Self-Regularization}. For smaller and/or
older DNNs, this Implicit Self-Regularization is like traditional Tikhonov
regularization, in that there is a `size scale' separating signal from noise.
For state-of-the-art DNNs, however, we identify a novel form of
\emph{Heavy-Tailed Self-Regularization}, similar to the self-organization seen
in the statistical physics of disordered systems. This implicit
Self-Regularization can depend strongly on the many knobs of the training
process. By exploiting the generalization gap phenomena, we demonstrate that we
can cause a small model to exhibit all 5+1 phases of training simply by
changing the batch size.
Multi-fidelity Bayesian Optimization with Max-value Entropy Search and its parallelization
Authors: Shion Takeno, Hitoshi Fukuoka, Yuhki Tsukada, Toshiyuki Koyama, Motoki Shiga, Ichiro Takeuchi, Masayuki Karasuyama
Date: 2019-01-24 00:00:00
Proceeding: ICML 2020 1
Tasks: Bayesian Optimization, Numerical Integration

In a standard setting of Bayesian optimization (BO), the objective function evaluation is assumed to be highly expensive. Multi-fidelity Bayesian optimization (MFBO) accelerates BO by incorporating lower fidelity observations available with a lower sampling cost. In this paper, we focus on the information-based approach, which is a popular and empirically successful approach in BO. For MFBO, however, existing information-based methods are plagued by difficulty in estimating the information gain. We propose an approach based on max-value entropy search (MES), which greatly facilitates computations by considering the entropy of the optimal function value instead of the optimal input point. We show that, in our multi-fidelity MES (MF-MES), most of additional computations, compared with usual MES, is reduced to analytical computations. Although an additional numerical integration is necessary for the information across different fidelities, this is only in one dimensional space, which can be performed efficiently and accurately. Further, we also propose parallelization of MF-MES. Since there exist a variety of different sampling costs, queries typically occur asynchronously in MFBO. We show that similar simple computations can be derived for asynchronous parallel MFBO. We demonstrate effectiveness of our approach by using benchmark datasets and a real-world application to materials science data.
A Novel Self-Intersection Penalty Term for Statistical Body Shape Models and Its Applications in 3D Pose Estimation
Authors: Zaiqiang Wu, Wei Jiang, Hao Luo, Lin Cheng
Date: 2019-01-24 00:00:00
Tasks: 3D Pose Estimation, 3D Reconstruction, Pose Estimation

Statistical body shape models are widely used in 3D pose estimation due to
their low-dimensional parameters representation. However, it is difficult to
avoid self-intersection between body parts accurately. Motivated by this fact,
we proposed a novel self-intersection penalty term for statistical body shape
models applied in 3D pose estimation. To avoid the trouble of computing
self-intersection for complex surfaces like the body meshes, the gradient of
our proposed self-intersection penalty term is manually derived from the
perspective of geometry. First, the self-intersection penalty term is defined
as the volume of the self-intersection region. To calculate the partial
derivatives with respect to the coordinates of the vertices, we employed
detection rays to divide vertices of statistical body shape models into
different groups depending on whether the vertex is in the region of
self-intersection. Second, the partial derivatives could be easily derived by
the normal vectors of neighboring triangles of the vertices. Finally, this
penalty term could be applied in gradient-based optimization algorithms to
remove the self-intersection of triangular meshes without using any
approximation. Qualitative and quantitative evaluations were conducted to
demonstrate the effectiveness and generality of our proposed method compared
with previous approaches. The experimental results show that our proposed
penalty term can avoid self-intersection to exclude unreasonable predictions
and improves the accuracy of 3D pose estimation indirectly. Further more, the
proposed method could be employed universally in triangular mesh based 3D
reconstruction.
QGAN: Quantized Generative Adversarial Networks
Authors: Peiqi Wang, Dongsheng Wang, Yu Ji, Xinfeng Xie, Haoxuan Song, XuXin Liu, Yongqiang Lyu, Yuan Xie
Date: 2019-01-24 00:00:00
Tasks: Quantization

The intensive computation and memory requirements of generative adversarial
neural networks (GANs) hinder its real-world deployment on edge devices such as
smartphones. Despite the success in model reduction of CNNs, neural network
quantization methods have not yet been studied on GANs, which are mainly faced
with the issues of both the effectiveness of quantization algorithms and the
instability of training GAN models. In this paper, we start with an extensive
study on applying existing successful methods to quantize GANs. Our observation
reveals that none of them generates samples with reasonable quality because of
the underrepresentation of quantized values in model weights, and the generator
and discriminator networks show different sensitivities upon quantization
methods. Motivated by these observations, we develop a novel quantization
method for GANs based on EM algorithms, named as QGAN. We also propose a
multi-precision algorithm to help find the optimal number of bits of quantized
GAN models in conjunction with corresponding result qualities. Experiments on
CIFAR-10 and CelebA show that QGAN can quantize GANs to even 1-bit or 2-bit
representations with results of quality comparable to original models.
FANDA: A Novel Approach to Perform Follow-up Query Analysis
Authors: Qian Liu, Bei Chen, Jian-Guang Lou, Ge Jin, Dongmei Zhang
Date: 2019-01-24 00:00:00

Recent work on Natural Language Interfaces to Databases (NLIDB) has attracted
considerable attention. NLIDB allow users to search databases using natural
language instead of SQL-like query languages. While saving the users from
having to learn query languages, multi-turn interaction with NLIDB usually
involves multiple queries where contextual information is vital to understand
the users' query intents. In this paper, we address a typical contextual
understanding problem, termed as follow-up query analysis. In spite of its
ubiquity, follow-up query analysis has not been well studied due to two primary
obstacles: the multifarious nature of follow-up query scenarios and the lack of
high-quality datasets. Our work summarizes typical follow-up query scenarios
and provides a new FollowUp dataset with $1000$ query triples on 120 tables.
Moreover, we propose a novel approach FANDA, which takes into account the
structures of queries and employs a ranking model with weakly supervised
max-margin learning. The experimental results on FollowUp demonstrate the
superiority of FANDA over multiple baselines across multiple metrics.
Large-Batch Training for LSTM and Beyond
Authors: Yang You, Jonathan Hseu, Chris Ying, James Demmel, Kurt Keutzer, Cho-Jui Hsieh
Date: 2019-01-24 00:00:00

Large-batch training approaches have enabled researchers to utilize
large-scale distributed processing and greatly accelerate deep-neural net (DNN)
training. For example, by scaling the batch size from 256 to 32K, researchers
have been able to reduce the training time of ResNet50 on ImageNet from 29
hours to 2.2 minutes (Ying et al., 2018). In this paper, we propose a new
approach called linear-epoch gradual-warmup (LEGW) for better large-batch
training. With LEGW, we are able to conduct large-batch training for both CNNs
and RNNs with the Sqrt Scaling scheme. LEGW enables Sqrt Scaling scheme to be
useful in practice and as a result we achieve much better results than the
Linear Scaling learning rate scheme. For LSTM applications, we are able to
scale the batch size by a factor of 64 without losing accuracy and without
tuning the hyper-parameters. For CNN applications, LEGW is able to achieve the
same accuracy even as we scale the batch size to 32K. LEGW works better than
previous large-batch auto-tuning techniques. LEGW achieves a 5.3X average
speedup over the baselines for four LSTM-based applications on the same
hardware. We also provide some theoretical explanations for LEGW.
Confidence-based Graph Convolutional Networks for Semi-Supervised Learning
Authors: Shikhar Vashishth, Prateek Yadav, Manik Bhandari, Partha Talukdar
Date: 2019-01-24 00:00:00

Predicting properties of nodes in a graph is an important problem with
applications in a variety of domains. Graph-based Semi-Supervised Learning
(SSL) methods aim to address this problem by labeling a small subset of the
nodes as seeds and then utilizing the graph structure to predict label scores
for the rest of the nodes in the graph. Recently, Graph Convolutional Networks
(GCNs) have achieved impressive performance on the graph-based SSL task. In
addition to label scores, it is also desirable to have confidence scores
associated with them. Unfortunately, confidence estimation in the context of
GCN has not been previously explored. We fill this important gap in this paper
and propose ConfGCN, which estimates labels scores along with their confidences
jointly in GCN-based setting. ConfGCN uses these estimated confidences to
determine the influence of one node on another during neighborhood aggregation,
thereby acquiring anisotropic capabilities. Through extensive analysis and
experiments on standard benchmarks, we find that ConfGCN is able to outperform
state-of-the-art baselines. We have made ConfGCN's source code available to
encourage reproducible research.
Machine Learning and Deep Learning Algorithms for Bearing Fault Diagnostics -- A Comprehensive Review
Authors: Shen Zhang, Shibo Zhang, Bingnan Wang, Thomas G. Habetler
Date: 2019-01-24 00:00:00
Tasks: BIG-bench Machine Learning, Time Series Analysis

In this survey paper, we systematically summarize existing literature on bearing fault diagnostics with machine learning (ML) and data mining techniques. While conventional ML methods, including artificial neural network (ANN), principal component analysis (PCA), support vector machines (SVM), etc., have been successfully applied to the detection and categorization of bearing faults for decades, recent developments in deep learning (DL) algorithms in the last five years have sparked renewed interest in both industry and academia for intelligent machine health monitoring. In this paper, we first provide a brief review of conventional ML methods, before taking a deep dive into the state-of-the-art DL algorithms for bearing fault applications. Specifically, the superiority of DL based methods over conventional ML methods are analyzed in terms of fault feature extraction and classification performances; many new functionalities enabled by DL techniques are also summarized. In addition, to obtain a more intuitive insight, a comparative study is conducted on the classification accuracy of different algorithms utilizing the open-source Case Western Reserve University (CWRU) bearing dataset. Finally, to facilitate the transition on applying various DL algorithms to bearing fault diagnostics, detailed recommendations and suggestions are provided for specific application conditions such as the setup environment, the data size, and the number of sensors and sensor types. Future research directions to further enhance the performance of DL algorithms on health monitoring are also discussed.
Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
Authors: Vardan Papyan
Date: 2019-01-24 00:00:00

We consider deep classifying neural networks. We expose a structure in the
derivative of the logits with respect to the parameters of the model, which is
used to explain the existence of outliers in the spectrum of the Hessian.
Previous works decomposed the Hessian into two components, attributing the
outliers to one of them, the so-called Covariance of gradients. We show this
term is not a Covariance but a second moment matrix, i.e., it is influenced by
means of gradients. These means possess an additive two-way structure that is
the source of the outliers in the spectrum. This structure can be used to
approximate the principal subspace of the Hessian using certain "averaging"
operations, avoiding the need for high-dimensional eigenanalysis. We
corroborate this claim across different datasets, architectures and sample
sizes.
Unsupervised Image-to-Image Translation with Self-Attention Networks
Authors: Taewon Kang, Kwang Hee Lee
Date: 2019-01-24 00:00:00
Tasks: Image-to-Image Translation, Style Transfer, Translation, Unsupervised Image-To-Image Translation

Unsupervised image translation aims to learn the transformation from a source domain to another target domain given unpaired training data. Several state-of-the-art works have yielded impressive results in the GANs-based unsupervised image-to-image translation. It fails to capture strong geometric or structural changes between domains, or it produces unsatisfactory result for complex scenes, compared to local texture mapping tasks such as style transfer. Recently, SAGAN (Han Zhang, 2018) showed that the self-attention network produces better results than the convolution-based GAN. However, the effectiveness of the self-attention network in unsupervised image-to-image translation tasks have not been verified. In this paper, we propose an unsupervised image-to-image translation with self-attention networks, in which long range dependency helps to not only capture strong geometric change but also generate details using cues from all feature locations. In experiments, we qualitatively and quantitatively show superiority of the proposed method compared to existing state-of-the-art unsupervised image-to-image translation task. The source code and our results are online: https://github.com/itsss/img2img_sa and http://itsc.kr/2019/01/24/2019_img2img_sa
Location reference identification from tweets during emergencies: A deep learning approach
Authors: Abhinav Kumar, Jyoti Prakash Singh
Date: 2019-01-24 00:00:00
Tasks: Management

Twitter is recently being used during crises to communicate with officials
and provide rescue and relief operation in real time. The geographical location
information of the event, as well as users, are vitally important in such
scenarios. The identification of geographic location is one of the challenging
tasks as the location information fields, such as user location and place name
of tweets are not reliable. The extraction of location information from tweet
text is difficult as it contains a lot of non-standard English, grammatical
errors, spelling mistakes, non-standard abbreviations, and so on. This research
aims to extract location words used in the tweet using a Convolutional Neural
Network (CNN) based model. We achieved the exact matching score of 0.929,
Hamming loss of 0.002, and $F_1$-score of 0.96 for the tweets related to the
earthquake. Our model was able to extract even three- to four-word long
location references which is also evident from the exact matching score of over
92\%. The findings of this paper can help in early event localization,
emergency situations, real-time road traffic management, localized
advertisement, and in various location-based services.
Visualizing Topographic Independent Component Analysis with Movies
Authors: Zhimin Chen, Darius Parvin, Maedbh King, Susan Hao
Date: 2019-01-24 00:00:00

Independent component analysis (ICA) has often been used as a tool to model
natural image statistics by separating multivariate signals in the image into
components that are assumed to be independent. However, these estimated
components oftentimes have higher order dependencies, such as co-activation of
components, that are not accounted for in the model. Topographic independent
component analysis(TICA), a modification of ICA, takes into account higher
order dependencies and orders components topographically as a function of
dependence. Here, we aim to visualize the time course of TICA basis activations
to movie stimuli. We find that the activity of TICA bases are often clustered
and move continuously, potentially resembling activity of topographically
organized cells in the visual cortex.
High Dimensional Robust $M$-Estimation: Arbitrary Corruption and Heavy Tails
Authors: Liu Liu, Tianyang Li, Constantine Caramanis
Date: 2019-01-24 00:00:00
Tasks: regression, Vocal Bursts Intensity Prediction

We consider the problem of sparsity-constrained $M$-estimation when both explanatory and response variables have heavy tails (bounded 4-th moments), or a fraction of arbitrary corruptions. We focus on the $k$-sparse, high-dimensional regime where the number of variables $d$ and the sample size $n$ are related through $n \sim k \log d$. We define a natural condition we call the Robust Descent Condition (RDC), and show that if a gradient estimator satisfies the RDC, then Robust Hard Thresholding (IHT using this gradient estimator), is guaranteed to obtain good statistical rates. The contribution of this paper is in showing that this RDC is a flexible enough concept to recover known results, and obtain new robustness results. Specifically, new results include: (a) For $k$-sparse high-dimensional linear- and logistic-regression with heavy tail (bounded 4-th moment) explanatory and response variables, a linear-time-computable median-of-means gradient estimator satisfies the RDC, and hence Robust Hard Thresholding is minimax optimal; (b) When instead of heavy tails we have $O(1/\sqrt{k}\log(nd))$-fraction of arbitrary corruptions in explanatory and response variables, a near linear-time computable trimmed gradient estimator satisfies the RDC, and hence Robust Hard Thresholding is minimax optimal. We demonstrate the effectiveness of our approach in sparse linear, logistic regression, and sparse precision matrix estimation on synthetic and real-world US equities data.
Reciprocal Translation between SAR and Optical Remote Sensing Images with Cascaded-Residual Adversarial Networks
Authors: Shilei Fu, Feng Xu, Ya-Qiu Jin
Date: 2019-01-24 00:00:00
Tasks: Translation

Despite the advantages of all-weather and all-day high-resolution imaging, synthetic aperture radar (SAR) images are much less viewed and used by general people because human vision is not adapted to microwave scattering phenomenon. However, expert interpreters can be trained by comparing side-by-side SAR and optical images to learn the mapping rules from SAR to optical. This paper attempts to develop machine intelligence that are trainable with large-volume co-registered SAR and optical images to translate SAR image to optical version for assisted SAR image interpretation. Reciprocal SAR-Optical image translation is a challenging task because it is raw data translation between two physically very different sensing modalities. This paper proposes a novel reciprocal adversarial network scheme where cascaded residual connections and hybrid L1-GAN loss are employed. It is trained and tested on both spaceborne GF-3 and airborne UAVSAR images. Results are presented for datasets of different resolutions and polarizations and compared with other state-of-the-art methods. The FID is used to quantitatively evaluate the translation performance. The possibility of unsupervised learning with unpaired SAR and optical images is also explored. Results show that the proposed translation network works well under many scenarios and it could potentially be used for assisted SAR interpretation.