pwc-archive/papers-with-abstracts huggingface dataset:
Incremental Classifier Learning with Generative Adversarial Networks
Authors: Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Zhengyou Zhang, Yun Fu
Date: 2018-02-02 00:00:00
Tasks: General Classification

In this paper, we address the incremental classifier learning problem, which
suffers from catastrophic forgetting. The main reason for catastrophic
forgetting is that the past data are not available during learning. Typical
approaches keep some exemplars for the past classes and use distillation
regularization to retain the classification capability on the past classes and
balance the past and new classes. However, there are four main problems with
these approaches. First, the loss function is not efficient for classification.
Second, there is unbalance problem between the past and new classes. Third, the
size of pre-decided exemplars is usually limited and they might not be
distinguishable from unseen new classes. Forth, the exemplars may not be
allowed to be kept for a long time due to privacy regulations. To address these
problems, we propose (a) a new loss function to combine the cross-entropy loss
and distillation loss, (b) a simple way to estimate and remove the unbalance
between the old and new classes , and (c) using Generative Adversarial Networks
(GANs) to generate historical data and select representative exemplars during
generation. We believe that the data generated by GANs have much less privacy
issues than real images because GANs do not directly copy any real image
patches. We evaluate the proposed method on CIFAR-100, Flower-102, and
MS-Celeb-1M-Base datasets and extensive experiments demonstrate the
effectiveness of our method.
Toward Scalable Verification for Safety-Critical Deep Networks
Authors: Lindsey Kuper, Guy Katz, Justin Gottschlich, Kyle Julian, Clark Barrett, Mykel Kochenderfer
Date: 2018-01-18 00:00:00
Tasks: Autonomous Driving, Deep Learning

The increasing use of deep neural networks for safety-critical applications,
such as autonomous driving and flight control, raises concerns about their
safety and reliability. Formal verification can address these concerns by
guaranteeing that a deep learning system operates as intended, but the state of
the art is limited to small systems. In this work-in-progress report we give an
overview of our work on mitigating this difficulty, by pursuing two
complementary directions: devising scalable verification techniques, and
identifying design choices that result in deep learning systems that are more
amenable to verification.
Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
Authors: Rohit Tripathy, Ilias Bilionis
Date: 2018-02-02 00:00:00
Tasks: Uncertainty Quantification

State-of-the-art computer codes for simulating real physical systems are
often characterized by a vast number of input parameters. Performing
uncertainty quantification (UQ) tasks with Monte Carlo (MC) methods is almost
always infeasible because of the need to perform hundreds of thousands or even
millions of forward model evaluations in order to obtain convergent statistics.
One, thus, tries to construct a cheap-to-evaluate surrogate model to replace
the forward model solver. For systems with large numbers of input parameters,
one has to deal with the curse of dimensionality - the exponential increase in
the volume of the input space, as the number of parameters increases linearly.
In this work, we demonstrate the use of deep neural networks (DNN) to construct
surrogate models for numerical simulators. We parameterize the structure of the
DNN in a manner that lends the DNN surrogate the interpretation of recovering a
low dimensional nonlinear manifold. The model response is a parameterized
nonlinear function of the low dimensional projections of the input. We think of
this low dimensional manifold as a nonlinear generalization of the notion of
the active subspace. Our approach is demonstrated with a problem on uncertainty
propagation in a stochastic elliptic partial differential equation (SPDE) with
uncertain diffusion coefficient. We deviate from traditional formulations of
the SPDE problem by not imposing a specific covariance structure on the random
diffusion coefficient. Instead, we attempt to solve a more challenging problem
of learning a map between an arbitrary snapshot of the diffusion field and the
response.
DLVM: A modern compiler infrastructure for deep learning systems
Authors: Richard Wei, Lane Schwartz, Vikram Adve
Date: 2017-11-08 00:00:00
Proceeding: ICLR 2018 1
Tasks: Code Generation, Deep Learning, GPU

Deep learning software demands reliability and performance. However, many of
the existing deep learning frameworks are software libraries that act as an
unsafe DSL in Python and a computation graph interpreter. We present DLVM, a
design and implementation of a compiler infrastructure with a linear algebra
intermediate representation, algorithmic differentiation by adjoint code
generation, domain-specific optimizations and a code generator targeting GPU
via LLVM. Designed as a modern compiler infrastructure inspired by LLVM, DLVM
is more modular and more generic than existing deep learning compiler
frameworks, and supports tensor DSLs with high expressivity. With our
prototypical staged DSL embedded in Swift, we argue that the DLVM system
enables a form of modular, safe and performant frameworks for deep learning.
Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing
Authors: Amir Rosenfeld, John K. Tsotsos
Date: 2018-02-02 00:00:00
Tasks: All

Training deep neural networks results in strong learned representations that
show good generalization capabilities. In most cases, training involves
iterative modification of all weights inside the network via back-propagation.
In Extreme Learning Machines, it has been suggested to set the first layer of a
network to fixed random values instead of learning it. In this paper, we
propose to take this approach a step further and fix almost all layers of a
deep convolutional neural network, allowing only a small portion of the weights
to be learned. As our experiments show, fixing even the majority of the
parameters of the network often results in performance which is on par with the
performance of learning all of them. The implications of this intriguing
property of deep neural networks are discussed and we suggest ways to harness
it to create more robust representations.
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
Authors: Jianbo Ye, Xin Lu, Zhe Lin, James Z. Wang
Date: 2018-02-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Computational Efficiency

Model pruning has become a useful technique that improves the computational
efficiency of deep learning, making it possible to deploy solutions in
resource-limited scenarios. A widely-used practice in relevant work assumes
that a smaller-norm parameter or feature plays a less informative role at the
inference time. In this paper, we propose a channel pruning technique for
accelerating the computations of deep convolutional neural networks (CNNs) that
does not critically rely on this assumption. Instead, it focuses on direct
simplification of the channel-to-channel computation graph of a CNN without the
need of performing a computationally difficult and not-always-useful task of
making high-dimensional tensors of CNN structured sparse. Our approach takes
two stages: first to adopt an end-to- end stochastic training method that
eventually forces the outputs of some channels to be constant, and then to
prune those constant channels from the original neural network by adjusting the
biases of their impacting layers such that the resulting compact model can be
quickly fine-tuned. Our approach is mathematically appealing from an
optimization perspective and easy to reproduce. We experimented our approach
through several image learning benchmarks and demonstrate its interesting
aspects and competitive performance.
Penalized Estimation of Directed Acyclic Graphs From Discrete Data
Authors: Jiaying Gu, Fei Fu, Qing Zhou
Date: 2014-03-10 00:00:00

Bayesian networks, with structure given by a directed acyclic graph (DAG),
are a popular class of graphical models. However, learning Bayesian networks
from discrete or categorical data is particularly challenging, due to the large
parameter space and the difficulty in searching for a sparse structure. In this
article, we develop a maximum penalized likelihood method to tackle this
problem. Instead of the commonly used multinomial distribution, we model the
conditional distribution of a node given its parents by multi-logit regression,
in which an edge is parameterized by a set of coefficient vectors with dummy
variables encoding the levels of a node. To obtain a sparse DAG, a group norm
penalty is employed, and a blockwise coordinate descent algorithm is developed
to maximize the penalized likelihood subject to the acyclicity constraint of a
DAG. When interventional data are available, our method constructs a causal
network, in which a directed edge represents a causal relation. We apply our
method to various simulated and real data sets. The results show that our
method is very competitive, compared to many existing methods, in DAG
estimation from both interventional and high-dimensional observational data.
VIBNN: Hardware Acceleration of Bayesian Neural Networks
Authors: Ruizhe Cai, Ao Ren, Ning Liu, Caiwen Ding, Luhao Wang, Xuehai Qian, Massoud Pedram, Yanzhi Wang
Date: 2018-02-02 00:00:00
Tasks: Small Data Image Classification, Variational Inference

Bayesian Neural Networks (BNNs) have been proposed to address the problem of
model uncertainty in training and inference. By introducing weights associated
with conditioned probability distributions, BNNs are capable of resolving the
overfitting issue commonly seen in conventional neural networks and allow for
small-data training, through the variational inference process. Frequent usage
of Gaussian random variables in this process requires a properly optimized
Gaussian Random Number Generator (GRNG). The high hardware cost of conventional
GRNG makes the hardware implementation of BNNs challenging.
  In this paper, we propose VIBNN, an FPGA-based hardware accelerator design
for variational inference on BNNs. We explore the design space for massive
amount of Gaussian variable sampling tasks in BNNs. Specifically, we introduce
two high performance Gaussian (pseudo) random number generators: the RAM-based
Linear Feedback Gaussian Random Number Generator (RLF-GRNG), which is inspired
by the properties of binomial distribution and linear feedback logics; and the
Bayesian Neural Network-oriented Wallace Gaussian Random Number Generator. To
achieve high scalability and efficient memory access, we propose a deep
pipelined accelerator architecture with fast execution and good hardware
utilization. Experimental results demonstrate that the proposed VIBNN
implementations on an FPGA can achieve throughput of 321,543.4 Images/s and
energy efficiency upto 52,694.8 Images/J while maintaining similar accuracy as
its software counterpart.
LOOP Descriptor: Local Optimal Oriented Pattern
Authors: Tapabrata Chakraborti, Brendan McCane, Steven Mills, Umapada Pal
Date: 2017-10-25 00:00:00

This letter introduces the LOOP binary descriptor (local optimal oriented
pattern) that encodes rotation invariance into the main formulation itself.
This makes any post processing stage for rotation invariance redundant and
improves on both accuracy and time complexity. We consider fine-grained
lepidoptera (moth/butterfly) species recognition as the representative problem
since it involves repetition of localized patterns and textures that may be
exploited for discrimination. We evaluate the performance of LOOP against its
predecessors as well as few other popular descriptors. Besides experiments on
standard benchmarks, we also introduce a new small image dataset on NZ
Lepidoptera. Loop performs as well or better on all datasets evaluated compared
to previous binary descriptors. The new dataset and demo code of the proposed
method are to be made available through the lead author's academic webpage and
GitHub.
Monte Carlo Structured SVI for Two-Level Non-Conjugate Models
Authors: Rishit Sheth, Roni Khardon
Date: 2016-12-12 00:00:00
Tasks: Gaussian Processes, Topic Models, Variational Inference, Vocal Bursts Valence Prediction

The stochastic variational inference (SVI) paradigm, which combines
variational inference, natural gradients, and stochastic updates, was recently
proposed for large-scale data analysis in conjugate Bayesian models and
demonstrated to be effective in several problems. This paper studies a family
of Bayesian latent variable models with two levels of hidden variables but
without any conjugacy requirements, making several contributions in this
context. The first is observing that SVI, with an improved structured
variational approximation, is applicable under more general conditions than
previously thought with the only requirement being that the approximating
variational distribution be in the same family as the prior. The resulting
approach, Monte Carlo Structured SVI (MC-SSVI), significantly extends the scope
of SVI, enabling large-scale learning in non-conjugate models. For models with
latent Gaussian variables we propose a hybrid algorithm, using both standard
and natural gradients, which is shown to improve stability and convergence.
Applications in mixed effects models, sparse Gaussian processes, probabilistic
matrix factorization and correlated topic models demonstrate the generality of
the approach and the advantages of the proposed algorithms.
Convergence Rates of Variational Posterior Distributions
Authors: Fengshuo Zhang, Chao GAO
Date: 2017-12-07 00:00:00

We study convergence rates of variational posterior distributions for nonparametric and high-dimensional inference. We formulate general conditions on prior, likelihood, and variational class that characterize the convergence rates. Under similar "prior mass and testing" conditions considered in the literature, the rate is found to be the sum of two terms. The first term stands for the convergence rate of the true posterior distribution, and the second term is contributed by the variational approximation error. For a class of priors that admit the structure of a mixture of product measures, we propose a novel prior mass condition, under which the variational approximation error of the mean-field class is dominated by convergence rate of the true posterior. We demonstrate the applicability of our general results for various models, prior distributions and variational classes by deriving convergence rates of the corresponding variational posteriors.
Green Stability Assumption: Unsupervised Learning for Statistics-Based Illumination Estimation
Authors: Nikola Banić, Sven Lončarić
Date: 2018-02-02 00:00:00
Tasks: Color Constancy

In the image processing pipeline of almost every digital camera there is a
part dedicated to computational color constancy i.e. to removing the influence
of illumination on the colors of the image scene. Some of the best known
illumination estimation methods are the so called statistics-based methods.
They are less accurate than the learning-based illumination estimation methods,
but they are faster and simpler to implement in embedded systems, which is one
of the reasons for their widespread usage. Although in the relevant literature
it often appears as if they require no training, this is not true because they
have parameter values that need to be fine-tuned in order to be more accurate.
In this paper it is first shown that the accuracy of statistics-based methods
reported in most papers was not obtained by means of the necessary
cross-validation, but by using the whole benchmark datasets for both training
and testing. After that the corrected results are given for the best known
benchmark datasets. Finally, the so called green stability assumption is
proposed that can be used to fine-tune the values of the parameters of the
statistics-based methods by using only non-calibrated images without known
ground-truth illumination. The obtained accuracy is practically the same as
when using calibrated training images, but the whole process is much faster.
The experimental results are presented and discussed. The source code is
available at http://www.fer.unizg.hr/ipg/resources/color_constancy/.
No Modes left behind: Capturing the data distribution effectively using GANs
Authors: Shashank Sharma, Vinay P. Namboodiri
Date: 2018-02-02 00:00:00
Tasks: Image Generation

Generative adversarial networks (GANs) while being very versatile in
realistic image synthesis, still are sensitive to the input distribution. Given
a set of data that has an imbalance in the distribution, the networks are
susceptible to missing modes and not capturing the data distribution. While
various methods have been tried to improve training of GANs, these have not
addressed the challenges of covering the full data distribution. Specifically,
a generator is not penalized for missing a mode. We show that these are
therefore still susceptible to not capturing the full data distribution.
  In this paper, we propose a simple approach that combines an encoder based
objective with novel loss functions for generator and discriminator that
improves the solution in terms of capturing missing modes. We validate that the
proposed method results in substantial improvements through its detailed
analysis on toy and real datasets. The quantitative and qualitative results
demonstrate that the proposed method improves the solution for the problem of
missing modes and improves training of GANs.
Order matters: Distributional properties of speech to young children bootstraps learning of semantic representations
Authors: Philip A Huebner, Jon A Willits
Date: 2018-02-02 00:00:00
Tasks: Language Acquisition

Some researchers claim that language acquisition is critically dependent on
experiencing linguistic input in order of increasing complexity. We set out to
test this hypothesis using a simple recurrent neural network (SRN) trained to
predict word sequences in CHILDES, a 5-million-word corpus of speech directed
to children. First, we demonstrated that age-ordered CHILDES exhibits a gradual
increase in linguistic complexity. Next, we compared the performance of two
groups of SRNs trained on CHILDES which had either been age-ordered or not.
Specifically, we assessed learning of grammatical and semantic structure and
showed that training on age-ordered input facilitates learning of semantic, but
not of sequential structure. We found that this advantage is eliminated when
the models were trained on input with utterance boundary information removed.
Full-pulse Tomographic Reconstruction with Deep Neural Networks
Authors: Diogo R. Ferreira, Pedro J. Carvalho, Horácio Fernandes
Date: 2018-02-02 00:00:00
Tasks: Tomographic Reconstructions

Plasma tomography consists in reconstructing the 2D radiation profile in a
poloidal cross-section of a fusion device, based on line-integrated
measurements along several lines of sight. The reconstruction process is
computationally intensive and, in practice, only a few reconstructions are
usually computed per pulse. In this work, we trained a deep neural network
based on a large collection of sample tomograms that have been produced at JET
over several years. Once trained, the network is able to reproduce those
results with high accuracy. More importantly, it can compute all the
tomographic reconstructions for a given pulse in just a few seconds. This makes
it possible to visualize several phenomena -- such as plasma heating,
disruptions and impurity transport -- over the course of a discharge.
Learning Attribute Representation for Human Activity Recognition
Authors: Fernando Moya Rueda, Gernot A. Fink
Date: 2018-02-02 00:00:00
Tasks: Activity Recognition, Attribute, Human Activity Recognition

Attribute representations became relevant in image recognition and word
spotting, providing support under the presence of unbalance and disjoint
datasets. However, for human activity recognition using sequential data from
on-body sensors, human-labeled attributes are lacking. This paper introduces a
search for attributes that represent favorably signal segments for recognizing
human activities. It presents three deep architectures, including
temporal-convolutions and an IMU centered design, for predicting attributes. An
empiric evaluation of random and learned attribute representations, and as well
as the networks is carried out on two datasets, outperforming the state-of-the
art.
Submodularity-Inspired Data Selection for Goal-Oriented Chatbot Training Based on Sentence Embeddings
Authors: Mladen Dimovski, Claudiu Musat, Vladimir Ilievski, Andreea Hossmann, Michael Baeriswyl
Date: 2018-02-02 00:00:00
Tasks: Active Learning, Chatbot, Natural Language Understanding, Sentence, Sentence Embeddings, Spoken Language Understanding

Spoken language understanding (SLU) systems, such as goal-oriented chatbots
or personal assistants, rely on an initial natural language understanding (NLU)
module to determine the intent and to extract the relevant information from the
user queries they take as input. SLU systems usually help users to solve
problems in relatively narrow domains and require a large amount of in-domain
training data. This leads to significant data availability issues that inhibit
the development of successful systems. To alleviate this problem, we propose a
technique of data selection in the low-data regime that enables us to train
with fewer labeled sentences, thus smaller labelling costs.
  We propose a submodularity-inspired data ranking function, the ratio-penalty
marginal gain, for selecting data points to label based only on the information
extracted from the textual embedding space. We show that the distances in the
embedding space are a viable source of information that can be used for data
selection. Our method outperforms two known active learning techniques and
enables cost-efficient training of the NLU unit. Moreover, our proposed
selection technique does not need the model to be retrained in between the
selection steps, making it time efficient as well.
Short-term Memory of Deep RNN
Authors: Claudio Gallicchio
Date: 2018-02-02 00:00:00

The extension of deep learning towards temporal data processing is gaining an
increasing research interest. In this paper we investigate the properties of
state dynamics developed in successive levels of deep recurrent neural networks
(RNNs) in terms of short-term memory abilities. Our results reveal interesting
insights that shed light on the nature of layering as a factor of RNN design.
Noticeably, higher layers in a hierarchically organized RNN architecture
results to be inherently biased towards longer memory spans even prior to
training of the recurrent connections. Moreover, in the context of Reservoir
Computing framework, our analysis also points out the benefit of a layered
recurrent organization as an efficient approach to improve the memory skills of
reservoir models.