pwc-archive/papers-with-abstracts huggingface dataset:
Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning
Authors: Wei Zhu, Qiang Qiu, Bao Wang, Jianfeng Lu, Guillermo Sapiro, Ingrid Daubechies
Date: 2018-05-18 00:00:00
Proceeding: ICLR 2019 5

Deep neural networks (DNNs) typically have enough capacity to fit random data
by brute force even when conventional data-dependent regularizations focusing
on the geometry of the features are imposed. We find out that the reason for
this is the inconsistency between the enforced geometry and the standard
softmax cross entropy loss. To resolve this, we propose a new framework for
data-dependent DNN regularization, the
Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During
training, the geometry enforced on one batch of features is simultaneously
validated on a separate batch using a validation loss consistent with the
geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank
Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative
features residing in orthogonal low-rank subspaces. Numerical experiments show
that OLE-GRSVNet outperforms DNNs with conventional regularization when trained
on real data. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses
to memorize random data or random labels, suggesting it only learns intrinsic
patterns by reducing the memorizing capacity of the baseline DNN.
Learning 3D Shape Completion under Weak Supervision
Authors: David Stutz, Andreas Geiger
Date: 2018-05-18 00:00:00
Tasks: Weakly-supervised Learning

We address the problem of 3D shape completion from sparse and noisy point
clouds, a fundamental problem in computer vision and robotics. Recent
approaches are either data-driven or learning-based: Data-driven approaches
rely on a shape model whose parameters are optimized to fit the observations;
Learning-based approaches, in contrast, avoid the expensive optimization step
by learning to directly predict complete shapes from incomplete observations in
a fully-supervised setting. However, full supervision is often not available in
practice. In this work, we propose a weakly-supervised learning-based approach
to 3D shape completion which neither requires slow optimization nor direct
supervision. While we also learn a shape prior on synthetic data, we amortize,
i.e., learn, maximum likelihood fitting using deep neural networks resulting in
efficient shape completion without sacrificing accuracy. On synthetic
benchmarks based on ShapeNet and ModelNet as well as on real robotics data from
KITTI and Kinect, we demonstrate that the proposed amortized maximum likelihood
approach is able to compete with recent fully supervised baselines and
outperforms data-driven approaches, while requiring less supervision and being
significantly faster.
XOGAN: One-to-Many Unsupervised Image-to-Image Translation
Authors: Yongqi Zhang
Date: 2018-05-18 00:00:00
Tasks: Image Generation, Image-to-Image Translation, Translation, Unsupervised Image-To-Image Translation

Unsupervised image-to-image translation aims at learning the relationship
between samples from two image domains without supervised pair information. The
relationship between two domain images can be one-to-one, one-to-many or
many-to-many. In this paper, we study the one-to-many unsupervised image
translation problem in which an input sample from one domain can correspond to
multiple samples in the other domain. To learn the complex relationship between
the two domains, we introduce an additional variable to control the variations
in our one-to-many mapping. A generative model with an XO-structure, called the
XOGAN, is proposed to learn the cross domain relationship among the two domains
and the ad- ditional variables. Not only can we learn to translate between the
two image domains, we can also handle the translated images with additional
variations. Experiments are performed on unpaired image generation tasks,
including edges-to-objects translation and facial image translation. We show
that the proposed XOGAN model can generate plausible images and control
variations, such as color and texture, of the generated images. Moreover, while
state-of-the-art unpaired image generation algorithms tend to generate images
with monotonous colors, XOGAN can generate more diverse results.
Neural Network Compression using Transform Coding and Clustering
Authors: Thorsten Laude, Yannick Richter, Jörn Ostermann
Date: 2018-05-18 00:00:00
Proceeding: NIPS Workshop CDNNRIA 2018
Tasks: Clustering, General Classification, image-classification, Image Classification, Neural Network Compression

With the deployment of neural networks on mobile devices and the necessity of
transmitting neural networks over limited or expensive channels, the file size
of the trained model was identified as bottleneck. In this paper, we propose a
codec for the compression of neural networks which is based on transform coding
for convolutional and dense layers and on clustering for biases and
normalizations. By using this codec, we achieve average compression factors
between 7.9-9.3 while the accuracy of the compressed networks for image
classification decreases only by 1%-2%, respectively.
Recognition of Activities from Eye Gaze and Egocentric Video
Authors: Anjith George, Aurobinda Routray
Date: 2018-05-18 00:00:00
Tasks: Classification, General Classification

This paper presents a framework for recognition of human activity from
egocentric video and eye tracking data obtained from a head-mounted eye
tracker. Three channels of information such as eye movement, ego-motion, and
visual features are combined for the classification of activities. Image
features were extracted using a pre-trained convolutional neural network. Eye
and ego-motion are quantized, and the windowed histograms are used as the
features. The combination of features obtains better accuracy for activity
classification as compared to individual features.
Learning and Inferring Movement with Deep Generative Model
Authors: Mingxuan Jing, Xiaojian Ma, Fuchun Sun, Huaping Liu
Date: 2018-05-18 00:00:00
Tasks: model, Motion Planning

Learning and inference movement is a very challenging problem due to its high
dimensionality and dependency to varied environments or tasks. In this paper,
we propose an effective probabilistic method for learning and inference of
basic movements. The motion planning problem is formulated as learning on a
directed graphic model and deep generative model is used to perform learning
and inference from demonstrations. An important characteristic of this method
is that it flexibly incorporates the task descriptors and context information
for long-term planning and it can be combined with dynamic systems for robot
control. The experimental validations on robotic approaching path planning
tasks show the advantages over the base methods with limited training data.
Dynamic learning rate using Mutual Information
Authors: Shrihari Vasudevan
Date: 2018-05-18 00:00:00

This paper demonstrates dynamic hyper-parameter setting, for deep neural
network training, using Mutual Information (MI). The specific hyper-parameter
studied in this paper is the learning rate. MI between the output layer and
true outcomes is used to dynamically set the learning rate of the network
through the training cycle; the idea is also extended to layer-wise setting of
learning rate. Two approaches are demonstrated - tracking relative change in
mutual information and, additionally tracking its value relative to a reference
measure. The paper does not attempt to recommend a specific learning rate
policy. Experiments demonstrate that mutual information may be effectively used
to dynamically set learning rate and achieve competitive to better outcomes in
competitive to better time.
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds
Authors: Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, Daniela Rus
Date: 2018-04-15 00:00:00
Proceeding: ICLR 2019 5
Tasks: Generalization Bounds, Neural Network Compression

We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network's output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones. We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters. Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks. We demonstrate the practical effectiveness of our algorithm on a variety of neural network configurations and real-world data sets.
Siamese Capsule Networks
Authors: James O' Neill
Date: 2018-05-18 00:00:00
Proceeding: ICLR 2019 5
Tasks: Face Verification, Few-Shot Learning

Capsule Networks have shown encouraging results on \textit{defacto} benchmark
computer vision datasets such as MNIST, CIFAR and smallNORB. Although, they are
yet to be tested on tasks where (1) the entities detected inherently have more
complex internal representations and (2) there are very few instances per class
to learn from and (3) where point-wise classification is not suitable. Hence,
this paper carries out experiments on face verification in both controlled and
uncontrolled settings that together address these points. In doing so we
introduce \textit{Siamese Capsule Networks}, a new variant that can be used for
pairwise learning tasks. The model is trained using contrastive loss with
$\ell_2$-normalized capsule encoded pose features. We find that \textit{Siamese
Capsule Networks} perform well against strong baselines on both pairwise
learning datasets, yielding best results in the few-shot learning setting where
image pairs in the test set contain unseen subjects.
A Study on Dialog Act Recognition using Character-Level Tokenization
Authors: Eugénio Ribeiro, Ricardo Ribeiro, David Martins de Matos
Date: 2018-05-18 00:00:00

Dialog act recognition is an important step for dialog systems since it
reveals the intention behind the uttered words. Most approaches on the task use
word-level tokenization. In contrast, this paper explores the use of
character-level tokenization. This is relevant since there is information at
the sub-word level that is related to the function of the words and, thus,
their intention. We also explore the use of different context windows around
each token, which are able to capture important elements, such as affixes.
Furthermore, we assess the importance of punctuation and capitalization. We
performed experiments on both the Switchboard Dialog Act Corpus and the DIHANA
Corpus. In both cases, the experiments not only show that character-level
tokenization leads to better performance than the typical word-level
approaches, but also that both approaches are able to capture complementary
information. Thus, the best results are achieved by combining tokenization at
both levels.
Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows
Authors: George Papamakarios, David C. Sterratt, Iain Murray
Date: 2018-05-18 00:00:00
Tasks: Bayesian Inference

We present Sequential Neural Likelihood (SNL), a new method for Bayesian
inference in simulator models, where the likelihood is intractable but
simulating data from the model is possible. SNL trains an autoregressive flow
on simulated data in order to learn a model of the likelihood in the region of
high posterior density. A sequential training procedure guides simulations and
reduces simulation cost by orders of magnitude. We show that SNL is more
robust, more accurate and requires less tuning than related neural-based
methods, and we discuss diagnostics for assessing calibration, convergence and
goodness-of-fit.
Approximate Bayesian inference in spatial environments
Authors: Atanas Mirchev, Baris Kayalibay, Maximilian Soelch, Patrick van der Smagt, Justin Bayer
Date: 2018-05-18 00:00:00
Tasks: Bayesian Inference, Decision Making, Variational Inference

Model-based approaches bear great promise for decision making of agents interacting with the physical world. In the context of spatial environments, different types of problems such as localisation, mapping, navigation or autonomous exploration are typically adressed with specialised methods, often relying on detailed knowledge of the system at hand. We express these tasks as probabilistic inference and planning under the umbrella of deep sequential generative models. Using the frameworks of variational inference and neural networks, our method inherits favourable properties such as flexibility, scalability and the ability to learn from data. The method performs comparably to specialised state-of-the-art methodology in two distinct simulated environments.
Mirrored Langevin Dynamics
Authors: Ya-Ping Hsieh, Ali Kavis, Paul Rolland, Volkan Cevher
Date: 2018-02-27 00:00:00
Proceeding: NeurIPS 2018 12

We consider the problem of sampling from constrained distributions, which has posed significant challenges to both non-asymptotic analysis and algorithmic design. We propose a unified framework, which is inspired by the classical mirror descent, to derive novel first-order sampling schemes. We prove that, for a general target distribution with strongly convex potential, our framework implies the existence of a first-order algorithm achieving $\tilde{O}(\epsilon^{-2}d)$ convergence, suggesting that the state-of-the-art $\tilde{O}(\epsilon^{-6}d^5)$ can be vastly improved. With the important Latent Dirichlet Allocation (LDA) application in mind, we specialize our algorithm to sample from Dirichlet posteriors, and derive the first non-asymptotic $\tilde{O}(\epsilon^{-2}d^2)$ rate for first-order sampling. We further extend our framework to the mini-batch setting and prove convergence rates when only stochastic gradients are available. Finally, we report promising experimental results for LDA on real datasets.
Distributionally Robust Inverse Covariance Estimation: The Wasserstein Shrinkage Estimator
Authors: Viet Anh Nguyen, Daniel Kuhn, Peyman Mohajerin Esfahani
Date: 2018-05-18 00:00:00

We introduce a distributionally robust maximum likelihood estimation model
with a Wasserstein ambiguity set to infer the inverse covariance matrix of a
$p$-dimensional Gaussian random vector from $n$ independent samples. The
proposed model minimizes the worst case (maximum) of Stein's loss across all
normal reference distributions within a prescribed Wasserstein distance from
the normal distribution characterized by the sample mean and the sample
covariance matrix. We prove that this estimation problem is equivalent to a
semidefinite program that is tractable in theory but beyond the reach of
general purpose solvers for practically relevant problem dimensions $p$. In the
absence of any prior structural information, the estimation problem has an
analytical solution that is naturally interpreted as a nonlinear shrinkage
estimator. Besides being invertible and well-conditioned even for $p>n$, the
new shrinkage estimator is rotation-equivariant and preserves the order of the
eigenvalues of the sample covariance matrix. These desirable properties are not
imposed ad hoc but emerge naturally from the underlying distributionally robust
optimization model. Finally, we develop a sequential quadratic approximation
algorithm for efficiently solving the general estimation problem subject to
conditional independence constraints typically encountered in Gaussian
graphical models.
Deep Convolutional Decision Jungle for Image Classification
Authors: Seungryul Baek, Kwang In Kim, Tae-Kyun Kim
Date: 2017-06-06 00:00:00
Tasks: Classification, Face Verification, General Classification, image-classification, Image Classification

We propose a novel method called deep convolutional decision jungle (CDJ) and
its learning algorithm for image classification. The CDJ maintains the
structure of standard convolutional neural networks (CNNs), i.e. multiple
layers of multiple response maps fully connected. Each response map-or node-in
both the convolutional and fully-connected layers selectively respond to class
labels s.t. each data sample travels via a specific soft route of those
activated nodes. The proposed method CDJ automatically learns features, whereas
decision forests and jungles require pre-defined feature sets. Compared to
CNNs, the method embeds the benefits of using data-dependent discriminative
functions, which better handles multi-modal/heterogeneous data; further,the
method offers more diverse sparse network responses, which in turn can be used
for cost-effective learning/classification. The network is learnt by combining
conventional softmax and proposed entropy losses in each layer. The entropy
loss,as used in decision tree growing, measures the purity of data activation
according to the class label distribution. The back-propagation rule for the
proposed loss function is derived from stochastic gradient descent (SGD)
optimization of CNNs. We show that our proposed method outperforms
state-of-the-art methods on three public image classification benchmarks and
one face verification dataset. We also demonstrate the use of auxiliary data
labels, when available, which helps our method to learn more discriminative
routing and representations and leads to improved classification.
Approximate Model Counting by Partial Knowledge Compilation
Authors: Yong Lai
Date: 2018-05-18 00:00:00
Tasks: model

Model counting is the problem of computing the number of satisfying
assignments of a given propositional formula. Although exact model counters can
be naturally furnished by most of the knowledge compilation (KC) methods, in
practice, they fail to generate the compiled results for the exact counting of
models for certain formulas due to the explosion in sizes. Decision-DNNF is an
important KC language that captures most of the practical compilers. We propose
a generalized Decision-DNNF (referred to as partial Decision-DNNF) via
introducing a class of new leaf vertices (called unknown vertices), and then
propose an algorithm called PartialKC to generate randomly partial
Decision-DNNF formulas from the given formulas. An unbiased estimate of the
model number can be computed via a randomly partial Decision-DNNF formula. Each
calling of PartialKC consists of multiple callings of MicroKC, while each of
the latter callings is a process of importance sampling equipped with KC
technologies. The experimental results show that PartialKC is more accurate
than both SampleSearch and SearchTreeSampler, PartialKC scales better than
SearchTreeSampler, and the KC technologies can obviously accelerate sampling.
A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese
Authors: Shiyu Zhou, Linhao Dong, Shuang Xu, Bo Xu
Date: 2018-05-16 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Decoder, Language Modeling, Language Modelling, Sequence-To-Sequence Speech Recognition, speech-recognition, Speech Recognition

The choice of modeling units is critical to automatic speech recognition
(ASR) tasks. Conventional ASR systems typically choose context-dependent states
(CD-states) or context-dependent phonemes (CD-phonemes) as their modeling
units. However, it has been challenged by sequence-to-sequence attention-based
models, which integrate an acoustic, pronunciation and language model into a
single neural network. On English ASR tasks, previous attempts have already
shown that the modeling unit of graphemes can outperform that of phonemes by
sequence-to-sequence attention-based model.
  In this paper, we are concerned with modeling units on Mandarin Chinese ASR
tasks using sequence-to-sequence attention-based models with the Transformer.
Five modeling units are explored including context-independent phonemes
(CI-phonemes), syllables, words, sub-words and characters. Experiments on HKUST
datasets demonstrate that the lexicon free modeling units can outperform
lexicon related modeling units in terms of character error rate (CER). Among
five modeling units, character based model performs best and establishes a new
state-of-the-art CER of $26.64\%$ on HKUST datasets without a hand-designed
lexicon and an extra language model integration, which corresponds to a $4.8\%$
relative improvement over the existing best CER of $28.0\%$ by the joint
CTC-attention based encoder-decoder network.
Recurrent knowledge distillation
Authors: Silvia L. Pintea, Yue Liu, Jan C. van Gemert
Date: 2018-05-18 00:00:00
Tasks: Knowledge Distillation

Knowledge distillation compacts deep networks by letting a small student
network learn from a large teacher network. The accuracy of knowledge
distillation recently benefited from adding residual layers. We propose to
reduce the size of the student network even further by recasting multiple
residual layers in the teacher network into a single recurrent student layer.
We propose three variants of adding recurrent connections into the student
network, and show experimentally on CIFAR-10, Scenes and MiniPlaces, that we
can reduce the number of parameters at little loss in accuracy.
Low-Cost Recurrent Neural Network Expected Performance Evaluation
Authors: Andrés Camero, Jamal Toutouh, Enrique Alba
Date: 2018-05-18 00:00:00

Recurrent neural networks are a powerful tool, but they are very sensitive to
their hyper-parameter configuration. Moreover, training properly a recurrent
neural network is a tough task, therefore selecting an appropriate
configuration is critical. Varied strategies have been proposed to tackle this
issue. However, most of them are still impractical because of the
time/resources needed. In this study, we propose a low computational cost model
to evaluate the expected performance of a given architecture based on the
distribution of the error of random samples of the weights. We empirically
validate our proposal using three use cases. The results suggest that this is a
promising alternative to reduce the cost of exploration for hyper-parameter
optimization.
Style Obfuscation by Invariance
Authors: Chris Emmery, Enrique Manjavacas, Grzegorz Chrupała
Date: 2018-05-18 00:00:00
Proceeding: COLING 2018 8
Tasks: Style Transfer

The task of obfuscating writing style using sequence models has previously
been investigated under the framework of obfuscation-by-transfer, where the
input text is explicitly rewritten in another style. These approaches also
often lead to major alterations to the semantic content of the input. In this
work, we propose obfuscation-by-invariance, and investigate to what extent
models trained to be explicitly style-invariant preserve semantics. We evaluate
our architectures on parallel and non-parallel corpora, and compare automatic
and human evaluations on the obfuscated sentences. Our experiments show that
style classifier performance can be reduced to chance level, whilst the
automatic evaluation of the output is seemingly equal to models applying
style-transfer. However, based on human evaluation we demonstrate a trade-off
between the level of obfuscation and the observed quality of the output in
terms of meaning preservation and grammaticality.