pwc-archive/papers-with-abstracts huggingface dataset:
Significance of Softmax-based Features in Comparison to Distance Metric Learning-based Features
Authors: Shota Horiguchi, Daiki Ikami, Kiyoharu Aizawa
Date: 2017-12-29 00:00:00
Tasks: Metric Learning

The extraction of useful deep features is important for many computer vision
tasks. Deep features extracted from classification networks have proved to
perform well in those tasks. To obtain features of greater usefulness,
end-to-end distance metric learning (DML) has been applied to train the feature
extractor directly. However, in these DML studies, there were no equitable
comparisons between features extracted from a DML-based network and those from
a softmax-based network. In this paper, by presenting objective comparisons
between these two approaches under the same network architecture, we show that
the softmax-based features perform competitive, or even better, to the
state-of-the-art DML features when the size of the dataset, that is, the number
of training samples per class, is large. The results suggest that softmax-based
features should be properly taken into account when evaluating the performance
of deep features.
Learning Deep and Compact Models for Gesture Recognition
Authors: Koustav Mullick, Anoop M. Namboodiri
Date: 2017-12-29 00:00:00
Tasks: Gesture Recognition, Knowledge Distillation, Model Compression

We look at the problem of developing a compact and accurate model for gesture
recognition from videos in a deep-learning framework. Towards this we propose a
joint 3DCNN-LSTM model that is end-to-end trainable and is shown to be better
suited to capture the dynamic information in actions. The solution achieves
close to state-of-the-art accuracy on the ChaLearn dataset, with only half the
model size. We also explore ways to derive a much more compact representation
in a knowledge distillation framework followed by model compression. The final
model is less than $1~MB$ in size, which is less than one hundredth of our
initial model, with a drop of $7\%$ in accuracy, and is suitable for real-time
gesture recognition on mobile devices.
The Multilinear Structure of ReLU Networks
Authors: Thomas Laurent, James Von Brecht
Date: 2017-12-29 00:00:00
Proceeding: ICML 2018 7

We study the loss surface of neural networks equipped with a hinge loss
criterion and ReLU or leaky ReLU nonlinearities. Any such network defines a
piecewise multilinear form in parameter space. By appealing to harmonic
analysis we show that all local minima of such network are non-differentiable,
except for those minima that occur in a region of parameter space where the
loss surface is perfectly flat. Non-differentiable minima are therefore not
technicalities or pathologies; they are heart of the problem when investigating
the loss of ReLU networks. As a consequence, we must employ techniques from
nonsmooth analysis to study these loss surfaces. We show how to apply these
techniques in some illustrative cases.
On Usage of Autoencoders and Siamese Networks for Online Handwritten Signature Verification
Authors: Kian Ahrabian, Bagher BabaAli
Date: 2017-12-07 00:00:00
Tasks: GPU

In this paper, we propose a novel writer-independent global feature
extraction framework for the task of automatic signature verification which
aims to make robust systems for automatically distinguishing negative and
positive samples. Our method consists of an autoencoder for modeling the sample
space into a fixed length latent space and a Siamese Network for classifying
the fixed-length samples obtained from the autoencoder based on the reference
samples of a subject as being "Genuine" or "Forged." During our experiments,
usage of Attention Mechanism and applying Downsampling significantly improved
the accuracy of the proposed framework. We evaluated our proposed framework
using SigWiComp2013 Japanese and GPDSsyntheticOnLineOffLineSignature datasets.
On the SigWiComp2013 Japanese dataset, we achieved 8.65% EER that means 1.2%
relative improvement compared to the best-reported result. Furthermore, on the
GPDSsyntheticOnLineOffLineSignature dataset, we achieved average EERs of 0.13%,
0.12%, 0.21% and 0.25% respectively for 150, 300, 1000 and 2000 test subjects
which indicates improvement of relative EER on the best-reported result by
95.67%, 95.26%, 92.9% and 91.52% respectively. Apart from the accuracy gain,
because of the nature of our proposed framework which is based on neural
networks and consequently is as simple as some consecutive matrix
multiplications, it has less computational cost than conventional methods such
as DTW and could be used concurrently on devices such as GPU, TPU, etc.
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Authors: Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, Trevor Darrell
Date: 2017-11-08 00:00:00
Proceeding: ICML 2018 7
Tasks: Domain Adaptation, Image-to-Image Translation, Semantic Segmentation, Synthetic-to-Real Translation, Unsupervised Image-To-Image Translation

Domain adaptation is critical for success in new, unseen environments.
Adversarial adaptation models applied in feature spaces discover domain
invariant representations, but are difficult to visualize and sometimes fail to
capture pixel-level and low-level domain shifts. Recent work has shown that
generative adversarial networks combined with cycle-consistency constraints are
surprisingly effective at mapping images between domains, even without the use
of aligned image pairs. We propose a novel discriminatively-trained
Cycle-Consistent Adversarial Domain Adaptation model. CyCADA adapts
representations at both the pixel-level and feature-level, enforces
cycle-consistency while leveraging a task loss, and does not require aligned
pairs. Our model can be applied in a variety of visual recognition and
prediction settings. We show new state-of-the-art results across multiple
adaptation tasks, including digit classification and semantic segmentation of
road scenes demonstrating transfer from synthetic to real world domains.
Convergence Analysis of Distributed Inference with Vector-Valued Gaussian Belief Propagation
Authors: Jian Du, Shaodan Ma, Yik-Chung Wu, Soummya Kar, José M. F. Moura
Date: 2016-11-07 00:00:00

This paper considers inference over distributed linear Gaussian models using
factor graphs and Gaussian belief propagation (BP). The distributed inference
algorithm involves only local computation of the information matrix and of the
mean vector, and message passing between neighbors. Under broad conditions, it
is shown that the message information matrix converges to a unique positive
definite limit matrix for arbitrary positive semidefinite initialization, and
it approaches an arbitrarily small neighborhood of this limit matrix at a
doubly exponential rate. A necessary and sufficient convergence condition for
the belief mean vector to converge to the optimal centralized estimator is
provided under the assumption that the message information matrix is
initialized as a positive semidefinite matrix. Further, it is shown that
Gaussian BP always converges when the underlying factor graph is given by the
union of a forest and a single loop. The proposed convergence condition in the
setup of distributed linear Gaussian models is shown to be strictly weaker than
other existing convergence conditions and requirements, including the Gaussian
Markov random field based walk-summability condition, and applicable to a large
class of scenarios.
Objective evaluation metrics for automatic classification of EEG events
Authors: Saeedeh Ziyabari, Vinit Shah, Meysam Golmohammadi, Iyad Obeid, Joseph Picone
Date: 2017-12-29 00:00:00
Tasks: Classification, EEG, Electroencephalogram (EEG), Event Detection, General Classification, Seizure Detection, Specificity

The evaluation of machine learning algorithms in biomedical fields for applications involving sequential data lacks standardization. Common quantitative scalar evaluation metrics such as sensitivity and specificity can often be misleading depending on the requirements of the application. Evaluation metrics must ultimately reflect the needs of users yet be sufficiently sensitive to guide algorithm development. Feedback from critical care clinicians who use automated event detection software in clinical applications has been overwhelmingly emphatic that a low false alarm rate, typically measured in units of the number of errors per 24 hours, is the single most important criterion for user acceptance. Though using a single metric is not often as insightful as examining performance over a range of operating conditions, there is a need for a single scalar figure of merit. In this paper, we discuss the deficiencies of existing metrics for a seizure detection task and propose several new metrics that offer a more balanced view of performance. We demonstrate these metrics on a seizure detection task based on the TUH EEG Corpus. We show that two promising metrics are a measure based on a concept borrowed from the spoken term detection literature, Actual Term-Weighted Value (ATWV), and a new metric, Time-Aligned Event Scoring (TAES), that accounts for the temporal alignment of the hypothesis to the reference annotation. We also demonstrate that state of the art technology based on deep learning, though impressive in its performance, still needs significant improvement before it meets very strict user acceptance criteria.
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
Authors: Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Deep Learning

High network communication cost for synchronizing gradients and parameters is
the well-known bottleneck of distributed training. In this work, we propose
TernGrad that uses ternary gradients to accelerate distributed deep learning in
data parallelism. Our approach requires only three numerical levels {-1,0,1},
which can aggressively reduce the communication time. We mathematically prove
the convergence of TernGrad under the assumption of a bound on gradients.
Guided by the bound, we propose layer-wise ternarizing and gradient clipping to
improve its convergence. Our experiments show that applying TernGrad on AlexNet
does not incur any accuracy loss and can even improve accuracy. The accuracy
loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a
performance model is proposed to study the scalability of TernGrad. Experiments
show significant speed gains for various deep neural networks. Our source code
is available.
Finite-sample risk bounds for maximum likelihood estimation with arbitrary penalties
Authors: W. D. Brinda, Jason M. Klusowski
Date: 2017-12-29 00:00:00

The MDL two-part coding $ \textit{index of resolvability} $ provides a
finite-sample upper bound on the statistical risk of penalized likelihood
estimators over countable models. However, the bound does not apply to
unpenalized maximum likelihood estimation or procedures with exceedingly small
penalties. In this paper, we point out a more general inequality that holds for
arbitrary penalties. In addition, this approach makes it possible to derive
exact risk bounds of order $1/n$ for iid parametric models, which improves on
the order $(\log n)/n$ resolvability bounds. We conclude by discussing
implications for adaptive estimation.
Differential Generative Adversarial Networks: Synthesizing Non-linear Facial Variations with Limited Number of Training Data
Authors: Geonmo Gu, Seong Tae Kim, Kihyun Kim, Wissam J. Baddar, Yong Man Ro
Date: 2017-11-28 00:00:00
Tasks: Face Generation

In face-related applications with a public available dataset, synthesizing
non-linear facial variations (e.g., facial expression, head-pose, illumination,
etc.) through a generative model is helpful in addressing the lack of training
data. In reality, however, there is insufficient data to even train the
generative model for face synthesis. In this paper, we propose Differential
Generative Adversarial Networks (D-GAN) that can perform photo-realistic face
synthesis even when training data is small. Two discriminators are devised to
ensure the generator to approximate a face manifold, which can express face
changes as it wants. Experimental results demonstrate that the proposed method
is robust to the amount of training data and synthesized images are useful to
improve the performance of a face expression classifier.
Accelerated Method for Stochastic Composition Optimization with Nonsmooth Regularization
Authors: Zhouyuan Huo, Bin Gu, Ji Liu, Heng Huang
Date: 2017-11-10 00:00:00
Tasks: Management, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Stochastic composition optimization draws much attention recently and has
been successful in many emerging applications of machine learning, statistical
analysis, and reinforcement learning. In this paper, we focus on the
composition problem with nonsmooth regularization penalty. Previous works
either have slow convergence rate or do not provide complete convergence
analysis for the general problem. In this paper, we tackle these two issues by
proposing a new stochastic composition optimization method for composition
problem with nonsmooth regularization penalty. In our method, we apply variance
reduction technique to accelerate the speed of convergence. To the best of our
knowledge, our method admits the fastest convergence rate for stochastic
composition optimization: for strongly convex composition problem, our
algorithm is proved to admit linear convergence; for general composition
problem, our algorithm significantly improves the state-of-the-art convergence
rate from $O(T^{-1/2})$ to $O((n_1+n_2)^{{2}/{3}}T^{-1})$. Finally, we apply
our proposed algorithm to portfolio management and policy evaluation in
reinforcement learning. Experimental results verify our theoretical analysis.
Reinforcement Learning with Analogical Similarity to Guide Schema Induction and Attention
Authors: James M. Foster, Matt Jones
Date: 2017-12-28 00:00:00
Tasks: Analogical Similarity, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Research in analogical reasoning suggests that higher-order cognitive
functions such as abstract reasoning, far transfer, and creativity are founded
on recognizing structural similarities among relational systems. Here we
integrate theories of analogy with the computational framework of reinforcement
learning (RL). We propose a psychology theory that is a computational synergy
between analogy and RL, in which analogical comparison provides the RL learning
algorithm with a measure of relational similarity, and RL provides feedback
signals that can drive analogical learning. Simulation results support the
power of this approach.
Sparse Representation Classification Beyond L1 Minimization and the Subspace Assumption
Authors: Cencheng Shen, Li Chen, Yuexiao Dong, Carey E. Priebe
Date: 2015-02-04 00:00:00
Tasks: Classification, Classification Consistency, General Classification

The sparse representation classifier (SRC) has been utilized in various classification problems, which makes use of L1 minimization and works well for image recognition satisfying a subspace assumption. In this paper we propose a new implementation of SRC via screening, establish its equivalence to the original SRC under regularity conditions, and prove its classification consistency under a latent subspace model and contamination. The results are demonstrated via simulations and real data experiments, where the new algorithm achieves comparable numerical performance and significantly faster.
Multi-timescale memory dynamics in a reinforcement learning network with attention-gated memory
Authors: Marco Martinolli, Wulfram Gerstner, Aditya Gilra
Date: 2017-12-28 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Learning and memory are intertwined in our brain and their relationship is at
the core of several recent neural network models. In particular, the
Attention-Gated MEmory Tagging model (AuGMEnT) is a reinforcement learning
network with an emphasis on biological plausibility of memory dynamics and
learning. We find that the AuGMEnT network does not solve some hierarchical
tasks, where higher-level stimuli have to be maintained over a long time, while
lower-level stimuli need to be remembered and forgotten over a shorter
timescale. To overcome this limitation, we introduce hybrid AuGMEnT, with leaky
or short-timescale and non-leaky or long-timescale units in memory, that allow
to exchange lower-level information while maintaining higher-level one, thus
solving both hierarchical and distractor tasks.
Corpus specificity in LSA and Word2vec: the role of out-of-domain documents
Authors: Edgar Altszyler, Mariano Sigman, Diego Fernandez Slezak
Date: 2017-12-28 00:00:00
Proceeding: WS 2018 7
Tasks: Specificity, Word Embeddings

Latent Semantic Analysis (LSA) and Word2vec are some of the most widely used
word embeddings. Despite the popularity of these techniques, the precise
mechanisms by which they acquire new semantic relations between words remain
unclear. In the present article we investigate whether LSA and Word2vec
capacity to identify relevant semantic dimensions increases with size of
corpus. One intuitive hypothesis is that the capacity to identify relevant
dimensions should increase as the amount of data increases. However, if corpus
size grow in topics which are not specific to the domain of interest, signal to
noise ratio may weaken. Here we set to examine and distinguish these
alternative hypothesis. To investigate the effect of corpus specificity and
size in word-embeddings we study two ways for progressive elimination of
documents: the elimination of random documents vs. the elimination of documents
unrelated to a specific task. We show that Word2vec can take advantage of all
the documents, obtaining its best performance when it is trained with the whole
corpus. On the contrary, the specialization (removal of out-of-domain
documents) of the training corpus, accompanied by a decrease of dimensionality,
can increase LSA word-representation quality while speeding up the processing
time. Furthermore, we show that the specialization without the decrease in LSA
dimensionality can produce a strong performance reduction in specific tasks.
From a cognitive-modeling point of view, we point out that LSA's word-knowledge
acquisitions may not be efficiently exploiting higher-order co-occurrences and
global relations, whereas Word2vec does.
Kernel Robust Bias-Aware Prediction under Covariate Shift
Authors: Anqi Liu, Rizal Fathony, Brian D. Ziebart
Date: 2017-12-28 00:00:00
Tasks: Prediction

Under covariate shift, training (source) data and testing (target) data
differ in input space distribution, but share the same conditional label
distribution. This poses a challenging machine learning task. Robust Bias-Aware
(RBA) prediction provides the conditional label distribution that is robust to
the worstcase logarithmic loss for the target distribution while matching
feature expectation constraints from the source distribution. However,
employing RBA with insufficient feature constraints may result in high
certainty predictions for much of the source data, while leaving too much
uncertainty for target data predictions. To overcome this issue, we extend the
representer theorem to the RBA setting, enabling minimization of regularized
expected target risk by a reweighted kernel expectation under the source
distribution. By applying kernel methods, we establish consistency guarantees
and demonstrate better performance of the RBA classifier than competing methods
on synthetically biased UCI datasets as well as datasets that have natural
covariate shift.
Robust Covariate Shift Prediction with General Losses and Feature Views
Authors: Anqi Liu, Brian D. Ziebart
Date: 2017-12-28 00:00:00

Covariate shift relaxes the widely-employed independent and identically
distributed (IID) assumption by allowing different training and testing input
distributions. Unfortunately, common methods for addressing covariate shift by
trying to remove the bias between training and testing distributions using
importance weighting often provide poor performance guarantees in theory and
unreliable predictions with high variance in practice. Recently developed
methods that construct a predictor that is inherently robust to the
difficulties of learning under covariate shift are restricted to minimizing
logloss and can be too conservative when faced with high-dimensional learning
tasks. We address these limitations in two ways: by robustly minimizing various
loss functions, including non-convex ones, under the testing distribution; and
by separately shaping the influence of covariate shift according to different
feature-based views of the relationship between input variables and example
labels. These generalizations make robust covariate shift prediction applicable
to more task scenarios. We demonstrate the benefits on classification under
covariate shift tasks.
Discriminative and Geometry Aware Unsupervised Domain Adaptation
Authors: Lingkun Luo, Liming Chen, Shiqiang Hu, Ying Lu, Xiaofang Wang
Date: 2017-12-28 00:00:00
Tasks: Domain Adaptation, image-classification, Image Classification, Unsupervised Domain Adaptation

Domain adaptation (DA) aims to generalize a learning model across training
and testing data despite the mismatch of their data distributions. In light of
a theoretical estimation of upper error bound, we argue in this paper that an
effective DA method should 1) search a shared feature subspace where source and
target data are not only aligned in terms of distributions as most state of the
art DA methods do, but also discriminative in that instances of different
classes are well separated; 2) account for the geometric structure of the
underlying data manifold when inferring data labels on the target domain. In
comparison with a baseline DA method which only cares about data distribution
alignment between source and target, we derive three different DA models,
namely CDDA, GA-DA, and DGA-DA, to highlight the contribution of Close yet
Discriminative DA(CDDA) based on 1), Geometry Aware DA (GA-DA) based on 2), and
finally Discriminative and Geometry Aware DA (DGA-DA) implementing jointly 1)
and 2). Using both synthetic and real data, we show the effectiveness of the
proposed approach which consistently outperforms state of the art DA methods
over 36 image classification DA tasks through 6 popular benchmarks. We further
carry out in-depth analysis of the proposed DA method in quantifying the
contribution of each term of our DA model and provide insights into the
proposed DA methods in visualizing both real and synthetic data.
Debiased Machine Learning of Set-Identified Linear Models
Authors: Vira Semenova
Date: 2017-12-28 00:00:00
Tasks: BIG-bench Machine Learning, Model Selection

This paper provides estimation and inference methods for an identified set's boundary (i.e., support function) where the selection among a very large number of covariates is based on modern regularized tools. I characterize the boundary using a semiparametric moment equation. Combining Neyman-orthogonality and sample splitting ideas, I construct a root-N consistent, uniformly asymptotically Gaussian estimator of the boundary and propose a multiplier bootstrap procedure to conduct inference. I apply this result to the partially linear model, the partially linear IV model and the average partial derivative with an interval-valued outcome.