pwc-archive/papers-with-abstracts huggingface dataset:
Transfer entropy-based feedback improves performance in artificial neural networks
Authors: Sebastian Herzog, Christian Tetzlaff, Florentin Wörgötter
Date: 2017-06-13 00:00:00

The structure of the majority of modern deep neural networks is characterized
by uni- directional feed-forward connectivity across a very large number of
layers. By contrast, the architecture of the cortex of vertebrates contains
fewer hierarchical levels but many recurrent and feedback connections. Here we
show that a small, few-layer artificial neural network that employs feedback
will reach top level performance on a standard benchmark task, otherwise only
obtained by large feed-forward structures. To achieve this we use feed-forward
transfer entropy between neurons to structure feedback connectivity. Transfer
entropy can here intuitively be understood as a measure for the relevance of
certain pathways in the network, which are then amplified by feedback. Feedback
may therefore be key for high network performance in small brain-like
architectures.
Theory II: Landscape of the Empirical Risk in Deep Learning
Authors: Qianli Liao, Tomaso Poggio
Date: 2017-03-28 00:00:00
Tasks: Deep Learning

Previous theoretical work on deep learning and neural network optimization
tend to focus on avoiding saddle points and local minima. However, the
practical observation is that, at least in the case of the most successful Deep
Convolutional Neural Networks (DCNNs), practitioners can always increase the
network size to fit the training data (an extreme example would be [1]). The
most successful DCNNs such as VGG and ResNets are best used with a degree of
"overparametrization". In this work, we characterize with a mix of theory and
experiments, the landscape of the empirical risk of overparametrized DCNNs. We
first prove in the regression framework the existence of a large number of
degenerate global minimizers with zero empirical error (modulo inconsistent
equations). The argument that relies on the use of Bezout theorem is rigorous
when the RELUs are replaced by a polynomial nonlinearity (which empirically
works as well). As described in our Theory III [2] paper, the same minimizers
are degenerate and thus very likely to be found by SGD that will furthermore
select with higher probability the most robust zero-minimizer. We further
experimentally explored and visualized the landscape of empirical risk of a
DCNN on CIFAR-10 during the entire training process and especially the global
minima. Finally, based on our theoretical and experimental results, we propose
an intuitive model of the landscape of DCNN's empirical loss surface, which
might not be as complicated as people commonly believe.
Synthesis of Near-regular Natural Textures
Authors: V. Asha
Date: 2017-06-22 00:00:00
Tasks: Texture Synthesis

Texture synthesis is widely used in the field of computer graphics, vision,
and image processing. In the present paper, a texture synthesis algorithm is
proposed for near-regular natural textures with the help of a representative
periodic pattern extracted from the input textures using distance matching
function. Local texture statistics is then analyzed against global texture
statistics for non-overlapping windows of size same as periodic pattern size
and a representative periodic pattern is extracted from the image and used for
texture synthesis, while preserving the global regularity and visual
appearance. Validation of the algorithm based on experiments with synthetic
textures whose periodic pattern sizes are known and containing camouflages /
defects proves the strength of the algorithm for texture synthesis and its
application in detection of camouflages / defects in textures.
Optimizing Differentiable Relaxations of Coreference Evaluation Metrics
Authors: Phong Le, Ivan Titov
Date: 2017-04-14 00:00:00
Proceeding: CONLL 2017 8
Tasks: Imitation Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Coreference evaluation metrics are hard to optimize directly as they are
non-differentiable functions, not easily decomposable into elementary
decisions. Consequently, most approaches optimize objectives only indirectly
related to the end goal, resulting in suboptimal performance. Instead, we
propose a differentiable relaxation that lends itself to gradient-based
optimisation, thus bypassing the need for reinforcement learning or heuristic
modification of cross-entropy. We show that by modifying the training objective
of a competitive neural coreference system, we obtain a substantial gain in
performance. This suggests that our approach can be regarded as a viable
alternative to using reinforcement learning or more computationally expensive
imitation learning.
Continuum Limit of Posteriors in Graph Bayesian Inverse Problems
Authors: Nicolas Garcia Trillos, Daniel Sanz-Alonso
Date: 2017-06-22 00:00:00
Tasks: Uncertainty Quantification

We consider the problem of recovering a function input of a differential
equation formulated on an unknown domain $M$. We assume to have access to a
discrete domain $M_n=\{x_1, \dots, x_n\} \subset M$, and to noisy measurements
of the output solution at $p\le n$ of those points. We introduce a graph-based
Bayesian inverse problem, and show that the graph-posterior measures over
functions in $M_n$ converge, in the large $n$ limit, to a posterior over
functions in $M$ that solves a Bayesian inverse problem with known domain.
  The proofs rely on the variational formulation of the Bayesian update, and on
a new topology for the study of convergence of measures over functions on point
clouds to a measure over functions on the continuum. Our framework, techniques,
and results may serve to lay the foundations of robust uncertainty
quantification of graph-based tasks in machine learning. The ideas are
presented in the concrete setting of recovering the initial condition of the
heat equation on an unknown manifold.
Shape recognition of volcanic ash by simple convolutional neural network
Authors: Daigo Shoji, Rina Noguchi
Date: 2017-06-22 00:00:00

Shape analyses of tephra grains result in understanding eruption mechanism of
volcanoes. However, we have to define and select parameter set such as
convexity for the precise discrimination of tephra grains. Selection of the
best parameter set for the recognition of tephra shapes is complicated.
Actually, many shape parameters have been suggested. Recently, neural network
has made a great success in the field of machine learning. Convolutional neural
network can recognize the shape of images without human bias and shape
parameters. We applied the simple convolutional neural network developed for
the handwritten digits to the recognition of tephra shapes. The network was
trained by Morphologi tephra images, and it can recognize the tephra shapes
with approximately 90% of accuracy.
Frank-Wolfe Optimization for Symmetric-NMF under Simplicial Constraint
Authors: Han Zhao, Geoff Gordon
Date: 2017-06-20 00:00:00
Tasks: Clustering

Symmetric nonnegative matrix factorization has found abundant applications in
various domains by providing a symmetric low-rank decomposition of nonnegative
matrices. In this paper we propose a Frank-Wolfe (FW) solver to optimize the
symmetric nonnegative matrix factorization problem under a simplicial
constraint, which has recently been proposed for probabilistic clustering.
Compared with existing solutions, this algorithm is simple to implement, and
has no hyperparameters to be tuned. Building on the recent advances of FW
algorithms in nonconvex optimization, we prove an $O(1/\varepsilon^2)$
convergence rate to $\varepsilon$-approximate KKT points, via a tight bound
$\Theta(n^2)$ on the curvature constant, which matches the best known result in
unconstrained nonconvex setting using gradient methods. Numerical results
demonstrate the effectiveness of our algorithm. As a side contribution, we
construct a simple nonsmooth convex problem where the FW algorithm fails to
converge to the optimum. This result raises an interesting question about
necessary conditions of the success of the FW algorithm on convex problems.
Curvature-aware Manifold Learning
Authors: Yangyang Li
Date: 2017-06-22 00:00:00

Traditional manifold learning algorithms assumed that the embedded manifold
is globally or locally isometric to Euclidean space. Under this assumption,
they divided manifold into a set of overlapping local patches which are locally
isometric to linear subsets of Euclidean space. By analyzing the global or
local isometry assumptions it can be shown that the learnt manifold is a flat
manifold with zero Riemannian curvature tensor. In general, manifolds may not
satisfy these hypotheses. One major limitation of traditional manifold learning
is that it does not consider the curvature information of manifold. In order to
remove these limitations, we present our curvature-aware manifold learning
algorithm called CAML. The purpose of our algorithm is to break the local
isometry assumption and to reduce the dimension of the general manifold which
is not isometric to Euclidean space. Thus, our method adds the curvature
information to the process of manifold learning. The experiments have shown
that our method CAML is more stable than other manifold learning algorithms by
comparing the neighborhood preserving ratios.
Deep Speaker Verification: Do We Need End to End?
Authors: Dong Wang, Lantian Li, Zhiyuan Tang, Thomas Fang Zheng
Date: 2017-06-22 00:00:00
Tasks: Speaker Verification

End-to-end learning treats the entire system as a whole adaptable black box,
which, if sufficient data are available, may learn a system that works very
well for the target task. This principle has recently been applied to several
prototype research on speaker verification (SV), where the feature learning and
classifier are learned together with an objective function that is consistent
with the evaluation metric. An opposite approach to end-to-end is feature
learning, which firstly trains a feature learning model, and then constructs a
back-end classifier separately to perform SV. Recently, both approaches
achieved significant performance gains on SV, mainly attributed to the smart
utilization of deep neural networks. However, the two approaches have not been
carefully compared, and their respective advantages have not been well
discussed. In this paper, we compare the end-to-end and feature learning
approaches on a text-independent SV task. Our experiments on a dataset sampled
from the Fisher database and involving 5,000 speakers demonstrated that the
feature learning approach outperformed the end-to-end approach. This is a
strong support for the feature learning approach, at least with data and
computation resources similar to ours.
Cross-lingual Speaker Verification with Deep Feature Learning
Authors: Lantian Li, Dong Wang, Askar Rozi, Thomas Fang Zheng
Date: 2017-06-22 00:00:00
Tasks: Speaker Verification

Existing speaker verification (SV) systems often suffer from performance
degradation if there is any language mismatch between model training, speaker
enrollment, and test. A major cause of this degradation is that most existing
SV methods rely on a probabilistic model to infer the speaker factor, so any
significant change on the distribution of the speech signal will impact the
inference. Recently, we proposed a deep learning model that can learn how to
extract the speaker factor by a deep neural network (DNN). By this feature
learning, an SV system can be constructed with a very simple back-end model. In
this paper, we investigate the robustness of the feature-based SV system in
situations with language mismatch. Our experiments were conducted on a complex
cross-lingual scenario, where the model training was in English, and the
enrollment and test were in Chinese or Uyghur. The experiments demonstrated
that the feature-based system outperformed the i-vector system with a large
margin, particularly with language mismatch between enrollment and test.
Speaker Recognition with Cough, Laugh and "Wei"
Authors: Miao Zhang, Yixiang Chen, Lantian Li, Dong Wang
Date: 2017-06-22 00:00:00
Tasks: Speaker Recognition

This paper proposes a speaker recognition (SRE) task with trivial speech
events, such as cough and laugh. These trivial events are ubiquitous in
conversations and less subjected to intentional change, therefore offering
valuable particularities to discover the genuine speaker from disguised speech.
However, trivial events are often short and idiocratic in spectral patterns,
making SRE extremely difficult. Fortunately, we found a very powerful deep
feature learning structure that can extract highly speaker-sensitive features.
By employing this tool, we studied the SRE performance on three types of
trivial events: cough, laugh and "Wei" (a short Chinese "Hello"). The results
show that there is rich speaker information within these trivial events, even
for cough that is intuitively less speaker distinguishable. With the deep
feature approach, the EER can reach 10%-14% with the three trivial events,
despite their extremely short durations (0.2-1.0 seconds).
Unsupervised Neural-Symbolic Integration
Authors: Son N. Tran
Date: 2017-06-06 00:00:00

Symbolic has been long considered as a language of human intelligence while
neural networks have advantages of robust computation and dealing with noisy
data. The integration of neural-symbolic can offer better learning and
reasoning while providing a means for interpretability through the
representation of symbolic knowledge. Although previous works focus intensively
on supervised feedforward neural networks, little has been done for the
unsupervised counterparts. In this paper we show how to integrate symbolic
knowledge into unsupervised neural networks. We exemplify our approach with
knowledge in different forms, including propositional logic for DNA promoter
prediction and first-order logic for understanding family relationship.
A Novel VHR Image Change Detection Algorithm Based on Image Fusion and Fuzzy C-Means Clustering
Authors: Rongcui Dong, Haoxiang Wang
Date: 2017-06-22 00:00:00
Tasks: Change Detection, Clustering

This thesis describes a study to perform change detection on Very High
Resolution satellite images using image fusion based on 2D Discrete Wavelet
Transform and Fuzzy C-Means clustering algorithm. Multiple other methods are
also quantitatively and qualitatively compared in this study.
Multi-View 3D Object Detection Network for Autonomous Driving
Authors: Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Tian Xia
Date: 2016-11-23 00:00:00
Proceeding: CVPR 2017 7
Tasks: 3D Object Detection, Autonomous Driving, Object, object-detection, Object Detection, Object Proposal Generation

This paper aims at high-accuracy 3D object detection in autonomous driving
scenario. We propose Multi-View 3D networks (MV3D), a sensory-fusion framework
that takes both LIDAR point cloud and RGB images as input and predicts oriented
3D bounding boxes. We encode the sparse 3D point cloud with a compact
multi-view representation. The network is composed of two subnetworks: one for
3D object proposal generation and another for multi-view feature fusion. The
proposal network generates 3D candidate boxes efficiently from the bird's eye
view representation of 3D point cloud. We design a deep fusion scheme to
combine region-wise features from multiple views and enable interactions
between intermediate layers of different paths. Experiments on the challenging
KITTI benchmark show that our approach outperforms the state-of-the-art by
around 25% and 30% AP on the tasks of 3D localization and 3D detection. In
addition, for 2D detection, our approach obtains 10.3% higher AP than the
state-of-the-art on the hard data among the LIDAR-based methods.
Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networks
Authors: M. Huzaifah
Date: 2017-06-22 00:00:00
Tasks: Audio Classification, Classification, Environmental Sound Classification, General Classification, Sound Classification, speech-recognition, Speech Recognition

Recent successful applications of convolutional neural networks (CNNs) to
audio classification and speech recognition have motivated the search for
better input representations for more efficient training. Visual displays of an
audio signal, through various time-frequency representations such as
spectrograms offer a rich representation of the temporal and spectral structure
of the original signal. In this letter, we compare various popular signal
processing methods to obtain this representation, such as short-time Fourier
transform (STFT) with linear and Mel scales, constant-Q transform (CQT) and
continuous Wavelet transform (CWT), and assess their impact on the
classification performance of two environmental sound datasets using CNNs. This
study supports the hypothesis that time-frequency representations are valuable
in learning useful features for sound classification. Moreover, the actual
transformation used is shown to impact the classification accuracy, with
Mel-scaled STFT outperforming the other discussed methods slightly and baseline
MFCC features to a large degree. Additionally, we observe that the optimal
window size during transformation is dependent on the characteristics of the
audio signal and architecturally, 2D convolution yielded better results in most
cases compared to 1D.
A Useful Motif for Flexible Task Learning in an Embodied Two-Dimensional Visual Environment
Authors: Kevin T. Feigelis, Daniel L. K. Yamins
Date: 2017-06-22 00:00:00

Animals (especially humans) have an amazing ability to learn new tasks
quickly, and switch between them flexibly. How brains support this ability is
largely unknown, both neuroscientifically and algorithmically. One reasonable
supposition is that modules drawing on an underlying general-purpose sensory
representation are dynamically allocated on a per-task basis. Recent results
from neuroscience and artificial intelligence suggest the role of the general
purpose visual representation may be played by a deep convolutional neural
network, and give some clues how task modules based on such a representation
might be discovered and constructed. In this work, we investigate module
architectures in an embodied two-dimensional touchscreen environment, in which
an agent's learning must occur via interactions with an environment that emits
images and rewards, and accepts touches as input. This environment is designed
to capture the physical structure of the task environments that are commonly
deployed in visual neuroscience and psychophysics. We show that in this
context, very simple changes in the nonlinear activations used by such a module
can significantly influence how fast it is at learning visual tasks and how
suitable it is for switching to new tasks.
Learning Domain-Invariant Subspace using Domain Features and Independence Maximization
Authors: Ke Yan, Lu Kou, David Zhang
Date: 2016-03-15 00:00:00
Tasks: Domain Adaptation

Domain adaptation algorithms are useful when the distributions of the
training and the test data are different. In this paper, we focus on the
problem of instrumental variation and time-varying drift in the field of
sensors and measurement, which can be viewed as discrete and continuous
distributional change in the feature space. We propose maximum independence
domain adaptation (MIDA) and semi-supervised MIDA (SMIDA) to address this
problem. Domain features are first defined to describe the background
information of a sample, such as the device label and acquisition time. Then,
MIDA learns a subspace which has maximum independence with the domain features,
so as to reduce the inter-domain discrepancy in distributions. A feature
augmentation strategy is also designed to project samples according to their
backgrounds so as to improve the adaptation. The proposed algorithms are
flexible and fast. Their effectiveness is verified by experiments on synthetic
datasets and four real-world ones on sensors, measurement, and computer vision.
They can greatly enhance the practicability of sensor systems, as well as
extend the application scope of existing domain adaptation algorithms by
uniformly handling different kinds of distributional change.
Learning Localized Geometric Features Using 3D-CNN: An Application to Manufacturability Analysis of Drilled Holes
Authors: Aditya Balu, Sambit Ghadai, Kin Gwn Lore, Gavin Young, Adarsh Krishnamurthy, Soumik Sarkar
Date: 2016-12-07 00:00:00
Tasks: Decision Making, Object, Object Recognition

3D convolutional neural networks (3D-CNN) have been used for object
recognition based on the voxelized shape of an object. In this paper, we
present a 3D-CNN based method to learn distinct local geometric features of
interest within an object. In this context, the voxelized representation may
not be sufficient to capture the distinguishing information about such local
features. To enable efficient learning, we augment the voxel data with surface
normals of the object boundary. We then train a 3D-CNN with this augmented data
and identify the local features critical for decision-making using 3D
gradient-weighted class activation maps. An application of this feature
identification framework is to recognize difficult-to-manufacture drilled hole
features in a complex CAD geometry. The framework can be extended to identify
difficult-to-manufacture features at multiple spatial scales leading to a
real-time decision support system for design for manufacturability.
Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
Authors: Shuchang Zhou, Yuzhi Wang, He Wen, Qinyao He, Yuheng Zou
Date: 2017-06-22 00:00:00
Tasks: Quantization

Quantized Neural Networks (QNNs), which use low bitwidth numbers for
representing parameters and performing computations, have been proposed to
reduce the computation complexity, storage size and memory usage. In QNNs,
parameters and activations are uniformly quantized, such that the
multiplications and additions can be accelerated by bitwise operations.
However, distributions of parameters in Neural Networks are often imbalanced,
such that the uniform quantization determined from extremal values may under
utilize available bitwidth. In this paper, we propose a novel quantization
method that can ensure the balance of distributions of quantized values. Our
method first recursively partitions the parameters by percentiles into balanced
bins, and then applies uniform quantization. We also introduce computationally
cheaper approximations of percentiles to reduce the computation overhead
introduced. Overall, our method improves the prediction accuracies of QNNs
without introducing extra computation during inference, has negligible impact
on training speed, and is applicable to both Convolutional Neural Networks and
Recurrent Neural Networks. Experiments on standard datasets including ImageNet
and Penn Treebank confirm the effectiveness of our method. On ImageNet, the
top-5 error rate of our 4-bit quantized GoogLeNet model is 12.7\%, which is
superior to the state-of-the-arts of QNNs.
Generating Long-term Trajectories Using Deep Hierarchical Networks
Authors: Stephan Zheng, Yisong Yue, Patrick Lucey
Date: 2017-06-21 00:00:00
Proceeding: NeurIPS 2016 12

We study the problem of modeling spatiotemporal trajectories over long time
horizons using expert demonstrations. For instance, in sports, agents often
choose action sequences with long-term goals in mind, such as achieving a
certain strategic position. Conventional policy learning approaches, such as
those based on Markov decision processes, generally fail at learning cohesive
long-term behavior in such high-dimensional state spaces, and are only
effective when myopic modeling lead to the desired behavior. The key difficulty
is that conventional approaches are "shallow" models that only learn a single
state-action policy. We instead propose a hierarchical policy class that
automatically reasons about both long-term and short-term goals, which we
instantiate as a hierarchical neural network. We showcase our approach in a
case study on learning to imitate demonstrated basketball trajectories, and
show that it generates significantly more realistic trajectories compared to
non-hierarchical baselines as judged by professional sports analysts.