pwc-archive/papers-with-abstracts huggingface dataset:
Exponential Moving Average Model in Parallel Speech Recognition Training
Authors: Xu Tian, Jun Zhang, Zejun Ma, Yi He, Juan Wei
Date: 2017-03-03 00:00:00
Tasks: speech-recognition, Speech Recognition

As training data rapid growth, large-scale parallel training with multi-GPUs
cluster is widely applied in the neural network model learning currently.We
present a new approach that applies exponential moving average method in
large-scale parallel training of neural network model. It is a non-interference
strategy that the exponential moving average model is not broadcasted to
distributed workers to update their local models after model synchronization in
the training process, and it is implemented as the final model of the training
system. Fully-connected feed-forward neural networks (DNNs) and deep
unidirectional Long short-term memory (LSTM) recurrent neural networks (RNNs)
are successfully trained with proposed method for large vocabulary continuous
speech recognition on Shenma voice search data in Mandarin. The character error
rate (CER) of Mandarin speech recognition further degrades than
state-of-the-art approaches of parallel training.
On the Origin of Deep Learning
Authors: Haohan Wang, Bhiksha Raj
Date: 2017-02-24 00:00:00
Tasks: Deep Learning

This paper is a review of the evolutionary history of deep learning models.
It covers from the genesis of neural networks when associationism modeling of
the brain is studied, to the models that dominate the last decade of research
in deep learning like convolutional neural networks, deep belief networks, and
recurrent neural networks. In addition to a review of these models, this paper
primarily focuses on the precedents of the models above, examining how the
initial ideas are assembled to construct the early models and how these
preliminary models are developed into their current forms. Many of these
evolutionary paths last more than half a century and have a diversity of
directions. For example, CNN is built on prior knowledge of biological vision
system; DBN is evolved from a trade-off of modeling power and computation
complexity of graphical models and many nowadays models are neural counterparts
of ancient linear models. This paper reviews these evolutionary paths and
offers a concise thought flow of how these models are developed, and aims to
provide a thorough background for deep learning. More importantly, along with
the path, this paper summarizes the gist behind these milestones and proposes
many directions to guide the future research of deep learning.
Image denoising via group sparsity residual constraint
Authors: Zhiyuan Zha, Xin Liu, Ziheng Zhou, Xiaohua Huang, Jingang Shi, Zhenhong Shang, Lan Tang, Yechao Bai, Qiong Wang, Xinggan Zhang
Date: 2016-09-12 00:00:00
Tasks: Deblurring, Denoising, Image Denoising

Group sparsity has shown great potential in various low-level vision tasks
(e.g, image denoising, deblurring and inpainting). In this paper, we propose a
new prior model for image denoising via group sparsity residual constraint
(GSRC). To enhance the performance of group sparse-based image denoising, the
concept of group sparsity residual is proposed, and thus, the problem of image
denoising is translated into one that reduces the group sparsity residual. To
reduce the residual, we first obtain some good estimation of the group sparse
coefficients of the original image by the first-pass estimation of noisy image,
and then centralize the group sparse coefficients of noisy image to the
estimation. Experimental results have demonstrated that the proposed method not
only outperforms many state-of-the-art denoising methods such as BM3D and WNNM,
but results in a faster speed.
Scalable Deep Traffic Flow Neural Networks for Urban Traffic Congestion Prediction
Authors: Mohammadhani Fouladgar, Mostafa Parchami, Ramez Elmasri, Amir Ghaderi
Date: 2017-03-03 00:00:00
Tasks: Management, Prediction

Tracking congestion throughout the network road is a critical component of
Intelligent transportation network management systems. Understanding how the
traffic flows and short-term prediction of congestion occurrence due to
rush-hour or incidents can be beneficial to such systems to effectively manage
and direct the traffic to the most appropriate detours. Many of the current
traffic flow prediction systems are designed by utilizing a central processing
component where the prediction is carried out through aggregation of the
information gathered from all measuring stations. However, centralized systems
are not scalable and fail provide real-time feedback to the system whereas in a
decentralized scheme, each node is responsible to predict its own short-term
congestion based on the local current measurements in neighboring nodes.
  We propose a decentralized deep learning-based method where each node
accurately predicts its own congestion state in real-time based on the
congestion state of the neighboring stations. Moreover, historical data from
the deployment site is not required, which makes the proposed method more
suitable for newly installed stations. In order to achieve higher performance,
we introduce a regularized Euclidean loss function that favors high congestion
samples over low congestion samples to avoid the impact of the unbalanced
training dataset. A novel dataset for this purpose is designed based on the
traffic data obtained from traffic control stations in northern California.
Extensive experiments conducted on the designed benchmark reflect a successful
congestion prediction.
Co-Clustering for Multitask Learning
Authors: Keerthiram Murugesan, Jaime Carbonell, Yiming Yang
Date: 2017-03-03 00:00:00
Tasks: Clustering

This paper presents a new multitask learning framework that learns a shared
representation among the tasks, incorporating both task and feature clusters.
The jointly-induced clusters yield a shared latent subspace where task
relationships are learned more effectively and more generally than in
state-of-the-art multitask learning methods. The proposed general framework
enables the derivation of more specific or restricted state-of-the-art
multitask methods. The paper also proposes a highly-scalable multitask learning
algorithm, based on the new framework, using conjugate gradient descent and
generalized \textit{Sylvester equations}. Experimental results on synthetic and
benchmark datasets show that the proposed method systematically outperforms
several state-of-the-art multitask learning methods.
A Comparative Study of Word Embeddings for Reading Comprehension
Authors: Bhuwan Dhingra, Hanxiao Liu, Ruslan Salakhutdinov, William W. Cohen
Date: 2017-03-02 00:00:00
Tasks: BIG-bench Machine Learning, Reading Comprehension, Word Embeddings

The focus of past machine learning research for Reading Comprehension tasks
has been primarily on the design of novel deep learning architectures. Here we
show that seemingly minor choices made on (1) the use of pre-trained word
embeddings, and (2) the representation of out-of-vocabulary tokens at test
time, can turn out to have a larger impact than architectural choices on the
final performance. We systematically explore several options for these choices,
and provide recommendations to researchers working in this area.
Estimating the resolution of real images
Authors: Ryuta Mizutani, Rino Saiga, Susumu Takekoshi, Chie Inomoto, Naoya Nakamura, Makoto Arai, Kenichi Oshima, Masanari Itokawa, Akihisa Takeuchi, Kentaro Uesugi, Yasuko Terada, Yoshio Suzuki
Date: 2017-03-02 00:00:00

Image resolvability is the primary concern in imaging. This paper reports an
estimation of the full width at half maximum of the point spread function from
a Fourier domain plot of real sample images by neither using test objects, nor
defining a threshold criterion. We suggest that this method can be applied to
any type of image, independently of the imaging modality.
Optimization of distributions differences for classification
Authors: Mohammad Reza Bonyadi, Quang M. Tieng, David C. Reutens
Date: 2017-03-02 00:00:00
Tasks: Classification, General Classification, Multiobjective Optimization

In this paper we introduce a new classification algorithm called Optimization
of Distributions Differences (ODD). The algorithm aims to find a transformation
from the feature space to a new space where the instances in the same class are
as close as possible to one another while the gravity centers of these classes
are as far as possible from one another. This aim is formulated as a
multiobjective optimization problem that is solved by a hybrid of an
evolutionary strategy and the Quasi-Newton method. The choice of the
transformation function is flexible and could be any continuous space function.
We experiment with a linear and a non-linear transformation in this paper. We
show that the algorithm can outperform 6 other state-of-the-art classification
methods, namely naive Bayes, support vector machines, linear discriminant
analysis, multi-layer perceptrons, decision trees, and k-nearest neighbors, in
12 standard classification datasets. Our results show that the method is less
sensitive to the imbalanced number of instances comparing to these methods. We
also show that ODD maintains its performance better than other classification
methods in these datasets, hence, offers a better generalization ability.
Belief Propagation in Conditional RBMs for Structured Prediction
Authors: Wei Ping, Alexander Ihler
Date: 2017-03-02 00:00:00
Tasks: Prediction, Structured Prediction

Restricted Boltzmann machines~(RBMs) and conditional RBMs~(CRBMs) are popular
models for a wide range of applications. In previous work, learning on such
models has been dominated by contrastive divergence~(CD) and its variants.
Belief propagation~(BP) algorithms are believed to be slow for structured
prediction on conditional RBMs~(e.g., Mnih et al. [2011]), and not as good as
CD when applied in learning~(e.g., Larochelle et al. [2012]). In this work, we
present a matrix-based implementation of belief propagation algorithms on
CRBMs, which is easily scalable to tens of thousands of visible and hidden
units. We demonstrate that, in both maximum likelihood and max-margin learning,
training conditional RBMs with BP as the inference routine can provide
significantly better results than current state-of-the-art CD methods on
structured prediction problems. We also include practical guidelines on
training CRBMs with BP, and some insights on the interaction of learning and
inference algorithms for CRBMs.
Where is my Phone ? Personal Object Retrieval from Egocentric Images
Authors: Cristian Reyes, Eva Mohedano, Kevin McGuinness, Noel E. O'Connor, Xavier Giro-i-Nieto
Date: 2016-08-29 00:00:00
Tasks: Retrieval

This work presents a retrieval pipeline and evaluation scheme for the problem
of finding the last appearance of personal objects in a large dataset of images
captured from a wearable camera. Each personal object is modelled by a small
set of images that define a query for a visual search engine.The retrieved
results are reranked considering the temporal timestamps of the images to
increase the relevance of the later detections. Finally, a temporal
interleaving of the results is introduced for robustness against false
detections. The Mean Reciprocal Rank is proposed as a metric to evaluate this
problem. This application could help into developing personal assistants
capable of helping users when they do not remember where they left their
personal belongings.
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
Authors: Alberto Montes, Amaia Salvador, Santiago Pascual, Xavier Giro-i-Nieto
Date: 2016-08-29 00:00:00
Tasks: Action Detection, Activity Detection

This thesis explore different approaches using Convolutional and Recurrent
Neural Networks to classify and temporally localize activities on videos,
furthermore an implementation to achieve it has been proposed. As the first
step, features have been extracted from video frames using an state of the art
3D Convolutional Neural Network. This features are fed in a recurrent neural
network that solves the activity classification and temporally location tasks
in a simple and flexible way. Different architectures and configurations have
been tested in order to achieve the best performance and learning of the video
dataset provided. In addition it has been studied different kind of post
processing over the trained network's output to achieve a better results on the
temporally localization of activities on the videos. The results provided by
the neural network developed in this thesis have been submitted to the
ActivityNet Challenge 2016 of the CVPR, achieving competitive results using a
simple and flexible architecture.
A Restaurant Process Mixture Model for Connectivity Based Parcellation of the Cortex
Authors: Daniel Moyer, Boris A. Gutman, Neda Jahanshad, Paul M. Thompson
Date: 2017-03-02 00:00:00

One of the primary objectives of human brain mapping is the division of the
cortical surface into functionally distinct regions, i.e. parcellation. While
it is generally agreed that at macro-scale different regions of the cortex have
different functions, the exact number and configuration of these regions is not
known. Methods for the discovery of these regions are thus important,
particularly as the volume of available information grows. Towards this end, we
present a parcellation method based on a Bayesian non-parametric mixture model
of cortical connectivity.
Multi-Task Multiple Kernel Relationship Learning
Authors: Keerthiram Murugesan, Jaime Carbonell
Date: 2016-11-10 00:00:00

This paper presents a novel multitask multiple kernel learning framework that
efficiently learns the kernel weights leveraging the relationship across
multiple tasks. The idea is to automatically infer this task relationship in
the \textit{RKHS} space corresponding to the given base kernels. The problem is
formulated as a regularization-based approach called \textit{Multi-Task
Multiple Kernel Relationship Learning} (\textit{MK-MTRL}), which models the
task relationship matrix from the weights learned from latent feature spaces of
task-specific base kernels. Unlike in previous work, the proposed formulation
allows one to incorporate prior knowledge for simultaneously learning several
related tasks. We propose an alternating minimization algorithm to learn the
model parameters, kernel weights and task relationship matrix. In order to
tackle large-scale problems, we further propose a two-stage \textit{MK-MTRL}
online learning algorithm and show that it significantly reduces the
computational time, and also achieves performance comparable to that of the
joint learning framework. Experimental results on benchmark datasets show that
the proposed formulations outperform several state-of-the-art multitask
learning methods.
Asymptotically exact inference in differentiable generative models
Authors: Matthew M. Graham, Amos J. Storkey
Date: 2016-05-25 00:00:00

Many generative models can be expressed as a differentiable function of
random inputs drawn from some simple probability density. This framework
includes both deep generative architectures such as Variational Autoencoders
and a large class of procedurally defined simulator models. We present a method
for performing efficient MCMC inference in such models when conditioning on
observations of the model output. For some models this offers an asymptotically
exact inference method where Approximate Bayesian Computation might otherwise
be employed. We use the intuition that inference corresponds to integrating a
density across the manifold corresponding to the set of inputs consistent with
the observed outputs. This motivates the use of a constrained variant of
Hamiltonian Monte Carlo which leverages the smooth geometry of the manifold to
coherently move between inputs exactly consistent with observations. We
validate the method by performing inference tasks in a diverse set of models.
Emergence of Compositional Representations in Restricted Boltzmann Machines
Authors: Jérôme Tubiana, Rémi Monasson
Date: 2016-11-21 00:00:00

Extracting automatically the complex set of features composing real
high-dimensional data is crucial for achieving high performance in
machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically
known to be efficient for this purpose, and to be able to generate distributed
and graded representations of the data. We characterize the structural
conditions (sparsity of the weights, low effective temperature, nonlinearities
in the activation functions of hidden units, and adaptation of fields
maintaining the activity in the visible layer) allowing RBM to operate in such
a compositional phase. Evidence is provided by the replica analysis of an
adequate statistical ensemble of random RBMs and by RBM trained on the
handwritten digits dataset MNIST.
Generative Multi-Adversarial Networks
Authors: Ishan Durugkar, Ian Gemp, Sridhar Mahadevan
Date: 2016-11-05 00:00:00
Tasks: Image Generation

Generative adversarial networks (GANs) are a framework for producing a
generative model by way of a two-player minimax game. In this paper, we propose
the \emph{Generative Multi-Adversarial Network} (GMAN), a framework that
extends GANs to multiple discriminators. In previous work, the successful
training of GANs requires modifying the minimax objective to accelerate
training early on. In contrast, GMAN can be reliably trained with the original,
untampered objective. We explore a number of design perspectives with the
discriminator role ranging from formidable adversary to forgiving teacher.
Image generation tasks comparing the proposed framework to standard GANs
demonstrate GMAN produces higher quality samples in a fraction of the
iterations when measured by a pairwise GAM-type metric.
DAWT: Densely Annotated Wikipedia Texts across multiple languages
Authors: Nemanja Spasojevic, Preeti Bhargava, Guoning Hu
Date: 2017-03-02 00:00:00
Tasks: Articles, Entity Embeddings, Information Retrieval, Retrieval

In this work, we open up the DAWT dataset - Densely Annotated Wikipedia Texts
across multiple languages. The annotations include labeled text mentions
mapping to entities (represented by their Freebase machine ids) as well as the
type of the entity. The data set contains total of 13.6M articles, 5.0B tokens,
13.8M mention entity co-occurrences. DAWT contains 4.8 times more anchor text
to entity links than originally present in the Wikipedia markup. Moreover, it
spans several languages including English, Spanish, Italian, German, French and
Arabic. We also present the methodology used to generate the dataset which
enriches Wikipedia markup in order to increase number of links. In addition to
the main dataset, we open up several derived datasets including mention entity
co-occurrence counts and entity embeddings, as well as mappings between
Freebase ids and Wikidata item ids. We also discuss two applications of these
datasets and hope that opening them up would prove useful for the Natural
Language Processing and Information Retrieval communities, as well as
facilitate multi-lingual research.
Simultaneous Learning of Trees and Representations for Extreme Classification and Density Estimation
Authors: Yacine Jernite, Anna Choromanska, David Sontag
Date: 2016-10-14 00:00:00
Proceeding: ICML 2017 8
Tasks: Classification, Density Estimation, General Classification, Multi-class Classification, Representation Learning, text-classification, Text Classification

We consider multi-class classification where the predictor has a hierarchical
structure that allows for a very large number of labels both at train and test
time. The predictive power of such models can heavily depend on the structure
of the tree, and although past work showed how to learn the tree structure, it
expected that the feature vectors remained static. We provide a novel algorithm
to simultaneously perform representation learning for the input data and
learning of the hierarchi- cal predictor. Our approach optimizes an objec- tive
function which favors balanced and easily- separable multi-way node partitions.
We theoret- ically analyze this objective, showing that it gives rise to a
boosting style property and a bound on classification error. We next show how
to extend the algorithm to conditional density estimation. We empirically
validate both variants of the al- gorithm on text classification and language
mod- eling, respectively, and show that they compare favorably to common
baselines in terms of accu- racy and running time.
Dialog state tracking, a machine reading approach using Memory Network
Authors: Julien Perez, Fei Liu
Date: 2016-06-13 00:00:00
Proceeding: EACL 2017 4
Tasks: dialog state tracking, Management, Natural Language Understanding, Question Answering, Reading Comprehension, speech-recognition, Speech Recognition

In an end-to-end dialog system, the aim of dialog state tracking is to
accurately estimate a compact representation of the current dialog status from
a sequence of noisy observations produced by the speech recognition and the
natural language understanding modules. This paper introduces a novel method of
dialog state tracking based on the general paradigm of machine reading and
proposes to solve it using an End-to-End Memory Network, MemN2N, a
memory-enhanced neural network architecture. We evaluate the proposed approach
on the second Dialog State Tracking Challenge (DSTC-2) dataset. The corpus has
been converted for the occasion in order to frame the hidden state variable
inference as a question-answering task based on a sequence of utterances
extracted from a dialog. We show that the proposed tracker gives encouraging
results. Then, we propose to extend the DSTC-2 dataset with specific reasoning
capabilities requirement like counting, list maintenance, yes-no question
answering and indefinite knowledge management. Finally, we present encouraging
results using our proposed MemN2N based tracking model.
Variable Computation in Recurrent Neural Networks
Authors: Yacine Jernite, Edouard Grave, Armand Joulin, Tomas Mikolov
Date: 2016-11-18 00:00:00
Tasks: Language Modelling

Recurrent neural networks (RNNs) have been used extensively and with
increasing success to model various types of sequential data. Much of this
progress has been achieved through devising recurrent units and architectures
with the flexibility to capture complex statistics in the data, such as long
range dependency or localized attention phenomena. However, while many
sequential data (such as video, speech or language) can have highly variable
information flow, most recurrent models still consume input features at a
constant rate and perform a constant number of computations per time step,
which can be detrimental to both speed and model capacity. In this paper, we
explore a modification to existing recurrent units which allows them to learn
to vary the amount of computation they perform at each step, without prior
knowledge of the sequence's time structure. We show experimentally that not
only do our models require fewer operations, they also lead to better
performance overall on evaluation tasks.
Towards A Rigorous Science of Interpretable Machine Learning
Authors: Finale Doshi-Velez, Been Kim
Date: 2017-02-28 00:00:00
Tasks: BIG-bench Machine Learning, Interpretable Machine Learning, Position

As machine learning systems become ubiquitous, there has been a surge of
interest in interpretable machine learning: systems that provide explanation
for their outputs. These explanations are often used to qualitatively assess
other criteria such as safety or non-discrimination. However, despite the
interest in interpretability, there is very little consensus on what
interpretable machine learning is and how it should be measured. In this
position paper, we first define interpretability and describe when
interpretability is needed (and when it is not). Next, we suggest a taxonomy
for rigorous evaluation and expose open questions towards a more rigorous
science of interpretable machine learning.