pwc-archive/papers-with-abstracts huggingface dataset:
A Tour of TensorFlow
Authors: Peter Goldsborough
Date: 2016-10-01 00:00:00
Tasks: Deep Learning, speech-recognition, Speech Recognition

Deep learning is a branch of artificial intelligence employing deep neural
network architectures that has significantly advanced the state-of-the-art in
computer vision, speech recognition, natural language processing and other
domains. In November 2015, Google released $\textit{TensorFlow}$, an open
source deep learning software library for defining, training and deploying
machine learning models. In this paper, we review TensorFlow and put it in
context of modern deep learning concepts and software. We discuss its basic
computational paradigms and distributed execution model, its programming
interface as well as accompanying visualization toolkits. We then compare
TensorFlow to alternative libraries such as Theano, Torch or Caffe on a
qualitative as well as quantitative basis and finally comment on observed
use-cases of TensorFlow in academia and industry.
Very Deep Convolutional Neural Networks for Raw Waveforms
Authors: Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, Samarjit Das
Date: 2016-10-01 00:00:00
Tasks: Representation Learning

Learning acoustic models directly from the raw waveform data with minimal
processing is challenging. Current waveform-based models have generally used
very few (~2) convolutional layers, which might be insufficient for building
high-level discriminative features. In this work, we propose very deep
convolutional neural networks (CNNs) that directly use time-domain waveforms as
inputs. Our CNNs, with up to 34 weight layers, are efficient to optimize over
very long sequences (e.g., vector of size 32000), necessary for processing
acoustic waveforms. This is achieved through batch normalization, residual
learning, and a careful design of down-sampling in the initial layers. Our
networks are fully convolutional, without the use of fully connected layers and
dropout, to maximize representation learning. We use a large receptive field in
the first convolutional layer to mimic bandpass filters, but very small
receptive fields subsequently to control the model capacity. We demonstrate the
performance gains with the deeper models. Our evaluation shows that the CNN
with 18 weight layers outperform the CNN with 3 weight layers by over 15% in
absolute accuracy for an environmental sound recognition task and matches the
performance of models using log-mel features.
Consistency Ensuring in Social Web Services Based on Commitments Structure
Authors: Marzieh Adelnia, Mohammad Reza Khayyambashi
Date: 2016-10-01 00:00:00

Web Service is one of the most significant current discussions in information
sharing technologies and one of the examples of service oriented processing. To
ensure accurate execution of web services operations, it must be adaptable with
policies of the social networks in which it signs up. This adaptation
implements using controls called 'Commitment'. This paper describes commitments
structure and existing research about commitments and social web services, then
suggests an algorithm for consistency of commitments in social web services. As
regards the commitments may be executed concurrently, a key challenge in web
services execution based on commitment structure is consistency ensuring in
execution time. The purpose of this research is providing an algorithm for
consistency ensuring between web services operations based on commitments
structure.
Latent Tree Analysis
Authors: Nevin L. Zhang, Leonard K. M. Poon
Date: 2016-10-01 00:00:00
Tasks: BIG-bench Machine Learning

Latent tree analysis seeks to model the correlations among a set of random
variables using a tree of latent variables. It was proposed as an improvement
to latent class analysis --- a method widely used in social sciences and
medicine to identify homogeneous subgroups in a population. It provides new and
fruitful perspectives on a number of machine learning areas, including cluster
analysis, topic detection, and deep probabilistic modeling. This paper gives an
overview of the research on latent tree analysis and various ways it is used in
practice.
Data-Driven Learning of a Union of Sparsifying Transforms Model for Blind Compressed Sensing
Authors: Saiprasad Ravishankar, Yoram Bresler
Date: 2015-11-04 00:00:00
Tasks: compressed sensing, Diversity, Image Reconstruction

Compressed sensing is a powerful tool in applications such as magnetic
resonance imaging (MRI). It enables accurate recovery of images from highly
undersampled measurements by exploiting the sparsity of the images or image
patches in a transform domain or dictionary. In this work, we focus on blind
compressed sensing (BCS), where the underlying sparse signal model is a priori
unknown, and propose a framework to simultaneously reconstruct the underlying
image as well as the unknown model from highly undersampled measurements.
Specifically, our model is that the patches of the underlying image(s) are
approximately sparse in a transform domain. We also extend this model to a
union of transforms model that better captures the diversity of features in
natural images. The proposed block coordinate descent type algorithms for blind
compressed sensing are highly efficient, and are guaranteed to converge to at
least the partial global and partial local minimizers of the highly non-convex
BCS problems. Our numerical experiments show that the proposed framework
usually leads to better quality of image reconstructions in MRI compared to
several recent image reconstruction methods. Importantly, the learning of a
union of sparsifying transforms leads to better image reconstructions than a
single adaptive transform.
DeepSpark: A Spark-Based Distributed Deep Learning Framework for Commodity Clusters
Authors: Hanjoo Kim, Jae-hong Park, Jaehee Jang, Sungroh Yoon
Date: 2016-02-26 00:00:00
Tasks: Distributed Computing

The increasing complexity of deep neural networks (DNNs) has made it
challenging to exploit existing large-scale data processing pipelines for
handling massive data and parameters involved in DNN training. Distributed
computing platforms and GPGPU-based acceleration provide a mainstream solution
to this computational challenge. In this paper, we propose DeepSpark, a
distributed and parallel deep learning framework that exploits Apache Spark on
commodity clusters. To support parallel operations, DeepSpark automatically
distributes workloads and parameters to Caffe/Tensorflow-running nodes using
Spark, and iteratively aggregates training results by a novel lock-free
asynchronous variant of the popular elastic averaging stochastic gradient
descent based update scheme, effectively complementing the synchronized
processing capabilities of Spark. DeepSpark is an on-going project, and the
current release is available at http://deepspark.snu.ac.kr.
Vocabulary Selection Strategies for Neural Machine Translation
Authors: Gurvan L'Hostis, David Grangier, Michael Auli
Date: 2016-10-01 00:00:00
Tasks: CPU, Machine Translation, Sentence, Translation

Classical translation models constrain the space of possible outputs by
selecting a subset of translation rules based on the input sentence. Recent
work on improving the efficiency of neural translation models adopted a similar
strategy by restricting the output vocabulary to a subset of likely candidates
given the source. In this paper we experiment with context and embedding-based
selection methods and extend previous work by examining speed and accuracy
trade-offs in more detail. We show that decoding time on CPUs can be reduced by
up to 90% and training time by 25% on the WMT15 English-German and WMT16
English-Romanian tasks at the same or only negligible change in accuracy. This
brings the time to decode with a state of the art neural translation system to
just over 140 msec per sentence on a single CPU core for English-German.
Deep Image Set Hashing
Authors: Jie Feng, Svebor Karaman, I-Hong Jhuo, Shih-Fu Chang
Date: 2016-06-16 00:00:00
Tasks: Image Retrieval, Retrieval, Triplet

In applications involving matching of image sets, the information from
multiple images must be effectively exploited to represent each set.
State-of-the-art methods use probabilistic distribution or subspace to model a
set and use specific distance measure to compare two sets. These methods are
slow to compute and not compact to use in a large scale scenario.
Learning-based hashing is often used in large scale image retrieval as they
provide a compact representation of each sample and the Hamming distance can be
used to efficiently compare two samples. However, most hashing methods encode
each image separately and discard knowledge that multiple images in the same
set represent the same object or person. We investigate the set hashing problem
by combining both set representation and hashing in a single deep neural
network. An image set is first passed to a CNN module to extract image
features, then these features are aggregated using two types of set feature to
capture both set specific and database-wide distribution information. The
computed set feature is then fed into a multilayer perceptron to learn a
compact binary embedding. Triplet loss is used to train the network by forming
set similarity relations using class labels. We extensively evaluate our
approach on datasets used for image matching and show highly competitive
performance compared to state-of-the-art methods.
Faster Kernels for Graphs with Continuous Attributes via Hashing
Authors: Christopher Morris, Nils M. Kriege, Kristian Kersting, Petra Mutzel
Date: 2016-10-01 00:00:00

While state-of-the-art kernels for graphs with discrete labels scale well to
graphs with thousands of nodes, the few existing kernels for graphs with
continuous attributes, unfortunately, do not scale well. To overcome this
limitation, we present hash graph kernels, a general framework to derive
kernels for graphs with continuous attributes from discrete ones. The idea is
to iteratively turn continuous attributes into discrete labels using randomized
hash functions. We illustrate hash graph kernels for the Weisfeiler-Lehman
subtree kernel and for the shortest-path kernel. The resulting novel graph
kernels are shown to be, both, able to handle graphs with continuous attributes
and scalable to large graphs and data sets. This is supported by our
theoretical analysis and demonstrated by an extensive experimental evaluation.
Outlier Detection from Network Data with Subnetwork Interpretation
Authors: Xuan-Hong Dang, Arlei Silva, Ambuj Singh, Ananthram Swami, Prithwish Basu
Date: 2016-09-30 00:00:00
Tasks: Outlier Detection

Detecting a small number of outliers from a set of data observations is
always challenging. This problem is more difficult in the setting of multiple
network samples, where computing the anomalous degree of a network sample is
generally not sufficient. In fact, explaining why the network is exceptional,
expressed in the form of subnetwork, is also equally important. In this paper,
we develop a novel algorithm to address these two key problems. We treat each
network sample as a potential outlier and identify subnetworks that mostly
discriminate it from nearby regular samples. The algorithm is developed in the
framework of network regression combined with the constraints on both network
topology and L1-norm shrinkage to perform subnetwork discovery. Our method thus
goes beyond subspace/subgraph discovery and we show that it converges to a
global optimum. Evaluation on various real-world network datasets demonstrates
that our algorithm not only outperforms baselines in both network and high
dimensional setting, but also discovers highly relevant and interpretable local
subnetworks, further enhancing our understanding of anomalous networks.
Winograd Schemas and Machine Translation
Authors: Ernest Davis
Date: 2016-08-05 00:00:00
Tasks: Machine Translation, Sentence, Translation, World Knowledge

A Winograd schema is a pair of sentences that differ in a single word and
that contain an ambiguous pronoun whose referent is different in the two
sentences and requires the use of commonsense knowledge or world knowledge to
disambiguate. This paper discusses how Winograd schemas and other sentence
pairs could be used as challenges for machine translation using distinctions
between pronouns, such as gender, that appear in the target language but not in
the source.
Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models
Authors: Viktoriya Krakovna, Finale Doshi-Velez
Date: 2016-06-16 00:00:00
Tasks: speech-recognition, Speech Recognition, Translation

As deep neural networks continue to revolutionize various application
domains, there is increasing interest in making these powerful models more
understandable and interpretable, and narrowing down the causes of good and bad
predictions. We focus on recurrent neural networks (RNNs), state of the art
models in speech recognition and translation. Our approach to increasing
interpretability is by combining an RNN with a hidden Markov model (HMM), a
simpler and more transparent model. We explore various combinations of RNNs and
HMMs: an HMM trained on LSTM states; a hybrid model where an HMM is trained
first, then a small LSTM is given HMM state distributions and trained to fill
in gaps in the HMM's performance; and a jointly trained hybrid model. We find
that the LSTM and HMM learn complementary information about the features in the
text.
Discriminating Similar Languages: Evaluations and Explorations
Authors: Cyril Goutte, Serge Léger, Shervin Malmasi, Marcos Zampieri
Date: 2016-09-30 00:00:00
Proceeding: LREC 2016 5
Tasks: BIG-bench Machine Learning

We present an analysis of the performance of machine learning classifiers on
discriminating between similar languages and language varieties. We carried out
a number of experiments using the results of the two editions of the
Discriminating between Similar Languages (DSL) shared task. We investigate the
progress made between the two tasks, estimate an upper bound on possible
performance using ensemble and oracle combination, and provide learning curves
to help us understand which languages are more challenging. A number of
difficult sentences are identified and investigated further with human
annotation.
Modeling Language Change in Historical Corpora: The Case of Portuguese
Authors: Marcos Zampieri, Shervin Malmasi, Mark Dras
Date: 2016-09-30 00:00:00
Proceeding: LREC 2016 5
Tasks: General Classification, POS, text-classification, Text Classification

This paper presents a number of experiments to model changes in a historical
Portuguese corpus composed of literary texts for the purpose of temporal text
classification. Algorithms were trained to classify texts with respect to their
publication date taking into account lexical variation represented as word
n-grams, and morphosyntactic variation represented by part-of-speech (POS)
distribution. We report results of 99.8% accuracy using word unigram features
with a Support Vector Machines classifier to predict the publication date of
documents in time intervals of both one century and half a century. A feature
analysis is performed to investigate the most informative features for this
task and how they are linked to language change.
Function-Specific Mixing Times and Concentration Away from Equilibrium
Authors: Maxim Rabinovich, Aaditya Ramdas, Michael. I. Jordan, Martin J. Wainwright
Date: 2016-05-06 00:00:00

Slow mixing is the central hurdle when working with Markov chains, especially
those used for Monte Carlo approximations (MCMC). In many applications, it is
only of interest to estimate the stationary expectations of a small set of
functions, and so the usual definition of mixing based on total variation
convergence may be too conservative. Accordingly, we introduce
function-specific analogs of mixing times and spectral gaps, and use them to
prove Hoeffding-like function-specific concentration inequalities. These
results show that it is possible for empirical expectations of functions to
concentrate long before the underlying chain has mixed in the classical sense,
and we show that the concentration rates we achieve are optimal up to
constants. We use our techniques to derive confidence intervals that are
sharper than those implied by both classical Markov chain Hoeffding bounds and
Berry-Esseen-corrected CLT bounds. For applications that require testing,
rather than point estimation, we show similar improvements over recent
sequential testing results for MCMC. We conclude by applying our framework to
real data examples of MCMC, providing evidence that our theory is both accurate
and relevant to practice.
Recovery of non-linear cause-effect relationships from linearly mixed neuroimaging data
Authors: Sebastian Weichwald, Arthur Gretton, Bernhard Schölkopf, Moritz Grosse-Wentrup
Date: 2016-05-02 00:00:00
Tasks: Causal Inference

Causal inference concerns the identification of cause-effect relationships
between variables. However, often only linear combinations of variables
constitute meaningful causal variables. For example, recovering the signal of a
cortical source from electroencephalography requires a well-tuned combination
of signals recorded at multiple electrodes. We recently introduced the MERLiN
(Mixture Effect Recovery in Linear Networks) algorithm that is able to recover,
from an observed linear mixture, a causal variable that is a linear effect of
another given variable. Here we relax the assumption of this cause-effect
relationship being linear and present an extended algorithm that can pick up
non-linear cause-effect relationships. Thus, the main contribution is an
algorithm (and ready to use code) that has broader applicability and allows for
a richer model class. Furthermore, a comparative analysis indicates that the
assumption of linear cause-effect relationships is not restrictive in analysing
electroencephalographic data.
Technical Report: Graph-Structured Sparse Optimization for Connected Subgraph Detection
Authors: Baojian Zhou, Feng Chen
Date: 2016-09-30 00:00:00
Tasks: Astronomy

Structured sparse optimization is an important and challenging problem for
analyzing high-dimensional data in a variety of applications such as
bioinformatics, medical imaging, social networks, and astronomy. Although a
number of structured sparsity models have been explored, such as trees, groups,
clusters, and paths, connected subgraphs have been rarely explored in the
current literature. One of the main technical challenges is that there is no
structured sparsity-inducing norm that can directly model the space of
connected subgraphs, and there is no exact implementation of a projection
oracle for connected subgraphs due to its NP-hardness. In this paper, we
explore efficient approximate projection oracles for connected subgraphs, and
propose two new efficient algorithms, namely, Graph-IHT and Graph-GHTP, to
optimize a generic nonlinear objective function subject to connectivity
constraint on the support of the variables. Our proposed algorithms enjoy
strong guarantees analogous to several current methods for sparsity-constrained
optimization, such as Projected Gradient Descent (PGD), Approximate Model
Iterative Hard Thresholding (AM-IHT), and Gradient Hard Thresholding Pursuit
(GHTP) with respect to convergence rate and approximation accuracy. We apply
our proposed algorithms to optimize several well-known graph scan statistics in
several applications of connected subgraph detection as a case study, and the
experimental results demonstrate that our proposed algorithms outperform
state-of-the-art methods.
Unsupervised Learning from Continuous Video in a Scalable Predictive Recurrent Network
Authors: Filip Piekniewski, Patryk Laurent, Csaba Petre, Micah Richert, Dimitry Fisher, Todd Hylton
Date: 2016-07-22 00:00:00
Tasks: Common Sense Reasoning, Visual Tracking

Understanding visual reality involves acquiring common-sense knowledge about
countless regularities in the visual world, e.g., how illumination alters the
appearance of objects in a scene, and how motion changes their apparent spatial
relationship. These regularities are hard to label for training supervised
machine learning algorithms; consequently, algorithms need to learn these
regularities from the real world in an unsupervised way. We present a novel
network meta-architecture that can learn world dynamics from raw, continuous
video. The components of this network can be implemented using any algorithm
that possesses three key capabilities: prediction of a signal over time,
reduction of signal dimensionality (compression), and the ability to use
supplementary contextual information to inform the prediction. The presented
architecture is highly-parallelized and scalable, and is implemented using
localized connectivity, processing, and learning. We demonstrate an
implementation of this architecture where the components are built from
multi-layer perceptrons. We apply the implementation to create a system capable
of stable and robust visual tracking of objects as seen by a moving camera.
Results show performance on par with or exceeding state-of-the-art tracking
algorithms. The tracker can be trained in either fully supervised or
unsupervised-then-briefly-supervised regimes. Success of the briefly-supervised
regime suggests that the unsupervised portion of the model extracts useful
information about visual reality. The results suggest a new class of AI
algorithms that uniquely combine prediction and scalability in a way that makes
them suitable for learning from and --- and eventually acting within --- the
real world.
On the Worst-case Communication Overhead for Distributed Data Shuffling
Authors: Mohamed Attia, Ravi Tandon
Date: 2016-09-30 00:00:00

Distributed learning platforms for processing large scale data-sets are
becoming increasingly prevalent. In typical distributed implementations, a
centralized master node breaks the data-set into smaller batches for parallel
processing across distributed workers to achieve speed-up and efficiency.
Several computational tasks are of sequential nature, and involve multiple
passes over the data. At each iteration over the data, it is common practice to
randomly re-shuffle the data at the master node, assigning different batches
for each worker to process. This random re-shuffling operation comes at the
cost of extra communication overhead, since at each shuffle, new data points
need to be delivered to the distributed workers.
  In this paper, we focus on characterizing the information theoretically
optimal communication overhead for the distributed data shuffling problem. We
propose a novel coded data delivery scheme for the case of no excess storage,
where every worker can only store the assigned data batches under processing.
Our scheme exploits a new type of coding opportunity and is applicable to any
arbitrary shuffle, and for any number of workers. We also present an
information theoretic lower bound on the minimum communication overhead for
data shuffling, and show that the proposed scheme matches this lower bound for
the worst-case communication overhead.