pwc-archive/papers-with-abstracts huggingface dataset:
Attention-based Wav2Text with Feature Transfer Learning
Authors: Andros Tjandra, Sakriani Sakti, Satoshi Nakamura
Date: 2017-09-22 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Decoder, speech-recognition, Speech Recognition, Transfer Learning

Conventional automatic speech recognition (ASR) typically performs
multi-level pattern recognition tasks that map the acoustic speech waveform
into a hierarchy of speech units. But, it is widely known that information loss
in the earlier stage can propagate through the later stages. After the
resurgence of deep learning, interest has emerged in the possibility of
developing a purely end-to-end ASR system from the raw waveform to the
transcription without any predefined alignments and hand-engineered models.
However, the successful attempts in end-to-end architecture still used
spectral-based features, while the successful attempts in using raw waveform
were still based on the hybrid deep neural network - Hidden Markov model
(DNN-HMM) framework. In this paper, we construct the first end-to-end
attention-based encoder-decoder model to process directly from raw speech
waveform to the text transcription. We called the model as "Attention-based
Wav2Text". To assist the training process of the end-to-end model, we propose
to utilize a feature transfer learning. Experimental results also reveal that
the proposed Attention-based Wav2Text model directly with raw waveform could
achieve a better result in comparison with the attentional encoder-decoder
model trained on standard front-end filterbank features.
Neural Machine Translation
Authors: Philipp Koehn
Date: 2017-09-22 00:00:00
Tasks: Machine Translation, Translation

Draft of textbook chapter on neural machine translation. a comprehensive
treatment of the topic, ranging from introduction to neural networks,
computation graphs, description of the currently dominant attentional
sequence-to-sequence model, recent refinements, alternative architectures and
challenges. Written as chapter for the textbook Statistical Machine
Translation. Used in the JHU Fall 2017 class on machine translation.
Neural Optimizer Search with Reinforcement Learning
Authors: Irwan Bello, Barret Zoph, Vijay Vasudevan, Quoc V. Le
Date: 2017-09-21 00:00:00
Tasks: Machine Translation, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Translation

We present an approach to automate the process of discovering optimization
methods, with a focus on deep learning architectures. We train a Recurrent
Neural Network controller to generate a string in a domain specific language
that describes a mathematical update equation based on a list of primitive
functions, such as the gradient, running average of the gradient, etc. The
controller is trained with Reinforcement Learning to maximize the performance
of a model after a few epochs. On CIFAR-10, our method discovers several update
rules that are better than many commonly used optimizers, such as Adam,
RMSProp, or SGD with and without Momentum on a ConvNet model. We introduce two
new optimizers, named PowerSign and AddSign, which we show transfer well and
improve training on a variety of different tasks and architectures, including
ImageNet classification and Google's neural machine translation system.
Quantum Memristors in Quantum Photonics
Authors: M. Sanz, L. Lamata, E. Solano
Date: 2017-09-22 00:00:00

We propose a method to build quantum memristors in quantum photonic
platforms. We firstly design an effective beam splitter, which is tunable in
real-time, by means of a Mach-Zehnder-type array with two equal 50:50 beam
splitters and a tunable retarder, which allows us to control its reflectivity.
Then, we show that this tunable beam splitter, when equipped with weak
measurements and classical feedback, behaves as a quantum memristor. Indeed, in
order to prove its quantumness, we show how to codify quantum information in
the coherent beams. Moreover, we estimate the memory capability of the quantum
memristor. Finally, we show the feasibility of the proposed setup in integrated
quantum photonics.
Deconvolutional Paragraph Representation Learning
Authors: Yizhe Zhang, Dinghan Shen, Guoyin Wang, Zhe Gan, Ricardo Henao, Lawrence Carin
Date: 2017-08-16 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: General Classification, Representation Learning, Semi-Supervised Text Classification, Text Classification

Learning latent representations from long text sequences is an important
first step in many natural language processing applications. Recurrent Neural
Networks (RNNs) have become a cornerstone for this challenging task. However,
the quality of sentences during RNN-based decoding (reconstruction) decreases
with the length of the text. We propose a sequence-to-sequence, purely
convolutional and deconvolutional autoencoding framework that is free of the
above issue, while also being computationally efficient. The proposed method is
simple, easy to implement and can be leveraged as a building block for many
applications. We show empirically that compared to RNNs, our framework is
better at reconstructing and correcting long paragraphs. Quantitative
evaluation on semi-supervised text classification and summarization tasks
demonstrate the potential for better utilization of long unlabeled text data.
On overfitting and asymptotic bias in batch reinforcement learning with partial observability
Authors: Vincent Francois-Lavet, Guillaume Rabusseau, Joelle Pineau, Damien Ernst, Raphael Fonteneau
Date: 2017-09-22 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

This paper provides an analysis of the tradeoff between asymptotic bias
(suboptimality with unlimited data) and overfitting (additional suboptimality
due to limited data) in the context of reinforcement learning with partial
observability. Our theoretical analysis formally characterizes that while
potentially increasing the asymptotic bias, a smaller state representation
decreases the risk of overfitting. This analysis relies on expressing the
quality of a state representation by bounding L1 error terms of the associated
belief states. Theoretical results are empirically illustrated when the state
representation is a truncated history of observations, both on synthetic POMDPs
and on a large-scale POMDP in the context of smartgrids, with real-world data.
Finally, similarly to known results in the fully observable setting, we also
briefly discuss and empirically illustrate how using function approximators and
adapting the discount factor may enhance the tradeoff between asymptotic bias
and overfitting in the partially observable context.
Tropical Land Use Land Cover Mapping in Pará (Brazil) using Discriminative Markov Random Fields and Multi-temporal TerraSAR-X Data
Authors: Ron Hagensieker, Ribana Roscher, Johannes Rosentreter, Benjamin Jakimow, Björn Waske
Date: 2017-09-22 00:00:00

Remote sensing satellite data offer the unique possibility to map land use
land cover transformations by providing spatially explicit information.
However, detection of short-term processes and land use patterns of high
spatial-temporal variability is a challenging task. We present a novel
framework using multi-temporal TerraSAR-X data and machine learning techniques,
namely Discriminative Markov Random Fields with spatio-temporal priors, and
Import Vector Machines, in order to advance the mapping of land cover
characterized by short-term changes. Our study region covers a current
deforestation frontier in the Brazilian state Par\'{a} with land cover
dominated by primary forests, different types of pasture land and secondary
vegetation, and land use dominated by short-term processes such as
slash-and-burn activities. The data set comprises multi-temporal TerraSAR-X
imagery acquired over the course of the 2014 dry season, as well as optical
data (RapidEye, Landsat) for reference. Results show that land use land cover
is reliably mapped, resulting in spatially adjusted overall accuracies of up to
$79\%$ in a five class setting, yet limitations for the differentiation of
different pasture types remain. The proposed method is applicable on
multi-temporal data sets, and constitutes a feasible approach to map land use
land cover in regions that are affected by high-frequent temporal changes.
Humanoid Robots as Agents of Human Consciousness Expansion
Authors: Ben Goertzel, Julia Mossbridge, Eddie Monroe, David Hanson, Gino Yu
Date: 2017-09-22 00:00:00

The "Loving AI" project involves developing software enabling humanoid robots
to interact with people in loving and compassionate ways, and to promote
people' self-understanding and self-transcendence. Currently the project
centers on the Hanson Robotics robot "Sophia" -- specifically, on supplying
Sophia with personality content and cognitive, linguistic, perceptual and
behavioral content aimed at enabling loving interactions supportive of human
self-transcendence. In September 2017 a small pilot study was conducted,
involving the Sophia robot leading human subjects through dialogues and
exercises focused on meditation, visualization and relaxation. The pilot was an
apparent success, qualitatively demonstrating the viability of the approach and
the ability of appropriate human-robot interaction to increase human well-being
and advance human consciousness.
Edge Preserving and Multi-Scale Contextual Neural Network for Salient Object Detection
Authors: Xiang Wang, Huimin Ma, Xiaozhi Chen, ShaoDi You
Date: 2016-08-29 00:00:00
Tasks: Object, object-detection, Object Detection, RGB Salient Object Detection, Saliency Detection, Salient Object Detection

In this paper, we propose a novel edge preserving and multi-scale contextual
neural network for salient object detection. The proposed framework is aiming
to address two limits of the existing CNN based methods. First, region-based
CNN methods lack sufficient context to accurately locate salient object since
they deal with each region independently. Second, pixel-based CNN methods
suffer from blurry boundaries due to the presence of convolutional and pooling
layers. Motivated by these, we first propose an end-to-end edge-preserved
neural network based on Fast R-CNN framework (named RegionNet) to efficiently
generate saliency map with sharp object boundaries. Later, to further improve
it, multi-scale spatial context is attached to RegionNet to consider the
relationship between regions and the global scenes. Furthermore, our method can
be generally applied to RGB-D saliency detection by depth refinement. The
proposed framework achieves both clear detection boundary and multi-scale
contextual robustness simultaneously for the first time, and thus achieves an
optimized performance. Experiments on six RGB and two RGB-D benchmark datasets
demonstrate that the proposed method achieves state-of-the-art performance.
A Compressive Sensing Approach to Community Detection with Applications
Authors: Ming-Jun Lai, Daniel Mckenzie
Date: 2017-08-30 00:00:00
Tasks: Clustering, Community Detection, Compressive Sensing, Stochastic Block Model

The community detection problem for graphs asks one to partition the n
vertices V of a graph G into k communities, or clusters, such that there are
many intracluster edges and few intercluster edges. Of course this is
equivalent to finding a permutation matrix P such that, if A denotes the
adjacency matrix of G, then PAP^T is approximately block diagonal. As there are
k^n possible partitions of n vertices into k subsets, directly determining the
optimal clustering is clearly infeasible. Instead one seeks to solve a more
tractable approximation to the clustering problem. In this paper we reformulate
the community detection problem via sparse solution of a linear system
associated with the Laplacian of a graph G and then develop a two-stage
approach based on a thresholding technique and a compressive sensing algorithm
to find a sparse solution which corresponds to the community containing a
vertex of interest in G. Crucially, our approach results in an algorithm which
is able to find a single cluster of size n_0 in O(nlog(n)n_0) operations and
all k clusters in fewer than O(n^2ln(n)) operations. This is a marked
improvement over the classic spectral clustering algorithm, which is unable to
find a single cluster at a time and takes approximately O(n^3) operations to
find all k clusters. Moreover, we are able to provide robust guarantees of
success for the case where G is drawn at random from the Stochastic Block
Model, a popular model for graphs with clusters. Extensive numerical results
are also provided, showing the efficacy of our algorithm on both synthetic and
real-world data sets.
Improving Language Modelling with Noise-contrastive estimation
Authors: Farhana Ferdousi Liza, Marek Grzes
Date: 2017-09-22 00:00:00
Tasks: Language Modelling, Machine Translation, Translation

Neural language models do not scale well when the vocabulary is large.
Noise-contrastive estimation (NCE) is a sampling-based method that allows for
fast learning with large vocabularies. Although NCE has shown promising
performance in neural machine translation, it was considered to be an
unsuccessful approach for language modelling. A sufficient investigation of the
hyperparameters in the NCE-based neural language models was also missing. In
this paper, we showed that NCE can be a successful approach in neural language
modelling when the hyperparameters of a neural network are tuned appropriately.
We introduced the 'search-then-converge' learning rate schedule for NCE and
designed a heuristic that specifies how to use this schedule. The impact of the
other important hyperparameters, such as the dropout rate and the weight
initialisation range, was also demonstrated. We showed that appropriate tuning
of NCE-based neural language models outperforms the state-of-the-art
single-model methods on a popular benchmark.
Estimate Exchange over Network is Good for Distributed Hard Thresholding Pursuit
Authors: Ahmed Zaki, Partha P. Mitra, Lars K. Rasmussen, Saikat Chatterjee
Date: 2017-09-22 00:00:00

We investigate an existing distributed algorithm for learning sparse signals
or data over networks. The algorithm is iterative and exchanges intermediate
estimates of a sparse signal over a network. This learning strategy using
exchange of intermediate estimates over the network requires a limited
communication overhead for information transmission. Our objective in this
article is to show that the strategy is good for learning in spite of limited
communication. In pursuit of this objective, we first provide a restricted
isometry property (RIP)-based theoretical analysis on convergence of the
iterative algorithm. Then, using simulations, we show that the algorithm
provides competitive performance in learning sparse signals vis-a-vis an
existing alternate distributed algorithm. The alternate distributed algorithm
exchanges more information including observations and system parameters.
Global Weisfeiler-Lehman Graph Kernels
Authors: Christopher Morris, Kristian Kersting, Petra Mutzel
Date: 2017-03-07 00:00:00
Tasks: General Classification, Graph Classification

Most state-of-the-art graph kernels only take local graph properties into
account, i.e., the kernel is computed with regard to properties of the
neighborhood of vertices or other small substructures. On the other hand,
kernels that do take global graph propertiesinto account may not scale well to
large graph databases. Here we propose to start exploring the space between
local and global graph kernels, striking the balance between both worlds.
Specifically, we introduce a novel graph kernel based on the $k$-dimensional
Weisfeiler-Lehman algorithm. Unfortunately, the $k$-dimensional
Weisfeiler-Lehman algorithm scales exponentially in $k$. Consequently, we
devise a stochastic version of the kernel with provable approximation
guarantees using conditional Rademacher averages. On bounded-degree graphs, it
can even be computed in constant time. We support our theoretical results with
experiments on several graph classification benchmarks, showing that our
kernels often outperform the state-of-the-art in terms of classification
accuracies.
The Size of a Hyperball in a Conceptual Space
Authors: Lucas Bechberger
Date: 2017-07-04 00:00:00

The cognitive framework of conceptual spaces [3] provides geometric means for
representing knowledge. A conceptual space is a high-dimensional space whose
dimensions are partitioned into so-called domains. Within each domain, the
Euclidean metric is used to compute distances. Distances in the overall space
are computed by applying the Manhattan metric to the intra-domain distances.
Instances are represented as points in this space and concepts are represented
by regions. In this paper, we derive a formula for the size of a hyperball
under the combined metric of a conceptual space. One can think of such a
hyperball as the set of all points having a certain minimal similarity to the
hyperball's center.
Boltzmann machines for time-series
Authors: Takayuki Osogami
Date: 2017-08-20 00:00:00
Tasks: Time Series, Time Series Analysis

We review Boltzmann machines extended for time-series. These models often
have recurrent structure, and back propagration through time (BPTT) is used to
learn their parameters. The per-step computational complexity of BPTT in online
learning, however, grows linearly with respect to the length of preceding
time-series (i.e., learning rule is not local in time), which limits the
applicability of BPTT in online learning. We then review dynamic Boltzmann
machines (DyBMs), whose learning rule is local in time. DyBM's learning rule
relates to spike-timing dependent plasticity (STDP), which has been postulated
and experimentally confirmed for biological neural networks.
STAR: Spatio-Temporal Altimeter Waveform Retracking using Sparse Representation and Conditional Random Fields
Authors: Ribana Roscher, Bernd Uebbing, Jürgen Kusche
Date: 2017-09-22 00:00:00

Satellite radar altimetry is one of the most powerful techniques for
measuring sea surface height variations, with applications ranging from
operational oceanography to climate research. Over open oceans, altimeter
return waveforms generally correspond to the Brown model, and by inversion,
estimated shape parameters provide mean surface height and wind speed. However,
in coastal areas or over inland waters, the waveform shape is often distorted
by land influence, resulting in peaks or fast decaying trailing edges. As a
result, derived sea surface heights are then less accurate and waveforms need
to be reprocessed by sophisticated algorithms. To this end, this work suggests
a novel Spatio-Temporal Altimetry Retracking (STAR) technique. We show that
STAR enables the derivation of sea surface heights over the open ocean as well
as over coastal regions of at least the same quality as compared to existing
retracking methods, but for a larger number of cycles and thus retaining more
useful data. Novel elements of our method are (a) integrating information from
spatially and temporally neighboring waveforms through a conditional random
field approach, (b) sub-waveform detection, where relevant sub-waveforms are
separated from corrupted or non-relevant parts through a sparse representation
approach, and (c) identifying the final best set of sea surfaces heights from
multiple likely heights using Dijkstra's algorithm. We apply STAR to data from
the Jason-1, Jason-2 and Envisat missions for study sites in the Gulf of
Trieste, Italy and in the coastal region of the Ganges-Brahmaputra-Meghna
estuary, Bangladesh. We compare to several established and recent retracking
methods, as well as to tide gauge data. Our experiments suggest that the
obtained sea surface heights are significantly less affected by outliers when
compared to results obtained by other approaches.
On the Equivalence of Holographic and Complex Embeddings for Link Prediction
Authors: Katsuhiko Hayashi, Masashi Shimbo
Date: 2017-02-18 00:00:00
Proceeding: ACL 2017 7
Tasks: Knowledge Graph Completion, Link Prediction

We show the equivalence of two state-of-the-art link prediction/knowledge
graph completion methods: Nickel et al's holographic embedding and Trouillon et
al.'s complex embedding. We first consider a spectral version of the
holographic embedding, exploiting the frequency domain in the Fourier transform
for efficient computation. The analysis of the resulting method reveals that it
can be viewed as an instance of the complex embedding with certain constraints
cast on the initial vectors upon training. Conversely, any complex embedding
can be converted to an equivalent holographic embedding.
Semantic Segmentation from Limited Training Data
Authors: A. Milan, T. Pham, K. Vijay, D. Morrison, A. W. Tow, L. Liu, J. Erskine, R. Grinover, A. Gurman, T. Hunn, N. Kelly-Boxall, D. Lee, M. McTaggart, G. Rallos, A. Razjigaev, T. Rowntree, T. Shen, R. Smith, S. Wade-McCue, Z. Zhuang, C. Lehnert, G. Lin, I. Reid, P. Corke, J. Leitner
Date: 2017-09-22 00:00:00
Tasks: ARC, Boundary Detection, Metric Learning, Segmentation, Semantic Segmentation

We present our approach for robotic perception in cluttered scenes that led
to winning the recent Amazon Robotics Challenge (ARC) 2017. Next to small
objects with shiny and transparent surfaces, the biggest challenge of the 2017
competition was the introduction of unseen categories. In contrast to
traditional approaches which require large collections of annotated data and
many hours of training, the task here was to obtain a robust perception
pipeline with only few minutes of data acquisition and training time. To that
end, we present two strategies that we explored. One is a deep metric learning
approach that works in three separate steps: semantic-agnostic boundary
detection, patch classification and pixel-wise voting. The other is a
fully-supervised semantic segmentation approach with efficient dataset
collection. We conduct an extensive analysis of the two methods on our ARC 2017
dataset. Interestingly, only few examples of each class are sufficient to
fine-tune even very deep convolutional neural networks for this specific task.
Approximate Bayesian Inference in Linear State Space Models for Intermittent Demand Forecasting at Scale
Authors: Matthias Seeger, Syama Rangapuram, Yuyang Wang, David Salinas, Jan Gasthaus, Tim Januschowski, Valentin Flunkert
Date: 2017-09-22 00:00:00
Tasks: Bayesian Inference, Demand Forecasting, State Space Models

We present a scalable and robust Bayesian inference method for linear state
space models. The method is applied to demand forecasting in the context of a
large e-commerce platform, paying special attention to intermittent and bursty
target statistics. Inference is approximated by the Newton-Raphson algorithm,
reduced to linear-time Kalman smoothing, which allows us to operate on several
orders of magnitude larger problems than previous related work. In a study on
large real-world sales datasets, our method outperforms competing approaches on
fast and medium moving items.
BreathRNNet: Breathing Based Authentication on Resource-Constrained IoT Devices using RNNs
Authors: Jagmohan Chauhan, Suranga Seneviratne, Yining Hu, Archan Misra, Aruna Seneviratne, Youngki Lee
Date: 2017-09-22 00:00:00

Recurrent neural networks (RNNs) have shown promising results in audio and
speech processing applications due to their strong capabilities in modelling
sequential data. In many applications, RNNs tend to outperform conventional
models based on GMM/UBMs and i-vectors. Increasing popularity of IoT devices
makes a strong case for implementing RNN based inferences for applications such
as acoustics based authentication, voice commands, and edge analytics for smart
homes. Nonetheless, the feasibility and performance of RNN based inferences on
resources-constrained IoT devices remain largely unexplored. In this paper, we
investigate the feasibility of using RNNs for an end-to-end authentication
system based on breathing acoustics. We evaluate the performance of RNN models
on three types of devices; smartphone, smartwatch, and Raspberry Pi and show
that unlike CNN models, RNN models can be easily ported onto
resource-constrained devices without a significant loss in accuracy.