pwc-archive/papers-with-abstracts huggingface dataset:
Snap ML: A Hierarchical Framework for Machine Learning
Authors: Celestine Dünner, Thomas Parnell, Dimitrios Sarigiannis, Nikolas Ioannou, Andreea Anghel, Gummadi Ravi, Madhusudanan Kandasamy, Haralampos Pozidis
Date: 2018-03-16 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: BIG-bench Machine Learning, GPU

We describe a new software framework for fast training of generalized linear
models. The framework, named Snap Machine Learning (Snap ML), combines recent
advances in machine learning systems and algorithms in a nested manner to
reflect the hierarchical architecture of modern computing systems. We prove
theoretically that such a hierarchical system can accelerate training in
distributed environments where intra-node communication is cheaper than
inter-node communication. Additionally, we provide a review of the
implementation of Snap ML in terms of GPU acceleration, pipelining,
communication patterns and software architecture, highlighting aspects that
were critical for achieving high performance. We evaluate the performance of
Snap ML in both single-node and multi-node environments, quantifying the
benefit of the hierarchical scheme and the data streaming functionality, and
comparing with other widely-used machine learning software frameworks. Finally,
we present a logistic regression benchmark on the Criteo Terabyte Click Logs
dataset and show that Snap ML achieves the same test loss an order of magnitude
faster than any of the previously reported results, including those obtained
using TensorFlow and scikit-learn.
Multilingual bottleneck features for subword modeling in zero-resource languages
Authors: Enno Hermann, Sharon Goldwater
Date: 2018-03-23 00:00:00
Tasks: speech-recognition, Speech Recognition

How can we effectively develop speech technology for languages where no
transcribed data is available? Many existing approaches use no annotated
resources at all, yet it makes sense to leverage information from large
annotated corpora in other languages, for example in the form of multilingual
bottleneck features (BNFs) obtained from a supervised speech recognition
system. In this work, we evaluate the benefits of BNFs for subword modeling
(feature extraction) in six unseen languages on a word discrimination task.
First we establish a strong unsupervised baseline by combining two existing
methods: vocal tract length normalisation (VTLN) and the correspondence
autoencoder (cAE). We then show that BNFs trained on a single language already
beat this baseline; including up to 10 languages results in additional
improvements which cannot be matched by just adding more data from a single
language. Finally, we show that the cAE can improve further on the BNFs if
high-quality same-word pairs are available.
Learning to Write Stylized Chinese Characters by Reading a Handful of Examples
Authors: Danyang Sun, Tongzheng Ren, Chongxun Li, Hang Su, Jun Zhu
Date: 2017-12-06 00:00:00

Automatically writing stylized Chinese characters is an attractive yet
challenging task due to its wide applicabilities. In this paper, we propose a
novel framework named Style-Aware Variational Auto-Encoder (SA-VAE) to flexibly
generate Chinese characters. Specifically, we propose to capture the different
characteristics of a Chinese character by disentangling the latent features
into content-related and style-related components. Considering of the complex
shapes and structures, we incorporate the structure information as prior
knowledge into our framework to guide the generation. Our framework shows a
powerful one-shot/low-shot generalization ability by inferring the style
component given a character with unseen style. To the best of our knowledge,
this is the first attempt to learn to write new-style Chinese characters by
observing only one or a few examples. Extensive experiments demonstrate its
effectiveness in generating different stylized Chinese characters by fusing the
feature vectors corresponding to different contents and styles, which is of
significant importance in real-world applications.
iPose: Instance-Aware 6D Pose Estimation of Partly Occluded Objects
Authors: Omid Hosseini Jafari, Siva Karthik Mustikovela, Karl Pertsch, Eric Brachmann, Carsten Rother
Date: 2017-12-05 00:00:00
Tasks: 6D Pose Estimation, 6D Pose Estimation using RGB, Decoder, Instance Segmentation, Object, Pose Estimation, Semantic Segmentation

We address the task of 6D pose estimation of known rigid objects from single
input images in scenarios where the objects are partly occluded. Recent
RGB-D-based methods are robust to moderate degrees of occlusion. For RGB
inputs, no previous method works well for partly occluded objects. Our main
contribution is to present the first deep learning-based system that estimates
accurate poses for partly occluded objects from RGB-D and RGB input. We achieve
this with a new instance-aware pipeline that decomposes 6D object pose
estimation into a sequence of simpler steps, where each step removes specific
aspects of the problem. The first step localizes all known objects in the image
using an instance segmentation network, and hence eliminates surrounding
clutter and occluders. The second step densely maps pixels to 3D object surface
positions, so called object coordinates, using an encoder-decoder network, and
hence eliminates object appearance. The third, and final, step predicts the 6D
pose using geometric optimization. We demonstrate that we significantly
outperform the state-of-the-art for pose estimation of partly occluded objects
for both RGB and RGB-D input.
Uncertainty in multitask learning: joint representations for probabilistic MR-only radiotherapy planning
Authors: Felix J. S. Bragman, Ryutaro Tanno, Zach Eaton-Rosen, Wenqi Li, David J. Hawkes, Sebastien Ourselin, Daniel C. Alexander, Jamie R. McClelland, M. Jorge Cardoso
Date: 2018-06-18 00:00:00
Tasks: Bayesian Inference

Multi-task neural network architectures provide a mechanism that jointly
integrates information from distinct sources. It is ideal in the context of
MR-only radiotherapy planning as it can jointly regress a synthetic CT (synCT)
scan and segment organs-at-risk (OAR) from MRI. We propose a probabilistic
multi-task network that estimates: 1) intrinsic uncertainty through a
heteroscedastic noise model for spatially-adaptive task loss weighting and 2)
parameter uncertainty through approximate Bayesian inference. This allows
sampling of multiple segmentations and synCTs that share their network
representation. We test our model on prostate cancer scans and show that it
produces more accurate and consistent synCTs with a better estimation in the
variance of the errors, state of the art results in OAR segmentation and a
methodology for quality assurance in radiotherapy treatment planning.
Deep Recurrent Neural Network for Multi-target Filtering
Authors: Mehryar Emambakhsh, Alessandro Bay, Eduard Vazquez
Date: 2018-06-18 00:00:00

This paper addresses the problem of fixed motion and measurement models for
multi-target filtering using an adaptive learning framework. This is performed
by defining target tuples with random finite set terminology and utilisation of
recurrent neural networks with a long short-term memory architecture. A novel
data association algorithm compatible with the predicted tracklet tuples is
proposed, enabling the update of occluded targets, in addition to assigning
birth, survival and death of targets. The algorithm is evaluated over a
commonly used filtering simulation scenario, with highly promising results.
Low-Resource Speech-to-Text Translation
Authors: Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, Sharon Goldwater
Date: 2018-03-24 00:00:00
Tasks: Decoder, Machine Translation, speech-recognition, Speech Recognition, Speech-to-Text, Speech-to-Text Translation, Translation

Speech-to-text translation has many potential applications for low-resource
languages, but the typical approach of cascading speech recognition with
machine translation is often impossible, since the transcripts needed to train
a speech recognizer are usually not available for low-resource languages.
Recent work has found that neural encoder-decoder models can learn to directly
translate foreign speech in high-resource scenarios, without the need for
intermediate transcription. We investigate whether this approach also works in
settings where both data and computation are limited. To make the approach
efficient, we make several architectural changes, including a change from
character-level to word-level decoding. We find that this choice yields crucial
speed improvements that allow us to train with fewer computational resources,
yet still performs well on frequent words. We explore models trained on between
20 and 160 hours of data, and find that although models trained on less data
have considerably lower BLEU scores, they can still predict words with
relatively high precision and recall---around 50% for a model trained on 50
hours of data, versus around 60% for the full 160 hour model. Thus, they may
still be useful for some low-resource scenarios.
Computational Theories of Curiosity-Driven Learning
Authors: Pierre-Yves Oudeyer
Date: 2018-02-28 00:00:00
Tasks: BIG-bench Machine Learning, Lifelong learning

What are the functions of curiosity? What are the mechanisms of
curiosity-driven learning? We approach these questions about the living using
concepts and tools from machine learning and developmental robotics. We argue
that curiosity-driven learning enables organisms to make discoveries to solve
complex problems with rare or deceptive rewards. By fostering exploration and
discovery of a diversity of behavioural skills, and ignoring these rewards,
curiosity can be efficient to bootstrap learning when there is no information,
or deceptive information, about local improvement towards these problems. We
also explain the key role of curiosity for efficient learning of world models.
We review both normative and heuristic computational frameworks used to
understand the mechanisms of curiosity in humans, conceptualizing the child as
a sense-making organism. These frameworks enable us to discuss the
bi-directional causal links between curiosity and learning, and to provide new
hypotheses about the fundamental role of curiosity in self-organizing
developmental structures through curriculum learning. We present various
developmental robotics experiments that study these mechanisms in action, both
supporting these hypotheses to understand better curiosity in humans and
opening new research avenues in machine learning and artificial intelligence.
Finally, we discuss challenges for the design of experimental paradigms for
studying curiosity in psychology and cognitive neuroscience.
  Keywords: Curiosity, intrinsic motivation, lifelong learning, predictions,
world model, rewards, free-energy principle, learning progress, machine
learning, AI, developmental robotics, development, curriculum learning,
self-organization.
Nonparametric Topic Modeling with Neural Inference
Authors: Xuefei Ning, Yin Zheng, Zhuxi Jiang, Yu Wang, Huazhong Yang, Junzhou Huang
Date: 2018-06-18 00:00:00
Tasks: Topic Models

This work focuses on combining nonparametric topic models with Auto-Encoding
Variational Bayes (AEVB). Specifically, we first propose iTM-VAE, where the
topics are treated as trainable parameters and the document-specific topic
proportions are obtained by a stick-breaking construction. The inference of
iTM-VAE is modeled by neural networks such that it can be computed in a simple
feed-forward manner. We also describe how to introduce a hyper-prior into
iTM-VAE so as to model the uncertainty of the prior parameter. Actually, the
hyper-prior technique is quite general and we show that it can be applied to
other AEVB based models to alleviate the {\it collapse-to-prior} problem
elegantly. Moreover, we also propose HiTM-VAE, where the document-specific
topic distributions are generated in a hierarchical manner. HiTM-VAE is even
more flexible and can generate topic distributions with better variability.
Experimental results on 20News and Reuters RCV1-V2 datasets show that the
proposed models outperform the state-of-the-art baselines significantly. The
advantages of the hyper-prior technique and the hierarchical model construction
are also confirmed by experiments.
WSD algorithm based on a new method of vector-word contexts proximity calculation via epsilon-filtration
Authors: Alexander Kirillov, Natalia Krizhanovsky, Andrew Krizhanovsky
Date: 2018-05-24 00:00:00
Tasks: Sentence, Word Sense Disambiguation

The problem of word sense disambiguation (WSD) is considered in the article.
Given a set of synonyms (synsets) and sentences with these synonyms. It is
necessary to select the meaning of the word in the sentence automatically. 1285
sentences were tagged by experts, namely, one of the dictionary meanings was
selected by experts for target words. To solve the WSD-problem, an algorithm
based on a new method of vector-word contexts proximity calculation is
proposed. In order to achieve higher accuracy, a preliminary epsilon-filtering
of words is performed, both in the sentence and in the set of synonyms. An
extensive program of experiments was carried out. Four algorithms are
implemented, including a new algorithm. Experiments have shown that in a number
of cases the new algorithm shows better results. The developed software and the
tagged corpus have an open license and are available online. Wiktionary and
Wikisource are used. A brief description of this work can be viewed in slides
(https://goo.gl/9ak6Gt). Video lecture in Russian on this research is available
online (https://youtu.be/-DLmRkepf58).
The Kanerva Machine: A Generative Distributed Memory
Authors: Yan Wu, Greg Wayne, Alex Graves, Timothy Lillicrap
Date: 2018-04-05 00:00:00
Proceeding: ICLR 2018 1

We present an end-to-end trained memory system that quickly adapts to new
data and generates samples like them. Inspired by Kanerva's sparse distributed
memory, it has a robust distributed reading and writing mechanism. The memory
is analytically tractable, which enables optimal on-line compression via a
Bayesian update-rule. We formulate it as a hierarchical conditional generative
model, where memory provides a rich data-dependent prior distribution.
Consequently, the top-down memory and bottom-up perception are combined to
produce the code representing an observation. Empirically, we demonstrate that
the adaptive memory significantly improves generative models trained on both
the Omniglot and CIFAR datasets. Compared with the Differentiable Neural
Computer (DNC) and its variants, our memory model has greater capacity and is
significantly easier to train.
RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Authors: Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, Yong-Liang Yang
Date: 2018-06-18 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Inverse Rendering

Traditional computer graphics rendering pipeline is designed for procedurally
generating 2D quality images from 3D shapes with high performance. The
non-differentiability due to discrete operations such as visibility computation
makes it hard to explicitly correlate rendering parameters and the resulting
image, posing a significant challenge for inverse rendering tasks. Recent work
on differentiable rendering achieves differentiability either by designing
surrogate gradients for non-differentiable operations or via an approximate but
differentiable renderer. These methods, however, are still limited when it
comes to handling occlusion, and restricted to particular rendering effects. We
present RenderNet, a differentiable rendering convolutional network with a
novel projection unit that can render 2D images from 3D shapes. Spatial
occlusion and shading calculation are automatically encoded in the network. Our
experiments show that RenderNet can successfully learn to implement different
shaders, and can be used in inverse rendering tasks to estimate shape, pose,
lighting and texture from a single image.
Distributed learning with compressed gradients
Authors: Sarit Khirirat, Hamid Reza Feyzmahdavian, Mikael Johansson
Date: 2018-06-18 00:00:00
Tasks: BIG-bench Machine Learning, Distributed Optimization

Asynchronous computation and gradient compression have emerged as two key
techniques for achieving scalability in distributed optimization for
large-scale machine learning. This paper presents a unified analysis framework
for distributed gradient methods operating with staled and compressed
gradients. Non-asymptotic bounds on convergence rates and information exchange
are derived for several optimization algorithms. These bounds give explicit
expressions for step-sizes and characterize how the amount of asynchrony and
the compression accuracy affect iteration and communication complexity
guarantees. Numerical results highlight convergence properties of different
gradient compression algorithms and confirm that fast convergence under limited
information exchange is indeed possible.
SubGram: Extending Skip-gram Word Representation with Substrings
Authors: Tom Kocmi, Ondřej Bojar
Date: 2018-06-18 00:00:00

Skip-gram (word2vec) is a recent method for creating vector representations
of words ("distributed word representations") using a neural network. The
representation gained popularity in various areas of natural language
processing, because it seems to capture syntactic and semantic information
about words without any explicit supervision in this respect. We propose
SubGram, a refinement of the Skip-gram model to consider also the word
structure during the training process, achieving large gains on the Skip-gram
original test set.
Learning from Outside the Viability Kernel: Why we Should Build Robots that can Fall with Grace
Authors: Steve Heim, Alexander Spröwitz
Date: 2018-06-18 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Despite impressive results using reinforcement learning to solve complex
problems from scratch, in robotics this has still been largely limited to
model-based learning with very informative reward functions. One of the major
challenges is that the reward landscape often has large patches with no
gradient, making it difficult to sample gradients effectively. We show here
that the robot state-initialization can have a more important effect on the
reward landscape than is generally expected. In particular, we show the
counter-intuitive benefit of including initializations that are unviable, in
other words initializing in states that are doomed to fail.
ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing
Authors: Jian Zhang, Bernard Ghanem
Date: 2017-06-24 00:00:00
Proceeding: CVPR 2018 6
Tasks: Compressive Sensing

With the aim of developing a fast yet accurate algorithm for compressive
sensing (CS) reconstruction of natural images, we combine in this paper the
merits of two existing categories of CS methods: the structure insights of
traditional optimization-based methods and the speed of recent network-based
ones. Specifically, we propose a novel structured deep network, dubbed
ISTA-Net, which is inspired by the Iterative Shrinkage-Thresholding Algorithm
(ISTA) for optimizing a general $\ell_1$ norm CS reconstruction model. To cast
ISTA into deep network form, we develop an effective strategy to solve the
proximal mapping associated with the sparsity-inducing regularizer using
nonlinear transforms. All the parameters in ISTA-Net (\eg nonlinear transforms,
shrinkage thresholds, step sizes, etc.) are learned end-to-end, rather than
being hand-crafted. Moreover, considering that the residuals of natural images
are more compressible, an enhanced version of ISTA-Net in the residual domain,
dubbed {ISTA-Net}$^+$, is derived to further improve CS reconstruction.
Extensive CS experiments demonstrate that the proposed ISTA-Nets outperform
existing state-of-the-art optimization-based and network-based CS methods by
large margins, while maintaining fast computational speed. Our source codes are
available: \textsl{http://jianzhang.tech/projects/ISTA-Net}.
State Gradients for RNN Memory Analysis
Authors: Lyan Verwimp, Hugo Van hamme, Vincent Renkens, Patrick Wambacq
Date: 2018-05-11 00:00:00
Proceeding: WS 2018 11

We present a framework for analyzing what the state in RNNs remembers from
its input embeddings. Our approach is inspired by backpropagation, in the sense
that we compute the gradients of the states with respect to the input
embeddings. The gradient matrix is decomposed with Singular Value Decomposition
to analyze which directions in the embedding space are best transferred to the
hidden state space, characterized by the largest singular values. We apply our
approach to LSTM language models and investigate to what extent and for how
long certain classes of words are remembered on average for a certain corpus.
Additionally, the extent to which a specific property or relationship is
remembered by the RNN can be tracked by comparing a vector characterizing that
property with the direction(s) in embedding space that are best preserved in
hidden state space.
Convex Optimization with Unbounded Nonconvex Oracles using Simulated Annealing
Authors: Oren Mangoubi, Nisheeth K. Vishnoi
Date: 2017-11-07 00:00:00

We consider the problem of minimizing a convex objective function $F$ when
one can only evaluate its noisy approximation $\hat{F}$. Unless one assumes
some structure on the noise, $\hat{F}$ may be an arbitrary nonconvex function,
making the task of minimizing $F$ intractable. To overcome this, prior work has
often focused on the case when $F(x)-\hat{F}(x)$ is uniformly-bounded. In this
paper we study the more general case when the noise has magnitude $\alpha F(x)
+ \beta$ for some $\alpha, \beta > 0$, and present a polynomial time algorithm
that finds an approximate minimizer of $F$ for this noise model. Previously,
Markov chains, such as the stochastic gradient Langevin dynamics, have been
used to arrive at approximate solutions to these optimization problems.
However, for the noise model considered in this paper, no single temperature
allows such a Markov chain to both mix quickly and concentrate near the global
minimizer. We bypass this by combining "simulated annealing" with the
stochastic gradient Langevin dynamics, and gradually decreasing the temperature
of the chain in order to approach the global minimizer. As a corollary one can
approximately minimize a nonconvex function that is close to a convex function;
however, the closeness can deteriorate as one moves away from the optimum.
Incremental Sparse Bayesian Ordinal Regression
Authors: Chang Li, Maarten de Rijke
Date: 2018-06-18 00:00:00
Tasks: Multi-Label Learning, regression

Ordinal Regression (OR) aims to model the ordering information between
different data categories, which is a crucial topic in multi-label learning. An
important class of approaches to OR models the problem as a linear combination
of basis functions that map features to a high dimensional non-linear space.
However, most of the basis function-based algorithms are time consuming. We
propose an incremental sparse Bayesian approach to OR tasks and introduce an
algorithm to sequentially learn the relevant basis functions in the ordinal
scenario. Our method, called Incremental Sparse Bayesian Ordinal Regression
(ISBOR), automatically optimizes the hyper-parameters via the type-II maximum
likelihood method. By exploiting fast marginal likelihood optimization, ISBOR
can avoid big matrix inverses, which is the main bottleneck in applying basis
function-based algorithms to OR tasks on large-scale datasets. We show that
ISBOR can make accurate predictions with parsimonious basis functions while
offering automatic estimates of the prediction uncertainty. Extensive
experiments on synthetic and real word datasets demonstrate the efficiency and
effectiveness of ISBOR compared to other basis function-based OR approaches.