pwc-archive/papers-with-abstracts huggingface dataset:
Optimal Approximation with Sparsely Connected Deep Neural Networks
Authors: Helmut Bölcskei, Philipp Grohs, Gitta Kutyniok, Philipp Petersen
Date: 2017-05-04 00:00:00

We derive fundamental lower bounds on the connectivity and the memory
requirements of deep neural networks guaranteeing uniform approximation rates
for arbitrary function classes in $L^2(\mathbb R^d)$. In other words, we
establish a connection between the complexity of a function class and the
complexity of deep neural networks approximating functions from this class to
within a prescribed accuracy. Additionally, we prove that our lower bounds are
achievable for a broad family of function classes. Specifically, all function
classes that are optimally approximated by a general class of representation
systems---so-called \emph{affine systems}---can be approximated by deep neural
networks with minimal connectivity and memory requirements. Affine systems
encompass a wealth of representation systems from applied harmonic analysis
such as wavelets, ridgelets, curvelets, shearlets, $\alpha$-shearlets, and more
generally $\alpha$-molecules. Our central result elucidates a remarkable
universality property of neural networks and shows that they achieve the
optimum approximation properties of all affine systems combined. As a specific
example, we consider the class of $\alpha^{-1}$-cartoon-like functions, which
is approximated optimally by $\alpha$-shearlets. We also explain how our
results can be extended to the case of functions on low-dimensional immersed
manifolds. Finally, we present numerical experiments demonstrating that the
standard stochastic gradient descent algorithm generates deep neural networks
providing close-to-optimal approximation rates. Moreover, these results
indicate that stochastic gradient descent can actually learn approximations
that are sparse in the representation systems optimally sparsifying the
function class the network is trained on.
Color naming reflects both perceptual structure and communicative need
Authors: Noga Zaslavsky, Charles Kemp, Naftali Tishby, Terry Regier
Date: 2018-05-16 00:00:00

Gibson et al. (2017) argued that color naming is shaped by patterns of
communicative need. In support of this claim, they showed that color naming
systems across languages support more precise communication about warm colors
than cool colors, and that the objects we talk about tend to be warm-colored
rather than cool-colored. Here, we present new analyses that alter this
picture. We show that greater communicative precision for warm than for cool
colors, and greater communicative need, may both be explained by perceptual
structure. However, using an information-theoretic analysis, we also show that
color naming across languages bears signs of communicative need beyond what
would be predicted by perceptual structure alone. We conclude that color naming
is shaped both by perceptual structure, as has traditionally been argued, and
by patterns of communicative need, as argued by Gibson et al. - although for
reasons other than those they advanced.
Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Prediction
Authors: Onno Kampman, Elham J. Barezi, Dario Bertero, Pascale Fung
Date: 2018-05-02 00:00:00
Tasks: Emotional Intelligence

We propose a tri-modal architecture to predict Big Five personality trait
scores from video clips with different channels for audio, text, and video
data. For each channel, stacked Convolutional Neural Networks are employed. The
channels are fused both on decision-level and by concatenating their respective
fully connected layers. It is shown that a multimodal fusion approach
outperforms each single modality channel, with an improvement of 9.4\% over the
best individual modality (video). Full backpropagation is also shown to be
better than a linear combination of modalities, meaning complex interactions
between modalities can be leveraged to build better models. Furthermore, we can
see the prediction relevance of each modality for each trait. The described
model can be used to increase the emotional intelligence of virtual agents.
Semantic Structure and Interpretability of Word Embeddings
Authors: Lutfi Kerem Senel, Ihsan Utlu, Veysel Yucesoy, Aykut Koc, Tolga Cukur
Date: 2017-11-01 00:00:00
Tasks: Word Embeddings

Dense word embeddings, which encode semantic meanings of words to low
dimensional vector spaces have become very popular in natural language
processing (NLP) research due to their state-of-the-art performances in many
NLP tasks. Word embeddings are substantially successful in capturing semantic
relations among words, so a meaningful semantic structure must be present in
the respective vector spaces. However, in many cases, this semantic structure
is broadly and heterogeneously distributed across the embedding dimensions,
which makes interpretation a big challenge. In this study, we propose a
statistical method to uncover the latent semantic structure in the dense word
embeddings. To perform our analysis we introduce a new dataset (SEMCAT) that
contains more than 6500 words semantically grouped under 110 categories. We
further propose a method to quantify the interpretability of the word
embeddings; the proposed method is a practical alternative to the classical
word intrusion test that requires human intervention.
PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning
Authors: Aleksandra Faust, Oscar Ramirez, Marek Fiser, Kenneth Oslund, Anthony Francis, James Davidson, Lydia Tapia
Date: 2017-10-11 00:00:00
Tasks: Reinforcement Learning, Reinforcement Learning (RL)

We present PRM-RL, a hierarchical method for long-range navigation task
completion that combines sampling based path planning with reinforcement
learning (RL). The RL agents learn short-range, point-to-point navigation
policies that capture robot dynamics and task constraints without knowledge of
the large-scale topology. Next, the sampling-based planners provide roadmaps
which connect robot configurations that can be successfully navigated by the RL
agent. The same RL agents are used to control the robot under the direction of
the planning, enabling long-range navigation. We use the Probabilistic Roadmaps
(PRMs) for the sampling-based planner. The RL agents are constructed using
feature-based and deep neural net policies in continuous state and action
spaces. We evaluate PRM-RL, both in simulation and on-robot, on two navigation
tasks with non-trivial robot dynamics: end-to-end differential drive indoor
navigation in office environments, and aerial cargo delivery in urban
environments with load displacement constraints. Our results show improvement
in task completion over both RL agents on their own and traditional
sampling-based planners. In the indoor navigation task, PRM-RL successfully
completes up to 215 m long trajectories under noisy sensor conditions, and the
aerial cargo delivery completes flights over 1000 m without violating the task
constraints in an environment 63 million times larger than used in training.
Towards Dependable Deep Convolutional Neural Networks (CNNs) with Out-distribution Learning
Authors: Mahdieh Abbasi, Arezoo Rajabi, Christian Gagné, Rakesh B. Bobba
Date: 2018-04-24 00:00:00

Detection and rejection of adversarial examples in security sensitive and
safety-critical systems using deep CNNs is essential. In this paper, we propose
an approach to augment CNNs with out-distribution learning in order to reduce
misclassification rate by rejecting adversarial examples. We empirically show
that our augmented CNNs can either reject or classify correctly most
adversarial examples generated using well-known methods ( >95% for MNIST and
>75% for CIFAR-10 on average). Furthermore, we achieve this without requiring
to train using any specific type of adversarial examples and without
sacrificing the accuracy of models on clean samples significantly (< 4%).
Critical Points to Determine Persistence Homology
Authors: Charmin Asirimath, Jayampathy Ratnayake, Chathuranga Weeraddana
Date: 2018-05-16 00:00:00

Computation of the simplicial complexes of a large point cloud often relies
on extracting a sample, to reduce the associated computational burden. The
study considers sampling critical points of a Morse function associated to a
point cloud, to approximate the Vietoris-Rips complex or the witness complex
and compute persistence homology. The effectiveness of the novel approach is
compared with the farthest point sampling, in a context of classifying human
face images into ethnics groups using persistence homology.
Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning
Authors: Xianfu Chen, Honggang Zhang, Celimuge Wu, Shiwen Mao, Yusheng Ji, Mehdi Bennis
Date: 2018-05-16 00:00:00
Tasks: Deep Reinforcement Learning, Edge-computing, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

To improve the quality of computation experience for mobile devices,
mobile-edge computing (MEC) is a promising paradigm by providing computing
capabilities in close proximity within a sliced radio access network (RAN),
which supports both traditional communication and MEC services. Nevertheless,
the design of computation offloading policies for a virtual MEC system remains
challenging. Specifically, whether to execute a computation task at the mobile
device or to offload it for MEC server execution should adapt to the
time-varying network dynamics. In this paper, we consider MEC for a
representative mobile user in an ultra-dense sliced RAN, where multiple base
stations (BSs) are available to be selected for computation offloading. The
problem of solving an optimal computation offloading policy is modelled as a
Markov decision process, where our objective is to maximize the long-term
utility performance whereby an offloading decision is made based on the task
queue state, the energy queue state as well as the channel qualities between MU
and BSs. To break the curse of high dimensionality in state space, we first
propose a double deep Q-network (DQN) based strategic computation offloading
algorithm to learn the optimal policy without knowing a priori knowledge of
network dynamics. Then motivated by the additive structure of the utility
function, a Q-function decomposition technique is combined with the double DQN,
which leads to novel learning algorithm for the solving of stochastic
computation offloading. Numerical experiments show that our proposed learning
algorithms achieve a significant improvement in computation offloading
performance compared with the baseline policies.
Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension
Authors: Zhen Wang, Jiachen Liu, Xinyan Xiao, Yajuan Lyu, Tian Wu
Date: 2018-05-16 00:00:00
Proceeding: ACL 2018 7
Tasks: Answer Selection, Reading Comprehension, Reinforcement Learning

While sophisticated neural-based techniques have been developed in reading
comprehension, most approaches model the answer in an independent manner,
ignoring its relations with other answer candidates. This problem can be even
worse in open-domain scenarios, where candidates from multiple passages should
be combined to answer a single question. In this paper, we formulate reading
comprehension as an extract-then-select two-stage procedure. We first extract
answer candidates from passages, then select the final answer by combining
information from all the candidates. Furthermore, we regard candidate
extraction as a latent variable and train the two-stage process jointly with
reinforcement learning. As a result, our approach has improved the
state-of-the-art performance significantly on two challenging open-domain
reading comprehension datasets. Further analysis demonstrates the effectiveness
of our model components, especially the information fusion of all the
candidates and the joint training of the extract-then-select procedure.
Document Context Neural Machine Translation with Memory Networks
Authors: Sameen Maruf, Gholamreza Haffari
Date: 2017-11-10 00:00:00
Proceeding: ACL 2018 7
Tasks: Machine Translation, Structured Prediction, Translation

We present a document-level neural machine translation model which takes both
source and target document context into account using memory networks. We model
the problem as a structured prediction problem with interdependencies among the
observed and hidden variables, i.e., the source sentences and their unobserved
target translations in the document. The resulting structured prediction
problem is tackled with a neural translation model equipped with two memory
components, one each for the source and target side, to capture the documental
interdependencies. We train the model end-to-end, and propose an iterative
decoding algorithm based on block coordinate descent. Experimental results of
English translations from French, German, and Estonian documents show that our
model is effective in exploiting both source and target document context, and
statistically significantly outperforms the previous work in terms of BLEU and
METEOR.
Photorealistic Image Reconstruction from Hybrid Intensity and Event based Sensor
Authors: Prasan A Shedligeri, Kaushik Mitra
Date: 2018-05-16 00:00:00
Tasks: Image Reconstruction

Event sensors output a stream of asynchronous brightness changes (called
``events'') at a very high temporal rate. Previous works on recovering the lost
intensity information from the event sensor data have heavily relied on the
event stream, which makes the reconstructed images non-photorealistic and also
susceptible to noise in the event stream. We propose to reconstruct
photorealistic intensity images from a hybrid sensor consisting of a low frame
rate conventional camera, which has the scene texture information, along with
the event sensor. To accomplish our task, we warp the low frame rate intensity
images to temporally dense locations of the event data by estimating a
spatially dense scene depth and temporally dense sensor ego-motion. The results
obtained from our algorithm are more photorealistic compared to any of the
previous state-of-the-art algorithms. We also demonstrate our algorithm's
robustness to abrupt camera motion and noise in the event sensor data.
A Compact Network Learning Model for Distribution Regression
Authors: Connie Kou, Hwee Kuan Lee, Teck Khim Ng
Date: 2018-04-13 00:00:00
Tasks: model, regression

Despite the superior performance of deep learning in many applications,
challenges remain in the area of regression on function spaces. In particular,
neural networks are unable to encode function inputs compactly as each node
encodes just a real value. We propose a novel idea to address this shortcoming:
to encode an entire function in a single network node. To that end, we design a
compact network representation that encodes and propagates functions in single
nodes for the distribution regression task. Our proposed Distribution
Regression Network (DRN) achieves higher prediction accuracies while being much
more compact and uses fewer parameters than traditional neural networks.
Stochastic Approximation for Risk-aware Markov Decision Processes
Authors: Wenjie Huang, William B. Haskell
Date: 2018-05-11 00:00:00
Tasks: Q-Learning

We develop a stochastic approximation-type algorithm to solve finite state/action, infinite-horizon, risk-aware Markov decision processes. Our algorithm has two loops. The inner loop computes the risk by solving a stochastic saddle-point problem. The outer loop performs $Q$-learning to compute an optimal risk-aware policy. Several widely investigated risk measures (e.g. conditional value-at-risk, optimized certainty equivalent, and absolute semi-deviation) are covered by our algorithm. Almost sure convergence and the convergence rate of the algorithm are established. For an error tolerance $\epsilon>0$ for the optimal $Q$-value estimation gap and learning rate $k\in(1/2,\,1]$, the overall convergence rate of our algorithm is $\Omega((\ln(1/\delta\epsilon)/\epsilon^{2})^{1/k}+(\ln(1/\epsilon))^{1/(1-k)})$ with probability at least $1-\delta$.
Towards Robust Neural Machine Translation
Authors: Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, Yang Liu
Date: 2018-05-16 00:00:00
Proceeding: ACL 2018 7
Tasks: Decoder, Machine Translation, NMT, Translation

Small perturbations in the input can severely distort intermediate
representations and thus impact translation quality of neural machine
translation (NMT) models. In this paper, we propose to improve the robustness
of NMT models with adversarial stability training. The basic idea is to make
both the encoder and decoder in NMT models robust against input perturbations
by enabling them to behave similarly for the original input and its perturbed
counterpart. Experimental results on Chinese-English, English-German and
English-French translation tasks show that our approaches can not only achieve
significant improvements over strong NMT systems but also improve the
robustness of NMT models.
A Survey on Compiler Autotuning using Machine Learning
Authors: Amir H. Ashouri, William Killian, John Cavazos, Gianluca Palermo, Cristina Silvano
Date: 2018-01-13 00:00:00
Tasks: BIG-bench Machine Learning, Compiler Optimization, Survey

Since the mid-1990s, researchers have been trying to use machine-learning
based approaches to solve a number of different compiler optimization problems.
These techniques primarily enhance the quality of the obtained results and,
more importantly, make it feasible to tackle two main compiler optimization
problems: optimization selection (choosing which optimizations to apply) and
phase-ordering (choosing the order of applying optimizations). The compiler
optimization space continues to grow due to the advancement of applications,
increasing number of compiler optimizations, and new target architectures.
Generic optimization passes in compilers cannot fully leverage newly introduced
optimizations and, therefore, cannot keep up with the pace of increasing
options. This survey summarizes and classifies the recent advances in using
machine learning for the compiler optimization field, particularly on the two
major problems of (1) selecting the best optimizations and (2) the
phase-ordering of optimizations. The survey highlights the approaches taken so
far, the obtained results, the fine-grain classification among different
approaches and finally, the influential papers of the field.
Integrated Model, Batch and Domain Parallelism in Training Neural Networks
Authors: Amir Gholami, Ariful Azad, Peter Jin, Kurt Keutzer, Aydin Buluc
Date: 2017-12-12 00:00:00

We propose a new integrated method of exploiting model, batch and domain
parallelism for the training of deep neural networks (DNNs) on large
distributed-memory computers using minibatch stochastic gradient descent (SGD).
Our goal is to find an efficient parallelization strategy for a fixed batch
size using $P$ processes. Our method is inspired by the communication-avoiding
algorithms in numerical linear algebra. We see $P$ processes as logically
divided into a $P_r \times P_c$ grid where the $P_r$ dimension is implicitly
responsible for model/domain parallelism and the $P_c$ dimension is implicitly
responsible for batch parallelism. In practice, the integrated matrix-based
parallel algorithm encapsulates these types of parallelism automatically. We
analyze the communication complexity and analytically demonstrate that the
lowest communication costs are often achieved neither with pure model nor with
pure data parallelism. We also show how the domain parallel approach can help
in extending the theoretical scaling limit of the typical batch parallel
method.
Market Self-Learning of Signals, Impact and Optimal Trading: Invisible Hand Inference with Free Energy
Authors: Igor Halperin, Ilya Feldshteyn
Date: 2018-05-16 00:00:00
Tasks: Reinforcement Learning, Self-Learning

We present a simple model of a non-equilibrium self-organizing market where
asset prices are partially driven by investment decisions of a bounded-rational
agent. The agent acts in a stochastic market environment driven by various
exogenous "alpha" signals, agent's own actions (via market impact), and noise.
Unlike traditional agent-based models, our agent aggregates all traders in the
market, rather than being a representative agent. Therefore, it can be
identified with a bounded-rational component of the market itself, providing a
particular implementation of an Invisible Hand market mechanism. In such
setting, market dynamics are modeled as a fictitious self-play of such
bounded-rational market-agent in its adversarial stochastic environment. As
rewards obtained by such self-playing market agent are not observed from market
data, we formulate and solve a simple model of such market dynamics based on a
neuroscience-inspired Bounded Rational Information Theoretic Inverse
Reinforcement Learning (BRIT-IRL). This results in effective asset price
dynamics with a non-linear mean reversion - which in our model is generated
dynamically, rather than being postulated. We argue that our model can be used
in a similar way to the Black-Litterman model. In particular, it represents, in
a simple modeling framework, market views of common predictive signals, market
impacts and implied optimal dynamic portfolio allocations, and can be used to
assess values of private signals. Moreover, it allows one to quantify a
"market-implied" optimal investment strategy, along with a measure of market
rationality. Our approach is numerically light, and can be implemented using
standard off-the-shelf software such as TensorFlow.
InverseFaceNet: Deep Monocular Inverse Face Rendering
Authors: Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, Christian Theobalt
Date: 2017-03-31 00:00:00
Proceeding: CVPR 2018 6
Tasks: Face Reconstruction, Inverse Rendering

We introduce InverseFaceNet, a deep convolutional inverse rendering framework
for faces that jointly estimates facial pose, shape, expression, reflectance
and illumination from a single input image. By estimating all parameters from
just a single image, advanced editing possibilities on a single face image,
such as appearance editing and relighting, become feasible in real time. Most
previous learning-based face reconstruction approaches do not jointly recover
all dimensions, or are severely limited in terms of visual quality. In
contrast, we propose to recover high-quality facial pose, shape, expression,
reflectance and illumination using a deep neural network that is trained using
a large, synthetically created training corpus. Our approach builds on a novel
loss function that measures model-space similarity directly in parameter space
and significantly improves reconstruction accuracy. We further propose a
self-supervised bootstrapping process in the network training loop, which
iteratively updates the synthetic training corpus to better reflect the
distribution of real-world imagery. We demonstrate that this strategy
outperforms completely synthetically trained networks. Finally, we show
high-quality reconstructions and compare our approach to several
state-of-the-art approaches.
Narrative Modeling with Memory Chains and Semantic Supervision
Authors: Fei Liu, Trevor Cohn, Timothy Baldwin
Date: 2018-05-16 00:00:00
Proceeding: ACL 2018 7
Tasks: Cloze Test

Story comprehension requires a deep semantic understanding of the narrative,
making it a challenging task. Inspired by previous studies on ROC Story Cloze
Test, we propose a novel method, tracking various semantic aspects with
external neural memory chains while encouraging each to focus on a particular
semantic aspect. Evaluated on the task of story ending prediction, our model
demonstrates superior performance to a collection of competitive baselines,
setting a new state of the art.