pwc-archive/papers-with-abstracts huggingface dataset:
MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes
Authors: Henry Gouk, Bernhard Pfahringer, Eibe Frank, Michael Cree
Date: 2018-04-16 00:00:00

Effective regularisation of neural networks is essential to combat
overfitting due to the large number of parameters involved. We present an
empirical analogue to the Lipschitz constant of a feed-forward neural network,
which we refer to as the maximum gain. We hypothesise that constraining the
gain of a network will have a regularising effect, similar to how constraining
the Lipschitz constant of a network has been shown to improve generalisation. A
simple algorithm is provided that involves rescaling the weight matrix of each
layer after each parameter update. We conduct a series of studies on common
benchmark datasets, and also a novel dataset that we introduce to enable easier
significance testing for experiments using convolutional networks. Performance
on these datasets compares favourably with other common regularisation
techniques.
Bayesian Modeling via Goodness-of-fit
Authors: Subhadeep, Mukhopadhyay, Douglas Fletcher
Date: 2018-02-01 00:00:00

The two key issues of modern Bayesian statistics are: (i) establishing
principled approach for distilling statistical prior that is consistent with
the given data from an initial believable scientific prior; and (ii)
development of a Bayes-frequentist consolidated data analysis workflow that is
more effective than either of the two separately. In this paper, we propose the
idea of "Bayes via goodness of fit" as a framework for exploring these
fundamental questions, in a way that is general enough to embrace almost all of
the familiar probability models. Several illustrative examples show the benefit
of this new point of view as a practical data analysis tool. Relationship with
other Bayesian cultures is also discussed.
Annotation Artifacts in Natural Language Inference Data
Authors: Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, Noah A. Smith
Date: 2018-03-06 00:00:00
Proceeding: NAACL 2018 6
Tasks: Natural Language Inference, Negation, Sentence, Text Categorization

Large-scale datasets for natural language inference are created by presenting
crowd workers with a sentence (premise), and asking them to generate three new
sentences (hypotheses) that it entails, contradicts, or is logically neutral
with respect to. We show that, in a significant portion of such data, this
protocol leaves clues that make it possible to identify the label by looking
only at the hypothesis, without observing the premise. Specifically, we show
that a simple text categorization model can correctly classify the hypothesis
alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams
et. al, 2017). Our analysis reveals that specific linguistic phenomena such as
negation and vagueness are highly correlated with certain inference classes.
Our findings suggest that the success of natural language inference models to
date has been overestimated, and that the task remains a hard open problem.
Can Neural Machine Translation be Improved with User Feedback?
Authors: Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, Stefan Riezler
Date: 2018-04-16 00:00:00
Proceeding: NAACL 2018 6
Tasks: Machine Translation, NMT, Translation

We present the first real-world application of methods for improving neural
machine translation (NMT) with human reinforcement, based on explicit and
implicit user feedback collected on the eBay e-commerce platform. Previous work
has been confined to simulation experiments, whereas in this paper we work with
real logged feedback for offline bandit learning of NMT parameters. We conduct
a thorough analysis of the available explicit user judgments---five-star
ratings of translation quality---and show that they are not reliable enough to
yield significant improvements in bandit learning. In contrast, we successfully
utilize implicit task-based feedback collected in a cross-lingual search task
to improve task-specific and machine translation quality metrics.
Explainable Prediction of Medical Codes from Clinical Text
Authors: James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, Jacob Eisenstein
Date: 2018-02-15 00:00:00
Proceeding: NAACL 2018 6
Tasks: Medical Code Prediction, Prediction

Clinical notes are text documents that are created by clinicians for each
patient encounter. They are typically accompanied by medical codes, which
describe the diagnosis and treatment. Annotating these codes is labor intensive
and error prone; furthermore, the connection between the codes and the text is
not annotated, obscuring the reasons and details behind specific diagnoses and
treatments. We present an attentional convolutional network that predicts
medical codes from clinical text. Our method aggregates information across the
document using a convolutional neural network, and uses an attention mechanism
to select the most relevant segments for each of the thousands of possible
codes. The method is accurate, achieving precision@8 of 0.71 and a Micro-F1 of
0.54, which are both better than the prior state of the art. Furthermore,
through an interpretability evaluation by a physician, we show that the
attention mechanism identifies meaningful explanations for each code assignment
State-Augmentation Transformations for Risk-Sensitive Reinforcement Learning
Authors: Shuai Ma, Jia Yuan Yu
Date: 2018-04-16 00:00:00
Tasks: Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In the framework of MDP, although the general reward function takes three
arguments-current state, action, and successor state; it is often simplified to
a function of two arguments-current state and action. The former is called a
transition-based reward function, whereas the latter is called a state-based
reward function. When the objective involves the expected cumulative reward
only, this simplification works perfectly. However, when the objective is
risk-sensitive, this simplification leads to an incorrect value. We present
state-augmentation transformations (SATs), which preserve the reward sequences
as well as the reward distributions and the optimal policy in risk-sensitive
reinforcement learning. In risk-sensitive scenarios, firstly we prove that, for
every MDP with a stochastic transition-based reward function, there exists an
MDP with a deterministic state-based reward function, such that for any given
(randomized) policy for the first MDP, there exists a corresponding policy for
the second MDP, such that both Markov reward processes share the same reward
sequence. Secondly we illustrate that two situations require the proposed SATs
in an inventory control problem. One could be using Q-learning (or other
learning methods) on MDPs with transition-based reward functions, and the other
could be using methods, which are for the Markov processes with a deterministic
state-based reward functions, on the Markov processes with general reward
functions. We show the advantage of the SATs by considering Value-at-Risk as an
example, which is a risk measure on the reward distribution instead of the
measures (such as mean and variance) of the distribution. We illustrate the
error in the reward distribution estimation from the direct use of Q-learning,
and show how the SATs enable a variance formula to work on Markov processes
with general reward functions.
Near Human-Level Performance in Grammatical Error Correction with Hybrid Machine Translation
Authors: Roman Grundkiewicz, Marcin Junczys-Dowmunt
Date: 2018-04-16 00:00:00
Proceeding: NAACL 2018 6
Tasks: Grammatical Error Correction, Machine Translation, NMT, Translation

We combine two of the most popular approaches to automated Grammatical Error
Correction (GEC): GEC based on Statistical Machine Translation (SMT) and GEC
based on Neural Machine Translation (NMT). The hybrid system achieves new
state-of-the-art results on the CoNLL-2014 and JFLEG benchmarks. This GEC
system preserves the accuracy of SMT output and, at the same time, generates
more fluent sentences as it typical for NMT. Our analysis shows that the
created systems are closer to reaching human-level performance than any other
GEC system reported so far.
Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task
Authors: Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, Kenneth Heafield
Date: 2018-04-16 00:00:00
Proceeding: NAACL 2018 6
Tasks: Domain Adaptation, Grammatical Error Correction, Machine Translation, Transfer Learning, Translation

Previously, neural methods in grammatical error correction (GEC) did not
reach state-of-the-art results compared to phrase-based statistical machine
translation (SMT) baselines. We demonstrate parallels between neural GEC and
low-resource neural MT and successfully adapt several methods from low-resource
MT to neural GEC. We further establish guidelines for trustable results in
neural GEC and propose a set of model-independent methods for neural GEC that
can be easily applied in most GEC settings. Proposed methods include adding
source-side noise, domain-adaptation techniques, a GEC-specific
training-objective, transfer learning with monolingual data, and ensembling of
independently trained GEC models and language models. The combined effects of
these methods result in better than state-of-the-art neural GEC models that
outperform previously best neural GEC systems by more than 10% M$^2$ on the
CoNLL-2014 benchmark and 5.9% on the JFLEG test set. Non-neural
state-of-the-art systems are outperformed by more than 2% on the CoNLL-2014
benchmark and by 4% on JFLEG.
Isolating Sources of Disentanglement in Variational Autoencoders
Authors: Ricky T. Q. Chen, Xuechen Li, Roger Grosse, David Duvenaud
Date: 2018-02-14 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Disentanglement

We decompose the evidence lower bound to show the existence of a term
measuring the total correlation between latent variables. We use this to
motivate our $\beta$-TCVAE (Total Correlation Variational Autoencoder), a
refinement of the state-of-the-art $\beta$-VAE objective for learning
disentangled representations, requiring no additional hyperparameters during
training. We further propose a principled classifier-free measure of
disentanglement called the mutual information gap (MIG). We perform extensive
quantitative and qualitative experiments, in both restricted and non-restricted
settings, and show a strong relation between total correlation and
disentanglement, when the latent variables model is trained using our
framework.
Deep Learning as a Mixed Convex-Combinatorial Optimization Problem
Authors: Abram L. Friesen, Pedro Domingos
Date: 2017-10-31 00:00:00
Proceeding: ICLR 2018 1
Tasks: Combinatorial Optimization, Deep Learning, Quantization

As neural networks grow deeper and wider, learning networks with
hard-threshold activations is becoming increasingly important, both for network
quantization, which can drastically reduce time and energy requirements, and
for creating large integrated systems of deep networks, which may have
non-differentiable components and must avoid vanishing and exploding gradients
for effective learning. However, since gradient descent is not applicable to
hard-threshold functions, it is not clear how to learn networks of them in a
principled way. We address this problem by observing that setting targets for
hard-threshold hidden units in order to minimize loss is a discrete
optimization problem, and can be solved as such. The discrete optimization goal
is to find a set of targets such that each unit, including the output, has a
linearly separable problem to solve. Given these targets, the network
decomposes into individual perceptrons, which can then be learned with standard
convex approaches. Based on this, we develop a recursive mini-batch algorithm
for learning deep hard-threshold networks that includes the popular but poorly
justified straight-through estimator as a special case. Empirically, we show
that our algorithm improves classification accuracy in a number of settings,
including for AlexNet and ResNet-18 on ImageNet, when compared to the
straight-through estimator.
UCBoost: A Boosting Approach to Tame Complexity and Optimality for Stochastic Bandits
Authors: Fang Liu, Sinong Wang, Swapna Buccapatnam, Ness Shroff
Date: 2018-04-16 00:00:00
Tasks: Decision Making, Sequential Decision Making

In this work, we address the open problem of finding low-complexity
near-optimal multi-armed bandit algorithms for sequential decision making
problems. Existing bandit algorithms are either sub-optimal and computationally
simple (e.g., UCB1) or optimal and computationally complex (e.g., kl-UCB). We
propose a boosting approach to Upper Confidence Bound based algorithms for
stochastic bandits, that we call UCBoost. Specifically, we propose two types of
UCBoost algorithms. We show that UCBoost($D$) enjoys $O(1)$ complexity for each
arm per round as well as regret guarantee that is $1/e$-close to that of the
kl-UCB algorithm. We propose an approximation-based UCBoost algorithm,
UCBoost($\epsilon$), that enjoys a regret guarantee $\epsilon$-close to that of
kl-UCB as well as $O(\log(1/\epsilon))$ complexity for each arm per round.
Hence, our algorithms provide practitioners a practical way to trade optimality
with computational complexity. Finally, we present numerical results which show
that UCBoost($\epsilon$) can achieve the same regret performance as the
standard kl-UCB while incurring only $1\%$ of the computational cost of kl-UCB.
Neural Models for Reasoning over Multiple Mentions using Coreference
Authors: Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William W. Cohen, Ruslan Salakhutdinov
Date: 2018-04-16 00:00:00
Proceeding: NAACL 2018 6
Tasks: LAMBADA, Reading Comprehension

Many problems in NLP require aggregating information from multiple mentions
of the same entity which may be far apart in the text. Existing Recurrent
Neural Network (RNN) layers are biased towards short-term dependencies and
hence not suited to such tasks. We present a recurrent layer which is instead
biased towards coreferent dependencies. The layer uses coreference annotations
extracted from an external system to connect entity mentions belonging to the
same cluster. Incorporating this layer into a state-of-the-art reading
comprehension model improves performance on three datasets -- Wikihop, LAMBADA
and the bAbi AI tasks -- with large gains when training data is scarce.
Improving Implicit Discourse Relation Classification by Modeling Inter-dependencies of Discourse Units in a Paragraph
Authors: Zeyu Dai, Ruihong Huang
Date: 2018-04-16 00:00:00
Proceeding: NAACL 2018 6
Tasks: General Classification, Implicit Discourse Relation Classification, Relation, Relation Classification, Sentence

We argue that semantic meanings of a sentence or clause can not be
interpreted independently from the rest of a paragraph, or independently from
all discourse relations and the overall paragraph-level discourse structure.
With the goal of improving implicit discourse relation classification, we
introduce a paragraph-level neural networks that model inter-dependencies
between discourse units as well as discourse relation continuity and patterns,
and predict a sequence of discourse relations in a paragraph. Experimental
results show that our model outperforms the previous state-of-the-art systems
on the benchmark corpus of PDTB.
Heuristic Approaches for Goal Recognition in Incomplete Domain Models
Authors: Ramon Fraga Pereira, Felipe Meneguzzi
Date: 2018-04-16 00:00:00

Recent approaches to goal recognition have progressively relaxed the
assumptions about the amount and correctness of domain knowledge and available
observations, yielding accurate and efficient algorithms. These approaches,
however, assume completeness and correctness of the domain theory against which
their algorithms match observations: this is too strong for most real-world
domains. In this paper, we develop goal recognition techniques that are capable
of recognizing goals using \textit{incomplete} (and possibly incorrect) domain
theories. We show the efficiency and accuracy of our approaches empirically
against a large dataset of goal and plan recognition problems with incomplete
domains.
An information-theoretic on-line update principle for perception-action coupling
Authors: Zhen Peng, Tim Genewein, Felix Leibfried, Daniel A. Braun
Date: 2018-04-16 00:00:00

Inspired by findings of sensorimotor coupling in humans and animals, there
has recently been a growing interest in the interaction between action and
perception in robotic systems [Bogh et al., 2016]. Here we consider perception
and action as two serial information channels with limited
information-processing capacity. We follow [Genewein et al., 2015] and
formulate a constrained optimization problem that maximizes utility under
limited information-processing capacity in the two channels. As a solution we
obtain an optimal perceptual channel and an optimal action channel that are
coupled such that perceptual information is optimized with respect to
downstream processing in the action module. The main novelty of this study is
that we propose an online optimization procedure to find bounded-optimal
perception and action channels in parameterized serial perception-action
systems. In particular, we implement the perceptual channel as a multi-layer
neural network and the action channel as a multinomial distribution. We
illustrate our method in a NAO robot simulator with a simplified cup lifting
task.
Tree Morphology for Phenotyping from Semantics-Based Mapping in Orchard Environments
Authors: Wenbo Dong, Volkan Isler
Date: 2018-04-16 00:00:00

Measuring tree morphology for phenotyping is an essential but labor-intensive
activity in horticulture. Researchers often rely on manual measurements which
may not be accurate for example when measuring tree volume. Recent approaches
on automating the measurement process rely on LIDAR measurements coupled with
high-accuracy GPS. Usually each side of a row is reconstructed independently
and then merged using GPS information. Such approaches have two disadvantages:
(1) they rely on specialized and expensive equipment, and (2) since the
reconstruction process does not simultaneously use information from both sides,
side reconstructions may not be accurate. We also show that standard loop
closure methods do not necessarily align tree trunks well. In this paper, we
present a novel vision system that employs only an RGB-D camera to estimate
morphological parameters. A semantics-based mapping algorithm merges the
two-sides 3D models of tree rows, where integrated semantic information is
obtained and refined by robust fitting algorithms. We focus on measuring tree
height, canopy volume and trunk diameter from the optimized 3D model.
Experiments conducted in real orchards quantitatively demonstrate the accuracy
of our method.
When Image Denoising Meets High-Level Vision Tasks: A Deep Learning Approach
Authors: Ding Liu, Bihan Wen, Xianming Liu, Zhangyang Wang, Thomas S. Huang
Date: 2017-06-14 00:00:00
Tasks: Denoising, Image Denoising

Conventionally, image denoising and high-level vision tasks are handled
separately in computer vision. In this paper, we cope with the two jointly and
explore the mutual influence between them. First we propose a convolutional
neural network for image denoising which achieves the state-of-the-art
performance. Second we propose a deep neural network solution that cascades two
modules for image denoising and various high-level tasks, respectively, and use
the joint loss for updating only the denoising network via back-propagation. We
demonstrate that on one hand, the proposed denoiser has the generality to
overcome the performance degradation of different high-level vision tasks. On
the other hand, with the guidance of high-level vision information, the
denoising network can generate more visually appealing results. To the best of
our knowledge, this is the first work investigating the benefit of exploiting
image semantics simultaneously for image denoising and high-level vision tasks
via deep learning. The code is available online
https://github.com/Ding-Liu/DeepDenoising.
Densely Connected High Order Residual Network for Single Frame Image Super Resolution
Authors: Yiwen Huang, Ming Qin
Date: 2018-04-16 00:00:00
Tasks: Image Restoration, Image Super-Resolution, Super-Resolution

Deep convolutional neural networks (DCNN) have been widely adopted for
research on super resolution recently, however previous work focused mainly on
stacking as many layers as possible in their model, in this paper, we present a
new perspective regarding to image restoration problems that we can construct
the neural network model reflecting the physical significance of the image
restoration process, that is, embedding the a priori knowledge of image
restoration directly into the structure of our neural network model, we
employed a symmetric non-linear colorspace, the sigmoidal transfer, to replace
traditional transfers such as, sRGB, Rec.709, which are asymmetric non-linear
colorspaces, we also propose a "reuse plus patch" method to deal with super
resolution of different scaling factors, our proposed methods and model show
generally superior performance over previous work even though our model was
only roughly trained and could still be underfitting the training set.
Emergent Communication in a Multi-Modal, Multi-Step Referential Game
Authors: Katrina Evtimova, Andrew Drozdov, Douwe Kiela, Kyunghyun Cho
Date: 2017-05-29 00:00:00
Proceeding: ICLR 2018 1

Inspired by previous work on emergent communication in referential games, we
propose a novel multi-modal, multi-step referential game, where the sender and
receiver have access to distinct modalities of an object, and their information
exchange is bidirectional and of arbitrary duration. The multi-modal multi-step
setting allows agents to develop an internal communication significantly closer
to natural language, in that they share a single set of messages, and that the
length of the conversation may vary according to the difficulty of the task. We
examine these properties empirically using a dataset consisting of images and
textual descriptions of mammals, where the agents are tasked with identifying
the correct object. Our experiments indicate that a robust and efficient
communication protocol emerges, where gradual information exchange informs
better predictions and higher communication bandwidth improves generalization.
Defending against Adversarial Images using Basis Functions Transformations
Authors: Uri Shaham, James Garritano, Yutaro Yamada, Ethan Weinberger, Alex Cloninger, Xiuyuan Cheng, Kelly Stanton, Yuval Kluger
Date: 2018-03-28 00:00:00

We study the effectiveness of various approaches that defend against
adversarial attacks on deep networks via manipulations based on basis function
representations of images. Specifically, we experiment with low-pass filtering,
PCA, JPEG compression, low resolution wavelet approximation, and
soft-thresholding. We evaluate these defense techniques using three types of
popular attacks in black, gray and white-box settings. Our results show JPEG
compression tends to outperform the other tested defenses in most of the
settings considered, in addition to soft-thresholding, which performs well in
specific cases, and yields a more mild decrease in accuracy on benign examples.
In addition, we also mathematically derive a novel white-box attack in which
the adversarial perturbation is composed only of terms corresponding a to
pre-determined subset of the basis functions, of which a "low frequency attack"
is a special case.