pwc-archive/papers-with-abstracts huggingface dataset:
Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
Authors: Yoojin Choi, Mostafa El-Khamy, Jungwon Lee
Date: 2018-05-21 00:00:00
Tasks: Quantization

We consider the optimization of deep convolutional neural networks (CNNs)
such that they provide good performance while having reduced complexity if
deployed on either conventional systems utilizing spatial-domain convolution or
lower complexity systems designed for Winograd convolution. Furthermore, we
explore the universal quantization and compression of these networks. In
particular, the proposed framework produces one compressed model whose
convolutional filters can be made sparse either in the spatial domain or in the
Winograd domain. Hence, one compressed model can be deployed universally on any
platform, without need for re-training on the deployed platform, and the
sparsity of its convolutional filters can be exploited for further complexity
reduction in either domain. To get a better compression ratio, the sparse model
is compressed in the spatial domain which has a less number of parameters. From
our experiments, we obtain $24.2\times$, $47.7\times$ and $35.4\times$
compressed models for ResNet-18, AlexNet and CT-SRCNN, while their
computational cost is also reduced by $4.5\times$, $5.1\times$ and
$23.5\times$, respectively.
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Authors: Christy Y. Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing
Date: 2018-05-21 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Decision Making, Reinforcement Learning, Retrieval, Sentence

Generating long and coherent reports to describe medical images poses
challenges to bridging visual patterns with informative human linguistic
descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent
(HRGR-Agent) which reconciles traditional retrieval-based approaches populated
with human prior knowledge, with modern learning-based approaches to achieve
structured, robust, and diverse report generation. HRGR-Agent employs a
hierarchical decision-making procedure. For each sentence, a high-level
retrieval policy module chooses to either retrieve a template sentence from an
off-the-shelf template database, or invoke a low-level generation module to
generate a new sentence. HRGR-Agent is updated via reinforcement learning,
guided by sentence-level and word-level rewards. Experiments show that our
approach achieves the state-of-the-art results on two medical report datasets,
generating well-balanced structured sentences with robust coverage of
heterogeneous medical report contents. In addition, our model achieves the
highest detection accuracy of medical terminologies, and improved human
evaluation performance.
Character-based Neural Networks for Sentence Pair Modeling
Authors: Wuwei Lan, Wei Xu
Date: 2018-05-21 00:00:00
Proceeding: NAACL 2018 6
Tasks: Language Modeling, Language Modelling, Natural Language Inference, Paraphrase Identification, Semantic Textual Similarity, Sentence, Sentence Pair Modeling, Translation

Sentence pair modeling is critical for many NLP tasks, such as paraphrase
identification, semantic textual similarity, and natural language inference.
Most state-of-the-art neural models for these tasks rely on pretrained word
embedding and compose sentence-level semantics in varied ways; however, few
works have attempted to verify whether we really need pretrained embeddings in
these tasks. In this paper, we study how effective subword-level (character and
character n-gram) representations are in sentence pair modeling. Though it is
well-known that subword models are effective in tasks with single sentence
input, including language modeling and machine translation, they have not been
systematically studied in sentence pair modeling tasks where the semantic and
string similarities between texts matter. Our experiments show that subword
models without any pretrained word embedding can achieve new state-of-the-art
results on two social media datasets and competitive results on news data for
paraphrase identification.
Measuring and regularizing networks in function space
Authors: Ari S. Benjamin, David Rolnick, Konrad Kording
Date: 2018-05-21 00:00:00
Proceeding: ICLR 2019 5

To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a $L^2$ Hilbert space. We examine how typical networks behave in this space, and compare how parameter $\ell^2$ distances compare to function $L^2$ distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the $L^2/\ell^2$ ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the $L^2$ distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through $L^2$-space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature.
Variational limits of k-NN graph based functionals on data clouds
Authors: Nicolas Garcia Trillos
Date: 2016-07-03 00:00:00

This paper studies the large sample asymptotics of data analysis procedures
based on the optimization of functionals defined on $k$-NN graphs on point
clouds. The paper is framed in the context of minimization of balanced cut
functionals, but our techniques, ideas and results can be adapted to other
functionals of relevance. We rigorously show that provided the number of
neighbors in the graph $k:=k_n$ scales with the number of points in the cloud
as $n \gg k_n \gg \log(n)$, then with probability one, the solution to the
graph cut optimization problem converges towards the solution of an analogue
variational problem at the continuum level.
Multiple Causal Inference with Latent Confounding
Authors: Rajesh Ranganath, Adler Perotte
Date: 2018-05-21 00:00:00
Tasks: Causal Inference

Causal inference from observational data requires assumptions. These
assumptions range from measuring confounders to identifying instruments.
Traditionally, causal inference assumptions have focused on estimation of
effects for a single treatment. In this work, we construct techniques for
estimation with multiple treatments in the presence of unobserved confounding.
We develop two assumptions based on shared confounding between treatments and
independence of treatments given the confounder. Together, these assumptions
lead to a confounder estimator regularized by mutual information. For this
estimator, we develop a tractable lower bound. To recover treatment effects, we
use the residual information in the treatments independent of the confounder.
We validate on simulations and an example from clinical medicine.
Halo: Learning Semantics-Aware Representations for Cross-Lingual Information Extraction
Authors: Hongyuan Mei, Sheng Zhang, Kevin Duh, Benjamin Van Durme
Date: 2018-05-21 00:00:00
Proceeding: SEMEVAL 2018 6
Tasks: TAG

Cross-lingual information extraction (CLIE) is an important and challenging
task, especially in low resource scenarios. To tackle this challenge, we
propose a training method, called Halo, which enforces the local region of each
hidden state of a neural model to only generate target tokens with the same
semantic structure tag. This simple but powerful technique enables a neural
model to learn semantics-aware representations that are robust to noise,
without introducing any extra parameter, thus yielding better generalization in
both high and low resource settings.
On the Selection of Initialization and Activation Function for Deep Neural Networks
Authors: Soufiane Hayou, Arnaud Doucet, Judith Rousseau
Date: 2018-05-21 00:00:00
Proceeding: ICLR 2019 5

The weight initialization and the activation function of deep neural networks
have a crucial impact on the performance of the training procedure. An
inappropriate selection can lead to the loss of information of the input during
forward propagation and the exponential vanishing/exploding of gradients during
back-propagation. Understanding the theoretical properties of untrained random
networks is key to identifying which deep networks may be trained successfully
as recently demonstrated by Schoenholz et al. (2017) who showed that for deep
feedforward neural networks only a specific choice of hyperparameters known as
the `edge of chaos' can lead to good performance. We complete this analysis by
providing quantitative results showing that, for a class of ReLU-like
activation functions, the information propagates indeed deeper for an
initialization at the edge of chaos. By further extending this analysis, we
identify a class of activation functions that improve the information
propagation over ReLU-like functions. This class includes the Swish activation,
$\phi_{swish}(x) = x \cdot \text{sigmoid}(x)$, used in Hendrycks & Gimpel
(2016), Elfwing et al. (2017) and Ramachandran et al. (2017). This provides a
theoretical grounding for the excellent empirical performance of $\phi_{swish}$
observed in these contributions. We complement those previous results by
illustrating the benefit of using a random initialization on the edge of chaos
in this context.
Stochastic Backward Euler: An Implicit Gradient Descent Algorithm for $k$-means Clustering
Authors: Penghang Yin, Minh Pham, Adam Oberman, Stanley Osher
Date: 2017-10-21 00:00:00
Tasks: Clustering

In this paper, we propose an implicit gradient descent algorithm for the
classic $k$-means problem. The implicit gradient step or backward Euler is
solved via stochastic fixed-point iteration, in which we randomly sample a
mini-batch gradient in every iteration. It is the average of the fixed-point
trajectory that is carried over to the next gradient step. We draw connections
between the proposed stochastic backward Euler and the recent entropy
stochastic gradient descent (Entropy-SGD) for improving the training of deep
neural networks. Numerical experiments on various synthetic and real datasets
show that the proposed algorithm provides better clustering results compared to
$k$-means algorithms in the sense that it decreased the objective function (the
cluster) and is much more robust to initialization.
Robust Decentralized Learning Using ADMM with Unreliable Agents
Authors: Qunwei Li, Bhavya Kailkhura, Ryan Goldhahn, Priyadip Ray, Pramod K. Varshney
Date: 2017-10-14 00:00:00
Tasks: BIG-bench Machine Learning

Many machine learning problems can be formulated as consensus optimization
problems which can be solved efficiently via a cooperative multi-agent system.
However, the agents in the system can be unreliable due to a variety of
reasons: noise, faults and attacks. Providing erroneous updates leads the
optimization process in a wrong direction, and degrades the performance of
distributed machine learning algorithms. This paper considers the problem of
decentralized learning using ADMM in the presence of unreliable agents. First,
we rigorously analyze the effect of erroneous updates (in ADMM learning
iterations) on the convergence behavior of multi-agent system. We show that the
algorithm linearly converges to a neighborhood of the optimal solution under
certain conditions and characterize the neighborhood size analytically. Next,
we provide guidelines for network design to achieve a faster convergence. We
also provide conditions on the erroneous updates for exact convergence to the
optimal solution. Finally, to mitigate the influence of unreliable agents, we
propose \textsf{ROAD}, a robust variant of ADMM, and show its resilience to
unreliable agents with an exact convergence to the optimum.
Discovering Discrete Latent Topics with Neural Variational Inference
Authors: Yishu Miao, Edward Grefenstette, Phil Blunsom
Date: 2017-06-01 00:00:00
Proceeding: ICML 2017 8
Tasks: Topic Models, Variational Inference

Topic models have been widely explored as probabilistic generative models of
documents. Traditional inference methods have sought closed-form derivations
for updating the models, however as the expressiveness of these models grows,
so does the difficulty of performing fast and accurate inference over their
parameters. This paper presents alternative neural approaches to topic
modelling by providing parameterisable distributions over topics which permit
training by backpropagation in the framework of neural variational inference.
In addition, with the help of a stick-breaking construction, we propose a
recurrent network that is able to discover a notionally unbounded number of
topics, analogous to Bayesian non-parametric topic models. Experimental results
on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the
effectiveness and efficiency of these neural topic models.
Evolving Real-Time Heuristics Search Algorithms with Building Blocks
Authors: Md Solimul Chowdhury, Victor Silva
Date: 2018-05-21 00:00:00

The research area of real-time heuristics search has produced quite many
algorithms. In the landscape of real-time heuristics search research, it is not
rare to find that an algorithm X that appears to perform better than algorithm
Y on a group of problems, performed worse than Y for another group of problems.
If these published algorithms are combined to generate a more powerful space of
algorithms, then that novel space of algorithms may solve a distribution of
problems more efficiently. Based on this intuition, a recent work Bulitko 2016
has defined the task of finding a combination of heuristics search algorithms
as a survival task. In this evolutionary approach, a space of algorithms is
defined over a set of building blocks published algorithms and a simulated
evolution is used to recombine these building blocks to find out the best
algorithm from that space of algorithms.
  In this paper, we extend the set of building blocks by adding one published
algorithm, namely lookahead based A-star shaped local search space generation
method from LSSLRTA-star, plus an unpublished novel strategy to generate local
search space with Greedy Best First Search. Then we perform experiments in the
new space of algorithms, which show that the best algorithms selected by the
evolutionary process have the following property: the deeper is the lookahead
depth of an algorithm, the lower is its suboptimality and scrubbing complexity.
Sample Compression for Real-Valued Learners
Authors: Steve Hanneke, Aryeh Kontorovich, Menachem Sadigurschi
Date: 2018-05-21 00:00:00
Tasks: Open-Ended Question Answering, regression

We give an algorithmically efficient version of the learner-to-compression
scheme conversion in Moran and Yehudayoff (2016). In extending this technique
to real-valued hypotheses, we also obtain an efficient regression-to-bounded
sample compression converter. To our knowledge, this is the first general
compressed regression result (regardless of efficiency or boundedness)
guaranteeing uniform approximate reconstruction. Along the way, we develop a
generic procedure for constructing weak real-valued learners out of abstract
regressors; this may be of independent interest. In particular, this result
sheds new light on an open question of H. Simon (1997). We show applications to
two regression problems: learning Lipschitz and bounded-variation functions.
Classifier-agnostic saliency map extraction
Authors: Konrad Zolna, Krzysztof J. Geras, Kyunghyun Cho
Date: 2018-05-21 00:00:00
Proceeding: ICLR 2019 5
Tasks: General Classification

Currently available methods for extracting saliency maps identify parts of the input which are the most important to a specific fixed classifier. We show that this strong dependence on a given classifier hinders their performance. To address this problem, we propose classifier-agnostic saliency map extraction, which finds all parts of the image that any classifier could use, not just one given in advance. We observe that the proposed approach extracts higher quality saliency maps than prior work while being conceptually simple and easy to implement. The method sets the new state of the art result for localization task on the ImageNet data, outperforming all existing weakly-supervised localization techniques, despite not using the ground truth labels at the inference time. The code reproducing the results is available at https://github.com/kondiz/casme . The final version of this manuscript is published in Computer Vision and Image Understanding and is available online at https://doi.org/10.1016/j.cviu.2020.102969 .
Reinforcement Learning for Solving the Vehicle Routing Problem
Authors: Mohammadreza Nazari, Afshin Oroojlooy, Lawrence V. Snyder, Martin Takáč
Date: 2018-02-12 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Combinatorial Optimization, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We present an end-to-end framework for solving the Vehicle Routing Problem
(VRP) using reinforcement learning. In this approach, we train a single model
that finds near-optimal solutions for problem instances sampled from a given
distribution, only by observing the reward signals and following feasibility
rules. Our model represents a parameterized stochastic policy, and by applying
a policy gradient algorithm to optimize its parameters, the trained model
produces the solution as a sequence of consecutive actions in real time,
without the need to re-train for every new problem instance. On capacitated
VRP, our approach outperforms classical heuristics and Google's OR-Tools on
medium-sized instances in solution quality with comparable computation time
(after training). We demonstrate how our approach can handle problems with
split delivery and explore the effect of such deliveries on the solution
quality. Our proposed framework can be applied to other variants of the VRP
such as the stochastic VRP, and has the potential to be applied more generally
to combinatorial optimization problems.
Stochastic modified equations for the asynchronous stochastic gradient descent
Authors: Jing An, Jianfeng Lu, Lexing Ying
Date: 2018-05-21 00:00:00

We propose a stochastic modified equations (SME) for modeling the asynchronous stochastic gradient descent (ASGD) algorithms. The resulting SME of Langevin type extracts more information about the ASGD dynamics and elucidates the relationship between different types of stochastic gradient algorithms. We show the convergence of ASGD to the SME in the continuous time limit, as well as the SME's precise prediction to the trajectories of ASGD with various forcing terms. As an application of the SME, we propose an optimal mini-batching strategy for ASGD via solving the optimal control problem of the associated SME.
Sparse and Constrained Attention for Neural Machine Translation
Authors: Chaitanya Malaviya, Pedro Ferreira, André F. T. Martins
Date: 2018-05-21 00:00:00
Proceeding: ACL 2018 7
Tasks: Machine Translation, NMT, Translation

In NMT, words are sometimes dropped from the source or generated repeatedly
in the translation. We explore novel strategies to address the coverage problem
that change only the attention transformation. Our approach allocates
fertilities to source words, used to bound the attention each word can receive.
We experiment with various sparse and constrained attention transformations and
propose a new one, constrained sparsemax, shown to be differentiable and
sparse. Empirical evaluation is provided in three languages pairs.
The Roles of Supervised Machine Learning in Systems Neuroscience
Authors: Joshua I. Glaser, Ari S. Benjamin, Roozbeh Farhoodi, Konrad P. Kording
Date: 2018-05-21 00:00:00
Tasks: BIG-bench Machine Learning

Over the last several years, the use of machine learning (ML) in neuroscience
has been rapidly increasing. Here, we review ML's contributions, both realized
and potential, across several areas of systems neuroscience. We describe four
primary roles of ML within neuroscience: 1) creating solutions to engineering
problems, 2) identifying predictive variables, 3) setting benchmarks for simple
models of the brain, and 4) serving itself as a model for the brain. The
breadth and ease of its applicability suggests that machine learning should be
in the toolbox of most systems neuroscientists.
Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings
Authors: Bernd Bohnet, Ryan Mcdonald, Goncalo Simoes, Daniel Andor, Emily Pitler, Joshua Maynez
Date: 2018-05-21 00:00:00
Proceeding: ACL 2018 7
Tasks: Morphological Tagging, Part-Of-Speech Tagging, Sentence, Word Embeddings

The rise of neural networks, and particularly recurrent neural networks, has
produced significant advances in part-of-speech tagging accuracy. One
characteristic common among these models is the presence of rich initial word
encodings. These encodings typically are composed of a recurrent
character-based representation with learned and pre-trained word embeddings.
However, these encodings do not consider a context wider than a single word and
it is only through subsequent recurrent layers that word or sub-word
information interacts. In this paper, we investigate models that use recurrent
neural networks with sentence-level context for initial character and
word-based representations. In particular we show that optimal results are
obtained by integrating these context sensitive representations through
synchronized training with a meta-model that learns to combine their states. We
present results on part-of-speech and morphological tagging with
state-of-the-art performance on a number of languages.
Improving CNN classifiers by estimating test-time priors
Authors: Milan Sulc, Jiri Matas
Date: 2018-05-21 00:00:00
Tasks: Image Classification

The problem of different training and test set class priors is addressed in
the context of CNN classifiers. We compare two different approaches to
estimating the new priors: an existing Maximum Likelihood Estimation approach
(optimized by an EM algorithm or by projected gradient descend) and a proposed
Maximum a Posteriori approach, which increases the stability of the estimate by
introducing a Dirichlet hyper-prior on the class prior probabilities.
Experimental results show a significant improvement on the fine-grained
classification tasks using known evaluation-time priors, increasing the top-1
accuracy by 4.0% on the FGVC iNaturalist 2018 validation set and by 3.9% on the
FGVCx Fungi 2018 validation set. Estimation of the unknown test set priors
noticeably increases the accuracy on the PlantCLEF dataset, allowing a single
CNN model to achieve state-of-the-art results and outperform the
competition-winning ensemble of 12 CNNs. The proposed Maximum a Posteriori
estimation increases the prediction accuracy by 2.8% on PlantCLEF 2017 and by
1.8% on FGVCx Fungi, where the existing MLE method would lead to a decrease
accuracy.