pwc-archive/papers-with-abstracts huggingface dataset:
DCFNet: Deep Neural Network with Decomposed Convolutional Filters
Authors: Qiang Qiu, Xiuyuan Cheng, Robert Calderbank, Guillermo Sapiro
Date: 2018-02-12 00:00:00
Proceeding: ICML 2018 7
Tasks: General Classification, image-classification, Image Classification

Filters in a Convolutional Neural Network (CNN) contain model parameters
learned from enormous amounts of data. In this paper, we suggest to decompose
convolutional filters in CNN as a truncated expansion with pre-fixed bases,
namely the Decomposed Convolutional Filters network (DCFNet), where the
expansion coefficients remain learned from data. Such a structure not only
reduces the number of trainable parameters and computation, but also imposes
filter regularity by bases truncation. Through extensive experiments, we
consistently observe that DCFNet maintains accuracy for image classification
tasks with a significant reduction of model parameters, particularly with
Fourier-Bessel (FB) bases, and even with random bases. Theoretically, we
analyze the representation stability of DCFNet with respect to input
variations, and prove representation stability under generic assumptions on the
expansion coefficients. The analysis is consistent with the empirical
observations.
Free-breathing cardiac MRI using bandlimited manifold modelling
Authors: Sunrita Poddar, Yasir Mohsin, Deidra Ansah, Bijoy Thattaliyath, Ravi Ashwath, Mathews Jacob
Date: 2018-02-24 00:00:00

We introduce a novel bandlimited manifold framework and an algorithm to
recover freebreathing and ungated cardiac MR images from highly undersampled
measurements. The image frames in the free breathing and ungated dataset are
assumed to be points on a bandlimited manifold. We introduce a novel kernel
low-rank algorithm to estimate the manifold structure (Laplacian) from a
navigator-based acquisition scheme. The structure of the manifold is then used
to recover the images from highly undersampled measurements. A computationally
efficient algorithm, which relies on the bandlimited approximation of the
Laplacian matrix, is used to recover the images. The proposed scheme is
demonstrated on several patients with different breathing patterns and cardiac
rates, without requiring the need for manually tuning the reconstruction
parameters in each case. The proposed scheme enabled the recovery of
free-breathing and ungated data, providing reconstructions that are
qualitatively similar to breath-held scans performed on the same patients. This
shows the potential of the technique as a clinical protocol for free-breathing
cardiac scans.
Scalable Private Learning with PATE
Authors: Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, Úlfar Erlingsson
Date: 2018-02-24 00:00:00
Proceeding: ICLR 2018 1

Models and examples built with TensorFlow
Product Kernel Interpolation for Scalable Gaussian Processes
Authors: Jacob R. Gardner, Geoff Pleiss, Ruihan Wu, Kilian Q. Weinberger, Andrew Gordon Wilson
Date: 2018-02-24 00:00:00
Tasks: Gaussian Processes

Recent work shows that inference for Gaussian processes can be performed
efficiently using iterative methods that rely only on matrix-vector
multiplications (MVMs). Structured Kernel Interpolation (SKI) exploits these
techniques by deriving approximate kernels with very fast MVMs. Unfortunately,
such strategies suffer badly from the curse of dimensionality. We develop a new
technique for MVM based learning that exploits product kernel structure. We
demonstrate that this technique is broadly applicable, resulting in linear
rather than exponential runtime with dimension for SKI, as well as
state-of-the-art asymptotic complexity for multi-task GPs.
Synthetic and Natural Noise Both Break Neural Machine Translation
Authors: Yonatan Belinkov, Yonatan Bisk
Date: 2017-11-06 00:00:00
Proceeding: ICLR 2018 1
Tasks: Machine Translation, NMT, Translation

Character-based neural machine translation (NMT) models alleviate
out-of-vocabulary issues, learn morphology, and move us closer to completely
end-to-end translation systems. Unfortunately, they are also very brittle and
easily falter when presented with noisy data. In this paper, we confront NMT
models with synthetic and natural sources of noise. We find that
state-of-the-art models fail to translate even moderately noisy texts that
humans have no trouble comprehending. We explore two approaches to increase
model robustness: structure-invariant word representations and robust training
on noisy texts. We find that a model based on a character convolutional neural
network is able to simultaneously learn representations robust to multiple
kinds of noise.
Dimensionally Tight Bounds for Second-Order Hamiltonian Monte Carlo
Authors: Oren Mangoubi, Nisheeth K. Vishnoi
Date: 2018-02-24 00:00:00
Proceeding: NeurIPS 2018 12

Hamiltonian Monte Carlo (HMC) is a widely deployed method to sample from
high-dimensional distributions in Statistics and Machine learning. HMC is known
to run very efficiently in practice and its popular second-order "leapfrog"
implementation has long been conjectured to run in $d^{1/4}$ gradient
evaluations. Here we show that this conjecture is true when sampling from
strongly log-concave target distributions that satisfy a weak third-order
regularity property associated with the input data. Our regularity condition is
weaker than the Lipschitz Hessian property and allows us to show faster
convergence bounds for a much larger class of distributions than would be
possible with the usual Lipschitz Hessian constant alone. Important
distributions that satisfy our regularity condition include posterior
distributions used in Bayesian logistic regression for which the data satisfies
an "incoherence" property. Our result compares favorably with the best
available bounds for the class of strongly log-concave distributions, which
grow like $d^{{1}/{2}}$ gradient evaluations with the dimension. Moreover, our
simulations on synthetic data suggest that, when our regularity condition is
satisfied, leapfrog HMC performs better than its competitors -- both in terms
of accuracy and in terms of the number of gradient evaluations it requires.
Improving Recall of In Situ Sequencing by Self-Learned Features and a Graphical Model
Authors: Gabriele Partel, Giorgia Milli, Carolina Wählby
Date: 2018-02-24 00:00:00
Tasks: Sensitivity

Image-based sequencing of mRNA makes it possible to see where in a tissue
sample a given gene is active, and thus discern large numbers of different cell
types in parallel. This is crucial for gaining a better understanding of tissue
development and disease such as cancer. Signals are collected over multiple
staining and imaging cycles, and signal density together with noise makes
signal decoding challenging. Previous approaches have led to low signal recall
in efforts to maintain high sensitivity. We propose an approach where signal
candidates are generously included, and true-signal probability at the cycle
level is self-learned using a convolutional neural network. Signal candidates
and probability predictions are thereafter fed into a graphical model searching
for signal candidates across sequencing cycles. The graphical model combines
intensity, probability and spatial distance to find optimal paths representing
decoded signal sequences. We evaluate our approach in relation to
state-of-the-art, and show that we increase recall by $27\%$ at maintained
sensitivity. Furthermore, visual examination shows that most of the now
correctly resolved signals were previously lost due to high signal density.
Thus, the proposed approach has the potential to significantly improve further
analysis of spatial statistics in in situ sequencing experiments.
Cross-lingual Abstract Meaning Representation Parsing
Authors: Marco Damonte, Shay B. Cohen
Date: 2017-04-14 00:00:00
Proceeding: NAACL 2018 6
Tasks: Abstract Meaning Representation

Abstract Meaning Representation (AMR) annotation efforts have mostly focused
on English. In order to train parsers on other languages, we propose a method
based on annotation projection, which involves exploiting annotations in a
source language and a parallel corpus of the source language and a target
language. Using English as the source language, we show promising results for
Italian, Spanish, German and Chinese as target languages. Besides evaluating
the target parsers on non-gold datasets, we further propose an evaluation
method that exploits the English gold annotations and does not require access
to gold annotations for the target languages. This is achieved by inverting the
projection process: a new English parser is learned from the target language
parser and evaluated on the existing English gold standard.
N-GCN: Multi-scale Graph Convolution for Semi-supervised Node Classification
Authors: Sami Abu-El-Haija, Amol Kapoor, Bryan Perozzi, Joonseok Lee
Date: 2018-02-24 00:00:00
Tasks: General Classification, Node Classification

Graph Convolutional Networks (GCNs) have shown significant improvements in
semi-supervised learning on graph-structured data. Concurrently, unsupervised
learning of graph embeddings has benefited from the information contained in
random walks. In this paper, we propose a model: Network of GCNs (N-GCN), which
marries these two lines of work. At its core, N-GCN trains multiple instances
of GCNs over node pairs discovered at different distances in random walks, and
learns a combination of the instance outputs which optimizes the classification
objective. Our experiments show that our proposed N-GCN model improves
state-of-the-art baselines on all of the challenging node classification tasks
we consider: Cora, Citeseer, Pubmed, and PPI. In addition, our proposed method
has other desirable properties, including generalization to recently proposed
semi-supervised learning methods such as GraphSAGE, allowing us to propose
N-SAGE, and resilience to adversarial input perturbations.
A Block-wise, Asynchronous and Distributed ADMM Algorithm for General Form Consensus Optimization
Authors: Rui Zhu, Di Niu, Zongpeng Li
Date: 2018-02-24 00:00:00
Tasks: Form

Many machine learning models, including those with non-smooth regularizers,
can be formulated as consensus optimization problems, which can be solved by
the alternating direction method of multipliers (ADMM). Many recent efforts
have been made to develop asynchronous distributed ADMM to handle large amounts
of training data. However, all existing asynchronous distributed ADMM methods
are based on full model updates and require locking all global model parameters
to handle concurrency, which essentially serializes the updates from different
workers. In this paper, we present a novel block-wise, asynchronous and
distributed ADMM algorithm, which allows different blocks of model parameters
to be updated in parallel. The lock-free block-wise algorithm may greatly
speedup sparse optimization problems, a common scenario in reality, in which
most model updates only modify a subset of all decision variables. We
theoretically prove the convergence of our proposed algorithm to stationary
points for non-convex general form consensus problems with possibly non-smooth
regularizers. We implement the proposed ADMM algorithm on the Parameter Server
framework and demonstrate its convergence and near-linear speedup performance
as the number of workers increases.
Introduction to the SP theory of intelligence
Authors: J. Gerard Wolff
Date: 2018-02-24 00:00:00

This article provides a brief introduction to the "Theory of Intelligence"
and its realisation in the "SP Computer Model". The overall goal of the SP
programme of research, in accordance with long-established principles in
science, has been the simplification and integration of observations and
concepts across artificial intelligence, mainstream computing, mathematics, and
human learning, perception, and cognition. In broad terms, the SP system is a
brain-like system that takes in "New" information through its senses and stores
some or all of it as "Old" information. A central idea in the system is the
powerful concept of "SP-multiple-alignment", borrowed and adapted from
bioinformatics. This the key to the system's versatility in aspects of
intelligence, in the representation of diverse kinds of knowledge, and in the
seamless integration of diverse aspects of intelligence and diverse kinds of
knowledge, in any combination. There are many potential benefits and
applications of the SP system. It is envisaged that the system will be
developed as the "SP Machine", which will initially be a software virtual
machine, hosted on a high-performance computer, a vehicle for further research
and a step towards the development of an industrial-strength SP Machine.
Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization
Authors: Rui Zhu, Di Niu, Zongpeng Li
Date: 2018-02-24 00:00:00

We study stochastic algorithms for solving nonconvex optimization problems
with a convex yet possibly nonsmooth regularizer, which find wide applications
in many practical machine learning applications. However, compared to
asynchronous parallel stochastic gradient descent (AsynSGD), an algorithm
targeting smooth optimization, the understanding of the behavior of stochastic
algorithms for nonsmooth regularized optimization problems is limited,
especially when the objective function is nonconvex. To fill this theoretical
gap, in this paper, we propose and analyze asynchronous parallel stochastic
proximal gradient (Asyn-ProxSGD) methods for nonconvex problems. We establish
an ergodic convergence rate of $O(1/\sqrt{K})$ for the proposed Asyn-ProxSGD,
where $K$ is the number of updates made on the model, matching the convergence
rate currently known for AsynSGD (for smooth problems). To our knowledge, this
is the first work that provides convergence rates of asynchronous parallel
ProxSGD algorithms for nonconvex problems. Furthermore, our results are also
the first to show the convergence of any stochastic proximal methods without
assuming an increasing batch size or the use of additional variance reduction
techniques. We implement the proposed algorithms on Parameter Server and
demonstrate its convergence behavior and near-linear speedup, as the number of
workers increases, on two real-world datasets.
Tracking of enriched dialog states for flexible conversational information access
Authors: Yinpei Dai, Zhijian Ou, Dawei Ren, Pengfei Yu
Date: 2017-11-09 00:00:00
Tasks: Conversational Information Access, dialog state tracking, slot-filling, Slot Filling

Dialog state tracking (DST) is a crucial component in a task-oriented dialog
system for conversational information access. A common practice in current
dialog systems is to define the dialog state by a set of slot-value pairs. Such
representation of dialog states and the slot-filling based DST have been widely
employed, but suffer from three drawbacks. (1) The dialog state can contain
only a single value for a slot, and (2) can contain only users' affirmative
preference over the values for a slot. (3) Current task-based dialog systems
mainly focus on the searching task, while the enquiring task is also very
common in practice. The above observations motivate us to enrich current
representation of dialog states and collect a brand new dialog dataset about
movies, based upon which we build a new DST, called enriched DST (EDST), for
flexible accessing movie information. The EDST supports the searching task, the
enquiring task and their mixed task. We show that the new EDST method not only
achieves good results on Iqiyi dataset, but also outperforms other
state-of-the-art DST methods on the traditional dialog datasets, WOZ2.0 and
DSTC2.
Deep learning for conifer/deciduous classification of airborne LiDAR 3D point clouds representing individual trees
Authors: Hamid Hamraz, Nathan B. Jacobs, Marco A. Contreras, Chase H. Clark
Date: 2018-02-24 00:00:00
Tasks: Classification, Data Augmentation, General Classification

The purpose of this study was to investigate the use of deep learning for
coniferous/deciduous classification of individual trees from airborne LiDAR
data. To enable efficient processing by a deep convolutional neural network
(CNN), we designed two discrete representations using leaf-off and leaf-on
LiDAR data: a digital surface model with four channels (DSMx4) and a set of
four 2D views (4x2D). A training dataset of labeled tree crowns was generated
via segmentation of tree crowns, followed by co-registration with field data.
Potential mislabels due to GPS error or tree leaning were corrected using a
statistical ensemble filtering procedure. Because the training data was heavily
unbalanced (~8% conifers), we trained an ensemble of CNNs on random balanced
sub-samples of augmented data (180 rotational variations per instance). The
4x2D representation yielded similar classification accuracies to the DSMx4
representation (~82% coniferous and ~90% deciduous) while converging faster.
The data augmentation improved the classification accuracies, but more real
training instances (especially coniferous) likely results in much stronger
improvements. Leaf-off LiDAR data were the primary source of useful
information, which is likely due to the perennial nature of coniferous foliage.
LiDAR intensity values also proved to be useful, but normalization yielded no
significant improvements. Lastly, the classification accuracies of overstory
trees (~90%) were more balanced than those of understory trees (~90% deciduous
and ~65% coniferous), which is likely due to the incomplete capture of
understory tree crowns via airborne LiDAR. Automatic derivation of optimal
features via deep learning provide the opportunity for remarkable improvements
in prediction tasks where captured data are not friendly to human visual system
- likely yielding sub-optimal human-designed features.
Incremental Adversarial Domain Adaptation for Continually Changing Environments
Authors: Markus Wulfmeier, Alex Bewley, Ingmar Posner
Date: 2017-12-20 00:00:00
Tasks: Domain Adaptation, Generative Adversarial Network, Unsupervised Domain Adaptation

Continuous appearance shifts such as changes in weather and lighting
conditions can impact the performance of deployed machine learning models.
While unsupervised domain adaptation aims to address this challenge, current
approaches do not utilise the continuity of the occurring shifts. In
particular, many robotics applications exhibit these conditions and thus
facilitate the potential to incrementally adapt a learnt model over minor
shifts which integrate to massive differences over time. Our work presents an
adversarial approach for lifelong, incremental domain adaptation which benefits
from unsupervised alignment to a series of intermediate domains which
successively diverge from the labelled source domain. We empirically
demonstrate that our incremental approach improves handling of large appearance
changes, e.g. day to night, on a traversable-path segmentation task compared
with a direct, single alignment step approach. Furthermore, by approximating
the feature distribution for the source domain with a generative adversarial
network, the deployment module can be rendered fully independent of retaining
potentially large amounts of the related source training data for only a minor
reduction in performance.
Coding-theorem Like Behaviour and Emergence of the Universal Distribution from Resource-bounded Algorithmic Probability
Authors: Hector Zenil, Liliana Badillo, Santiago Hernández-Orozco, Francisco Hernández-Quiroz
Date: 2017-11-06 00:00:00

Previously referred to as `miraculous' in the scientific literature because
of its powerful properties and its wide application as optimal solution to the
problem of induction/inference, (approximations to) Algorithmic Probability
(AP) and the associated Universal Distribution are (or should be) of the
greatest importance in science. Here we investigate the emergence, the rates of
emergence and convergence, and the Coding-theorem like behaviour of AP in
Turing-subuniversal models of computation. We investigate empirical
distributions of computing models in the Chomsky hierarchy. We introduce
measures of algorithmic probability and algorithmic complexity based upon
resource-bounded computation, in contrast to previously thoroughly investigated
distributions produced from the output distribution of Turing machines. This
approach allows for numerical approximations to algorithmic
(Kolmogorov-Chaitin) complexity-based estimations at each of the levels of a
computational hierarchy. We demonstrate that all these estimations are
correlated in rank and that they converge both in rank and values as a function
of computational power, despite fundamental differences between computational
models. In the context of natural processes that operate below the Turing
universal level because of finite resources and physical degradation, the
investigation of natural biases stemming from algorithmic rules may shed light
on the distribution of outcomes. We show that up to 60\% of the
simplicity/complexity bias in distributions produced even by the weakest of the
computational models can be accounted for by Algorithmic Probability in its
approximation to the Universal Distribution.
One Big Net For Everything
Authors: Juergen Schmidhuber
Date: 2018-02-24 00:00:00
Tasks: Reinforcement Learning

I apply recent work on "learning to think" (2015) and on PowerPlay (2011) to
the incremental training of an increasingly general problem solver, continually
learning to solve new tasks without forgetting previous skills. The problem
solver is a single recurrent neural network (or similar general purpose
computer) called ONE. ONE is unusual in the sense that it is trained in various
ways, e.g., by black box optimization / reinforcement learning / artificial
evolution as well as supervised / unsupervised learning. For example, ONE may
learn through neuroevolution to control a robot through environment-changing
actions, and learn through unsupervised gradient descent to predict future
inputs and vector-valued reward signals as suggested in 1990. User-given tasks
can be defined through extra goal-defining input patterns, also proposed in
1990. Suppose ONE has already learned many skills. Now a copy of ONE can be
re-trained to learn a new skill, e.g., through neuroevolution without a
teacher. Here it may profit from re-using previously learned subroutines, but
it may also forget previous skills. Then ONE is retrained in PowerPlay style
(2011) on stored input/output traces of (a) ONE's copy executing the new skill
and (b) previous instances of ONE whose skills are still considered worth
memorizing. Simultaneously, ONE is retrained on old traces (even those of
unsuccessful trials) to become a better predictor, without additional expensive
interaction with the enviroment. More and more control and prediction skills
are thus collapsed into ONE, like in the chunker-automatizer system of the
neural history compressor (1991). This forces ONE to relate partially analogous
skills (with shared algorithmic information) to each other, creating common
subroutines in form of shared subnetworks of ONE, to greatly speed up
subsequent learning of additional, novel but algorithmically related skills.