pwc-archive/papers-with-abstracts huggingface dataset:
Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification
Authors: Woong Bae, Jaejun Yoo, Jong Chul Ye
Date: 2016-11-19 00:00:00
Tasks: Color Image Denoising, Denoising, Image Restoration, Image Super-Resolution, Super-Resolution

The latest deep learning approaches perform better than the state-of-the-art
signal processing approaches in various image restoration tasks. However, if an
image contains many patterns and structures, the performance of these CNNs is
still inferior. To address this issue, here we propose a novel feature space
deep residual learning algorithm that outperforms the existing residual
learning. The main idea is originated from the observation that the performance
of a learning algorithm can be improved if the input and/or label manifolds can
be made topologically simpler by an analytic mapping to a feature space. Our
extensive numerical studies using denoising experiments and NTIRE single-image
super-resolution (SISR) competition demonstrate that the proposed feature space
residual learning outperforms the existing state-of-the-art approaches.
Moreover, our algorithm was ranked third in NTIRE competition with 5-10 times
faster computational time compared to the top ranked teams. The source code is
available on page : https://github.com/iorism/CNN.git
Meta Networks
Authors: Tsendsuren Munkhdalai, Hong Yu
Date: 2017-03-02 00:00:00
Proceeding: ICML 2017 8
Tasks: Continual Learning, Meta-Learning

Neural networks have been successfully applied in applications with a large
amount of labeled data. However, the task of rapid generalization on new
concepts with small training data while preserving performances on previously
learned ones still presents a significant challenge to neural network models.
In this work, we introduce a novel meta learning method, Meta Networks
(MetaNet), that learns a meta-level knowledge across tasks and shifts its
inductive biases via fast parameterization for rapid generalization. When
evaluated on Omniglot and Mini-ImageNet benchmarks, our MetaNet models achieve
a near human-level performance and outperform the baseline approaches by up to
6% accuracy. We demonstrate several appealing properties of MetaNet relating to
generalization and continual learning.
A GPU-Outperforming FPGA Accelerator Architecture for Binary Convolutional Neural Networks
Authors: Yixing Li, Zichuan Liu, Kai Xu, Hao Yu, Fengbo Ren
Date: 2017-02-20 00:00:00
Tasks: GPU

FPGA-based hardware accelerators for convolutional neural networks (CNNs)
have obtained great attentions due to their higher energy efficiency than GPUs.
However, it is challenging for FPGA-based solutions to achieve a higher
throughput than GPU counterparts. In this paper, we demonstrate that FPGA
acceleration can be a superior solution in terms of both throughput and energy
efficiency when a CNN is trained with binary constraints on weights and
activations. Specifically, we propose an optimized FPGA accelerator
architecture tailored for bitwise convolution and normalization that features
massive spatial parallelism with deep pipelines stages. A key advantage of the
FPGA accelerator is that its performance is insensitive to data batch size,
while the performance of GPU acceleration varies largely depending on the batch
size of the data. Experiment results show that the proposed accelerator
architecture for binary CNNs running on a Virtex-7 FPGA is 8.3x faster and 75x
more energy-efficient than a Titan X GPU for processing online individual
requests in small batch sizes. For processing static data in large batch sizes,
the proposed solution is on a par with a Titan X GPU in terms of throughput
while delivering 9.5x higher energy efficiency.
Nuclear Discrepancy for Active Learning
Authors: Tom J. Viering, Jesse H. Krijthe, Marco Loog
Date: 2017-06-08 00:00:00
Tasks: Active Learning, Generalization Bounds

Active learning algorithms propose which unlabeled objects should be queried
for their labels to improve a predictive model the most. We study active
learners that minimize generalization bounds and uncover relationships between
these bounds that lead to an improved approach to active learning. In
particular we show the relation between the bound of the state-of-the-art
Maximum Mean Discrepancy (MMD) active learner, the bound of the Discrepancy,
and a new and looser bound that we refer to as the Nuclear Discrepancy bound.
We motivate this bound by a probabilistic argument: we show it considers
situations which are more likely to occur. Our experiments indicate that active
learning using the tightest Discrepancy bound performs the worst in terms of
the squared loss. Overall, our proposed loosest Nuclear Discrepancy
generalization bound performs the best. We confirm our probabilistic argument
empirically: the other bounds focus on more pessimistic scenarios that are
rarer in practice. We conclude that tightness of bounds is not always of main
importance and that active learning methods should concentrate on realistic
scenarios in order to improve performance.
CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks
Authors: Yuan-Fang Li, Ardavan Pedram
Date: 2017-06-01 00:00:00

Accelerating the inference of a trained DNN is a well studied subject. In
this paper we switch the focus to the training of DNNs. The training phase is
compute intensive, demands complicated data communication, and contains
multiple levels of data dependencies and parallelism. This paper presents an
algorithm/architecture space exploration of efficient accelerators to achieve
better network convergence rates and higher energy efficiency for training
DNNs. We further demonstrate that an architecture with hierarchical support for
collective communication semantics provides flexibility in training various
networks performing both stochastic and batched gradient descent based
techniques. Our results suggest that smaller networks favor non-batched
techniques while performance for larger networks is higher using batched
operations. At 45nm technology, CATERPILLAR achieves performance efficiencies
of 177 GFLOPS/W at over 80% utilization for SGD training on small networks and
211 GFLOPS/W at over 90% utilization for pipelined SGD/CP training on larger
networks using a total area of 103.2 mm$^2$ and 178.9 mm$^2$ respectively.
Making Neural QA as Simple as Possible but not Simpler
Authors: Dirk Weissenborn, Georg Wiese, Laura Seiffe
Date: 2017-03-14 00:00:00
Proceeding: CONLL 2017 8
Tasks: Question Answering, Reading Comprehension

Recent development of large-scale question answering (QA) datasets triggered
a substantial amount of research into end-to-end neural architectures for QA.
Increasingly complex systems have been conceived without comparison to simpler
neural baseline systems that would justify their complexity. In this work, we
propose a simple heuristic that guides the development of neural baseline
systems for the extractive QA task. We find that there are two ingredients
necessary for building a high-performing neural QA system: first, the awareness
of question words while processing the context and second, a composition
function that goes beyond simple bag-of-words modeling, such as recurrent
neural networks. Our results show that FastQA, a system that meets these two
requirements, can achieve very competitive performance compared with existing
models. We argue that this surprising finding puts results of previous systems
and the complexity of recent QA datasets into perspective.
Clustering with t-SNE, provably
Authors: George C. Linderman, Stefan Steinerberger
Date: 2017-06-08 00:00:00
Tasks: Clustering

t-distributed Stochastic Neighborhood Embedding (t-SNE), a clustering and
visualization method proposed by van der Maaten & Hinton in 2008, has rapidly
become a standard tool in a number of natural sciences. Despite its
overwhelming success, there is a distinct lack of mathematical foundations and
the inner workings of the algorithm are not well understood. The purpose of
this paper is to prove that t-SNE is able to recover well-separated clusters;
more precisely, we prove that t-SNE in the `early exaggeration' phase, an
optimization technique proposed by van der Maaten & Hinton (2008) and van der
Maaten (2014), can be rigorously analyzed. As a byproduct, the proof suggests
novel ways for setting the exaggeration parameter $\alpha$ and step size $h$.
Numerical examples illustrate the effectiveness of these rules: in particular,
the quality of embedding of topological structures (e.g. the swiss roll)
improves. We also discuss a connection to spectral clustering methods.
ToxTrac: a fast and robust software for tracking organisms
Authors: Alvaro Rodriquez, Hanqing Zhang, Jonatan Klaminder, Tomas Brodin, Patrik L. Andersson, Magnus Andersson
Date: 2017-06-08 00:00:00
Tasks: Camera Calibration, distortion correction

1. Behavioral analysis based on video recording is becoming increasingly
popular within research fields such as; ecology, medicine, ecotoxicology, and
toxicology. However, the programs available to analyze the data, which are;
free of cost, user-friendly, versatile, robust, fast and provide reliable
statistics for different organisms (invertebrates, vertebrates and mammals) are
significantly limited.
  2. We present an automated open-source executable software (ToxTrac) for
image-based tracking that can simultaneously handle several organisms monitored
in a laboratory environment. We compare the performance of ToxTrac with current
accessible programs on the web.
  3. The main advantages of ToxTrac are: i) no specific knowledge of the
geometry of the tracked bodies is needed; ii) processing speed, ToxTrac can
operate at a rate >25 frames per second in HD videos using modern desktop
computers; iii) simultaneous tracking of multiple organisms in multiple arenas;
iv) integrated distortion correction and camera calibration; v) robust against
false positives; vi) preservation of individual identification if crossing
occurs; vii) useful statistics and heat maps in real scale are exported in:
image, text and excel formats.
  4. ToxTrac can be used for high speed tracking of insects, fish, rodents or
other species, and provides useful locomotor information. We suggest using
ToxTrac for future studies of animal behavior independent of research area.
Download ToxTrac here: https://toxtrac.sourceforge.io
Pain-Free Random Differential Privacy with Sensitivity Sampling
Authors: Benjamin I. P. Rubinstein, Francesco Aldà
Date: 2017-06-08 00:00:00
Proceeding: ICML 2017 8
Tasks: Sensitivity

Popular approaches to differential privacy, such as the Laplace and
exponential mechanisms, calibrate randomised smoothing through global
sensitivity of the target non-private function. Bounding such sensitivity is
often a prohibitively complex analytic calculation. As an alternative, we
propose a straightforward sampler for estimating sensitivity of non-private
mechanisms. Since our sensitivity estimates hold with high probability, any
mechanism that would be $(\epsilon,\delta)$-differentially private under
bounded global sensitivity automatically achieves
$(\epsilon,\delta,\gamma)$-random differential privacy (Hall et al., 2012),
without any target-specific calculations required. We demonstrate on worked
example learners how our usable approach adopts a naturally-relaxed privacy
guarantee, while achieving more accurate releases even for non-private
functions that are black-box computer programs.
Surprise Search for Evolutionary Divergence
Authors: Daniele Gravina, Antonios Liapis, Georgios N. Yannakakis
Date: 2017-06-08 00:00:00

Inspired by the notion of surprise for unconventional discovery we introduce
a general search algorithm we name surprise search as a new method of
evolutionary divergent search. Surprise search is grounded in the divergent
search paradigm and is fabricated within the principles of evolutionary search.
The algorithm mimics the self-surprise cognitive process and equips
evolutionary search with the ability to seek for solutions that deviate from
the algorithm's expected behaviour. The predictive model of expected solutions
is based on historical trails of where the search has been and local
information about the search space. Surprise search is tested extensively in a
robot maze navigation task: experiments are held in four authored deceptive
mazes and in 60 generated mazes and compared against objective-based
evolutionary search and novelty search. The key findings of this study reveal
that surprise search is advantageous compared to the other two search
processes. In particular, it outperforms objective search and it is as
efficient as novelty search in all tasks examined. Most importantly, surprise
search is faster, on average, and more robust in solving the navigation problem
compared to any other algorithm examined. Finally, our analysis reveals that
surprise search explores the behavioural space more extensively and yields
higher population diversity compared to novelty search. What distinguishes
surprise search from other forms of divergent search, such as the search for
novelty, is its ability to diverge not from earlier and seen solutions but
rather from predicted and unseen points in the domain considered.
The Algorithmic Inflection of Russian and Generation of Grammatically Correct Text
Authors: T. M. Sadykov, T. A. Zhukov
Date: 2017-06-08 00:00:00

We present a deterministic algorithm for Russian inflection. This algorithm
is implemented in a publicly available web-service www.passare.ru which
provides functions for inflection of single words, word matching and synthesis
of grammatically correct Russian text. The inflectional functions have been
tested against the annotated corpus of Russian language OpenCorpora.
A Long Short-Term Memory Recurrent Neural Network Framework for Network Traffic Matrix Prediction
Authors: Abdelhadi Azzouni, Guy Pujolle
Date: 2017-05-16 00:00:00
Tasks: Management, Temporal Sequences, Time Series, Time Series Analysis

Network Traffic Matrix (TM) prediction is defined as the problem of
estimating future network traffic from the previous and achieved network
traffic data. It is widely used in network planning, resource management and
network security. Long Short-Term Memory (LSTM) is a specific recurrent neural
network (RNN) architecture that is well-suited to learn from experience to
classify, process and predict time series with time lags of unknown size. LSTMs
have been shown to model temporal sequences and their long-range dependencies
more accurately than conventional RNNs. In this paper, we propose a LSTM RNN
framework for predicting short and long term Traffic Matrix (TM) in large
networks. By validating our framework on real-world data from GEANT network, we
show that our LSTM models converge quickly and give state of the art TM
prediction performance for relatively small sized models.
Collaborative Filtering with Side Information: a Gaussian Process Perspective
Authors: Hyunjik Kim, Xiaoyu Lu, Seth Flaxman, Yee Whye Teh
Date: 2016-05-23 00:00:00
Tasks: Collaborative Filtering, regression

We tackle the problem of collaborative filtering (CF) with side information,
through the lens of Gaussian Process (GP) regression. Driven by the idea of
using the kernel to explicitly model user-item similarities, we formulate the
GP in a way that allows the incorporation of low-rank matrix factorisation,
arriving at our model, the Tucker Gaussian Process (TGP). Consequently, TGP
generalises classical Bayesian matrix factorisation models, and goes beyond
them to give a natural and elegant method for incorporating side information,
giving enhanced predictive performance for CF problems. Moreover we show that
it is a novel model for regression, especially well-suited to grid-structured
data and problems where the dependence on covariates is close to being
separable.
Responsible Autonomy
Authors: Virginia Dignum
Date: 2017-06-08 00:00:00
Tasks: Ethics

As intelligent systems are increasingly making decisions that directly affect
society, perhaps the most important upcoming research direction in AI is to
rethink the ethical implications of their actions. Means are needed to
integrate moral, societal and legal values with technological developments in
AI, both during the design process as well as part of the deliberation
algorithms employed by these systems. In this paper, we describe leading ethics
theories and propose alternative ways to ensure ethical behavior by artificial
systems. Given that ethics are dependent on the socio-cultural context and are
often only implicit in deliberation processes, methodologies are needed to
elicit the values held by designers and stakeholders, and to make these
explicit leading to better understanding and trust on artificial autonomous
systems.
Unlocking the Potential of Simulators: Design with RL in Mind
Authors: Rika Antonova, Silvia Cruciani
Date: 2017-06-08 00:00:00
Tasks: Decision Making, Friction, Reinforcement Learning, Reinforcement Learning (RL), Sequential Decision Making

Using Reinforcement Learning (RL) in simulation to construct policies useful
in real life is challenging. This is often attributed to the sequential
decision making aspect: inaccuracies in simulation accumulate over multiple
steps, hence the simulated trajectories diverge from what would happen in
reality.
  In our work we show the need to consider another important aspect: the
mismatch in simulating control. We bring attention to the need for modeling
control as well as dynamics, since oversimplifying assumptions about applying
actions of RL policies could make the policies fail on real-world systems.
  We design a simulator for solving a pivoting task (of interest in Robotics)
and demonstrate that even a simple simulator designed with RL in mind
outperforms high-fidelity simulators when it comes to learning a policy that is
to be deployed on a real robotic system. We show that a phenomenon that is hard
to model - friction - could be exploited successfully, even when RL is
performed using a simulator with a simple dynamics and noise model. Hence, we
demonstrate that as long as the main sources of uncertainty are identified, it
could be possible to learn policies applicable to real systems even using a
simple simulator.
  RL-compatible simulators could open the possibilities for applying a wide
range of RL algorithms in various fields. This is important, since currently
data sparsity in fields like healthcare and education frequently forces
researchers and engineers to only consider sample-efficient RL approaches.
Successful simulator-aided RL could increase flexibility of experimenting with
RL algorithms and help applying RL policies to real-world settings in fields
where data is scarce. We believe that lessons learned in Robotics could help
other fields design RL-compatible simulators, so we summarize our experience
and conclude with suggestions.
Context encoders as a simple but powerful extension of word2vec
Authors: Franziska Horn
Date: 2017-06-08 00:00:00
Proceeding: WS 2017 8
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Word Embeddings

With a simple architecture and the ability to learn meaningful word
embeddings efficiently from texts containing billions of words, word2vec
remains one of the most popular neural language models used today. However, as
only a single embedding is learned for every word in the vocabulary, the model
fails to optimally represent words with multiple meanings. Additionally, it is
not possible to create embeddings for new (out-of-vocabulary) words on the
spot. Based on an intuitive interpretation of the continuous bag-of-words
(CBOW) word2vec model's negative sampling training objective in terms of
predicting context based similarities, we motivate an extension of the model we
call context encoders (ConEc). By multiplying the matrix of trained word2vec
embeddings with a word's average context vector, out-of-vocabulary (OOV)
embeddings and representations for a word with multiple meanings can be created
based on the word's local contexts. The benefits of this approach are
illustrated by using these word embeddings as features in the CoNLL 2003 named
entity recognition (NER) task.
The Generalized Cross Validation Filter
Authors: Giulio Bottegal, Gianluigi Pillonetto
Date: 2017-06-08 00:00:00
Tasks: State Estimation

Generalized cross validation (GCV) is one of the most important approaches
used to estimate parameters in the context of inverse problems and
regularization techniques. A notable example is the determination of the
smoothness parameter in splines. When the data are generated by a state space
model, like in the spline case, efficient algorithms are available to evaluate
the GCV score with complexity that scales linearly in the data set size.
However, these methods are not amenable to on-line applications since they rely
on forward and backward recursions. Hence, if the objective has been evaluated
at time $t-1$ and new data arrive at time t, then O(t) operations are needed to
update the GCV score. In this paper we instead show that the update cost is
$O(1)$, thus paving the way to the on-line use of GCV. This result is obtained
by deriving the novel GCV filter which extends the classical Kalman filter
equations to efficiently propagate the GCV score over time. We also illustrate
applications of the new filter in the context of state estimation and on-line
regularized linear system identification.
Consistency Results for Stationary Autoregressive Processes with Constrained Coefficients
Authors: Alessio Sancetta
Date: 2017-06-08 00:00:00

We consider stationary autoregressive processes with coefficients restricted
to an ellipsoid, which includes autoregressive processes with absolutely
summable coefficients. We provide consistency results under different norms for
the estimation of such processes using constrained and penalized estimators. As
an application we show some weak form of universal consistency. Simulations
show that directly including the constraint in the estimation can lead to more
robust results.
Where is my forearm? Clustering of body parts from simultaneous tactile and linguistic input using sequential mapping
Authors: Karla Stepanova, Matej Hoffmann, Zdenek Straka, Frederico B. Klein, Angelo Cangelosi, Michal Vavrecka
Date: 2017-06-08 00:00:00
Tasks: Clustering

Humans and animals are constantly exposed to a continuous stream of sensory
information from different modalities. At the same time, they form more
compressed representations like concepts or symbols. In species that use
language, this process is further structured by this interaction, where a
mapping between the sensorimotor concepts and linguistic elements needs to be
established. There is evidence that children might be learning language by
simply disambiguating potential meanings based on multiple exposures to
utterances in different contexts (cross-situational learning). In existing
models, the mapping between modalities is usually found in a single step by
directly using frequencies of referent and meaning co-occurrences. In this
paper, we present an extension of this one-step mapping and introduce a newly
proposed sequential mapping algorithm together with a publicly available Matlab
implementation. For demonstration, we have chosen a less typical scenario:
instead of learning to associate objects with their names, we focus on body
representations. A humanoid robot is receiving tactile stimulations on its
body, while at the same time listening to utterances of the body part names
(e.g., hand, forearm and torso). With the goal at arriving at the correct "body
categories", we demonstrate how a sequential mapping algorithm outperforms
one-step mapping. In addition, the effect of data set size and noise in the
linguistic input are studied.