pwc-archive/papers-with-abstracts huggingface dataset:
WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia
Authors: Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, David Berthelot
Date: 2016-08-11 00:00:00
Proceeding: ACL 2016 8
Tasks: Articles, Document Classification, General Classification, Natural Language Understanding, Question Answering

We present WikiReading, a large-scale natural language understanding task and
publicly-available dataset with 18 million instances. The task is to predict
textual values from the structured knowledge base Wikidata by reading the text
of the corresponding Wikipedia articles. The task contains a rich variety of
challenging classification and extraction sub-tasks, making it well-suited for
end-to-end models such as deep neural networks (DNNs). We compare various
state-of-the-art DNN-based architectures for document classification,
information extraction, and question answering. We find that models supporting
a rich answer space, such as word or character sequences, perform best. Our
best-performing model, a word-level sequence to sequence model with a mechanism
to copy out-of-vocabulary words, obtains an accuracy of 71.8%.
Anomaly detection and classification for streaming data using PDEs
Authors: Bilal Abbasi, Jeff Calder, Adam M. Oberman
Date: 2016-08-15 00:00:00
Tasks: Anomaly Detection, Classification, General Classification

Nondominated sorting, also called Pareto Depth Analysis (PDA), is widely used
in multi-objective optimization and has recently found important applications
in multi-criteria anomaly detection. Recently, a partial differential equation
(PDE) continuum limit was discovered for nondominated sorting leading to a very
fast approximate sorting algorithm called PDE-based ranking. We propose in this
paper a fast real-time streaming version of the PDA algorithm for anomaly
detection that exploits the computational advantages of PDE continuum limits.
Furthermore, we derive new PDE continuum limits for sorting points within their
nondominated layers and show how the new PDEs can be used to classify anomalies
based on which criterion was more significantly violated. We also prove
statistical convergence rates for PDE-based ranking, and present the results of
numerical experiments with both synthetic and real data.
Illuminant Estimation using Ensembles of Multivariate Regression Trees
Authors: Peter van Beek, R. Wayne Oldford
Date: 2017-03-15 00:00:00
Tasks: regression

White balancing is a fundamental step in the image processing pipeline. The
process involves estimating the chromaticity of the illuminant or light source
and using the estimate to correct the image to remove any color cast. Given the
importance of the problem, there has been much previous work on illuminant
estimation. Recently, an approach based on ensembles of univariate regression
trees that are fit using the squared-error loss function has been proposed and
shown to give excellent performance. In this paper, we show that a simpler and
more accurate ensemble model can be learned by (i) using multivariate
regression trees to take into account that the chromaticity components of the
illuminant are correlated and constrained, and (ii) fitting each tree by
directly minimizing a loss function of interest---such as recovery angular
error or reproduction angular error---rather than indirectly using the
squared-error loss function as a surrogate. We show empirically that overall
our method leads to improved performance on diverse image sets.
Neural Machine Translation in Linear Time
Authors: Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, Koray Kavukcuoglu
Date: 2016-10-31 00:00:00
Tasks: Decoder, Language Modelling, Machine Translation, Memorization, Translation

We present a novel neural network for processing sequences. The ByteNet is a
one-dimensional convolutional neural network that is composed of two parts, one
to encode the source sequence and the other to decode the target sequence. The
two network parts are connected by stacking the decoder on top of the encoder
and preserving the temporal resolution of the sequences. To address the
differing lengths of the source and the target, we introduce an efficient
mechanism by which the decoder is dynamically unfolded over the representation
of the encoder. The ByteNet uses dilation in the convolutional layers to
increase its receptive field. The resulting network has two core properties: it
runs in time that is linear in the length of the sequences and it sidesteps the
need for excessive memorization. The ByteNet decoder attains state-of-the-art
performance on character-level language modelling and outperforms the previous
best results obtained with recurrent networks. The ByteNet also achieves
state-of-the-art performance on character-to-character machine translation on
the English-to-German WMT translation task, surpassing comparable neural
translation models that are based on recurrent networks with attentional
pooling and run in quadratic time. We find that the latent alignment structure
contained in the representations reflects the expected alignment between the
tokens.
Deep Embedding Forest: Forest-based Serving with Deep Embedding Features
Authors: Jie Zhu, Ying Shan, JC Mao, Dong Yu, Holakou Rahmanian, Yi Zhang
Date: 2017-03-15 00:00:00

Deep Neural Networks (DNN) have demonstrated superior ability to extract high
level embedding vectors from low level features. Despite the success, the
serving time is still the bottleneck due to expensive run-time computation of
multiple layers of dense matrices. GPGPU, FPGA, or ASIC-based serving systems
require additional hardware that are not in the mainstream design of most
commercial applications. In contrast, tree or forest-based models are widely
adopted because of low serving cost, but heavily depend on carefully engineered
features. This work proposes a Deep Embedding Forest model that benefits from
the best of both worlds. The model consists of a number of embedding layers and
a forest/tree layer. The former maps high dimensional (hundreds of thousands to
millions) and heterogeneous low-level features to the lower dimensional
(thousands) vectors, and the latter ensures fast serving.
  Built on top of a representative DNN model called Deep Crossing, and two
forest/tree-based models including XGBoost and LightGBM, a two-step Deep
Embedding Forest algorithm is demonstrated to achieve on-par or slightly better
performance as compared with the DNN counterpart, with only a fraction of
serving time on conventional hardware. After comparing with a joint
optimization algorithm called partial fuzzification, also proposed in this
paper, it is concluded that the two-step Deep Embedding Forest has achieved
near optimal performance. Experiments based on large scale data sets (up to 1
billion samples) from a major sponsored search engine proves the efficacy of
the proposed model.
Recurrent Orthogonal Networks and Long-Memory Tasks
Authors: Mikael Henaff, Arthur Szlam, Yann Lecun
Date: 2016-02-22 00:00:00

Although RNNs have been shown to be powerful tools for processing sequential
data, finding architectures or optimization strategies that allow them to model
very long term dependencies is still an active area of research. In this work,
we carefully analyze two synthetic datasets originally outlined in (Hochreiter
and Schmidhuber, 1997) which are used to evaluate the ability of RNNs to store
information over many time steps. We explicitly construct RNN solutions to
these problems, and using these constructions, illuminate both the problems
themselves and the way in which RNNs store different types of information in
their hidden states. These constructions furthermore explain the success of
recent methods that specify unitary initializations or constraints on the
transition matrices.
A clever elimination strategy for efficient minimal solvers
Authors: Zuzana Kukelova, Joe Kileel, Bernd Sturmfels, Tomas Pajdla
Date: 2017-03-15 00:00:00
Proceeding: CVPR 2017 7

We present a new insight into the systematic generation of minimal solvers in
computer vision, which leads to smaller and faster solvers. Many minimal
problem formulations are coupled sets of linear and polynomial equations where
image measurements enter the linear equations only. We show that it is useful
to solve such systems by first eliminating all the unknowns that do not appear
in the linear equations and then extending solutions to the rest of unknowns.
This can be generalized to fully non-linear systems by linearization via
lifting. We demonstrate that this approach leads to more efficient solvers in
three problems of partially calibrated relative camera pose computation with
unknown focal length and/or radial distortion. Our approach also generates new
interesting constraints on the fundamental matrices of partially calibrated
cameras, which were not known before.
Exact Simulation of Noncircular or Improper Complex-Valued Stationary Gaussian Processes using Circulant Embedding
Authors: Adam M. Sykulski, Donald B. Percival
Date: 2016-05-17 00:00:00
Tasks: Gaussian Processes

This paper provides an algorithm for simulating improper (or noncircular)
complex-valued stationary Gaussian processes. The technique utilizes recently
developed methods for multivariate Gaussian processes from the circulant
embedding literature. The method can be performed in $\mathcal{O}(n\log_2 n)$
operations, where $n$ is the length of the desired sequence. The method is
exact, except when eigenvalues of prescribed circulant matrices are negative.
We evaluate the performance of the algorithm empirically, and provide a
practical example where the method is guaranteed to be exact for all $n$, with
an improper fractional Gaussian noise process.
InScript: Narrative texts annotated with script information
Authors: Ashutosh Modi, Tatjana Anikina, Simon Ostermann, Manfred Pinkal
Date: 2017-03-15 00:00:00
Proceeding: LREC 2016 5

This paper presents the InScript corpus (Narrative Texts Instantiating Script
structure). InScript is a corpus of 1,000 stories centered around 10 different
scenarios. Verbs and noun phrases are annotated with event and participant
types, respectively. Additionally, the text is annotated with coreference
information. The corpus shows rich lexical variation and will serve as a unique
resource for the study of the role of script knowledge in natural language
processing.
Transfer Learning for Melanoma Detection: Participation in ISIC 2017 Skin Lesion Classification Challenge
Authors: Dennis H. Murphree, Che Ngufor
Date: 2017-03-15 00:00:00
Tasks: General Classification, image-classification, Image Classification, Lesion Classification, Skin Lesion Classification, Transfer Learning

This manuscript describes our participation in the International Skin Imaging
Collaboration's 2017 Skin Lesion Analysis Towards Melanoma Detection
competition. We participated in Part 3: Lesion Classification. The two stated
goals of this binary image classification challenge were to distinguish between
(a) melanoma and (b) nevus and seborrheic keratosis, followed by distinguishing
between (a) seborrheic keratosis and (b) nevus and melanoma. We chose a deep
neural network approach with a transfer learning strategy, using a pre-trained
Inception V3 network as both a feature extractor to provide input for a
multi-layer perceptron as well as fine-tuning an augmented Inception network.
This approach yielded validation set AUC's of 0.84 on the second task and 0.76
on the first task, for an average AUC of 0.80. We joined the competition
unfortunately late, and we look forward to improving on these results.
Fast Genetic Algorithms
Authors: Benjamin Doerr, Huu Phuoc Le, Régis Makhmara, Ta Duy Nguyen
Date: 2017-03-09 00:00:00

For genetic algorithms using a bit-string representation of length~$n$, the
general recommendation is to take $1/n$ as mutation rate. In this work, we
discuss whether this is really justified for multimodal functions. Taking jump
functions and the $(1+1)$ evolutionary algorithm as the simplest example, we
observe that larger mutation rates give significantly better runtimes. For the
$\jump_{m,n}$ function, any mutation rate between $2/n$ and $m/n$ leads to a
speed-up at least exponential in $m$ compared to the standard choice.
  The asymptotically best runtime, obtained from using the mutation rate $m/n$
and leading to a speed-up super-exponential in $m$, is very sensitive to small
changes of the mutation rate. Any deviation by a small $(1 \pm \eps)$ factor
leads to a slow-down exponential in $m$. Consequently, any fixed mutation rate
gives strongly sub-optimal results for most jump functions.
  Building on this observation, we propose to use a random mutation rate
$\alpha/n$, where $\alpha$ is chosen from a power-law distribution. We prove
that the $(1+1)$ EA with this heavy-tailed mutation rate optimizes any
$\jump_{m,n}$ function in a time that is only a small polynomial (in~$m$)
factor above the one stemming from the optimal rate for this $m$.
  Our heavy-tailed mutation operator yields similar speed-ups (over the best
known performance guarantees) for the vertex cover problem in bipartite graphs
and the matching problem in general graphs.
  Following the example of fast simulated annealing, fast evolution strategies,
and fast evolutionary programming, we propose to call genetic algorithms using
a heavy-tailed mutation operator \emph{fast genetic algorithms}.
Texture segmentation with Fully Convolutional Networks
Authors: Vincent Andrearczyk, Paul F. Whelan
Date: 2017-03-15 00:00:00
Tasks: Segmentation, Texture Classification

In the last decade, deep learning has contributed to advances in a wide range
computer vision tasks including texture analysis. This paper explores a new
approach for texture segmentation using deep convolutional neural networks,
sharing important ideas with classic filter bank based texture segmentation
methods. Several methods are developed to train Fully Convolutional Networks to
segment textures in various applications. We show in particular that these
networks can learn to recognize and segment a type of texture, e.g. wood and
grass from texture recognition datasets (no training segmentation). We
demonstrate that Fully Convolutional Networks can learn from repetitive
patterns to segment a particular texture from a single image or even a part of
an image. We take advantage of these findings to develop a method that is
evaluated on a series of supervised and unsupervised experiments and improve
the state of the art on the Prague texture segmentation datasets.
Exploiting Spatio-Temporal Structure with Recurrent Winner-Take-All Networks
Authors: Eder Santana, Matthew Emigh, Pablo Zegers, Jose C. Principe
Date: 2016-10-31 00:00:00
Tasks: All, Object Recognition, Time Series, Time Series Analysis

We propose a convolutional recurrent neural network, with Winner-Take-All
dropout for high dimensional unsupervised feature learning in multi-dimensional
time series. We apply the proposedmethod for object recognition with temporal
context in videos and obtain better results than comparable methods in the
literature, including the Deep Predictive Coding Networks previously proposed
by Chalasani and Principe.Our contributions can be summarized as a scalable
reinterpretation of the Deep Predictive Coding Networks trained end-to-end with
backpropagation through time, an extension of the previously proposed
Winner-Take-All Autoencoders to sequences in time, and a new technique for
initializing and regularizing convolutional-recurrent neural networks.
Haploid-Diploid Evolutionary Algorithms
Authors: Larry Bull
Date: 2016-08-19 00:00:00
Tasks: Evolutionary Algorithms

This paper uses the recent idea that the fundamental haploid-diploid
lifecycle of eukaryotic organisms implements a rudimentary form of learning
within evolution. A general approach for evolutionary computation is here
derived that differs from all previous known work using diploid
representations. The primary role of recombination is also changed from that
previously considered in both natural and artificial evolution under the new
view. Using well-known abstract tuneable models it is shown that varying
fitness landscape ruggedness varies the benefit of the new approach.
Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback
Authors: Kory W. Mathewson, Patrick M. Pilarski
Date: 2017-03-03 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

This paper contributes a first study into how different human users deliver
simultaneous control and feedback signals during human-robot interaction. As
part of this work, we formalize and present a general interactive learning
framework for online cooperation between humans and reinforcement learning
agents. In many human-machine interaction settings, there is a growing gap
between the degrees-of-freedom of complex semi-autonomous systems and the
number of human control channels. Simple human control and feedback mechanisms
are required to close this gap and allow for better collaboration between
humans and machines on complex tasks. To better inform the design of concurrent
control and feedback interfaces, we present experimental results from a
human-robot collaborative domain wherein the human must simultaneously deliver
both control and feedback signals to interactively train an actor-critic
reinforcement learning robot. We compare three experimental conditions: 1)
human delivered control signals, 2) reward-shaping feedback signals, and 3)
simultaneous control and feedback. Our results suggest that subjects provide
less feedback when simultaneously delivering feedback and control signals and
that control signal quality is not significantly diminished. Our data suggest
that subjects may also modify when and how they provide feedback. Through
algorithmic development and tuning informed by this study, we expect
semi-autonomous actions of robotic agents can be better shaped by human
feedback, allowing for seamless collaboration and improved performance in
difficult interactive domains.
Fuzzy Rankings: Properties and Applications
Authors: Jiří Mazurek
Date: 2017-03-15 00:00:00
Tasks: Decision Making

In practice, a ranking of objects with respect to given set of criteria is of
considerable importance. However, due to lack of knowledge, information of time
pressure, decision makers might not be able to provide a (crisp) ranking of
objects from the top to the bottom. Instead, some objects might be ranked
equally, or better than other objects only to some degree. In such cases, a
generalization of crisp rankings to fuzzy rankings can be more useful. The aim
of the article is to introduce the notion of a fuzzy ranking and to discuss its
several properties, namely orderings, similarity and indecisiveness. The
proposed approach can be used both for group decision making or multiple
criteria decision making when uncertainty is involved.
Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
Authors: Chen Yunpeng, Jin Xiaojie, Kang Bingyi, Feng Jiashi, Yan Shuicheng
Date: 2017-03-07 00:00:00
Tasks: General Classification

Residual units are wildly used for alleviating optimization difficulties when
building deep neural networks. However, the performance gain does not well
compensate the model size increase, indicating low parameter efficiency in
these residual units. In this work, we first revisit the residual function in
several variations of residual units and demonstrate that these residual
functions can actually be explained with a unified framework based on
generalized block term decomposition. Then, based on the new explanation, we
propose a new architecture, Collective Residual Unit (CRU), which enhances the
parameter efficiency of deep neural networks through collective tensor
factorization. CRU enables knowledge sharing across different residual units
using shared factors. Experimental results show that our proposed CRU Network
demonstrates outstanding parameter efficiency, achieving comparable
classification performance to ResNet-200 with the model size of ResNet-50. By
building a deeper network using CRU, we can achieve state-of-the-art single
model classification accuracy on ImageNet-1k and Places365-Standard benchmark
datasets. (Code and trained models are available on GitHub)
A Machine-Learning Framework for Design for Manufacturability
Authors: Aditya Balu, Sambit Ghadai, Gavin Young, Soumik Sarkar, Adarsh Krishnamurthy
Date: 2017-03-04 00:00:00
Tasks: BIG-bench Machine Learning

this is a duplicate submission(original is arXiv:1612.02141). Hence want to
withdraw it
Temporal Ensembling for Semi-Supervised Learning
Authors: Samuli Laine, Timo Aila
Date: 2016-10-07 00:00:00
Tasks: General Classification, Semi-Supervised Image Classification

In this paper, we present a simple and efficient method for training deep
neural networks in a semi-supervised setting where only a small portion of
training data is labeled. We introduce self-ensembling, where we form a
consensus prediction of the unknown labels using the outputs of the
network-in-training on different epochs, and most importantly, under different
regularization and input augmentation conditions. This ensemble prediction can
be expected to be a better predictor for the unknown labels than the output of
the network at the most recent training epoch, and can thus be used as a target
for training. Using our method, we set new records for two standard
semi-supervised learning benchmarks, reducing the (non-augmented)
classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from
18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16%
by enabling the standard augmentations. We additionally obtain a clear
improvement in CIFAR-100 classification accuracy by using random images from
the Tiny Images dataset as unlabeled extra inputs during training. Finally, we
demonstrate good tolerance to incorrect labels.
A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models
Authors: Ryan Spring, Anshumali Shrivastava
Date: 2017-03-15 00:00:00

Log-linear models are arguably the most successful class of graphical models
for large-scale applications because of their simplicity and tractability.
Learning and inference with these models require calculating the partition
function, which is a major bottleneck and intractable for large state spaces.
Importance Sampling (IS) and MCMC-based approaches are lucrative. However, the
condition of having a "good" proposal distribution is often not satisfied in
practice.
  In this paper, we add a new dimension to efficient estimation via sampling.
We propose a new sampling scheme and an unbiased estimator that estimates the
partition function accurately in sub-linear time. Our samples are generated in
near-constant time using locality sensitive hashing (LSH), and so are
correlated and unnormalized. We demonstrate the effectiveness of our proposed
approach by comparing the accuracy and speed of estimating the partition
function against other state-of-the-art estimation techniques including IS and
the efficient variant of Gumbel-Max sampling. With our efficient sampling
scheme, we accurately train real-world language models using only 1-2% of
computations.