pwc-archive/papers-with-abstracts huggingface dataset:
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval
Authors: Peng Xu, Yongye Huang, Tongtong Yuan, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, Zhanyu Ma, Jun Guo
Date: 2018-04-04 00:00:00
Proceeding: CVPR 2018 6
Tasks: Deep Hashing, Retrieval, Sketch Recognition

We propose a deep hashing framework for sketch retrieval that, for the first
time, works on a multi-million scale human sketch dataset. Leveraging on this
large dataset, we explore a few sketch-specific traits that were otherwise
under-studied in prior literature. Instead of following the conventional sketch
recognition task, we introduce the novel problem of sketch hashing retrieval
which is not only more challenging, but also offers a better testbed for
large-scale sketch analysis, since: (i) more fine-grained sketch feature
learning is required to accommodate the large variations in style and
abstraction, and (ii) a compact binary code needs to be learned at the same
time to enable efficient retrieval. Key to our network design is the embedding
of unique characteristics of human sketch, where (i) a two-branch CNN-RNN
architecture is adapted to explore the temporal ordering of strokes, and (ii) a
novel hashing loss is specifically designed to accommodate both the temporal
and abstract traits of sketches. By working with a 3.8M sketch dataset, we show
that state-of-the-art hashing models specifically engineered for static images
fail to perform well on temporal sketch data. Our network on the other hand not
only offers the best retrieval performance on various code sizes, but also
yields the best generalization performance under a zero-shot setting and when
re-purposed for sketch recognition. Such superior performances effectively
demonstrate the benefit of our sketch-specific design.
Transfer Learning with Neural AutoML
Authors: Catherine Wong, Neil Houlsby, Yifeng Lu, Andrea Gesmundo
Date: 2018-03-07 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: AutoML, General Classification, image-classification, Image Classification, Neural Architecture Search, Transfer Learning

We reduce the computational cost of Neural AutoML with transfer learning.
AutoML relieves human effort by automating the design of ML algorithms. Neural
AutoML has become popular for the design of deep learning architectures,
however, this method has a high computation cost. To address this we propose
Transfer Neural AutoML that uses knowledge from prior tasks to speed up network
design. We extend RL-based architecture search methods to support parallel
training on multiple tasks and then transfer the search strategy to new tasks.
On language and image classification tasks, Transfer Neural AutoML reduces
convergence time over single-task training by over an order of magnitude on
many tasks.
SpectralNet: Spectral Clustering using Deep Neural Networks
Authors: Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, Yuval Kluger
Date: 2018-01-04 00:00:00
Proceeding: ICLR 2018 1
Tasks: Clustering, Stochastic Optimization

Spectral clustering is a leading and popular technique in unsupervised data
analysis. Two of its major limitations are scalability and generalization of
the spectral embedding (i.e., out-of-sample-extension). In this paper we
introduce a deep learning approach to spectral clustering that overcomes the
above shortcomings. Our network, which we call SpectralNet, learns a map that
embeds input data points into the eigenspace of their associated graph
Laplacian matrix and subsequently clusters them. We train SpectralNet using a
procedure that involves constrained stochastic optimization. Stochastic
optimization allows it to scale to large datasets, while the constraints, which
are implemented using a special-purpose output layer, allow us to keep the
network output orthogonal. Moreover, the map learned by SpectralNet naturally
generalizes the spectral embedding to unseen data points. To further improve
the quality of the clustering, we replace the standard pairwise Gaussian
affinities with affinities leaned from unlabeled data using a Siamese network.
Additional improvement can be achieved by applying the network to code
representations produced, e.g., by standard autoencoders. Our end-to-end
learning procedure is fully unsupervised. In addition, we apply VC dimension
theory to derive a lower bound on the size of SpectralNet. State-of-the-art
clustering results are reported on the Reuters dataset. Our implementation is
publicly available at https://github.com/kstant0725/SpectralNet .
Gradient Boosting With Piece-Wise Linear Regression Trees
Authors: Yu Shi, Jian Li, Zhize Li
Date: 2018-02-15 00:00:00
Tasks: Ensemble Learning, regression

Gradient Boosted Decision Trees (GBDT) is a very successful ensemble learning algorithm widely used across a variety of applications. Recently, several variants of GBDT training algorithms and implementations have been designed and heavily optimized in some very popular open sourced toolkits including XGBoost, LightGBM and CatBoost. In this paper, we show that both the accuracy and efficiency of GBDT can be further enhanced by using more complex base learners. Specifically, we extend gradient boosting to use piecewise linear regression trees (PL Trees), instead of piecewise constant regression trees, as base learners. We show that PL Trees can accelerate convergence of GBDT and improve the accuracy. We also propose some optimization tricks to substantially reduce the training time of PL Trees, with little sacrifice of accuracy. Moreover, we propose several implementation techniques to speedup our algorithm on modern computer architectures with powerful Single Instruction Multiple Data (SIMD) parallelism. The experimental results show that GBDT with PL Trees can provide very competitive testing accuracy with comparable or less training time.
GoSGD: Distributed Optimization for Deep Learning with Gossip Exchange
Authors: Michael Blot, David Picard, Matthieu Cord
Date: 2018-04-04 00:00:00
Tasks: Deep Learning, Distributed Optimization

We address the issue of speeding up the training of convolutional neural
networks by studying a distributed method adapted to stochastic gradient
descent. Our parallel optimization setup uses several threads, each applying
individual gradient descents on a local variable. We propose a new way of
sharing information between different threads based on gossip algorithms that
show good consensus convergence properties. Our method called GoSGD has the
advantage to be fully asynchronous and decentralized.
A Projective Simulation Scheme for Partially-Observable Multi-Agent Systems
Authors: Rasoul Kheiri
Date: 2016-10-29 00:00:00

We introduce a kind of partial observability to the projective simulation (PS) learning method. It is done by adding a belief projection operator and an observability parameter to the original framework of the efficiency of the PS model. I provide theoretical formulations, network representations, and situated scenarios derived from the invasion toy problem as a starting point for some multi-agent PS models.
f-Divergence constrained policy improvement
Authors: Boris Belousov, Jan Peters
Date: 2017-12-29 00:00:00
Tasks: Reinforcement Learning

To ensure stability of learning, state-of-the-art generalized policy
iteration algorithms augment the policy improvement step with a trust region
constraint bounding the information loss. The size of the trust region is
commonly determined by the Kullback-Leibler (KL) divergence, which not only
captures the notion of distance well but also yields closed-form solutions. In
this paper, we consider a more general class of f-divergences and derive the
corresponding policy update rules. The generic solution is expressed through
the derivative of the convex conjugate function to f and includes the KL
solution as a special case. Within the class of f-divergences, we further focus
on a one-parameter family of $\alpha$-divergences to study effects of the
choice of divergence on policy improvement. Previously known as well as new
policy updates emerge for different values of $\alpha$. We show that every type
of policy update comes with a compatible policy evaluation resulting from the
chosen f-divergence. Interestingly, the mean-squared Bellman error minimization
is closely related to policy evaluation with the Pearson $\chi^2$-divergence
penalty, while the KL divergence results in the soft-max policy update and a
log-sum-exp critic. We carry out asymptotic analysis of the solutions for
different values of $\alpha$ and demonstrate the effects of using different
divergence functions on a multi-armed bandit problem and on common standard
reinforcement learning problems.
Boosting Handwriting Text Recognition in Small Databases with Transfer Learning
Authors: José Carlos Aradillas, Juan José Murillo-Fuentes, Pablo M. Olmos
Date: 2018-04-04 00:00:00
Tasks: HTR, Transfer Learning

In this paper we deal with the offline handwriting text recognition (HTR)
problem with reduced training datasets. Recent HTR solutions based on
artificial neural networks exhibit remarkable solutions in referenced
databases. These deep learning neural networks are composed of both
convolutional (CNN) and long short-term memory recurrent units (LSTM). In
addition, connectionist temporal classification (CTC) is the key to avoid
segmentation at character level, greatly facilitating the labeling task. One of
the main drawbacks of the CNNLSTM-CTC (CLC) solutions is that they need a
considerable part of the text to be transcribed for every type of calligraphy,
typically in the order of a few thousands of lines. Furthermore, in some
scenarios the text to transcribe is not that long, e.g. in the Washington
database. The CLC typically overfits for this reduced number of training
samples. Our proposal is based on the transfer learning (TL) from the
parameters learned with a bigger database. We first investigate, for a reduced
and fixed number of training samples, 350 lines, how the learning from a large
database, the IAM, can be transferred to the learning of the CLC of a reduced
database, Washington. We focus on which layers of the network could be not
re-trained. We conclude that the best solution is to re-train the whole CLC
parameters initialized to the values obtained after the training of the CLC
from the larger database. We also investigate results when the training size is
further reduced. The differences in the CER are more remarkable when training
with just 350 lines, a CER of 3.3% is achieved with TL while we have a CER of
18.2% when training from scratch. As a byproduct, the learning times are quite
reduced. Similar good results are obtained from the Parzival database when
trained with this reduced number of lines and this new approach.
Normalized Cut Loss for Weakly-supervised CNN Segmentation
Authors: Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, Christopher Schroers
Date: 2018-04-04 00:00:00
Proceeding: CVPR 2018 6
Tasks: Interactive Segmentation, Segmentation, Semantic Segmentation

Most recent semantic segmentation methods train deep convolutional neural
networks with fully annotated masks requiring pixel-accuracy for good quality
training. Common weakly-supervised approaches generate full masks from partial
input (e.g. scribbles or seeds) using standard interactive segmentation methods
as preprocessing. But, errors in such masks result in poorer training since
standard loss functions (e.g. cross-entropy) do not distinguish seeds from
potentially mislabeled other pixels. Inspired by the general ideas in
semi-supervised learning, we address these problems via a new principled loss
function evaluating network output with criteria standard in "shallow"
segmentation, e.g. normalized cut. Unlike prior work, the cross entropy part of
our loss evaluates only seeds where labels are known while normalized cut
softly evaluates consistency of all pixels. We focus on normalized cut loss
where dense Gaussian kernel is efficiently implemented in linear time by fast
Bilateral filtering. Our normalized cut loss approach to segmentation brings
the quality of weakly-supervised training significantly closer to fully
supervised methods.
Hidden Talents of the Variational Autoencoder
Authors: Bin Dai, Yu Wang, John Aston, Gang Hua, David Wipf
Date: 2017-06-16 00:00:00
Tasks: Dimensionality Reduction

Variational autoencoders (VAE) represent a popular, flexible form of deep generative model that can be stochastically fit to samples from a given random process using an information-theoretic variational bound on the true underlying distribution. Once so-obtained, the model can be putatively used to generate new samples from this distribution, or to provide a low-dimensional latent representation of existing samples. While quite effective in numerous application domains, certain important mechanisms which govern the behavior of the VAE are obfuscated by the intractable integrals and resulting stochastic approximations involved. Moreover, as a highly non-convex model, it remains unclear exactly how minima of the underlying energy relate to original design purposes. We attempt to better quantify these issues by analyzing a series of tractable special cases of increasing complexity. In doing so, we unveil interesting connections with more traditional dimensionality reduction models, as well as an intrinsic yet underappreciated propensity for robustly dismissing sparse outliers when estimating latent manifolds. With respect to the latter, we demonstrate that the VAE can be viewed as the natural evolution of recent robust PCA models, capable of learning nonlinear manifolds of unknown dimension obscured by gross corruptions.
Conformal Prediction in Learning Under Privileged Information Paradigm with Applications in Drug Discovery
Authors: Niharika Gauraha, Lars Carlsson, Ola Spjuth
Date: 2018-03-29 00:00:00
Tasks: Conformal Prediction, Drug Discovery, Prediction, Prediction Intervals, valid

This paper explores conformal prediction in the learning under privileged
information (LUPI) paradigm. We use the SVM+ realization of LUPI in an
inductive conformal predictor, and apply it to the MNIST benchmark dataset and
three datasets in drug discovery. The results show that using privileged
information produces valid models and improves efficiency compared to standard
SVM, however the improvement varies between the tested datasets and is not
substantial in the drug discovery applications. More importantly, using SVM+ in
a conformal prediction framework enables valid prediction intervals at
specified significance levels.
An Imprecise Probabilistic Estimator for the Transition Rate Matrix of a Continuous-Time Markov Chain
Authors: Thomas Krak, Alexander Erreygers, Jasper De Bock
Date: 2018-04-04 00:00:00

We consider the problem of estimating the transition rate matrix of a
continuous-time Markov chain from a finite-duration realisation of this
process. We approach this problem in an imprecise probabilistic framework,
using a set of prior distributions on the unknown transition rate matrix. The
resulting estimator is a set of transition rate matrices that, for reasons of
conjugacy, is easy to find. To determine the hyperparameters for our set of
priors, we reconsider the problem in discrete time, where we can use the
well-known Imprecise Dirichlet Model. In particular, we show how the limit of
the resulting discrete-time estimators is a continuous-time estimator. It
corresponds to a specific choice of hyperparameters and has an exceptionally
simple closed-form expression.
3DContextNet: K-d Tree Guided Hierarchical Learning of Point Clouds Using Local and Global Contextual Cues
Authors: Wei Zeng, Theo Gevers
Date: 2017-11-30 00:00:00
Tasks: Segmentation, Semantic Segmentation

Classification and segmentation of 3D point clouds are important tasks in
computer vision. Because of the irregular nature of point clouds, most of the
existing methods convert point clouds into regular 3D voxel grids before they
are used as input for ConvNets. Unfortunately, voxel representations are highly
insensitive to the geometrical nature of 3D data. More recent methods encode
point clouds to higher dimensional features to cover the global 3D space.
However, these models are not able to sufficiently capture the local structures
of point clouds.
  Therefore, in this paper, we propose a method that exploits both local and
global contextual cues imposed by the k-d tree. The method is designed to learn
representation vectors progressively along the tree structure. Experiments on
challenging benchmarks show that the proposed model provides discriminative
point set features. For the task of 3D scene semantic segmentation, our method
significantly outperforms the state-of-the-art on the Stanford Large-Scale 3D
Indoor Spaces Dataset (S3DIS).
Generative Visual Rationales
Authors: Jarrel Seah, Jennifer Tang, Andy Kitchen, Jonathan Seah
Date: 2018-04-04 00:00:00
Tasks: Prediction

Interpretability and small labelled datasets are key issues in the practical
application of deep learning, particularly in areas such as medicine. In this
paper, we present a semi-supervised technique that addresses both these issues
by leveraging large unlabelled datasets to encode and decode images into a
dense latent representation. Using chest radiography as an example, we apply
this encoder to other labelled datasets and apply simple models to the latent
vectors to learn algorithms to identify heart failure.
  For each prediction, we generate visual rationales by optimizing a latent
representation to minimize the prediction of disease while constrained by a
similarity measure in image space. Decoding the resultant latent representation
produces an image without apparent disease. The difference between the original
decoding and the altered image forms an interpretable visual rationale for the
algorithm's prediction on that image. We also apply our method to the MNIST
dataset and compare the generated rationales to other techniques described in
the literature.
A Multi-Stage Multi-Task Neural Network for Aerial Scene Interpretation and Geolocalization
Authors: Alina Marcu, Dragos Costea, Emil Slusanschi, Marius Leordeanu
Date: 2018-04-04 00:00:00
Tasks: Decoder, GPU, Segmentation, Semantic Segmentation

Semantic segmentation and vision-based geolocalization in aerial images are
challenging tasks in computer vision. Due to the advent of deep convolutional
nets and the availability of relatively low cost UAVs, they are currently
generating a growing attention in the field. We propose a novel multi-task
multi-stage neural network that is able to handle the two problems at the same
time, in a single forward pass. The first stage of our network predicts
pixelwise class labels, while the second stage provides a precise location
using two branches. One branch uses a regression network, while the other is
used to predict a location map trained as a segmentation task. From a
structural point of view, our architecture uses encoder-decoder modules at each
stage, having the same encoder structure re-used. Furthermore, its size is
limited to be tractable on an embedded GPU. We achieve commercial GPS-level
localization accuracy from satellite images with spatial resolution of 1 square
meter per pixel in a city-wide area of interest. On the task of semantic
segmentation, we obtain state-of-the-art results on two challenging datasets,
the Inria Aerial Image Labeling dataset and Massachusetts Buildings.
Denoising of image gradients and total generalized variation denoising
Authors: Birgit Komander, Dirk A. Lorenz, Lena Vestweber
Date: 2017-12-22 00:00:00
Tasks: Denoising, Image Reconstruction

We revisit total variation denoising and study an augmented model where we
assume that an estimate of the image gradient is available. We show that this
increases the image reconstruction quality and derive that the resulting model
resembles the total generalized variation denoising method, thus providing a
new motivation for this model. Further, we propose to use a constraint
denoising model and develop a variational denoising model that is basically
parameter free, i.e. all model parameters are estimated directly from the noisy
image.
  Moreover, we use Chambolle-Pock's primal dual method as well as the
Douglas-Rachford method for the new models. For the latter one has to solve
large discretizations of partial differential equations. We propose to do this
in an inexact manner using the preconditioned conjugate gradients method and
derive preconditioners for this. Numerical experiments show that the resulting
method has good denoising properties and also that preconditioning does
increase convergence speed significantly. Finally we analyze the duality gap of
different formulations of the TGV denoising problem and derive a simple
stopping criterion.
When Hypermutations and Ageing Enable Artificial Immune Systems to Outperform Evolutionary Algorithms
Authors: Dogan Corus, Pietro S. Oliveto, Donya Yazdani
Date: 2018-04-04 00:00:00
Tasks: Evolutionary Algorithms

We present a time complexity analysis of the Opt-IA artificial immune system
(AIS). We first highlight the power and limitations of its distinguishing
operators (i.e., hypermutations with mutation potential and ageing) by
analysing them in isolation. Recent work has shown that ageing combined with
local mutations can help escape local optima on a dynamic optimisation
benchmark function. We generalise this result by rigorously proving that,
compared to evolutionary algorithms (EAs), ageing leads to impressive speed-ups
on the standard Cliff benchmark function both when using local and global
mutations. Unless the stop at first constructive mutation (FCM) mechanism is
applied, we show that hypermutations require exponential expected runtime to
optimise any function with a polynomial number of optima. If instead FCM is
used, the expected runtime is at most a linear factor larger than the upper
bound achieved for any random local search algorithm using the artificial
fitness levels method. Nevertheless, we prove that algorithms using
hypermutations can be considerably faster than EAs at escaping local optima. An
analysis of the complete Opt-IA reveals that it is efficient on the previously
considered functions and highlights problems where the use of the full
algorithm is crucial. We complete the picture by presenting a class of
functions for which Opt-IA fails with overwhelming probability while standard
EAs are efficient.
Deep Cascade Multi-task Learning for Slot Filling in Online Shopping Assistant
Authors: Yu Gong, Xusheng Luo, Yu Zhu, Wenwu Ou, Zhao Li, Muhua Zhu, Kenny Q. Zhu, Lu Duan, Xi Chen
Date: 2018-03-30 00:00:00
Tasks: Multi-Task Learning, Natural Language Understanding, slot-filling, Slot Filling

Slot filling is a critical task in natural language understanding (NLU) for dialog systems. State-of-the-art approaches treat it as a sequence labeling problem and adopt such models as BiLSTM-CRF. While these models work relatively well on standard benchmark datasets, they face challenges in the context of E-commerce where the slot labels are more informative and carry richer expressions. In this work, inspired by the unique structure of E-commerce knowledge base, we propose a novel multi-task model with cascade and residual connections, which jointly learns segment tagging, named entity tagging and slot filling. Experiments show the effectiveness of the proposed cascade and residual structures. Our model has a 14.6% advantage in F1 score over the strong baseline methods on a new Chinese E-commerce shopping assistant dataset, while achieving competitive accuracies on a standard dataset. Furthermore, online test deployed on such dominant E-commerce platform shows 130% improvement on accuracy of understanding user utterances. Our model has already gone into production in the E-commerce platform.