pwc-archive/papers-with-abstracts huggingface dataset:
Saddle-free Hessian-free Optimization
Authors: Martin Arjovsky
Date: 2015-05-30 00:00:00
Tasks: Second-order methods

Nonconvex optimization problems such as the ones in training deep neural
networks suffer from a phenomenon called saddle point proliferation. This means
that there are a vast number of high error saddle points present in the loss
function. Second order methods have been tremendously successful and widely
adopted in the convex optimization community, while their usefulness in deep
learning remains limited. This is due to two problems: computational complexity
and the methods being driven towards the high error saddle points. We introduce
a novel algorithm specially designed to solve these two issues, providing a
crucial first step to take the widely known advantages of Newton's method to
the nonconvex optimization community, especially in high dimensional settings.
Integrating Topic Models and Latent Factors for Recommendation
Authors: Danis J. Wilson, Wei zhang
Date: 2016-10-28 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems, Topic Models

Nowadays, we have large amounts of online items in various web-based applications, which makes it an important task to build effective personalized recommender systems so as to save users' efforts in information seeking. One of the most extensively and successfully used methods for personalized recommendation is the Collaborative Filtering (CF) technique, which makes recommendation based on users' historical choices as well as those of the others'. The most popular CF method, like Latent Factor Model (LFM), is to model how users evaluate items by understanding the hidden dimension or factors of their opinions. How to model these hidden factors is key to improve the performance of recommender system. In this work, we consider the problem of hotel recommendation for travel planning services by integrating the location information and the user's preference for recommendation. The intuition is that user preferences may change dynamically over different locations, thus treating the historical decisions of a user as static or universally applicable can be infeasible in real-world applications. For example, users may prefer chain brand hotels with standard configurations when traveling for business, while they may prefer unique local hotels when traveling for entertainment. In this paper, we aim to provide trip-level personalization for users in recommendation.
Comparing learning algorithms in neural network for diagnosing cardiovascular disease
Authors: Mirmorsal Madani
Date: 2016-11-05 00:00:00
Tasks: Sensitivity

Today data mining techniques are exploited in medical science for diagnosing,
overcoming and treating diseases. Neural network is one of the techniques which
are widely used for diagnosis in medical field. In this article efficiency of
nine algorithms, which are basis of neural network learning in diagnosing
cardiovascular diseases, will be assessed. Algorithms are assessed in terms of
accuracy, sensitivity, transparency, AROC and convergence rate by means of 10
fold cross validation. The results suggest that in training phase, Lonberg-M
algorithm has the best efficiency in terms of all metrics, algorithm OSS has
maximum accuracy in testing phase, algorithm SCG has the maximum transparency
and algorithm CGB has the maximum sensitivity.
High-Dimensional $L_2$Boosting: Rate of Convergence
Authors: Ye Luo, Martin Spindler, Jannis Kück
Date: 2016-02-29 00:00:00
Tasks: Vocal Bursts Intensity Prediction

Boosting is one of the most significant developments in machine learning. This paper studies the rate of convergence of $L_2$Boosting, which is tailored for regression, in a high-dimensional setting. Moreover, we introduce so-called \textquotedblleft post-Boosting\textquotedblright. This is a post-selection estimator which applies ordinary least squares to the variables selected in the first stage by $L_2$Boosting. Another variant is \textquotedblleft Orthogonal Boosting\textquotedblright\ where after each step an orthogonal projection is conducted. We show that both post-$L_2$Boosting and the orthogonal boosting achieve the same rate of convergence as LASSO in a sparse, high-dimensional setting. We show that the rate of convergence of the classical $L_2$Boosting depends on the design matrix described by a sparse eigenvalue constant. To show the latter results, we derive new approximation results for the pure greedy algorithm, based on analyzing the revisiting behavior of $L_2$Boosting. We also introduce feasible rules for early stopping, which can be easily implemented and used in applied work. Our results also allow a direct comparison between LASSO and boosting which has been missing from the literature. Finally, we present simulation studies and applications to illustrate the relevance of our theoretical results and to provide insights into the practical aspects of boosting. In these simulation studies, post-$L_2$Boosting clearly outperforms LASSO.
A Differentiable Physics Engine for Deep Learning in Robotics
Authors: Jonas Degrave, Michiel Hermans, Joni Dambre, Francis wyffels
Date: 2016-11-05 00:00:00
Tasks: CPU, Deep Learning, Evolutionary Algorithms, GPU, Q-Learning, Reinforcement Learning

An important field in robotics is the optimization of controllers. Currently,
robots are often treated as a black box in this optimization process, which is
the reason why derivative-free optimization methods such as evolutionary
algorithms or reinforcement learning are omnipresent. When gradient-based
methods are used, models are kept small or rely on finite difference
approximations for the Jacobian. This method quickly grows expensive with
increasing numbers of parameters, such as found in deep learning. We propose
the implementation of a modern physics engine, which can differentiate control
parameters. This engine is implemented for both CPU and GPU. Firstly, this
paper shows how such an engine speeds up the optimization process, even for
small problems. Furthermore, it explains why this is an alternative approach to
deep Q-learning, for using deep learning in robotics. Finally, we argue that
this is a big step for deep learning in robotics, as it opens up new
possibilities to optimize robots, both in hardware and software.
Boosting Image Captioning with Attributes
Authors: Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei
Date: 2016-11-05 00:00:00
Proceeding: ICCV 2017 10
Tasks: Image Captioning

Automatically describing an image with a natural language has been an
emerging challenge in both fields of computer vision and natural language
processing. In this paper, we present Long Short-Term Memory with Attributes
(LSTM-A) - a novel architecture that integrates attributes into the successful
Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs)
image captioning framework, by training them in an end-to-end manner. To
incorporate attributes, we construct variants of architectures by feeding image
representations and attributes into RNNs in different ways to explore the
mutual but also fuzzy relationship between them. Extensive experiments are
conducted on COCO image captioning dataset and our framework achieves superior
results when compared to state-of-the-art deep models. Most remarkably, we
obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and
publicly available splits in (Karpathy & Fei-Fei, 2015) when extracting image
representations by GoogleNet and achieve to date top-1 performance on COCO
captioning Leaderboard.
GPU-based Pedestrian Detection for Autonomous Driving
Authors: Victor Campmany, Sergio Silva, Antonio Espinosa, Juan Carlos Moure, David Vázquez, Antonio M. López
Date: 2016-11-05 00:00:00
Tasks: Autonomous Driving, CPU, General Classification, GPU, Pedestrian Detection

We propose a real-time pedestrian detection system for the embedded Nvidia
Tegra X1 GPU-CPU hybrid platform. The pipeline is composed by the following
state-of-the-art algorithms: Histogram of Local Binary Patterns (LBP) and
Histograms of Oriented Gradients (HOG) features extracted from the input image;
Pyramidal Sliding Window technique for candidate generation; and Support Vector
Machine (SVM) for classification. Results show a 8x speedup in the target Tegra
X1 platform and a better performance/watt ratio than desktop CUDA platforms in
study.
What Is the Best Practice for CNNs Applied to Visual Instance Retrieval?
Authors: Jiedong Hao, Jing Dong, Wei Wang, Tieniu Tan
Date: 2016-11-05 00:00:00
Tasks: Image Retrieval, Retrieval

Previous work has shown that feature maps of deep convolutional neural
networks (CNNs) can be interpreted as feature representation of a particular
image region. Features aggregated from these feature maps have been exploited
for image retrieval tasks and achieved state-of-the-art performances in recent
years. The key to the success of such methods is the feature representation.
However, the different factors that impact the effectiveness of features are
still not explored thoroughly. There are much less discussion about the best
combination of them.
  The main contribution of our paper is the thorough evaluations of the various
factors that affect the discriminative ability of the features extracted from
CNNs. Based on the evaluation results, we also identify the best choices for
different factors and propose a new multi-scale image feature representation
method to encode the image effectively. Finally, we show that the proposed
method generalises well and outperforms the state-of-the-art methods on four
typical datasets used for visual instance retrieval.
Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
Authors: Frank S. He, Yang Liu, Alexander G. Schwing, Jian Peng
Date: 2016-11-05 00:00:00
Tasks: Atari Games, Deep Reinforcement Learning, Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We propose a novel training algorithm for reinforcement learning which
combines the strength of deep Q-learning with a constrained optimization
approach to tighten optimality and encourage faster reward propagation. Our
novel technique makes deep reinforcement learning more practical by drastically
reducing the training time. We evaluate the performance of our approach on the
49 games of the challenging Arcade Learning Environment, and report significant
improvements in both training time and accuracy.
Query Clustering using Segment Specific Context Embeddings
Authors: S. K Kolluru, Prasenjit Mukherjee
Date: 2016-08-03 00:00:00
Tasks: Clustering

This paper presents a novel query clustering approach to capture the broad
interest areas of users querying search engines. We make use of recent advances
in NLP - word2vec and extend it to get query2vec, vector representations of
queries, based on query contexts, obtained from the top search results for the
query and use a highly scalable Divide & Merge clustering algorithm on top of
the query vectors, to get the clusters. We have tried this approach on a
variety of segments, including Retail, Travel, Health, Phones and found the
clusters to be effective in discovering user's interest areas which have high
monetization potential.
Class-prior Estimation for Learning from Positive and Unlabeled Data
Authors: Marthinus C. du Plessis, Gang Niu, Masashi Sugiyama
Date: 2016-11-05 00:00:00

We consider the problem of estimating the class prior in an unlabeled
dataset. Under the assumption that an additional labeled dataset is available,
the class prior can be estimated by fitting a mixture of class-wise data
distributions to the unlabeled data distribution. However, in practice, such an
additional labeled dataset is often not available. In this paper, we show that,
with additional samples coming only from the positive class, the class prior of
the unlabeled dataset can be estimated correctly. Our key idea is to use
properly penalized divergences for model fitting to cancel the error caused by
the absence of negative samples. We further show that the use of the penalized
$L_1$-distance gives a computationally efficient algorithm with an analytic
solution. The consistency, stability, and estimation error are theoretically
analyzed. Finally, we experimentally demonstrate the usefulness of the proposed
method.
Semi-supervised Word Sense Disambiguation with Neural Models
Authors: Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, Eric Altendorf
Date: 2016-03-22 00:00:00
Proceeding: COLING 2016 12
Tasks: Language Modeling, Language Modelling, Word Sense Disambiguation

Determining the intended sense of words in text - word sense disambiguation
(WSD) - is a long standing problem in natural language processing. Recently,
researchers have shown promising results using word vectors extracted from a
neural network language model as features in WSD algorithms. However, a simple
average or concatenation of word vectors for each word in a text loses the
sequential and syntactic information of the text. In this paper, we study WSD
with a sequence learning neural net, LSTM, to better capture the sequential and
syntactic patterns of the text. To alleviate the lack of training data in
all-words WSD, we employ the same LSTM in a semi-supervised label propagation
classifier. We demonstrate state-of-the-art results, especially on verbs.
Incorporating long-range consistency in CNN-based texture generation
Authors: G. Berger, R. Memisevic
Date: 2016-06-03 00:00:00
Tasks: Image Generation, Texture Synthesis

Gatys et al. (2015) showed that pair-wise products of features in a
convolutional network are a very effective representation of image textures. We
propose a simple modification to that representation which makes it possible to
incorporate long-range structure into image generation, and to render images
that satisfy various symmetry constraints. We show how this can greatly improve
rendering of regular textures and of images that contain other kinds of
symmetric structure. We also present applications to inpainting and season
transfer.
Submodular Optimization under Noise
Authors: Avinatan Hassidim, Yaron Singer
Date: 2016-01-12 00:00:00

We consider the problem of maximizing a monotone submodular function under
noise. There has been a great deal of work on optimization of submodular
functions under various constraints, resulting in algorithms that provide
desirable approximation guarantees. In many applications, however, we do not
have access to the submodular function we aim to optimize, but rather to some
erroneous or noisy version of it. This raises the question of whether provable
guarantees are obtainable in presence of error and noise. We provide initial
answers, by focusing on the question of maximizing a monotone submodular
function under a cardinality constraint when given access to a noisy oracle of
the function. We show that:
  - For a cardinality constraint $k \geq 2$, there is an approximation
algorithm whose approximation ratio is arbitrarily close to $1-1/e$;
  - For $k=1$ there is an algorithm whose approximation ratio is arbitrarily
close to $1/2$. No randomized algorithm can obtain an approximation ratio
better than $1/2+o(1)$;
  -If the noise is adversarial, no non-trivial approximation guarantee can be
obtained.
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Authors: Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, Kurt Keutzer
Date: 2016-02-24 00:00:00
Tasks: Image Classification, Model Compression, Network Pruning

Recent research on deep neural networks has focused primarily on improving
accuracy. For a given accuracy level, it is typically possible to identify
multiple DNN architectures that achieve that accuracy level. With equivalent
accuracy, smaller DNN architectures offer at least three advantages: (1)
Smaller DNNs require less communication across servers during distributed
training. (2) Smaller DNNs require less bandwidth to export a new model from
the cloud to an autonomous car. (3) Smaller DNNs are more feasible to deploy on
FPGAs and other hardware with limited memory. To provide all of these
advantages, we propose a small DNN architecture called SqueezeNet. SqueezeNet
achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters.
Additionally, with model compression techniques we are able to compress
SqueezeNet to less than 0.5MB (510x smaller than AlexNet).
  The SqueezeNet architecture is available for download here:
https://github.com/DeepScale/SqueezeNet
Classification with Ultrahigh-Dimensional Features
Authors: Yanming Li, Hyokyoung Hong, Jian Kang, Kevin He, Ji Zhu, Yi Li
Date: 2016-11-04 00:00:00
Tasks: Classification, General Classification

Although much progress has been made in classification with high-dimensional
features \citep{Fan_Fan:2008, JGuo:2010, CaiSun:2014, PRXu:2014},
classification with ultrahigh-dimensional features, wherein the features much
outnumber the sample size, defies most existing work. This paper introduces a
novel and computationally feasible multivariate screening and classification
method for ultrahigh-dimensional data. Leveraging inter-feature correlations,
the proposed method enables detection of marginally weak and sparse signals and
recovery of the true informative feature set, and achieves asymptotic optimal
misclassification rates. We also show that the proposed procedure provides more
powerful discovery boundaries compared to those in \citet{CaiSun:2014} and
\citet{JJin:2009}. The performance of the proposed procedure is evaluated using
simulation studies and demonstrated via classification of patients with
different post-transplantation renal functional types.
Extensions and Limitations of the Neural GPU
Authors: Eric Price, Wojciech Zaremba, Ilya Sutskever
Date: 2016-11-02 00:00:00
Tasks: GPU

The Neural GPU is a recent model that can learn algorithms such as
multi-digit binary addition and binary multiplication in a way that generalizes
to inputs of arbitrary length. We show that there are two simple ways of
improving the performance of the Neural GPU: by carefully designing a
curriculum, and by increasing model size. The latter requires a memory
efficient implementation, as a naive implementation of the Neural GPU is memory
intensive. We find that these techniques increase the set of algorithmic
problems that can be solved by the Neural GPU: we have been able to learn to
perform all the arithmetic operations (and generalize to arbitrarily long
numbers) when the arguments are given in the decimal representation (which,
surprisingly, has not been possible before). We have also been able to train
the Neural GPU to evaluate long arithmetic expressions with multiple operands
that require respecting the precedence order of the operands, although these
have succeeded only in their binary representation, and not with perfect
accuracy.
  In addition, we gain insight into the Neural GPU by investigating its failure
modes. We find that Neural GPUs that correctly generalize to arbitrarily long
numbers still fail to compute the correct answer on highly-symmetric, atypical
inputs: for example, a Neural GPU that achieves near-perfect generalization on
decimal multiplication of up to 100-digit long numbers can fail on
$000000\dots002 \times 000000\dots002$ while succeeding at $2 \times 2$. These
failure modes are reminiscent of adversarial examples.
Estimating Causal Direction and Confounding of Two Discrete Variables
Authors: Krzysztof Chalupka, Frederick Eberhardt, Pietro Perona
Date: 2016-11-04 00:00:00
Tasks: Vocal Bursts Valence Prediction

We propose a method to classify the causal relationship between two discrete
variables given only the joint distribution of the variables, acknowledging
that the method is subject to an inherent baseline error. We assume that the
causal system is acyclicity, but we do allow for hidden common causes. Our
algorithm presupposes that the probability distributions $P(C)$ of a cause $C$
is independent from the probability distribution $P(E\mid C)$ of the
cause-effect mechanism. While our classifier is trained with a Bayesian
assumption of flat hyperpriors, we do not make this assumption about our test
data. This work connects to recent developments on the identifiability of
causal models over continuous variables under the assumption of "independent
mechanisms". Carefully-commented Python notebooks that reproduce all our
experiments are available online at
http://vision.caltech.edu/~kchalupk/code.html.
Protein Secondary Structure Prediction Using Deep Multi-scale Convolutional Neural Networks and Next-Step Conditioning
Authors: Akosua Busia, Jasmine Collins, Navdeep Jaitly
Date: 2016-11-04 00:00:00
Tasks: Protein Secondary Structure Prediction, Protein Structure Prediction

Recently developed deep learning techniques have significantly improved the
accuracy of various speech and image recognition systems. In this paper we
adapt some of these techniques for protein secondary structure prediction. We
first train a series of deep neural networks to predict eight-class secondary
structure labels given a protein's amino acid sequence information and find
that using recent methods for regularization, such as dropout and weight-norm
constraining, leads to measurable gains in accuracy. We then adapt recent
convolutional neural network architectures--Inception, ReSNet, and DenseNet
with Batch Normalization--to the problem of protein structure prediction. These
convolutional architectures make heavy use of multi-scale filter layers that
simultaneously compute features on several scales, and use residual connections
to prevent underfitting. Using a carefully modified version of these
architectures, we achieve state-of-the-art performance of 70.0% per amino acid
accuracy on the public CB513 benchmark dataset. Finally, we explore additions
from sequence-to-sequence learning, altering the model to make its predictions
conditioned on both the protein's amino acid sequence and its past secondary
structure labels. We introduce a new method of ensembling such a conditional
model with our convolutional model, an approach which reaches 70.6% Q8 accuracy
on CB513. We argue that these results can be further refined for larger boosts
in prediction accuracy through more sophisticated attempts to control
overfitting of conditional models. We aim to release the code for these
experiments as part of the TensorFlow repository.
Contextual Semibandits via Supervised Learning Oracles
Authors: Akshay Krishnamurthy, Alekh Agarwal, Miroslav Dudik
Date: 2015-02-20 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Decision Making, Learning-To-Rank

We study an online decision making problem where on each round a learner
chooses a list of items based on some side information, receives a scalar
feedback value for each individual item, and a reward that is linearly related
to this feedback. These problems, known as contextual semibandits, arise in
crowdsourcing, recommendation, and many other domains. This paper reduces
contextual semibandits to supervised learning, allowing us to leverage powerful
supervised learning methods in this partial-feedback setting. Our first
reduction applies when the mapping from feedback to reward is known and leads
to a computationally efficient algorithm with near-optimal regret. We show that
this algorithm outperforms state-of-the-art approaches on real-world
learning-to-rank datasets, demonstrating the advantage of oracle-based
algorithms. Our second reduction applies to the previously unstudied setting
when the linear mapping from feedback to reward is unknown. Our regret
guarantees are superior to prior techniques that ignore the feedback.