pwc-archive/papers-with-abstracts huggingface dataset:
Extreme Classification in Log Memory
Authors: Qixuan Huang, Yiqiu Wang, Tharun Medini, Anshumali Shrivastava
Date: 2018-10-09 00:00:00
Tasks: Classification, General Classification, GPU

We present Merged-Averaged Classifiers via Hashing (MACH) for
K-classification with ultra-large values of K. Compared to traditional
one-vs-all classifiers that require O(Kd) memory and inference cost, MACH only
need O(d log K) (d is dimensionality )memory while only requiring O(K log K + d
log K) operation for inference. MACH is a generic K-classification algorithm,
with provably theoretical guarantees, which requires O(log K) memory without
any assumption on the relationship between classes. MACH uses universal hashing
to reduce classification with a large number of classes to few independent
classification tasks with small (constant) number of classes. We provide
theoretical quantification of discriminability-memory tradeoff. With MACH we
can train ODP dataset with 100,000 classes and 400,000 features on a single
Titan X GPU, with the classification accuracy of 19.28%, which is the
best-reported accuracy on this dataset. Before this work, the best performing
baseline is a one-vs-all classifier that requires 40 billion parameters (160 GB
model size) and achieves 9% accuracy. In contrast, MACH can achieve 9% accuracy
with 480x reduction in the model size (of mere 0.3GB). With MACH, we also
demonstrate complete training of fine-grained imagenet dataset (compressed size
104GB), with 21,000 classes, on a single GPU. To the best of our knowledge,
this is the first work to demonstrate complete training of these extreme-class
datasets on a single Titan X.
Bird Species Classification using Transfer Learning with Multistage Training
Authors: Sourya Dipta Das, Akash Kumar
Date: 2018-10-09 00:00:00
Tasks: Classification, General Classification, Transfer Learning

Bird species classification has received more and more attention in the field
of computer vision, for its promising applications in biology and environmental
studies. Recognizing bird species is difficult due to the challenges of
discriminative region localization and fine-grained feature learning. In this
paper, we have introduced a Transfer learning based method with multistage
training. We have used both Pre-Trained Mask-RCNN and an ensemble model
consisting of Inception Nets (InceptionV3 & InceptionResNetV2 ) to get
localization and species of the bird from the images respectively. Our final
model achieves an F1 score of 0.5567 or 55.67 % on the dataset provided in CVIP
2018 Challenge.
Data-dependent compression of random features for large-scale kernel approximation
Authors: Raj Agrawal, Trevor Campbell, Jonathan H. Huggins, Tamara Broderick
Date: 2018-10-09 00:00:00
Tasks: feature selection

Kernel methods offer the flexibility to learn complex relationships in
modern, large data sets while enjoying strong theoretical guarantees on
quality. Unfortunately, these methods typically require cubic running time in
the data set size, a prohibitive cost in the large-data setting. Random feature
maps (RFMs) and the Nystrom method both consider low-rank approximations to the
kernel matrix as a potential solution. But, in order to achieve desirable
theoretical guarantees, the former may require a prohibitively large number of
features J+, and the latter may be prohibitively expensive for high-dimensional
problems. We propose to combine the simplicity and generality of RFMs with a
data-dependent feature selection scheme to achieve desirable theoretical
approximation properties of Nystrom with just O(log J+) features. Our key
insight is to begin with a large set of random features, then reduce them to a
small number of weighted features in a data-dependent, computationally
efficient way, while preserving the statistical guarantees of using the
original large set of features. We demonstrate the efficacy of our method with
theory and experiments--including on a data set with over 50 million
observations. In particular, we show that our method achieves small kernel
matrix approximation error and better test set accuracy with provably fewer
random features than state-of-the-art methods.
Feature Selection using Stochastic Gates
Authors: Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, Yuval Kluger
Date: 2018-10-09 00:00:00
Proceeding: ICML 2020 1
Tasks: feature selection

Feature selection problems have been extensively studied for linear estimation, for instance, Lasso, but less emphasis has been placed on feature selection for non-linear functions. In this study, we propose a method for feature selection in high-dimensional non-linear function estimation problems. The new procedure is based on minimizing the $\ell_0$ norm of the vector of indicator variables that represent if a feature is selected or not. Our approach relies on the continuous relaxation of Bernoulli distributions, which allows our model to learn the parameters of the approximate Bernoulli distributions via gradient descent. This general framework simultaneously minimizes a loss function while selecting relevant features. Furthermore, we provide an information-theoretic justification of incorporating Bernoulli distribution into our approach and demonstrate the potential of the approach on synthetic and real-life applications.
Deep clustering: On the link between discriminative models and K-means
Authors: Mohammed Jabi, Marco Pedersoli, Amar Mitiche, Ismail Ben Ayed
Date: 2018-10-09 00:00:00
Tasks: Clustering, Deep Clustering, Image Clustering, regression

In the context of recent deep clustering studies, discriminative models dominate the literature and report the most competitive performances. These models learn a deep discriminative neural network classifier in which the labels are latent. Typically, they use multinomial logistic regression posteriors and parameter regularization, as is very common in supervised learning. It is generally acknowledged that discriminative objective functions (e.g., those based on the mutual information or the KL divergence) are more flexible than generative approaches (e.g., K-means) in the sense that they make fewer assumptions about the data distributions and, typically, yield much better unsupervised deep learning results. On the surface, several recent discriminative models may seem unrelated to K-means. This study shows that these models are, in fact, equivalent to K-means under mild conditions and common posterior models and parameter regularization. We prove that, for the commonly used logistic regression posteriors, maximizing the $L_2$ regularized mutual information via an approximate alternating direction method (ADM) is equivalent to a soft and regularized K-means loss. Our theoretical analysis not only connects directly several recent state-of-the-art discriminative models to K-means, but also leads to a new soft and regularized deep K-means algorithm, which yields competitive performance on several image clustering benchmarks.
Distributed Wildfire Surveillance with Autonomous Aircraft using Deep Reinforcement Learning
Authors: Kyle D. Julian, Mykel J. Kochenderfer
Date: 2018-10-09 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Teams of autonomous unmanned aircraft can be used to monitor wildfires,
enabling firefighters to make informed decisions. However, controlling multiple
autonomous fixed-wing aircraft to maximize forest fire coverage is a complex
problem. The state space is high dimensional, the fire propagates
stochastically, the sensor information is imperfect, and the aircraft must
coordinate with each other to accomplish their mission. This work presents two
deep reinforcement learning approaches for training decentralized controllers
that accommodate the high dimensionality and uncertainty inherent in the
problem. The first approach controls the aircraft using immediate observations
of the individual aircraft. The second approach allows aircraft to collaborate
on a map of the wildfire's state and maintain a time history of locations
visited, which are used as inputs to the controller. Simulation results show
that both approaches allow the aircraft to accurately track wildfire expansions
and outperform an online receding horizon controller. Additional simulations
demonstrate that the approach scales with different numbers of aircraft and
generalizes to different wildfire shapes.
Deep Neural Network Compression for Aircraft Collision Avoidance Systems
Authors: Kyle D. Julian, Mykel J. Kochenderfer, Michael P. Owen
Date: 2018-10-09 00:00:00
Tasks: Collision Avoidance, Decision Making, Neural Network Compression

One approach to designing decision making logic for an aircraft collision
avoidance system frames the problem as a Markov decision process and optimizes
the system using dynamic programming. The resulting collision avoidance
strategy can be represented as a numeric table. This methodology has been used
in the development of the Airborne Collision Avoidance System X (ACAS X) family
of collision avoidance systems for manned and unmanned aircraft, but the high
dimensionality of the state space leads to very large tables. To improve
storage efficiency, a deep neural network is used to approximate the table.
With the use of an asymmetric loss function and a gradient descent algorithm,
the parameters for this network can be trained to provide accurate estimates of
table values while preserving the relative preferences of the possible
advisories for each state. By training multiple networks to represent
subtables, the network also decreases the required runtime for computing the
collision avoidance advisory. Simulation studies show that the network improves
the safety and efficiency of the collision avoidance system. Because only the
network parameters need to be stored, the required storage space is reduced by
a factor of 1000, enabling the collision avoidance system to operate using
current avionics systems.
Penetrating the Fog: the Path to Efficient CNN Models
Authors: Kun Wan, Boyuan Feng, Shu Yang, Yufei Ding
Date: 2018-10-09 00:00:00
Proceeding: ICLR 2019 5

With the increasing demand to deploy convolutional neural networks (CNNs) on
mobile platforms, the sparse kernel approach was proposed, which could save
more parameters than the standard convolution while maintaining accuracy.
However, despite the great potential, no prior research has pointed out how to
craft an sparse kernel design with such potential (i.e., effective design), and
all prior works just adopt simple combinations of existing sparse kernels such
as group convolution. Meanwhile due to the large design space it is also
impossible to try all combinations of existing sparse kernels. In this paper,
we are the first in the field to consider how to craft an effective sparse
kernel design by eliminating the large design space. Specifically, we present a
sparse kernel scheme to illustrate how to reduce the space from three aspects.
First, in terms of composition we remove designs composed of repeated layers.
Second, to remove designs with large accuracy degradation, we find an unified
property named information field behind various sparse kernel designs, which
could directly indicate the final accuracy. Last, we remove designs in two
cases where a better parameter efficiency could be achieved. Additionally, we
provide detailed efficiency analysis on the final four designs in our scheme.
Experimental results validate the idea of our scheme by showing that our scheme
is able to find designs which are more efficient in using parameters and
computation with similar or higher accuracy.
Learning under Misspecified Objective Spaces
Authors: Andreea Bobu, Andrea Bajcsy, Jaime F. Fisac, Anca D. Dragan
Date: 2018-10-11 00:00:00

Learning robot objective functions from human input has become increasingly
important, but state-of-the-art techniques assume that the human's desired
objective lies within the robot's hypothesis space. When this is not true, even
methods that keep track of uncertainty over the objective fail because they
reason about which hypothesis might be correct, and not whether any of the
hypotheses are correct. We focus specifically on learning from physical human
corrections during the robot's task execution, where not having a rich enough
hypothesis space leads to the robot updating its objective in ways that the
person did not actually intend. We observe that such corrections appear
irrelevant to the robot, because they are not the best way of achieving any of
the candidate objectives. Instead of naively trusting and learning from every
human interaction, we propose robots learn conservatively by reasoning in real
time about how relevant the human's correction is for the robot's hypothesis
space. We test our inference method in an experiment with human interaction
data, and demonstrate that this alleviates unintended learning in an in-person
user study with a 7DoF robot manipulator.
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
Authors: Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
Date: 2018-10-11 00:00:00
Tasks: Gaussian Processes, parameter estimation

There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical. As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case. We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.
Simple and Effective Text Simplification Using Semantic and Neural Methods
Authors: Elior Sulem, Omri Abend, Ari Rappoport
Date: 2018-10-11 00:00:00
Proceeding: ACL 2018 7
Tasks: Machine Translation, Semantic Parsing, Sentence, Text Simplification, Translation

Sentence splitting is a major simplification operator. Here we present a
simple and efficient splitting algorithm based on an automatic semantic parser.
After splitting, the text is amenable for further fine-tuned simplification
operations. In particular, we show that neural Machine Translation can be
effectively used in this situation. Previous application of Machine Translation
for simplification suffers from a considerable disadvantage in that they are
over-conservative, often failing to modify the source in any way. Splitting
based on semantic parsing, as proposed here, alleviates this issue. Extensive
automatic and human evaluation shows that the proposed method compares
favorably to the state-of-the-art in combined lexical and structural
simplification.
Neural Relation Extraction Within and Across Sentence Boundaries
Authors: Pankaj Gupta, Subburam Rajaram, Hinrich Schütze, Bernt Andrassy, Thomas Runkler
Date: 2018-10-11 00:00:00
Tasks: Relation, Relation Extraction, Sentence

Past work in relation extraction mostly focuses on binary relation between
entity pairs within single sentence. Recently, the NLP community has gained
interest in relation extraction in entity pairs spanning multiple sentences. In
this paper, we propose a novel architecture for this task: inter-sentential
dependency-based neural networks (iDepNN). iDepNN models the shortest and
augmented dependency paths via recurrent and recursive neural networks to
extract relationships within (intra-) and across (inter-) sentence boundaries.
Compared to SVM and neural network baselines, iDepNN is more robust to false
positives in relationships spanning sentences.
  We evaluate our models on four datasets from newswire (MUC6) and medical
(BioNLP shared task) domains that achieve state-of-the-art performance and show
a better balance in precision and recall for inter-sentential relationships. We
perform better than 11 teams participating in the BioNLP shared task 2016 and
achieve a gain of 5.2% (0.587 vs 0.558) in F1 over the winning team. We also
release the crosssentence annotations for MUC6.
Unbiased deep solvers for linear parametric PDEs
Authors: Marc Sabate Vidales, David Siska, Lukasz Szpruch
Date: 2018-10-11 00:00:00

We develop several deep learning algorithms for approximating families of parametric PDE solutions. The proposed algorithms approximate solutions together with their gradients, which in the context of mathematical finance means that the derivative prices and hedging strategies are computed simulatenously. Having approximated the gradient of the solution one can combine it with a Monte-Carlo simulation to remove the bias in the deep network approximation of the PDE solution (derivative price). This is achieved by leveraging the Martingale Representation Theorem and combining the Monte Carlo simulation with the neural network. The resulting algorithm is robust with respect to quality of the neural network approximation and consequently can be used as a black-box in case only limited a priori information about the underlying problem is available. We believe this is important as neural network based algorithms often require fair amount of tuning to produce satisfactory results. The methods are empirically shown to work for high-dimensional problems (e.g. 100 dimensions). We provide diagnostics that shed light on appropriate network architectures.
Taming the Cross Entropy Loss
Authors: Manuel Martinez, Rainer Stiefelhagen
Date: 2018-10-11 00:00:00
Tasks: General Classification

We present the Tamed Cross Entropy (TCE) loss function, a robust derivative
of the standard Cross Entropy (CE) loss used in deep learning for
classification tasks. However, unlike other robust losses, the TCE loss is
designed to exhibit the same training properties than the CE loss in noiseless
scenarios. Therefore, the TCE loss requires no modification on the training
regime compared to the CE loss and, in consequence, can be applied in all
applications where the CE loss is currently used. We evaluate the TCE loss
using the ResNet architecture on four image datasets that we artificially
contaminated with various levels of label noise. The TCE loss outperforms the
CE loss in every tested scenario.
Regularized Contextual Bandits
Authors: Xavier Fontaine, Quentin Berthet, Vianney Perchet
Date: 2018-10-11 00:00:00
Tasks: Multi-Armed Bandits

We consider the stochastic contextual bandit problem with additional regularization. The motivation comes from problems where the policy of the agent must be close to some baseline policy which is known to perform well on the task. To tackle this problem we use a nonparametric model and propose an algorithm splitting the context space into bins, and solving simultaneously - and independently - regularized multi-armed bandit instances on each bin. We derive slow and fast rates of convergence, depending on the unknown complexity of the problem. We also consider a new relevant margin condition to get problem-independent convergence rates, ending up in intermediate convergence rates interpolating between the aforementioned slow and fast rates.
A Theory-Based Evaluation of Nearest Neighbor Models Put Into Practice
Authors: Hendrik Fichtenberger, Dennis Rohde
Date: 2018-10-11 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Retrieval

In the $k$-nearest neighborhood model ($k$-NN), we are given a set of points
$P$, and we shall answer queries $q$ by returning the $k$ nearest neighbors of
$q$ in $P$ according to some metric. This concept is crucial in many areas of
data analysis and data processing, e.g., computer vision, document retrieval
and machine learning. Many $k$-NN algorithms have been published and
implemented, but often the relation between parameters and accuracy of the
computed $k$-NN is not explicit. We study property testing of $k$-NN graphs in
theory and evaluate it empirically: given a point set $P \subset
\mathbb{R}^\delta$ and a directed graph $G=(P,E)$, is $G$ a $k$-NN graph, i.e.,
every point $p \in P$ has outgoing edges to its $k$ nearest neighbors, or is it
$\epsilon$-far from being a $k$-NN graph? Here, $\epsilon$-far means that one
has to change more than an $\epsilon$-fraction of the edges in order to make
$G$ a $k$-NN graph. We develop a randomized algorithm with one-sided error that
decides this question, i.e., a property tester for the $k$-NN property, with
complexity $O(\sqrt{n} k^2 / \epsilon^2)$ measured in terms of the number of
vertices and edges it inspects, and we prove a lower bound of $\Omega(\sqrt{n /
\epsilon k})$. We evaluate our tester empirically on the $k$-NN models computed
by various algorithms and show that it can be used to detect $k$-NN models with
bad accuracy in significantly less time than the building time of the $k$-NN
model.
Identification of Invariant Sensorimotor Structures as a Prerequisite for the Discovery of Objects
Authors: Nicolas Le Hir, Olivier Sigaud, Alban Laflaquière
Date: 2018-10-11 00:00:00
Tasks: Clustering

Perceiving the surrounding environment in terms of objects is useful for any
general purpose intelligent agent. In this paper, we investigate a fundamental
mechanism making object perception possible, namely the identification of
spatio-temporally invariant structures in the sensorimotor experience of an
agent. We take inspiration from the Sensorimotor Contingencies Theory to define
a computational model of this mechanism through a sensorimotor, unsupervised
and predictive approach. Our model is based on processing the unsupervised
interaction of an artificial agent with its environment. We show how
spatio-temporally invariant structures in the environment induce regularities
in the sensorimotor experience of an agent, and how this agent, while building
a predictive model of its sensorimotor experience, can capture them as densely
connected subgraphs in a graph of sensory states connected by motor commands.
Our approach is focused on elementary mechanisms, and is illustrated with a set
of simple experiments in which an agent interacts with an environment. We show
how the agent can build an internal model of moving but spatio-temporally
invariant structures by performing a Spectral Clustering of the graph modeling
its overall sensorimotor experiences. We systematically examine properties of
the model, shedding light more globally on the specificities of the paradigm
with respect to methods based on the supervised processing of collections of
static images.
Deep Learning for Image Denoising: A Survey
Authors: Chunwei Tian, Yong Xu, Lunke Fei, Ke Yan
Date: 2018-10-11 00:00:00
Tasks: BIG-bench Machine Learning, Deep Learning, Denoising, GPU, Image Denoising, Survey

Since the proposal of big data analysis and Graphic Processing Unit (GPU),
the deep learning technology has received a great deal of attention and has
been widely applied in the field of imaging processing. In this paper, we have
an aim to completely review and summarize the deep learning technologies for
image denoising proposed in recent years. Morever, we systematically analyze
the conventional machine learning methods for image denoising. Finally, we
point out some research directions for the deep learning technologies in image
denoising.