pwc-archive/papers-with-abstracts huggingface dataset:
Subspace Clustering by Block Diagonal Representation
Authors: Canyi Lu, Jiashi Feng, Zhouchen Lin, Tao Mei, Shuicheng Yan
Date: 2018-05-23 00:00:00
Tasks: Clustering

This paper studies the subspace clustering problem. Given some data points
approximately drawn from a union of subspaces, the goal is to group these data
points into their underlying subspaces. Many subspace clustering methods have
been proposed and among which sparse subspace clustering and low-rank
representation are two representative ones. Despite the different motivations,
we observe that many existing methods own the common block diagonal property,
which possibly leads to correct clustering, yet with their proofs given case by
case. In this work, we consider a general formulation and provide a unified
theoretical guarantee of the block diagonal property. The block diagonal
property of many existing methods falls into our special case. Second, we
observe that many existing methods approximate the block diagonal
representation matrix by using different structure priors, e.g., sparsity and
low-rankness, which are indirect. We propose the first block diagonal matrix
induced regularizer for directly pursuing the block diagonal matrix. With this
regularizer, we solve the subspace clustering problem by Block Diagonal
Representation (BDR), which uses the block diagonal structure prior. The BDR
model is nonconvex and we propose an alternating minimization solver and prove
its convergence. Experiments on real datasets demonstrate the effectiveness of
BDR.
Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
Authors: Ji Gao, Jack Lanchantin, Mary Lou Soffa, Yanjun Qi
Date: 2018-01-13 00:00:00
Tasks: Adversarial Text, General Classification, Sentiment Analysis, Spam detection, text-classification, Text Classification, Text Classification (Sentiment Analysis)

Although various techniques have been proposed to generate adversarial
samples for white-box attacks on text, little attention has been paid to
black-box attacks, which are more realistic scenarios. In this paper, we
present a novel algorithm, DeepWordBug, to effectively generate small text
perturbations in a black-box setting that forces a deep-learning classifier to
misclassify a text input. We employ novel scoring strategies to identify the
critical tokens that, if modified, cause the classifier to make an incorrect
prediction. Simple character-level transformations are applied to the
highest-ranked tokens in order to minimize the edit distance of the
perturbation, yet change the original classification. We evaluated DeepWordBug
on eight real-world text datasets, including text classification, sentiment
analysis, and spam detection. We compare the result of DeepWordBug with two
baselines: Random (Black-box) and Gradient (White-box). Our experimental
results indicate that DeepWordBug reduces the prediction accuracy of current
state-of-the-art deep-learning models, including a decrease of 68\% on average
for a Word-LSTM model and 48\% on average for a Char-CNN model.
Highway State Gating for Recurrent Highway Networks: improving information flow through time
Authors: Ron Shoham, Haim Permuter
Date: 2018-05-23 00:00:00

Recurrent Neural Networks (RNNs) play a major role in the field of sequential
learning, and have outperformed traditional algorithms on many benchmarks.
Training deep RNNs still remains a challenge, and most of the state-of-the-art
models are structured with a transition depth of 2-4 layers. Recurrent Highway
Networks (RHNs) were introduced in order to tackle this issue. These have
achieved state-of-the-art performance on a few benchmarks using a depth of 10
layers. However, the performance of this architecture suffers from a
bottleneck, and ceases to improve when an attempt is made to add more layers.
In this work, we analyze the causes for this, and postulate that the main
source is the way that the information flows through time. We introduce a novel
and simple variation for the RHN cell, called Highway State Gating (HSG), which
allows adding more layers, while continuing to improve performance. By using a
gating mechanism for the state, we allow the net to "choose" whether to pass
information directly through time, or to gate it. This mechanism also allows
the gradient to back-propagate directly through time and, therefore, results in
a slightly faster convergence. We use the Penn Treebank (PTB) dataset as a
platform for empirical proof of concept. Empirical results show that the
improvement due to Highway State Gating is for all depths, and as the depth
increases, the improvement also increases.
Cramer-Wold AutoEncoder
Authors: Szymon Knop, Jacek Tabor, Przemysław Spurek, Igor Podolak, Marcin Mazur, Stanisław Jastrzębski
Date: 2018-05-23 00:00:00
Proceeding: ICLR 2019 5

We propose a new generative model, Cramer-Wold Autoencoder (CWAE). Following WAE, we directly encourage normality of the latent space. Our paper uses also the recent idea from Sliced WAE (SWAE) model, which uses one-dimensional projections as a method of verifying closeness of two distributions. The crucial new ingredient is the introduction of a new (Cramer-Wold) metric in the space of densities, which replaces the Wasserstein metric used in SWAE. We show that the Cramer-Wold metric between Gaussian mixtures is given by a simple analytic formula, which results in the removal of sampling necessary to estimate the cost function in WAE and SWAE models. As a consequence, while drastically simplifying the optimization procedure, CWAE produces samples of a matching perceptual quality to other SOTA models.
Segmentation of Liver Lesions with Reduced Complexity Deep Models
Authors: Ram Krishna Pandey, Aswin Vasan, A. G. Ramakrishnan
Date: 2018-05-23 00:00:00
Tasks: Tumor Segmentation

We propose a computationally efficient architecture that learns to segment
lesions from CT images of the liver. The proposed architecture uses bilinear
interpolation with sub-pixel convolution at the last layer to upscale the
course feature in bottle neck architecture. Since bilinear interpolation and
sub-pixel convolution do not have any learnable parameter, our overall model is
faster and occupies less memory footprint than the traditional U-net. We
evaluate our proposed architecture on the highly competitive dataset of 2017
Liver Tumor Segmentation (LiTS) Challenge. Our method achieves competitive
results while reducing the number of learnable parameters roughly by a factor
of 13.8 compared to the original UNet model.
SymmSLIC: Symmetry Aware Superpixel Segmentation and its Applications
Authors: Rajendra Nagar, Shanmuganathan Raman
Date: 2018-05-23 00:00:00
Tasks: Semantic Segmentation, Superpixels

Over-segmentation of an image into superpixels has become a useful tool for
solving various problems in image processing and computer vision. Reflection
symmetry is quite prevalent in both natural and man-made objects and is an
essential cue in understanding and grouping the objects in natural scenes.
Existing algorithms for estimating superpixels do not preserve the reflection
symmetry of an object which leads to different sizes and shapes of superpixels
across the symmetry axis. In this work, we propose an algorithm to over-segment
an image through the propagation of reflection symmetry evident at the pixel
level to superpixel boundaries. In order to achieve this goal, we first find
the reflection symmetry in the image and represent it by a set of pairs of
pixels which are mirror reflections of each other. We partition the image into
superpixels while preserving this reflection symmetry through an iterative
algorithm. We compare the proposed method with state-of-the-art superpixel
generation methods and show the effectiveness in preserving the size and shape
of superpixel boundaries across the reflection symmetry axes. We also present
two applications, symmetry axes detection and unsupervised symmetric object
segmentation, to illustrate the effectiveness of the proposed approach.
Improving LSTM-CTC based ASR performance in domains with limited training data
Authors: Jayadev Billa
Date: 2017-07-03 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Data Augmentation, General Classification, speech-recognition, Speech Recognition

This paper addresses the observed performance gap between automatic speech
recognition (ASR) systems based on Long Short Term Memory (LSTM) neural
networks trained with the connectionist temporal classification (CTC) loss
function and systems based on hybrid Deep Neural Networks (DNNs) trained with
the cross entropy (CE) loss function on domains with limited data. We step
through a number of experiments that show incremental improvements on a
baseline EESEN toolkit based LSTM-CTC ASR system trained on the Librispeech
100hr (train-clean-100) corpus. Our results show that with effective
combination of data augmentation and regularization, a LSTM-CTC based system
can exceed the performance of a strong Kaldi based baseline trained on the same
data.
Monte Carlo Tree Search for Asymmetric Trees
Authors: Thomas M. Moerland, Joost Broekens, Aske Plaat, Catholijn M. Jonker
Date: 2018-05-23 00:00:00
Tasks: Atari Games, OpenAI Gym

We present an extension of Monte Carlo Tree Search (MCTS) that strongly
increases its efficiency for trees with asymmetry and/or loops. Asymmetric
termination of search trees introduces a type of uncertainty for which the
standard upper confidence bound (UCB) formula does not account. Our first
algorithm (MCTS-T), which assumes a non-stochastic environment, backs-up tree
structure uncertainty and leverages it for exploration in a modified UCB
formula. Results show vastly improved efficiency in a well-known asymmetric
domain in which MCTS performs arbitrarily bad. Next, we connect the ideas about
asymmetric termination to the presence of loops in the tree, where the same
state appears multiple times in a single trace. An extension to our algorithm
(MCTS-T+), which in addition to non-stochasticity assumes full state
observability, further increases search efficiency for domains with loops as
well. Benchmark testing on a set of OpenAI Gym and Atari 2600 games indicates
that our algorithms always perform better than or at least equivalent to
standard MCTS, and could be first-choice tree search algorithms for
non-stochastic, fully-observable environments.
Tight Bounds for Collaborative PAC Learning via Multiplicative Weights
Authors: Jiecao Chen, Qin Zhang, Yuan Zhou
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: PAC learning

We study the collaborative PAC learning problem recently proposed in Blum et
al.~\cite{BHPQ17}, in which we have $k$ players and they want to learn a target
function collaboratively, such that the learned function approximates the
target function well on all players' distributions simultaneously. The quality
of the collaborative learning algorithm is measured by the ratio between the
sample complexity of the algorithm and that of the learning algorithm for a
single distribution (called the overhead). We obtain a collaborative learning
algorithm with overhead $O(\ln k)$, improving the one with overhead $O(\ln^2
k)$ in \cite{BHPQ17}. We also show that an $\Omega(\ln k)$ overhead is
inevitable when $k$ is polynomial bounded by the VC dimension of the hypothesis
class. Finally, our experimental study has demonstrated the superiority of our
algorithm compared with the one in Blum et al. on real-world datasets.
A Unified Framework for Training Neural Networks
Authors: Hadi Ghauch, Hossein Shokri-Ghadikolaei, Carlo Fischione, Mikael Skoglund
Date: 2018-05-23 00:00:00
Tasks: General Classification, regression

The lack of mathematical tractability of Deep Neural Networks (DNNs) has
hindered progress towards having a unified convergence analysis of training
algorithms, in the general setting. We propose a unified optimization framework
for training different types of DNNs, and establish its convergence for
arbitrary loss, activation, and regularization functions, assumed to be smooth.
We show that framework generalizes well-known first- and second-order training
methods, and thus allows us to show the convergence of these methods for
various DNN architectures and learning tasks, as a special case of our
approach. We discuss some of its applications in training various DNN
architectures (e.g., feed-forward, convolutional, linear networks), to
regression and classification tasks.
Learning latent variable structured prediction models with Gaussian perturbations
Authors: Kevin Bello, Jean Honorio
Date: 2018-05-23 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Decoder, Generalization Bounds, Prediction, Structured Prediction

The standard margin-based structured prediction commonly uses a maximum loss
over all possible structured outputs. The large-margin formulation including
latent variables not only results in a non-convex formulation but also
increases the search space by a factor of the size of the latent space. Recent
work has proposed the use of the maximum loss over random structured outputs
sampled independently from some proposal distribution, with theoretical
guarantees. We extend this work by including latent variables. We study a new
family of loss functions under Gaussian perturbations and analyze the effect of
the latent space on the generalization bounds. We show that the non-convexity
of learning with latent variables originates naturally, as it relates to a
tight upper bound of the Gibbs decoder distortion with respect to the latent
space. Finally, we provide a formulation using random samples that produces a
tighter upper bound of the Gibbs decoder distortion up to a statistical
accuracy, which enables a faster evaluation of the objective function. We
illustrate the method with synthetic experiments and a computer vision
application.
Learning Pose Specific Representations by Predicting Different Views
Authors: Georg Poier, David Schinagl, Horst Bischof
Date: 2018-04-10 00:00:00
Proceeding: CVPR 2018 6
Tasks: Hand Pose Estimation, Object, Pose Estimation

The labeled data required to learn pose estimation for articulated objects is
difficult to provide in the desired quantity, realism, density, and accuracy.
To address this issue, we develop a method to learn representations, which are
very specific for articulated poses, without the need for labeled training
data. We exploit the observation that the object pose of a known object is
predictive for the appearance in any known view. That is, given only the pose
and shape parameters of a hand, the hand's appearance from any viewpoint can be
approximated. To exploit this observation, we train a model that -- given input
from one view -- estimates a latent representation, which is trained to be
predictive for the appearance of the object when captured from another
viewpoint. Thus, the only necessary supervision is the second view. The
training process of this model reveals an implicit pose representation in the
latent space. Importantly, at test time the pose representation can be inferred
using only a single view. In qualitative and quantitative experiments we show
that the learned representations capture detailed pose information. Moreover,
when training the proposed method jointly with labeled and unlabeled data, it
consistently surpasses the performance of its fully supervised counterpart,
while reducing the amount of needed labeled samples by at least one order of
magnitude.
Abstractive Text Classification Using Sequence-to-convolution Neural Networks
Authors: Taehoon Kim, Jihoon Yang
Date: 2018-05-20 00:00:00
Tasks: Classification, Data Augmentation, General Classification, text-classification, Text Classification

We propose a new deep neural network model and its training scheme for text classification. Our model Sequence-to-convolution Neural Networks(Seq2CNN) consists of two blocks: Sequential Block that summarizes input texts and Convolution Block that receives summary of input and classifies it to a label. Seq2CNN is trained end-to-end to classify various-length texts without preprocessing inputs into fixed length. We also present Gradual Weight Shift(GWS) method that stabilizes training. GWS is applied to our model's loss function. We compared our model with word-based TextCNN trained with different data preprocessing methods. We obtained significant improvement in classification accuracy over word-based TextCNN without any ensemble or data augmentation.
How much does a word weigh? Weighting word embeddings for word sense induction
Authors: Nikolay Arefyev, Pavel Ermolaev, Alexander Panchenko
Date: 2018-05-23 00:00:00
Tasks: Clustering, Machine Translation, Translation, Word Embeddings, Word Sense Induction

The paper describes our participation in the first shared task on word sense
induction and disambiguation for the Russian language RUSSE'2018 (Panchenko et
al., 2018). For each of several dozens of ambiguous words, the participants
were asked to group text fragments containing it according to the senses of
this word, which were not provided beforehand, therefore the "induction" part
of the task. For instance, a word "bank" and a set of text fragments (also
known as "contexts") in which this word occurs, e.g. "bank is a financial
institution that accepts deposits" and "river bank is a slope beside a body of
water" were given. A participant was asked to cluster such contexts in the
unknown in advance number of clusters corresponding to, in this case, the
"company" and the "area" senses of the word "bank". The organizers proposed
three evaluation datasets of varying complexity and text genres based
respectively on texts of Wikipedia, Web pages, and a dictionary of the Russian
language. We present two experiments: a positive and a negative one, based
respectively on clustering of contexts represented as a weighted average of
word embeddings and on machine translation using two state-of-the-art
production neural machine translation systems. Our team showed the second best
result on two datasets and the third best result on the remaining one dataset
among 18 participating teams. We managed to substantially outperform
competitive state-of-the-art baselines from the previous years based on sense
embeddings.
Pushing the bounds of dropout
Authors: Gábor Melis, Charles Blundell, Tomáš Kočiský, Karl Moritz Hermann, Chris Dyer, Phil Blunsom
Date: 2018-05-23 00:00:00
Proceeding: ICLR 2019 5
Tasks: Language Modelling

We show that dropout training is best understood as performing MAP estimation
concurrently for a family of conditional models whose objectives are themselves
lower bounded by the original dropout objective. This discovery allows us to
pick any model from this family after training, which leads to a substantial
improvement on regularisation-heavy language modelling. The family includes
models that compute a power mean over the sampled dropout masks, and their less
stochastic subvariants with tighter and higher lower bounds than the fully
stochastic dropout objective. We argue that since the deterministic
subvariant's bound is equal to its objective, and the highest amongst these
models, the predominant view of it as a good approximation to MC averaging is
misleading. Rather, deterministic dropout is the best available approximation
to the true objective.
Attributes in Multiple Facial Images
Authors: Xudong Liu, Guodong Guo
Date: 2018-05-23 00:00:00
Tasks: Attribute, Face Recognition

Facial attribute recognition is conventionally computed from a single image.
In practice, each subject may have multiple face images. Taking the eye size as
an example, it should not change, but it may have different estimation in
multiple images, which would make a negative impact on face recognition. Thus,
how to compute these attributes corresponding to each subject rather than each
single image is a profound work. To address this question, we deploy deep
training for facial attributes prediction, and we explore the inconsistency
issue among the attributes computed from each single image. Then, we develop
two approaches to address the inconsistency issue. Experimental results show
that the proposed methods can handle facial attribute estimation on either
multiple still images or video frames, and can correct the incorrectly
annotated labels. The experiments are conducted on two large public databases
with annotations of facial attributes.
Fidelity-Weighted Learning
Authors: Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, Bernhard Schölkopf
Date: 2017-11-08 00:00:00
Proceeding: ICLR 2018 1
Tasks: Ad-Hoc Information Retrieval, Information Retrieval, Retrieval

Training deep neural networks requires many training samples, but in practice
training labels are expensive to obtain and may be of varying quality, as some
may be from trusted expert labelers while others might be from heuristics or
other sources of weak supervision such as crowd-sourcing. This creates a
fundamental quality versus-quantity trade-off in the learning process. Do we
learn from the small amount of high-quality data or the potentially large
amount of weakly-labeled data? We argue that if the learner could somehow know
and take the label-quality into account when learning the data representation,
we could get the best of both worlds. To this end, we propose
"fidelity-weighted learning" (FWL), a semi-supervised student-teacher approach
for training deep neural networks using weakly-labeled data. FWL modulates the
parameter updates to a student network (trained on the task we care about) on a
per-sample basis according to the posterior confidence of its label-quality
estimated by a teacher (who has access to the high-quality labels). Both
student and teacher are learned from the data. We evaluate FWL on two tasks in
information retrieval and natural language processing where we outperform
state-of-the-art alternative semi-supervised methods, indicating that our
approach makes better use of strong and weak labels, and leads to better
task-dependent data representations.
Towards the first adversarially robust neural network model on MNIST
Authors: Lukas Schott, Jonas Rauber, Matthias Bethge, Wieland Brendel
Date: 2018-05-23 00:00:00
Proceeding: ICLR 2019 5
Tasks: Adversarial Robustness, Binarization, Robust classification

Despite much effort, deep neural networks remain highly susceptible to tiny
input perturbations and even for MNIST, one of the most common toy datasets in
computer vision, no neural network model exists for which adversarial
perturbations are large and make semantic sense to humans. We show that even
the widely recognized and by far most successful defense by Madry et al. (1)
overfits on the L-infinity metric (it's highly susceptible to L2 and L0
perturbations), (2) classifies unrecognizable images with high certainty, (3)
performs not much better than simple input binarization and (4) features
adversarial perturbations that make little sense to humans. These results
suggest that MNIST is far from being solved in terms of adversarial robustness.
We present a novel robust classification model that performs analysis by
synthesis using learned class-conditional data distributions. We derive bounds
on the robustness and go to great length to empirically evaluate our model
using maximally effective adversarial attacks by (a) applying decision-based,
score-based, gradient-based and transfer-based attacks for several different Lp
norms, (b) by designing a new attack that exploits the structure of our
defended model and (c) by devising a novel decision-based attack that seeks to
minimize the number of perturbed pixels (L0). The results suggest that our
approach yields state-of-the-art robustness on MNIST against L0, L2 and
L-infinity perturbations and we demonstrate that most adversarial examples are
strongly perturbed towards the perceptual boundary between the original and the
adversarial class.