pwc-archive/papers-with-abstracts huggingface dataset:
A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
Authors: Yongqiang Cai, Qianxiao Li, Zuowei Shen
Date: 2018-09-29 00:00:00
Proceeding: ICLR 2019 5

Despite its empirical success and recent theoretical progress, there generally lacks a quantitative analysis of the effect of batch normalization (BN) on the convergence and stability of gradient descent. In this paper, we provide such an analysis on the simple problem of ordinary least squares (OLS). Since precise dynamical properties of gradient descent (GD) is completely known for the OLS problem, it allows us to isolate and compare the additional effects of BN. More precisely, we show that unlike GD, gradient descent with BN (BNGD) converges for arbitrary learning rates for the weights, and the convergence remains linear under mild conditions. Moreover, we quantify two different sources of acceleration of BNGD over GD -- one due to over-parameterization which improves the effective condition number and another due having a large range of learning rates giving rise to fast descent. These phenomena set BNGD apart from GD and could account for much of its robustness properties. These findings are confirmed quantitatively by numerical experiments, which further show that many of the uncovered properties of BNGD in OLS are also observed qualitatively in more complex supervised learning problems.
Visual Object Tracking based on Adaptive Siamese and Motion Estimation Network
Authors: Hossein Kashiani, Shahriar B. Shokouhi
Date: 2018-09-29 00:00:00
Tasks: Motion Estimation, Object Tracking, Visual Object Tracking

Recently, convolutional neural network (CNN) has attracted much attention in
different areas of computer vision, due to its powerful abstract feature
representation. Visual object tracking is one of the interesting and important
areas in computer vision that achieves remarkable improvements in recent years.
In this work, we aim to improve both the motion and observation models in
visual object tracking by leveraging representation power of CNNs. To this end,
a motion estimation network (named MEN) is utilized to seek the most likely
locations of the target and prepare a further clue in addition to the previous
target position. Hence the motion estimation would be enhanced by generating a
small number of candidates near two plausible positions. The generated
candidates are then fed into a trained Siamese network to detect the most
probable candidate. Each candidate is compared to an adaptable buffer, which is
updated under a predefined condition. To take into account the target
appearance changes, a weighting CNN (called WCNN) adaptively assigns weights to
the final similarity scores of the Siamese network using sequence-specific
information. Evaluation results on well-known benchmark datasets (OTB100, OTB50
and OTB2013) prove that the proposed tracker outperforms the state-of-the-art
competitors.
Improved Gradient-Based Optimization Over Discrete Distributions
Authors: Evgeny Andriyash, Arash Vahdat, Bill Macready
Date: 2018-09-29 00:00:00
Tasks: Variational Inference

In many applications we seek to maximize an expectation with respect to a distribution over discrete variables. Estimating gradients of such objectives with respect to the distribution parameters is a challenging problem. We analyze existing solutions including finite-difference (FD) estimators and continuous relaxation (CR) estimators in terms of bias and variance. We show that the commonly used Gumbel-Softmax estimator is biased and propose a simple method to reduce it. We also derive a simpler piece-wise linear continuous relaxation that also possesses reduced bias. We demonstrate empirically that reduced bias leads to a better performance in variational inference and on binary optimization tasks.
Domain-Adversarial Multi-Task Framework for Novel Therapeutic Property Prediction of Compounds
Authors: Lingwei Xie, Song He, Shu Yang, Boyuan Feng, Kun Wan, Zhongnan Zhang, Xiaochen Bo, Yufei Ding
Date: 2018-09-28 00:00:00
Tasks: Property Prediction

With the rapid development of high-throughput technologies, parallel
acquisition of large-scale drug-informatics data provides huge opportunities to
improve pharmaceutical research and development. One significant application is
the purpose prediction of small molecule compounds, aiming to specify
therapeutic properties of extensive purpose-unknown compounds and to repurpose
novel therapeutic properties of FDA-approved drugs. Such problem is very
challenging since compound attributes contain heterogeneous data with various
feature patterns such as drug fingerprint, drug physicochemical property, drug
perturbation gene expression. Moreover, there is complex nonlinear dependency
among heterogeneous data. In this paper, we propose a novel domain-adversarial
multi-task framework for integrating shared knowledge from multiple domains.
The framework utilizes the adversarial strategy to effectively learn target
representations and models their nonlinear dependency. Experiments on two
real-world datasets illustrate that the performance of our approach obtains an
obvious improvement over competitive baselines. The novel therapeutic
properties of purpose-unknown compounds we predicted are mostly reported or
brought to the clinics. Furthermore, our framework can integrate various
attributes beyond the three domains examined here and can be applied in the
industry for screening the purpose of huge amounts of as yet unidentified
compounds. Source codes of this paper are available on Github.
Predicting the Generalization Gap in Deep Networks with Margin Distributions
Authors: Yiding Jiang, Dilip Krishnan, Hossein Mobahi, Samy Bengio
Date: 2018-09-28 00:00:00
Proceeding: ICLR 2019 5

As shown in recent research, deep neural networks can perfectly fit randomly labeled data, but with very poor accuracy on held out data. This phenomenon indicates that loss functions such as cross-entropy are not a reliable indicator of generalization. This leads to the crucial question of how generalization gap should be predicted from the training data and network parameters. In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap. Our measure is based on the concept of margin distribution, which are the distances of training points to the decision boundary. We find that it is necessary to use margin distributions at multiple layers of a deep network. On the CIFAR-10 and the CIFAR-100 datasets, our proposed measure correlates very strongly with the generalization gap. In addition, we find the following other factors to be of importance: normalizing margin values for scale independence, using characterizations of margin distribution rather than just the margin (closest distance to decision boundary), and working in log space instead of linear space (effectively using a product of margins rather than a sum). Our measure can be easily applied to feedforward deep networks with any architecture and may point towards new training loss functions that could enable better generalization.
DeepSSM: A Deep Learning Framework for Statistical Shape Modeling from Raw Images
Authors: Riddhish Bhalodia, Shireen Y. Elhabian, Ladislav Kavan, Ross T. Whitaker
Date: 2018-09-28 00:00:00
Tasks: Data Augmentation

Statistical shape modeling is an important tool to characterize variation in
anatomical morphology. Typical shapes of interest are measured using 3D imaging
and a subsequent pipeline of registration, segmentation, and some extraction of
shape features or projections onto some lower-dimensional shape space, which
facilitates subsequent statistical analysis. Many methods for constructing
compact shape representations have been proposed, but are often impractical due
to the sequence of image preprocessing operations, which involve significant
parameter tuning, manual delineation, and/or quality control by the users. We
propose DeepSSM: a deep learning approach to extract a low-dimensional shape
representation directly from 3D images, requiring virtually no parameter tuning
or user assistance. DeepSSM uses a convolutional neural network (CNN) that
simultaneously localizes the biological structure of interest, establishes
correspondences, and projects these points onto a low-dimensional shape
representation in the form of PCA loadings within a point distribution model.
To overcome the challenge of the limited availability of training images, we
present a novel data augmentation procedure that uses existing correspondences
on a relatively small set of processed images with shape statistics to create
plausible training samples with known shape parameters. Hence, we leverage the
limited CT/MRI scans (40-50) into thousands of images needed to train a CNN.
After the training, the CNN automatically produces accurate low-dimensional
shape representations for unseen images. We validate DeepSSM for three
different applications pertaining to modeling pediatric cranial CT for
characterization of metopic craniosynostosis, femur CT scans identifying
morphologic deformities of the hip due to femoroacetabular impingement, and
left atrium MRI scans for atrial fibrillation recurrence prediction.
Open-Ended Content-Style Recombination Via Leakage Filtering
Authors: Karl Ridgeway, Michael C. Mozer
Date: 2018-09-28 00:00:00
Proceeding: ICLR 2019 5
Tasks: Few-Shot Learning, Metric Learning

We consider visual domains in which a class label specifies the content of an
image, and class-irrelevant properties that differentiate instances constitute
the style. We present a domain-independent method that permits the open-ended
recombination of style of one image with the content of another. Open ended
simply means that the method generalizes to style and content not present in
the training data. The method starts by constructing a content embedding using
an existing deep metric-learning technique. This trained content encoder is
incorporated into a variational autoencoder (VAE), paired with a to-be-trained
style encoder. The VAE reconstruction loss alone is inadequate to ensure a
decomposition of the latent representation into style and content. Our method
thus includes an auxiliary loss, leakage filtering, which ensures that no style
information remaining in the content representation is used for reconstruction
and vice versa. We synthesize novel images by decoding the style representation
obtained from one image with the content representation from another. Using
this method for data-set augmentation, we obtain state-of-the-art performance
on few-shot learning tasks.
Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture
Authors: Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, Maja Pantic
Date: 2018-09-28 00:00:00
Tasks: Audio-Visual Speech Recognition, Automatic Speech Recognition (ASR), Lipreading, speech-recognition, Speech Recognition, Visual Speech Recognition

Recent works in speech recognition rely either on connectionist temporal
classification (CTC) or sequence-to-sequence models for character-level
recognition. CTC assumes conditional independence of individual characters,
whereas attention-based models can provide nonsequential alignments. Therefore,
we could use a CTC loss in combination with an attention-based model in order
to force monotonic alignments and at the same time get rid of the conditional
independence assumption. In this paper, we use the recently proposed hybrid
CTC/attention architecture for audio-visual recognition of speech in-the-wild.
To the best of our knowledge, this is the first time that such a hybrid
architecture architecture is used for audio-visual recognition of speech. We
use the LRS2 database and show that the proposed audio-visual model leads to an
1.3% absolute decrease in word error rate over the audio-only model and
achieves the new state-of-the-art performance on LRS2 database (7% word error
rate). We also observe that the audio-visual model significantly outperforms
the audio-based model (up to 32.9% absolute improvement in word error rate) for
several different types of noise as the signal-to-noise ratio decreases.
Superimposition-guided Facial Reconstruction from Skull
Authors: Celong Liu, Xin Li
Date: 2018-09-28 00:00:00
Tasks: Facial Inpainting

We develop a new algorithm to perform facial reconstruction from a given
skull. This technique has forensic application in helping the identification of
skeletal remains when other information is unavailable. Unlike most existing
strategies that directly reconstruct the face from the skull, we utilize a
database of portrait photos to create many face candidates, then perform a
superimposition to get a well matched face, and then revise it according to the
superimposition. To support this pipeline, we build an effective autoencoder
for image-based facial reconstruction, and a generative model for constrained
face inpainting. Our experiments have demonstrated that the proposed pipeline
is stable and accurate.
Predicting Destinations by Nearest Neighbor Search on Training Vessel Routes
Authors: Valentin Roşca, Emanuel Onica, Paul Diac, Ciprian Amariei
Date: 2018-09-28 00:00:00

The DEBS Grand Challenge 2018 is set in the context of maritime route
prediction. Vessel routes are modeled as streams of Automatic Identification
System (AIS) data points selected from real-world tracking data. The challenge
requires to correctly estimate the destination ports and arrival times of
vessel trips, as early as possible. Our proposed solution partitions the
training vessel routes by reported destination port and uses a nearest neighbor
search to find the training routes that are closer to the query AIS point.
Particular improvements have been included as well, such as a way to avoid
changing the predicted ports frequently within one query route and automating
the parameters tuning by the use of a genetic algorithm. This leads to
significant improvements on the final score.
The Partially Observable Games We Play for Cyber Deception
Authors: Mohamadreza Ahmadi, Murat Cubuktepe, Nils Jansen, Sebastian Junges, Joost-Pieter Katoen, Ufuk Topcu
Date: 2018-09-28 00:00:00

Progressively intricate cyber infiltration mechanisms have made conventional
means of defense, such as firewalls and malware detectors, incompetent. These
sophisticated infiltration mechanisms can study the defender's behavior,
identify security caveats, and modify their actions adaptively. To tackle these
security challenges, cyber-infrastructures require active defense techniques
that incorporate cyber deception, in which the defender (deceiver) implements a
strategy to mislead the infiltrator. To this end, we use a two-player partially
observable stochastic game (POSG) framework, wherein the deceiver has full
observability over the states of the POSG, and the infiltrator has partial
observability. Then, the deception problem is to compute a strategy for the
deceiver that minimizes the expected cost of deception against all strategies
of the infiltrator. We first show that the underlying problem is a robust
mixed-integer linear program, which is intractable to solve in general. Towards
a scalable approach, we compute optimal finite-memory strategies for the
infiltrator by a reduction to a series of synthesis problems for parametric
Markov decision processes. We use these infiltration strategies to find robust
strategies for the deceiver using mixed-integer linear programming. We
illustrate the performance of our technique on a POSG model for network
security. Our experiments demonstrate that the proposed approach handles
scenarios considerably larger than those of the state-of-the-art methods.
Reconciling Feature-Reuse and Overfitting in DenseNet with Specialized Dropout
Authors: Kun Wan, Boyuan Feng, Lingwei Xie, Yufei Ding
Date: 2018-09-28 00:00:00
Proceeding: ICLR 2019 5

Recently convolutional neural networks (CNNs) achieve great accuracy in
visual recognition tasks. DenseNet becomes one of the most popular CNN models
due to its effectiveness in feature-reuse. However, like other CNN models,
DenseNets also face overfitting problem if not severer. Existing dropout method
can be applied but not as effective due to the introduced nonlinear
connections. In particular, the property of feature-reuse in DenseNet will be
impeded, and the dropout effect will be weakened by the spatial correlation
inside feature maps. To address these problems, we craft the design of a
specialized dropout method from three aspects, dropout location, dropout
granularity, and dropout probability. The insights attained here could
potentially be applied as a general approach for boosting the accuracy of other
CNN models with similar nonlinear connections. Experimental results show that
DenseNets with our specialized dropout method yield better accuracy compared to
vanilla DenseNet and state-of-the-art CNN models, and such accuracy boost
increases with the model depth.
Cell Grid Architecture for Maritime Route Prediction on AIS Data Streams
Authors: Ciprian Amariei, Paul Diac, Emanuel Onica, Valentin Roşca
Date: 2018-09-28 00:00:00

The 2018 Grand Challenge targets the problem of accurate predictions on data
streams produced by automatic identification system (AIS) equipment, describing
naval traffic. This paper reports the technical details of a custom solution,
which exposes multiple tuning parameters, making its configurability one of the
main strengths. Our solution employs a cell grid architecture essentially based
on a sequence of hash tables, specifically built for the targeted use case.
This makes it particularly effective in prediction on AIS data, obtaining a
high accuracy and scalable performance results. Moreover, the architecture
proposed accommodates also an optionally semi-supervised learning process
besides the basic supervised mode.
Deep Residual Network for Off-Resonance Artifact Correction with Application to Pediatric Body Magnetic Resonance Angiography with 3D Cones
Authors: David Y Zeng, Jamil Shaikh, Dwight G. Nishimura, Shreyas S. Vasanawala, Joseph Y. Cheng
Date: 2018-09-28 00:00:00
Tasks: SSIM

Purpose: Off-resonance artifact correction by deep-learning, to facilitate
rapid pediatric body imaging with a scan time efficient 3D cones trajectory.
Methods: A residual convolutional neural network to correct off-resonance
artifacts (Off-ResNet) was trained with a prospective study of 30 pediatric
magnetic resonance angiography exams. Each exam acquired a short-readout scan
(1.18 ms +- 0.38) and a long-readout scan (3.35 ms +- 0.74) at 3T.
Short-readout scans, with longer scan times but negligible off-resonance
blurring, were used as reference images and augmented with additional
off-resonance for supervised training examples. Long-readout scans, with
greater off-resonance artifacts but shorter scan time, were corrected by
autofocus and Off-ResNet and compared to short-readout scans by normalized
root-mean-square error (NRMSE), structural similarity index (SSIM), and peak
signal-to-noise ratio (PSNR). Scans were also compared by scoring on eight
anatomical features by two radiologists, using analysis of variance with
post-hoc Tukey's test. Reader agreement was determined with intraclass
correlation. Results: Long-readout scans were on average 59.3% shorter than
short-readout scans. Images from Off-ResNet had superior NRMSE, SSIM, and PSNR
compared to uncorrected images across +-1kHz off-resonance (P<0.01). The
proposed method had superior NRMSE over -677Hz to +1kHz and superior SSIM and
PSNR over +-1kHz compared to autofocus (P<0.01). Radiologic scoring
demonstrated that long-readout scans corrected with Off-ResNet were
non-inferior to short-readout scans (P<0.01). Conclusion: The proposed method
can correct off-resonance artifacts from rapid long-readout 3D cones scans to a
non-inferior image quality compared to diagnostically standard short-readout
scans.
Adversarial Attacks and Defences: A Survey
Authors: Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, Debdeep Mukhopadhyay
Date: 2018-09-28 00:00:00
Tasks: Deep Learning, Survey

Deep learning has emerged as a strong and efficient framework that can be
applied to a broad spectrum of complex learning problems which were difficult
to solve using the traditional machine learning techniques in the past. In the
last few years, deep learning has advanced radically in such a way that it can
surpass human-level performance on a number of tasks. As a consequence, deep
learning is being extensively used in most of the recent day-to-day
applications. However, security of deep learning systems are vulnerable to
crafted adversarial examples, which may be imperceptible to the human eye, but
can lead the model to misclassify the output. In recent times, different types
of adversaries based on their threat model leverage these vulnerabilities to
compromise a deep learning system where adversaries have high incentives.
Hence, it is extremely important to provide robustness to deep learning
algorithms against these adversaries. However, there are only a few strong
countermeasures which can be used in all types of attack scenarios to design a
robust deep learning system. In this paper, we attempt to provide a detailed
discussion on different types of adversarial attacks with various threat models
and also elaborate the efficiency and challenges of recent countermeasures
against them.
Differentially Private Contextual Linear Bandits
Authors: Roshan Shariff, Or Sheffet
Date: 2018-09-28 00:00:00
Proceeding: NeurIPS 2018 12

We study the contextual linear bandit problem, a version of the standard
stochastic multi-armed bandit (MAB) problem where a learner sequentially
selects actions to maximize a reward which depends also on a user provided
per-round context. Though the context is chosen arbitrarily or adversarially,
the reward is assumed to be a stochastic function of a feature vector that
encodes the context and selected action. Our goal is to devise private learners
for the contextual linear bandit problem.
  We first show that using the standard definition of differential privacy
results in linear regret. So instead, we adopt the notion of joint differential
privacy, where we assume that the action chosen on day $t$ is only revealed to
user $t$ and thus needn't be kept private that day, only on following days. We
give a general scheme converting the classic linear-UCB algorithm into a joint
differentially private algorithm using the tree-based algorithm. We then apply
either Gaussian noise or Wishart noise to achieve joint-differentially private
algorithms and bound the resulting algorithms' regrets. In addition, we give
the first lower bound on the additional regret any private algorithms for the
MAB problem must incur.