pwc-archive/papers-with-abstracts huggingface dataset:
Convergence of Contrastive Divergence Algorithm in Exponential Family
Authors: Bai Jiang, Tung-Yu Wu, Yifan Jin, Wing H. Wong
Date: 2016-03-17 00:00:00

The Contrastive Divergence (CD) algorithm has achieved notable success in
training energy-based models including Restricted Boltzmann Machines and played
a key role in the emergence of deep learning. The idea of this algorithm is to
approximate the intractable term in the exact gradient of the log-likelihood
function by using short Markov chain Monte Carlo (MCMC) runs. The approximate
gradient is computationally-cheap but biased. Whether and why the CD algorithm
provides an asymptotically consistent estimate are still open questions. This
paper studies the asymptotic properties of the CD algorithm in canonical
exponential families, which are special cases of the energy-based model.
Suppose the CD algorithm runs $m$ MCMC transition steps at each iteration $t$
and iteratively generates a sequence of parameter estimates $\{\theta_t\}_{t
\ge 0}$ given an i.i.d. data sample $\{X_i\}_{i=1}^n \sim p_{\theta_\star}$.
Under conditions which are commonly obeyed by the CD algorithm in practice, we
prove the existence of some bounded $m$ such that any limit point of the time
average $\left. \sum_{s=0}^{t-1} \theta_s \right/ t$ as $t \to \infty$ is a
consistent estimate for the true parameter $\theta_\star$. Our proof is based
on the fact that $\{\theta_t\}_{t \ge 0}$ is a homogenous Markov chain
conditional on the data sample $\{X_i\}_{i=1}^n$. This chain meets the
Foster-Lyapunov drift criterion and converges to a random walk around the
Maximum Likelihood Estimate. The range of the random walk shrinks to zero at
rate $\mathcal{O}(1/\sqrt[3]{n})$ as the sample size $n \to \infty$.
Testing for Feature Relevance: The HARVEST Algorithm
Authors: Herbert Weisberg, Victor Pontes, Mathis Thoma
Date: 2017-09-30 00:00:00
Tasks: feature selection

Feature selection with high-dimensional data and a very small proportion of
relevant features poses a severe challenge to standard statistical methods. We
have developed a new approach (HARVEST) that is straightforward to apply,
albeit somewhat computer-intensive. This algorithm can be used to pre-screen a
large number of features to identify those that are potentially useful. The
basic idea is to evaluate each feature in the context of many random subsets of
other features. HARVEST is predicated on the assumption that an irrelevant
feature can add no real predictive value, regardless of which other features
are included in the subset. Motivated by this idea, we have derived a simple
statistical test for feature relevance. Empirical analyses and simulations
produced so far indicate that the HARVEST algorithm is highly effective in
predictive analytics, both in science and business.
Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
Authors: Randall Balestriero, Herve Glotin, Richard Baraniuk
Date: 2018-02-27 00:00:00

Deep Neural Networks (DNNs) provide state-of-the-art solutions in several
difficult machine perceptual tasks. However, their performance relies on the
availability of a large set of labeled training data, which limits the breadth
of their applicability. Hence, there is a need for new {\em semi-supervised
learning} methods for DNNs that can leverage both (a small amount of) labeled
and unlabeled training data. In this paper, we develop a general loss function
enabling DNNs of any topology to be trained in a semi-supervised manner without
extra hyper-parameters. As opposed to current semi-supervised techniques based
on topology-specific or unstable approaches, ours is both robust and general.
We demonstrate that our approach reaches state-of-the-art performance on the
SVHN ($9.82\%$ test error, with $500$ labels and wide Resnet) and CIFAR10
(16.38% test error, with 8000 labels and sigmoid convolutional neural network)
data sets.
Tell Me Where to Look: Guided Attention Inference Network
Authors: Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, Yun Fu
Date: 2018-02-27 00:00:00
Proceeding: CVPR 2018 6
Tasks: Object Localization, Semantic Segmentation, Weakly-supervised Learning

Weakly supervised learning with only coarse labels can obtain visual
explanations of deep neural network such as attention maps by back-propagating
gradients. These attention maps are then available as priors for tasks such as
object localization and semantic segmentation. In one common framework we
address three shortcomings of previous approaches in modeling such attention
maps: We (1) first time make attention maps an explicit and natural component
of the end-to-end training, (2) provide self-guidance directly on these maps by
exploring supervision form the network itself to improve them, and (3)
seamlessly bridge the gap between using weak and extra supervision if
available. Despite its simplicity, experiments on the semantic segmentation
task demonstrate the effectiveness of our methods. We clearly surpass the
state-of-the-art on Pascal VOC 2012 val. and test set. Besides, the proposed
framework provides a way not only explaining the focus of the learner but also
feeding back with direct guidance towards specific tasks. Under mild
assumptions our method can also be understood as a plug-in to existing weakly
supervised learners to improve their generalization performance.
ADMM-based Networked Stochastic Variational Inference
Authors: Hamza Anwar, Quanyan Zhu
Date: 2018-02-27 00:00:00
Tasks: Articles, Document Classification, General Classification, Stochastic Optimization, Variational Inference

Owing to the recent advances in "Big Data" modeling and prediction tasks,
variational Bayesian estimation has gained popularity due to their ability to
provide exact solutions to approximate posteriors. One key technique for
approximate inference is stochastic variational inference (SVI). SVI poses
variational inference as a stochastic optimization problem and solves it
iteratively using noisy gradient estimates. It aims to handle massive data for
predictive and classification tasks by applying complex Bayesian models that
have observed as well as latent variables. This paper aims to decentralize it
allowing parallel computation, secure learning and robustness benefits. We use
Alternating Direction Method of Multipliers in a top-down setting to develop a
distributed SVI algorithm such that independent learners running inference
algorithms only require sharing the estimated model parameters instead of their
private datasets. Our work extends the distributed SVI-ADMM algorithm that we
first propose, to an ADMM-based networked SVI algorithm in which not only are
the learners working distributively but they share information according to
rules of a graph by which they form a network. This kind of work lies under the
umbrella of `deep learning over networks' and we verify our algorithm for a
topic-modeling problem for corpus of Wikipedia articles. We illustrate the
results on latent Dirichlet allocation (LDA) topic model in large document
classification, compare performance with the centralized algorithm, and use
numerical experiments to corroborate the analytical results.
Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter
Authors: Ziqi Zhang, Lei Luo
Date: 2018-02-27 00:00:00
Tasks: Hate Speech Detection

In recent years, the increasing propagation of hate speech on social media
and the urgent need for effective counter-measures have drawn significant
investment from governments, companies, and researchers. A large number of
methods have been developed for automated hate speech detection online. This
aims to classify textual content into non-hate or hate speech, in which case
the method may also identify the targeting characteristics (i.e., types of
hate, such as race, and religion) in the hate speech. However, we notice
significant difference between the performance of the two (i.e., non-hate v.s.
hate). In this work, we argue for a focus on the latter problem for practical
reasons. We show that it is a much more challenging task, as our analysis of
the language in the typical datasets shows that hate speech lacks unique,
discriminative features and therefore is found in the 'long tail' in a dataset
that is difficult to discover. We then propose Deep Neural Network structures
serving as feature extractors that are particularly effective for capturing the
semantics of hate speech. Our methods are evaluated on the largest collection
of hate speech datasets based on Twitter, and are shown to be able to
outperform the best performing method by up to 5 percentage points in
macro-average F1, or 8 percentage points in the more challenging case of
identifying hateful content.
Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
Authors: Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, Aaron Courville
Date: 2018-02-27 00:00:00
Proceeding: ICML 2018 7
Tasks: Image Segmentation, Semantic Segmentation, Structured Prediction

Learning inter-domain mappings from unpaired data can improve performance in
structured prediction tasks, such as image segmentation, by reducing the need
for paired data. CycleGAN was recently proposed for this problem, but
critically assumes the underlying inter-domain mapping is approximately
deterministic and one-to-one. This assumption renders the model ineffective for
tasks requiring flexible, many-to-many mappings. We propose a new model, called
Augmented CycleGAN, which learns many-to-many mappings between domains. We
examine Augmented CycleGAN qualitatively and quantitatively on several image
datasets.
Extractive Text Summarization using Neural Networks
Authors: Aakash Sinha, Abhishek Yadav, Akshay Gahlot
Date: 2018-02-27 00:00:00
Tasks: Document Summarization, Extractive Text Summarization, Feature Engineering, Text Summarization

Text Summarization has been an extensively studied problem. Traditional
approaches to text summarization rely heavily on feature engineering. In
contrast to this, we propose a fully data-driven approach using feedforward
neural networks for single document summarization. We train and evaluate the
model on standard DUC 2002 dataset which shows results comparable to the state
of the art models. The proposed model is scalable and is able to produce the
summary of arbitrarily sized documents by breaking the original document into
fixed sized parts and then feeding it recursively to the network.
Detecting Statistical Interactions from Neural Network Weights
Authors: Michael Tsang, Dehua Cheng, Yan Liu
Date: 2017-05-14 00:00:00
Proceeding: ICLR 2018 1

Interpreting neural networks is a crucial and challenging task in machine
learning. In this paper, we develop a novel framework for detecting statistical
interactions captured by a feedforward multilayer neural network by directly
interpreting its learned weights. Depending on the desired interactions, our
method can achieve significantly better or similar interaction detection
performance compared to the state-of-the-art without searching an exponential
solution space of possible interactions. We obtain this accuracy and efficiency
by observing that interactions between input features are created by the
non-additive effect of nonlinear activation functions, and that interacting
paths are encoded in weight matrices. We demonstrate the performance of our
method and the importance of discovered interactions via experimental results
on both synthetic datasets and real-world application datasets.
Reconstruction of partially sampled multi-band images - Application to STEM-EELS imaging
Authors: Étienne Monier, Thomas Oberlin, Nathalie Brun, Marcel Tencé, Marta de Frutos, Nicolas Dobigeon
Date: 2018-02-27 00:00:00

Electron microscopy has shown to be a very powerful tool to map the chemical
nature of samples at various scales down to atomic resolution. However, many
samples can not be analyzed with an acceptable signal-to-noise ratio because of
the radiation damage induced by the electron beam. This is particularly crucial
for electron energy loss spectroscopy (EELS) which acquires spectral-spatial
data and requires high beam intensity. Since scanning transmission electron
microscopes (STEM) are able to acquire data cubes by scanning the electron
probe over the sample and recording a spectrum for each spatial position, it is
possible to design the scan pattern and to sample only specific pixels. As a
consequence, partial acquisition schemes are now conceivable, provided a
reconstruction of the full data cube is conducted as a post-processing step.
This paper proposes two reconstruction algorithms for multi-band images
acquired by STEM-EELS which exploits the spectral structure and the spatial
smoothness of the image. The performance of the proposed schemes is illustrated
thanks to experiments conducted on a realistic phantom dataset as well as real
EELS spectrum-images.
Domain Modelling in Computational Persuasion for Behaviour Change in Healthcare
Authors: Lisa Chalaguine, Emmanuel Hadoux, Fiona Hamilton, Andrew Hayward, Anthony Hunter, Sylwia Polberg, Henry W. W. Potts
Date: 2018-02-27 00:00:00

The aim of behaviour change is to help people to change aspects of their
behaviour for the better (e.g., to decrease calorie intake, to drink in
moderation, to take more exercise, to complete a course of antibiotics once
started, etc.). In current persuasion technology for behaviour change, the
emphasis is on helping people to explore their issues (e.g., through
questionnaires or game playing) or to remember to follow a behaviour change
plan (e.g., diaries and email reminders). However, recent developments in
computational persuasion are leading to an argument-centric approach to
persuasion that can potentially be harnessed in behaviour change applications.
In this paper, we review developments in computational persuasion, and then
focus on domain modelling as a key component. We present a multi-dimensional
approach to domain modelling. At the core of this proposal is an ontology which
provides a representation of key factors, in particular kinds of belief, which
we have identified in the behaviour change literature as being important in
diverse behaviour change initiatives. Our proposal for domain modelling is
intended to facilitate the acquisition and representation of the arguments that
can be used in persuasion dialogues, together with meta-level information about
them which can be used by the persuader to make strategic choices of argument
to present.
Generating High Quality Visible Images from SAR Images Using CNNs
Authors: Puyang Wang, Vishal M. Patel
Date: 2018-02-27 00:00:00
Tasks: Colorization, Generative Adversarial Network, Image Colorization, Vocal Bursts Intensity Prediction

We propose a novel approach for generating high quality visible-like images
from Synthetic Aperture Radar (SAR) images using Deep Convolutional Generative
Adversarial Network (GAN) architectures. The proposed approach is based on a
cascaded network of convolutional neural nets (CNNs) for despeckling and image
colorization. The cascaded structure results in faster convergence during
training and produces high quality visible images from the corresponding SAR
images. Experimental results on both simulated and real SAR images show that
the proposed method can produce visible-like images better compared to the
recent state-of-the-art deep learning-based methods.
Improving OCR Accuracy on Early Printed Books using Deep Convolutional Networks
Authors: Christoph Wick, Christian Reul, Frank Puppe
Date: 2018-02-27 00:00:00
Tasks: Optical Character Recognition (OCR)

This paper proposes a combination of a convolutional and a LSTM network to
improve the accuracy of OCR on early printed books. While the standard model of
line based OCR uses a single LSTM layer, we utilize a CNN- and Pooling-Layer
combination in advance of an LSTM layer. Due to the higher amount of trainable
parameters the performance of the network relies on a high amount of training
examples to unleash its power. Hereby, the error is reduced by a factor of up
to 44%, yielding a CER of 1% and below. To further improve the results we use a
voting mechanism to achieve character error rates (CER) below $0.5%$. The
runtime of the deep model for training and prediction of a book behaves very
similar to a shallow network.
Simultaneous Traffic Sign Detection and Boundary Estimation using Convolutional Neural Network
Authors: Hee Seok Lee, Kang Kim
Date: 2018-02-27 00:00:00
Tasks: Image Segmentation, Semantic Segmentation, Traffic Sign Detection

We propose a novel traffic sign detection system that simultaneously
estimates the location and precise boundary of traffic signs using
convolutional neural network (CNN). Estimating the precise boundary of traffic
signs is important in navigation systems for intelligent vehicles where traffic
signs can be used as 3D landmarks for road environment. Previous traffic sign
detection systems, including recent methods based on CNN, only provide bounding
boxes of traffic signs as output, and thus requires additional processes such
as contour estimation or image segmentation to obtain the precise sign
boundary. In this work, the boundary estimation of traffic signs is formulated
as a 2D pose and shape class prediction problem, and this is effectively solved
by a single CNN. With the predicted 2D pose and the shape class of a target
traffic sign in an input image, we estimate the actual boundary of the target
sign by projecting the boundary of a corresponding template sign image into the
input image plane. By formulating the boundary estimation problem as a
CNN-based pose and shape prediction task, our method is end-to-end trainable,
and more robust to occlusion and small targets than other boundary estimation
methods that rely on contour estimation or image segmentation. The proposed
method with architectural optimization provides an accurate traffic sign
boundary estimation which is also efficient in compute, showing a detection
frame rate higher than 7 frames per second on low-power mobile platforms.
Improving precision and recall of face recognition in SIPP with combination of modified mean search and LSH
Authors: Xihua Li
Date: 2017-09-09 00:00:00
Tasks: Face Recognition

Although face recognition has been improved much as the development of Deep
Neural Networks, SIPP(Single Image Per Person) problem in face recognition has
not been better solved, especially in practical applications where searching
over complicated database. In this paper, a combination of modified mean search
and LSH method would be introduced orderly to improve the precision and recall
of SIPP face recognition without retrain of the DNN model. First, a modified
SVD based augmentation method would be introduced to get more intra-class
variations even for person with only one image. Second, an unique rule based
combination of modified mean search and LSH method was proposed the first time
to help get the most similar personID in a complicated dataset, and some
theoretical explaining followed. Third, we would like to emphasize, no need to
retrain of the DNN model and would easy to be extended without much efforts. We
do some practical testing in competition of Msceleb challenge-2 2017 which was
hold by Microsoft Research, great improvement of coverage from 13.39% to
19.25%, 29.94%, 42.11%, 47.52% at precision 99%(P99) would be shown latter,
coverage reach 94.2% and 100% at precision 97%(P97) and 95%(P95) respectively.
As far as we known, this is the only paper who do not fine-tuning on
competition dataset and ranked top-10. A similar test on CASIA WebFace dataset
also demonstrated the same improvements on both precision and recall.
Deep Learning Architectures for Face Recognition in Video Surveillance
Authors: Saman Bashbaghi, Eric Granger, Robert Sabourin, Mostafa Parchami
Date: 2018-02-27 00:00:00
Tasks: Deep Learning, Face Recognition, Triplet

Face recognition (FR) systems for video surveillance (VS) applications
attempt to accurately detect the presence of target individuals over a
distributed network of cameras. In video-based FR systems, facial models of
target individuals are designed a priori during enrollment using a limited
number of reference still images or video data. These facial models are not
typically representative of faces being observed during operations due to large
variations in illumination, pose, scale, occlusion, blur, and to camera
inter-operability. Specifically, in still-to-video FR application, a single
high-quality reference still image captured with still camera under controlled
conditions is employed to generate a facial model to be matched later against
lower-quality faces captured with video cameras under uncontrolled conditions.
Current video-based FR systems can perform well on controlled scenarios, while
their performance is not satisfactory in uncontrolled scenarios mainly because
of the differences between the source (enrollment) and the target (operational)
domains. Most of the efforts in this area have been toward the design of robust
video-based FR systems in unconstrained surveillance environments. This chapter
presents an overview of recent advances in still-to-video FR scenario through
deep convolutional neural networks (CNNs). In particular, deep learning
architectures proposed in the literature based on triplet-loss function (e.g.,
cross-correlation matching CNN, trunk-branch ensemble CNN and HaarNet) and
supervised autoencoders (e.g., canonical face representation CNN) are reviewed
and compared in terms of accuracy and computational complexity.
Mostly Exploration-Free Algorithms for Contextual Bandits
Authors: Hamsa Bastani, Mohsen Bayati, Khashayar Khosravi
Date: 2017-04-28 00:00:00
Tasks: Diversity, Multi-Armed Bandits, Thompson Sampling

The contextual bandit literature has traditionally focused on algorithms that address the exploration-exploitation tradeoff. In particular, greedy algorithms that exploit current estimates without any exploration may be sub-optimal in general. However, exploration-free greedy algorithms are desirable in practical settings where exploration may be costly or unethical (e.g., clinical trials). Surprisingly, we find that a simple greedy algorithm can be rate optimal (achieves asymptotically optimal regret) if there is sufficient randomness in the observed contexts (covariates). We prove that this is always the case for a two-armed bandit under a general class of context distributions that satisfy a condition we term covariate diversity. Furthermore, even absent this condition, we show that a greedy algorithm can be rate optimal with positive probability. Thus, standard bandit algorithms may unnecessarily explore. Motivated by these results, we introduce Greedy-First, a new algorithm that uses only observed contexts and rewards to determine whether to follow a greedy algorithm or to explore. We prove that this algorithm is rate optimal without any additional assumptions on the context distribution or the number of arms. Extensive simulations demonstrate that Greedy-First successfully reduces exploration and outperforms existing (exploration-based) contextual bandit algorithms such as Thompson sampling or upper confidence bound (UCB).
The Emergence of Spectral Universality in Deep Networks
Authors: Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
Date: 2018-02-27 00:00:00

Recent work has shown that tight concentration of the entire spectrum of
singular values of a deep network's input-output Jacobian around one at
initialization can speed up learning by orders of magnitude. Therefore, to
guide important design choices, it is important to build a full theoretical
understanding of the spectra of Jacobians at initialization. To this end, we
leverage powerful tools from free probability theory to provide a detailed
analytic understanding of how a deep network's Jacobian spectrum depends on
various hyperparameters including the nonlinearity, the weight and bias
distributions, and the depth. For a variety of nonlinearities, our work reveals
the emergence of new universal limiting spectral distributions that remain
concentrated around one even as the depth goes to infinity.