pwc-archive/papers-with-abstracts huggingface dataset:
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Authors: Behrooz Ghorbani, Shankar Krishnan, Ying Xiao
Date: 2019-01-29 00:00:00

To understand the dynamics of optimization in deep neural networks, we
develop a tool to study the evolution of the entire Hessian spectrum throughout
the optimization process. Using this, we study a number of hypotheses
concerning smoothness, curvature, and sharpness in the deep learning
literature. We then thoroughly analyze a crucial structural feature of the
spectra: in non-batch normalized networks, we observe the rapid appearance of
large isolated eigenvalues in the spectrum, along with a surprising
concentration of the gradient in the corresponding eigenspaces. In batch
normalized networks, these two effects are almost absent. We characterize these
effects, and explain how they affect optimization speed through both theory and
experiments. As part of this work, we adapt advanced tools from numerical
linear algebra that allow scalable and accurate estimation of the entire
Hessian spectrum of ImageNet-scale neural networks; this technique may be of
independent interest in other applications.
Revisiting Sample Selection Approach to Positive-Unlabeled Learning: Turning Unlabeled Data into Positive rather than Negative
Authors: Miao Xu, Bingcong Li, Gang Niu, Bo Han, Masashi Sugiyama
Date: 2019-01-29 00:00:00
Tasks: Memorization

In the early history of positive-unlabeled (PU) learning, the sample
selection approach, which heuristically selects negative (N) data from U data,
was explored extensively. However, this approach was later dominated by the
importance reweighting approach, which carefully treats all U data as N data.
May there be a new sample selection method that can outperform the latest
importance reweighting method in the deep learning age? This paper is devoted
to answering this question affirmatively---we propose to label large-loss U
data as P, based on the memorization properties of deep networks. Since P data
selected in such a way are biased, we develop a novel learning objective that
can handle such biased P data properly. Experiments confirm the superiority of
the proposed method.
Simultaneous prediction of multiple outcomes using revised stacking algorithms
Authors: Li Xing, Mary Lesperance, Xuekui Zhang
Date: 2019-01-29 00:00:00
Tasks: Prediction

Motivation: HIV is difficult to treat because its virus mutates at a high
rate and mutated viruses easily develop resistance to existing drugs. If the
relationships between mutations and drug resistances can be determined from
historical data, patients can be provided personalized treatment according to
their own mutation information. The HIV Drug Resistance Database was built to
investigate the relationships. Our goal is to build a model using data in this
database, which simultaneously predicts the resistance of multiple drugs using
mutation information from sequences of viruses for any new patient.
  Results: We propose two variations of a stacking algorithm which borrow
information among multiple prediction tasks to improve multivariate prediction
performance. The most attractive feature of our proposed methods is the
flexibility with which complex multivariate prediction models can be
constructed using any univariate prediction models. Using cross-validation
studies, we show that our proposed methods outperform other popular
multivariate prediction methods.
  Availability: An R package will be made available.
Learning to Validate the Quality of Detected Landmarks
Authors: Wolfgang Fuhl, Enkelejda Kasneci
Date: 2019-01-29 00:00:00
Tasks: Head Pose Estimation, Pose Estimation

We present a new loss function for the validation of image landmarks detected via Convolutional Neural Networks (CNN). The network learns to estimate how accurate its landmark estimation is. This loss function is applicable to all regression-based location estimations and allows the exclusion of unreliable landmarks from further processing. In addition, we formulate a novel batch balancing approach which weights the importance of samples based on their produced loss. This is done by computing a probability distribution mapping on an interval from which samples can be selected using a uniform random selection scheme. We conducted experiments on the 300W, AFLW, and WFLW facial landmark datasets. In the first experiments, the influence of our batch balancing approach is evaluated by comparing it against uniform sampling. In addition, we evaluated the impact of the validation loss on the landmark accuracy based on uniform sampling. The last experiments evaluate the correlation of the validation signal with the landmark accuracy. All experiments were performed for all three datasets.
Harnessing GANs for Zero-shot Learning of New Classes in Visual Speech Recognition
Authors: Yaman Kumar, Dhruva Sahrawat, Shubham Maheshwari, Debanjan Mahata, Amanda Stent, Yifang Yin, Rajiv Ratn Shah, Roger Zimmermann
Date: 2019-01-29 00:00:00
Tasks: speech-recognition, Speech Recognition, Visual Speech Recognition, Zero-Shot Learning

Visual Speech Recognition (VSR) is the process of recognizing or interpreting speech by watching the lip movements of the speaker. Recent machine learning based approaches model VSR as a classification problem; however, the scarcity of training data leads to error-prone systems with very low accuracies in predicting unseen classes. To solve this problem, we present a novel approach to zero-shot learning by generating new classes using Generative Adversarial Networks (GANs), and show how the addition of unseen class samples increases the accuracy of a VSR system by a significant margin of 27% and allows it to handle speaker-independent out-of-vocabulary phrases. We also show that our models are language agnostic and therefore capable of seamlessly generating, using English training data, videos for a new language (Hindi). To the best of our knowledge, this is the first work to show empirical evidence of the use of GANs for generating training samples of unseen classes in the domain of VSR, hence facilitating zero-shot learning. We make the added videos for new classes publicly available along with our code.
Identifiability of Gaussian Structural Equation Models with Homogeneous and Heterogeneous Error Variances
Authors: Gunwoong Park, Younghwan Kim
Date: 2019-01-29 00:00:00

In this work, we consider the identifiability assumption of Gaussian linear structural equation models (SEMs) in which each variable is determined by a linear function of its parents plus normally distributed error. It has been shown that linear Gaussian structural equation models are fully identifiable if all error variances are the same or known. Hence, this work proves the identifiability of Gaussian SEMs with both homogeneous and heterogeneous unknown error variances. Our new identifiability assumption exploits not only error variances, but edge weights; hence, it is strictly milder than prior work on the identifiability result. We further provide a structure learning algorithm that is statistically consistent and computationally feasible, based on our new assumption. The proposed algorithm assumes that all relevant variables are observed, while it does not assume causal minimality and faithfulness. We verify our theoretical findings through simulations and real multivariate data, and compare our algorithm to state-of-the-art PC, GES and GDS algorithms.
Structuring an unordered text document
Authors: Shashank Yadav, Tejas Shimpi, C. Ravindranath Chowdary, Prashant Sharma, Deepansh Agrawal, Shivang Agarwal
Date: 2019-01-29 00:00:00
Tasks: Document Summarization, Question Answering, Sentence

Segmenting an unordered text document into different sections is a very
useful task in many text processing applications like multiple document
summarization, question answering, etc. This paper proposes structuring of an
unordered text document based on the keywords in the document. We test our
approach on Wikipedia documents using both statistical and predictive methods
such as the TextRank algorithm and Google's USE (Universal Sentence Encoder).
From our experimental results, we show that the proposed model can effectively
structure an unordered document into sections.
Adversarial Adaptation of Scene Graph Models for Understanding Civic Issues
Authors: Shanu Kumar, Shubham Atreja, Anjali Singh, Mohit Jain
Date: 2019-01-29 00:00:00

Citizen engagement and technology usage are two emerging trends driven by
smart city initiatives. Governments around the world are adopting technology
for faster resolution of civic issues. Typically, citizens report issues, such
as broken roads, garbage dumps, etc. through web portals and mobile apps, in
order for the government authorities to take appropriate actions. Several
mediums -- text, image, audio, video -- are used to report these issues.
Through a user study with 13 citizens and 3 authorities, we found that image is
the most preferred medium to report civic issues. However, analyzing civic
issue related images is challenging for the authorities as it requires manual
effort. Moreover, previous works have been limited to identifying a specific
set of issues from images. In this work, given an image, we propose to generate
a Civic Issue Graph consisting of a set of objects and the semantic relations
between them, which are representative of the underlying civic issue. We also
release two multi-modal (text and images) datasets, that can help in further
analysis of civic issues from images. We present a novel approach for
adversarial training of existing scene graph models that enables the use of
scene graphs for new applications in the absence of any labelled training data.
We conduct several experiments to analyze the efficacy of our approach, and
using human evaluation, we establish the appropriateness of our model at
representing different civic issues.
Optimising Clifford Circuits with Quantomatic
Authors: Andrew Fagan, Ross Duncan
Date: 2019-01-29 00:00:00

We present a system of equations between Clifford circuits, all derivable in
the ZX-calculus, and formalised as rewrite rules in the Quantomatic proof
assistant. By combining these rules with some non-trivial simplification
procedures defined in the Quantomatic tactic language, we demonstrate the use
of Quantomatic as a circuit optimisation tool. We prove that the system always
reduces Clifford circuits of one or two qubits to their minimal form, and give
numerical results demonstrating its performance on larger Clifford circuits.
Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
Authors: Dongqi Han, Kenji Doya, Jun Tani
Date: 2019-01-29 00:00:00
Tasks: continuous-control, Continuous Control, Meta-Learning, Reinforcement Learning, Reinforcement Learning (RL)

Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.
Evaluating Generalization Ability of Convolutional Neural Networks and Capsule Networks for Image Classification via Top-2 Classification
Authors: Hao Ren, Jianlin Su, Hong Lu
Date: 2019-01-29 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification

Image classification is a challenging problem which aims to identify the category of object in the image. In recent years, deep Convolutional Neural Networks (CNNs) have been applied to handle this task, and impressive improvement has been achieved. However, some research showed the output of CNNs can be easily altered by adding relatively small perturbations to the input image, such as modifying few pixels. Recently, Capsule Networks (CapsNets) are proposed, which can help eliminating this limitation. Experiments on MNIST dataset revealed that capsules can better characterize the features of object than CNNs. But it's hard to find a suitable quantitative method to compare the generalization ability of CNNs and CapsNets. In this paper, we propose a new image classification task called Top-2 classification to evaluate the generalization ability of CNNs and CapsNets. The models are trained on single label image samples same as the traditional image classification task. But in the test stage, we randomly concatenate two test image samples which contain different labels, and then use the trained models to predict the top-2 labels on the unseen newly-created two label image samples. This task can provide us precise quantitative results to compare the generalization ability of CNNs and CapsNets. Back to the CapsNet, because it uses Full Connectivity (FC) mechanism among all capsules, it requires many parameters. To reduce the number of parameters, we introduce the Parameter-Sharing (PS) mechanism between capsules. Experiments on five widely used benchmark image datasets demonstrate the method significantly reduces the number of parameters, without losing the effectiveness of extracting features. Further, on the Top-2 classification task, the proposed PS CapsNets obtain impressive higher accuracy compared to the traditional CNNs and FC CapsNets by a large margin.
Deep-dust: Predicting concentrations of fine dust in Seoul using LSTM
Authors: Sookyung Kim, Jungmin M. Lee, Jiwoo Lee, Jihoon Seo
Date: 2019-01-29 00:00:00
Tasks: Time Series, Time Series Analysis

Polluting fine dusts in South Korea which are mainly consisted of biomass
burning and fugitive dust blown from dust belt is significant problem these
days. Predicting concentrations of fine dust particles in Seoul is challenging
because they are product of complicate chemical reactions among gaseous
pollutants and also influenced by dynamical interactions between pollutants and
multiple climate variables. Elaborating state-of-art time series analysis
techniques using deep learning, non-linear interactions between multiple
variables can be captured and used to predict future dust concentration. In
this work, we propose the LSTM based model to predict hourly concentration of
fine dust at target location in Seoul based on previous concentration of
pollutants, dust concentrations and climate variables in surrounding area. Our
results show that proposed model successfully predicts future dust
concentrations at 25 target districts(Gu) in Seoul.
Sparse Least Squares Low Rank Kernel Machines
Authors: Di Xu, Manjing Fang, Xia Hong, Junbin Gao
Date: 2019-01-29 00:00:00
Tasks: Computational Efficiency

A general framework of least squares support vector machine with low rank kernels, referred to as LR-LSSVM, is introduced in this paper. The special structure of low rank kernels with a controlled model size brings sparsity as well as computational efficiency to the proposed model. Meanwhile, a two-step optimization algorithm with three different criteria is proposed and various experiments are carried out using the example of the so-call robust RBF kernel to validate the model. The experiment results show that the performance of the proposed algorithm is comparable or superior to several existing kernel machines.
A maximum principle argument for the uniform convergence of graph Laplacian regressors
Authors: Nicolas Garcia Trillos, Ryan Murray
Date: 2019-01-29 00:00:00
Tasks: regression

This paper investigates the use of methods from partial differential equations and the Calculus of variations to study learning problems that are regularized using graph Laplacians. Graph Laplacians are a powerful, flexible method for capturing local and global geometry in many classes of learning problems, and the techniques developed in this paper help to broaden the methodology of studying such problems. In particular, we develop the use of maximum principle arguments to establish asymptotic consistency guarantees within the context of noise corrupted, non-parametric regression with samples living on an unknown manifold embedded in $\mathbb{R}^d$. The maximum principle arguments provide a new technical tool which informs parameter selection by giving concrete error estimates in terms of various regularization parameters. A review of learning algorithms which utilize graph Laplacians, as well as previous developments in the use of differential equation and variational techniques to study those algorithms, is given. In addition, new connections are drawn between Laplacian methods and other machine learning techniques, such as kernel regression and k-nearest neighbor methods.
A Parallel Projection Method for Metric Constrained Optimization
Authors: Cameron Ruggles, Nate Veldt, David F. Gleich
Date: 2019-01-29 00:00:00
Tasks: Clustering

Many clustering applications in machine learning and data mining rely on
solving metric-constrained optimization problems. These problems are
characterized by $O(n^3)$ constraints that enforce triangle inequalities on
distance variables associated with $n$ objects in a large dataset. Despite its
usefulness, metric-constrained optimization is challenging in practice due to
the cubic number of constraints and the high-memory requirements of standard
optimization software. Recent work has shown that iterative projection methods
are able to solve metric-constrained optimization problems on a much larger
scale than was previously possible, thanks to their comparatively low memory
requirement. However, the major limitation of projection methods is their slow
convergence rate. In this paper we present a parallel projection method for
metric-constrained optimization which allows us to speed up the convergence
rate in practice. The key to our approach is a new parallel execution schedule
that allows us to perform projections at multiple metric constraints
simultaneously without any conflicts or locking of variables. We illustrate the
effectiveness of this execution schedule by implementing and testing a parallel
projection method for solving the metric-constrained linear programming
relaxation of correlation clustering. We show numerous experimental results on
problems involving up to 2.9 trillion constraints.
Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning
Authors: Nicolas Garcia Trillos, Zach Kaplan, Daniel Sanz-Alonso
Date: 2019-01-29 00:00:00

The aim of this paper is to provide new theoretical and computational
understanding on two loss regularizations employed in deep learning, known as
local entropy and heat regularization. For both regularized losses we introduce
variational characterizations that naturally suggest a two-step scheme for
their optimization, based on the iterative shift of a probability density and
the calculation of a best Gaussian approximation in Kullback-Leibler
divergence. Under this unified light, the optimization schemes for local
entropy and heat regularized loss differ only over which argument of the
Kullback-Leibler divergence is used to find the best Gaussian approximation.
Local entropy corresponds to minimizing over the second argument, and the
solution is given by moment matching. This allows to replace traditional
back-propagation calculation of gradients by sampling algorithms, opening an
avenue for gradient-free, parallelizable training of neural networks.
General Fair Empirical Risk Minimization
Authors: Luca Oneto, Michele Donini, Massimiliano Pontil
Date: 2019-01-29 00:00:00
Tasks: Fairness, regression

We tackle the problem of algorithmic fairness, where the goal is to avoid the unfairly influence of sensitive information, in the general context of regression with possible continuous sensitive attributes. We extend the framework of fair empirical risk minimization to this general scenario, covering in this way the whole standard supervised learning setting. Our generalized fairness measure reduces to well known notions of fairness available in literature. We derive learning guarantees for our method, that imply in particular its statistical consistency, both in terms of the risk and the fairness measure. We then specialize our approach to kernel methods and propose a convex fair estimator in that setting. We test the estimator on a commonly used benchmark dataset (Communities and Crime) and on a new dataset collected at the University of Genova, containing the information of the academic career of five thousand students. The latter dataset provides a challenging real case scenario of unfair behaviour of standard regression methods that benefits from our methodology. The experimental results show that our estimator is effective at mitigating the trade-off between accuracy and fairness requirements.
Active learning for binary classification with variable selection
Authors: Zhanfeng Wang, Yumi Kwon, Yuan-Chin Ivan Chang
Date: 2019-01-29 00:00:00
Tasks: Active Learning, Binary Classification, Classification, General Classification, Variable Selection

Modern computing and communication technologies can make data collection
procedures very efficient. However, our ability to analyze large data sets
and/or to extract information out from them is hard-pressed to keep up with our
capacities for data collection. Among these huge data sets, some of them are
not collected for any particular research purpose. For a classification
problem, this means that the essential label information may not be readily
obtainable, in the data set in hands, and an extra labeling procedure is
required such that we can have enough label information to be used for
constructing a classification model. When the size of a data set is huge, to
label each subject in it will cost a lot in both capital and time. Thus, it is
an important issue to decide which subjects should be labeled first in order to
efficiently reduce the training cost/time. Active learning method is a
promising outlet for this situation, because with the active learning ideas, we
can select the unlabeled subjects sequentially without knowing their label
information. In addition, there will be no confirmed information about the
essential variables for constructing an efficient classification rule. Thus,
how to merge a variable selection scheme with an active learning procedure is
of interest. In this paper, we propose a procedure for building binary
classification models when the complete label information is not available in
the beginning of the training stage. We study an model-based active learning
procedure with sequential variable selection schemes, and discuss the results
of the proposed procedure from both theoretical and numerical aspects.
Cloud-Net: An end-to-end Cloud Detection Algorithm for Landsat 8 Imagery
Authors: Sorour Mohajerani, Parvaneh Saeedi
Date: 2019-01-29 00:00:00
Proceeding: Conference: 2019 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 2019 1
Tasks: Cloud Detection

Cloud detection in satellite images is an important first-step in many remote
sensing applications. This problem is more challenging when only a limited
number of spectral bands are available. To address this problem, a deep
learning-based algorithm is proposed in this paper. This algorithm consists of
a Fully Convolutional Network (FCN) that is trained by multiple patches of
Landsat 8 images. This network, which is called Cloud-Net, is capable of
capturing global and local cloud features in an image using its convolutional
blocks. Since the proposed method is an end-to-end solution, no complicated
pre-processing step is required. Our experimental results prove that the
proposed method outperforms the state-of-the-art method over a benchmark
dataset by 8.7\% in Jaccard Index.