pwc-archive/papers-with-abstracts huggingface dataset:
Minimum Spectral Connectivity Projection Pursuit
Authors: David P. Hofmeyr, Nicos G. Pavlidis, Idris A. Eckley
Date: 2015-09-04 00:00:00
Tasks: Clustering, Dimensionality Reduction

We study the problem of determining the optimal low dimensional projection
for maximising the separability of a binary partition of an unlabelled dataset,
as measured by spectral graph theory. This is achieved by finding projections
which minimise the second eigenvalue of the graph Laplacian of the projected
data, which corresponds to a non-convex, non-smooth optimisation problem. We
show that the optimal univariate projection based on spectral connectivity
converges to the vector normal to the maximum margin hyperplane through the
data, as the scaling parameter is reduced to zero. This establishes a
connection between connectivity as measured by spectral graph theory and
maximal Euclidean separation. The computational cost associated with each
eigen-problem is quadratic in the number of data. To mitigate this issue, we
propose an approximation method using microclusters with provable approximation
error bounds. Combining multiple binary partitions within a divisive
hierarchical model allows us to construct clustering solutions admitting
clusters with varying scales and lying within different subspaces. We evaluate
the performance of the proposed method on a large collection of benchmark
datasets and find that it compares favourably with existing methods for
projection pursuit and dimension reduction for data clustering.
Variance Reduced methods for Non-convex Composition Optimization
Authors: Liu Liu, Ji Liu, DaCheng Tao
Date: 2017-11-13 00:00:00
Tasks: Reinforcement Learning

This paper explores the non-convex composition optimization in the form
including inner and outer finite-sum functions with a large number of component
functions. This problem arises in some important applications such as nonlinear
embedding and reinforcement learning. Although existing approaches such as
stochastic gradient descent (SGD) and stochastic variance reduced gradient
(SVRG) descent can be applied to solve this problem, their query complexity
tends to be high, especially when the number of inner component functions is
large. In this paper, we apply the variance-reduced technique to derive two
variance reduced algorithms that significantly improve the query complexity if
the number of inner component functions is large. To the best of our knowledge,
this is the first work that establishes the query complexity analysis for
non-convex stochastic composition. Experiments validate the proposed algorithms
and theoretical analysis.
Convolutional Neural Network with Word Embeddings for Chinese Word Segmentation
Authors: Chunqi Wang, Bo Xu
Date: 2017-11-13 00:00:00
Proceeding: IJCNLP 2017 11
Tasks: Chinese Word Segmentation, Feature Engineering, Word Embeddings

Character-based sequence labeling framework is flexible and efficient for
Chinese word segmentation (CWS). Recently, many character-based neural models
have been applied to CWS. While they obtain good performance, they have two
obvious weaknesses. The first is that they heavily rely on manually designed
bigram feature, i.e. they are not good at capturing n-gram features
automatically. The second is that they make no use of full word information.
For the first weakness, we propose a convolutional neural model, which is able
to capture rich n-gram features without any feature engineering. For the second
one, we propose an effective approach to integrate the proposed model with word
embeddings. We evaluate the model on two benchmark datasets: PKU and MSR.
Without any feature engineering, the model obtains competitive performance --
95.7% on PKU and 97.3% on MSR. Armed with word embeddings, the model achieves
state-of-the-art performance on both datasets -- 96.5% on PKU and 98.0% on MSR,
without using any external labeled resource.
Sparse-TDA: Sparse Realization of Topological Data Analysis for Multi-Way Classification
Authors: Wei Guo, Krithika Manohar, Steven L. Brunton, Ashis G. Banerjee
Date: 2017-01-12 00:00:00
Tasks: General Classification, Texture Classification, Topological Data Analysis

Topological data analysis (TDA) has emerged as one of the most promising
techniques to reconstruct the unknown shapes of high-dimensional spaces from
observed data samples. TDA, thus, yields key shape descriptors in the form of
persistent topological features that can be used for any supervised or
unsupervised learning task, including multi-way classification. Sparse
sampling, on the other hand, provides a highly efficient technique to
reconstruct signals in the spatial-temporal domain from just a few
carefully-chosen samples. Here, we present a new method, referred to as the
Sparse-TDA algorithm, that combines favorable aspects of the two techniques.
This combination is realized by selecting an optimal set of sparse pixel
samples from the persistent features generated by a vector-based TDA algorithm.
These sparse samples are selected from a low-rank matrix representation of
persistent features using QR pivoting. We show that the Sparse-TDA method
demonstrates promising performance on three benchmark problems related to human
posture recognition and image texture classification.
Should You Derive, Or Let the Data Drive? An Optimization Framework for Hybrid First-Principles Data-Driven Modeling
Authors: Remi R. Lam, Lior Horesh, Haim Avron, Karen E. Willcox
Date: 2017-11-12 00:00:00
Tasks: Decision Making

Mathematical models are used extensively for diverse tasks including
analysis, optimization, and decision making. Frequently, those models are
principled but imperfect representations of reality. This is either due to
incomplete physical description of the underlying phenomenon (simplified
governing equations, defective boundary conditions, etc.), or due to numerical
approximations (discretization, linearization, round-off error, etc.). Model
misspecification can lead to erroneous model predictions, and respectively
suboptimal decisions associated with the intended end-goal task. To mitigate
this effect, one can amend the available model using limited data produced by
experiments or higher fidelity models. A large body of research has focused on
estimating explicit model parameters. This work takes a different perspective
and targets the construction of a correction model operator with implicit
attributes. We investigate the case where the end-goal is inversion and
illustrate how appropriate choices of properties imposed upon the correction
and corrected operator lead to improved end-goal insights.
A unified framework for hard and soft clustering with regularized optimal transport
Authors: Jean-Frédéric Diebold, Nicolas Papadakis, Arnaud Dessein, Charles-Alban Deledalle
Date: 2017-11-12 00:00:00
Tasks: Clustering, Relation

In this paper, we formulate the problem of inferring a Finite Mixture Model from discrete data as an optimal transport problem with entropic regularization of parameter $\lambda\geq 0$. Our method unifies hard and soft clustering, the Expectation-Maximization (EM) algorithm being exactly recovered for $\lambda=1$. The family of clustering algorithm we propose rely on the resolution of nonconvex problems using alternating minimization. We study the convergence property of our generalized $\lambda-$EM algorithms and show that each step in the minimization process has a closed form solution when inferring finite mixture models of exponential families. Experiments highlight the benefits of taking a parameter $\lambda>1$ to improve the inference performance and $\lambda\to 0$ for classification.
Fast Reading Comprehension with ConvNets
Authors: Felix Wu, Ni Lao, John Blitzer, Guandao Yang, Kilian Weinberger
Date: 2017-11-12 00:00:00
Proceeding: ICLR 2018 1
Tasks: Question Answering, Reading Comprehension

State-of-the-art deep reading comprehension models are dominated by recurrent
neural nets. Their sequential nature is a natural fit for language, but it also
precludes parallelization within an instances and often becomes the bottleneck
for deploying such models to latency critical scenarios. This is particularly
problematic for longer texts. Here we present a convolutional architecture as
an alternative to these recurrent architectures. Using simple dilated
convolutional units in place of recurrent ones, we achieve results comparable
to the state of the art on two question answering tasks, while at the same time
achieving up to two orders of magnitude speedups for question answering.
Filtering Variational Objectives
Authors: Chris J. Maddison, Dieterich Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, andriy mnih, Arnaud Doucet, Yee Whye Teh
Date: 2017-05-25 00:00:00
Proceeding: NeurIPS 2017 12

When used as a surrogate objective for maximum likelihood estimation in
latent variable models, the evidence lower bound (ELBO) produces
state-of-the-art results. Inspired by this, we consider the extension of the
ELBO to a family of lower bounds defined by a particle filter's estimator of
the marginal likelihood, the filtering variational objectives (FIVOs). FIVOs
take the same arguments as the ELBO, but can exploit a model's sequential
structure to form tighter bounds. We present results that relate the tightness
of FIVO's bound to the variance of the particle filter's estimator by
considering the generic case of bounds defined as log-transformed likelihood
estimators. Experimentally, we show that training with FIVO results in
substantial improvements over training the same model architecture with the
ELBO on sequential data.
Analysis of the Self Projected Matching Pursuit Algorithm
Authors: Laura Rebollo-Neira, Miroslav Rozloznik, Pradip Sasmal
Date: 2016-08-31 00:00:00

The convergence and numerical analysis of a low memory implementation of the Orthogonal Matching Pursuit greedy strategy, which is termed Self Projected Matching Pursuit, is presented. This approach renders an iterative way of solving the least squares problem with much less storage requirement than direct linear algebra techniques. Hence, it appropriate for solving large linear systems. The analysis highlights its suitability within the class of well posed problems.
Stochastic Nonconvex Optimization with Large Minibatches
Authors: Weiran Wang, Nathan Srebro
Date: 2017-09-25 00:00:00
Tasks: Stochastic Optimization

We study stochastic optimization of nonconvex loss functions, which are
typical objectives for training neural networks. We propose stochastic
approximation algorithms which optimize a series of regularized, nonlinearized
losses on large minibatches of samples, using only first-order gradient
information. Our algorithms provably converge to an approximate critical point
of the expected objective with faster rates than minibatch stochastic gradient
descent, and facilitate better parallelization by allowing larger minibatches.
Alpha-Divergences in Variational Dropout
Authors: Bogdan Mazoure, Riashat Islam
Date: 2017-11-12 00:00:00
Tasks: Variational Inference

We investigate the use of alternative divergences to Kullback-Leibler (KL) in
variational inference(VI), based on the Variational Dropout \cite{kingma2015}.
Stochastic gradient variational Bayes (SGVB) \cite{aevb} is a general framework
for estimating the evidence lower bound (ELBO) in Variational Bayes. In this
work, we extend the SGVB estimator with using Alpha-Divergences, which are
alternative to divergences to VI' KL objective. The Gaussian dropout can be
seen as a local reparametrization trick of the SGVB objective. We extend the
Variational Dropout to use alpha divergences for variational inference. Our
results compare $\alpha$-divergence variational dropout with standard
variational dropout with correlated and uncorrelated weight noise. We show that
the $\alpha$-divergence with $\alpha \rightarrow 1$ (or KL divergence) is still
a good measure for use in variational inference, in spite of the efficient use
of Alpha-divergences for Dropout VI \cite{Li17}. $\alpha \rightarrow 1$ can
yield the lowest training error, and optimizes a good lower bound for the
evidence lower bound (ELBO) among all values of the parameter $\alpha \in
[0,\infty)$.
A generalization of the Jensen divergence: The chord gap divergence
Authors: Frank Nielsen
Date: 2017-09-29 00:00:00
Tasks: Clustering

We introduce a novel family of distances, called the chord gap divergences,
that generalizes the Jensen divergences (also called the Burbea-Rao distances),
and study its properties. It follows a generalization of the celebrated
statistical Bhattacharyya distance that is frequently met in applications. We
report an iterative concave-convex procedure for computing centroids, and
analyze the performance of the $k$-means++ clustering with respect to that new
dissimilarity measure by introducing the Taylor-Lagrange remainder form of the
skew Jensen divergences.
Variational Inference via $χ$-Upper Bound Minimization
Authors: Adji B. Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, David M. Blei
Date: 2016-11-01 00:00:00
Tasks: Variational Inference

Variational inference (VI) is widely used as an efficient alternative to
Markov chain Monte Carlo. It posits a family of approximating distributions $q$
and finds the closest member to the exact posterior $p$. Closeness is usually
measured via a divergence $D(q || p)$ from $q$ to $p$. While successful, this
approach also has problems. Notably, it typically leads to underestimation of
the posterior variance. In this paper we propose CHIVI, a black-box variational
inference algorithm that minimizes $D_{\chi}(p || q)$, the $\chi$-divergence
from $p$ to $q$. CHIVI minimizes an upper bound of the model evidence, which we
term the $\chi$ upper bound (CUBO). Minimizing the CUBO leads to improved
posterior uncertainty, and it can also be used with the classical VI lower
bound (ELBO) to provide a sandwich estimate of the model evidence. We study
CHIVI on three models: probit regression, Gaussian process classification, and
a Cox process model of basketball plays. When compared to expectation
propagation and classical VI, CHIVI produces better error rates and more
accurate estimates of posterior variance.
MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving
Authors: Mennatullah Siam, Heba Mahgoub, Mohamed Zahran, Senthil Yogamani, Martin Jagersand, Ahmad El-Sallab
Date: 2017-09-14 00:00:00
Tasks: Autonomous Driving, Computational Efficiency, Motion Detection, Motion Segmentation, Moving Object Detection, Multi-Task Learning, object-detection, Object Detection, Segmentation, Semantic Segmentation, vehicle detection

We propose a novel multi-task learning system that combines appearance and
motion cues for a better semantic reasoning of the environment. A unified
architecture for joint vehicle detection and motion segmentation is introduced.
In this architecture, a two-stream encoder is shared among both tasks. In order
to evaluate our method in autonomous driving setting, KITTI annotated sequences
with detection and odometry ground truth are used to automatically generate
static/dynamic annotations on the vehicles. This dataset is called KITTI Moving
Object Detection dataset (KITTI MOD). The dataset will be made publicly
available to act as a benchmark for the motion detection task. Our experiments
show that the proposed method outperforms state of the art methods that utilize
motion cue only with 21.5% in mAP on KITTI MOD. Our method performs on par with
the state of the art unsupervised methods on DAVIS benchmark for generic object
segmentation. One of our interesting conclusions is that joint training of
motion segmentation and vehicle detection benefits motion segmentation. Motion
segmentation has relatively fewer data, unlike the detection task. However, the
shared fusion encoder benefits from joint training to learn a generalized
representation. The proposed method runs in 120 ms per frame, which beats the
state of the art motion detection/segmentation in computational efficiency.
Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
Authors: Takuya Akiba, Shuji Suzuki, Keisuke Fukuda
Date: 2017-11-12 00:00:00

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be
achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by
using a large minibatch size of 32k. To maintain accuracy with this large
minibatch size, we employed several techniques such as RMSprop warm-up, batch
normalization without moving averages, and a slow-start learning rate schedule.
This paper also describes the details of the hardware and software of the
system used to achieve the above performance.
High-Order Attention Models for Visual Question Answering
Authors: Idan Schwartz, Alexander G. Schwing, Tamir Hazan
Date: 2017-11-12 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA), Vocal Bursts Intensity Prediction

The quest for algorithms that enable cognitive abilities is an important part
of machine learning. A common trait in many recently investigated
cognitive-like tasks is that they take into account different data modalities,
such as visual and textual input. In this paper we propose a novel and
generally applicable form of attention mechanism that learns high-order
correlations between various data modalities. We show that high-order
correlations effectively direct the appropriate attention to the relevant
elements in the different data modalities that are required to solve the joint
task. We demonstrate the effectiveness of our high-order attention mechanism on
the task of visual question answering (VQA), where we achieve state-of-the-art
performance on the standard VQA dataset.
Hyperspectral Image Classification with Markov Random Fields and a Convolutional Neural Network
Authors: Xiangyong Cao, Feng Zhou, Lin Xu, Deyu Meng, Zongben Xu, John Paisley
Date: 2017-05-01 00:00:00
Tasks: Classification, General Classification, Hyperspectral Image Classification, image-classification, Image Classification

This paper presents a new supervised classification algorithm for remotely
sensed hyperspectral image (HSI) which integrates spectral and spatial
information in a unified Bayesian framework. First, we formulate the HSI
classification problem from a Bayesian perspective. Then, we adopt a
convolutional neural network (CNN) to learn the posterior class distributions
using a patch-wise training strategy to better use the spatial information.
Next, spatial information is further considered by placing a spatial smoothness
prior on the labels. Finally, we iteratively update the CNN parameters using
stochastic gradient decent (SGD) and update the class labels of all pixel
vectors using an alpha-expansion min-cut-based algorithm. Compared with other
state-of-the-art methods, the proposed classification method achieves better
performance on one synthetic dataset and two benchmark HSI datasets in a number
of experimental settings.
A machine learning approach for efficient uncertainty quantification using multiscale methods
Authors: Shing Chan, Ahmed H. Elsheikh
Date: 2017-11-12 00:00:00
Tasks: Attribute, BIG-bench Machine Learning, Uncertainty Quantification

Several multiscale methods account for sub-grid scale features using coarse
scale basis functions. For example, in the Multiscale Finite Volume method the
coarse scale basis functions are obtained by solving a set of local problems
over dual-grid cells. We introduce a data-driven approach for the estimation of
these coarse scale basis functions. Specifically, we employ a neural network
predictor fitted using a set of solution samples from which it learns to
generate subsequent basis functions at a lower computational cost than solving
the local problems. The computational advantage of this approach is realized
for uncertainty quantification tasks where a large number of realizations has
to be evaluated. We attribute the ability to learn these basis functions to the
modularity of the local problems and the redundancy of the permeability patches
between samples. The proposed method is evaluated on elliptic problems yielding
very promising results.
Semi-Supervised Learning via New Deep Network Inversion
Authors: Randall Balestriero, Vincent Roger, Herve G. Glotin, Richard G. Baraniuk
Date: 2017-11-12 00:00:00

We exploit a recently derived inversion scheme for arbitrary deep neural
networks to develop a new semi-supervised learning framework that applies to a
wide range of systems and problems. The approach outperforms current
state-of-the-art methods on MNIST reaching $99.14\%$ of test set accuracy while
using $5$ labeled examples per class. Experiments with one-dimensional signals
highlight the generality of the method. Importantly, our approach is simple,
efficient, and requires no change in the deep network architecture.
Learning from Complementary Labels
Authors: Takashi Ishida, Gang Niu, Weihua Hu, Masashi Sugiyama
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Classification, General Classification, Multi-class Classification

Collecting labeled data is costly and thus a critical bottleneck in
real-world classification tasks. To mitigate this problem, we propose a novel
setting, namely learning from complementary labels for multi-class
classification. A complementary label specifies a class that a pattern does not
belong to. Collecting complementary labels would be less laborious than
collecting ordinary labels, since users do not have to carefully choose the
correct class from a long list of candidate classes. However, complementary
labels are less informative than ordinary labels and thus a suitable approach
is needed to better learn from them. In this paper, we show that an unbiased
estimator to the classification risk can be obtained only from complementarily
labeled data, if a loss function satisfies a particular symmetric condition. We
derive estimation error bounds for the proposed method and prove that the
optimal parametric convergence rate is achieved. We further show that learning
from complementary labels can be easily combined with learning from ordinary
labels (i.e., ordinary supervised learning), providing a highly practical
implementation of the proposed method. Finally, we experimentally demonstrate
the usefulness of the proposed methods.
A Sequence-Based Mesh Classifier for the Prediction of Protein-Protein Interactions
Authors: Edgar D. Coelho, Igor N. Cruz, André Santiago, José Luis Oliveira, António Dourado, Joel P. Arrais
Date: 2017-11-12 00:00:00

The worldwide surge of multiresistant microbial strains has propelled the
search for alternative treatment options. The study of Protein-Protein
Interactions (PPIs) has been a cornerstone in the clarification of complex
physiological and pathogenic processes, thus being a priority for the
identification of vital components and mechanisms in pathogens. Despite the
advances of laboratorial techniques, computational models allow the screening
of protein interactions between entire proteomes in a fast and inexpensive
manner. Here, we present a supervised machine learning model for the prediction
of PPIs based on the protein sequence. We cluster amino acids regarding their
physicochemical properties, and use the discrete cosine transform to represent
protein sequences. A mesh of classifiers was constructed to create
hyper-specialised classifiers dedicated to the most relevant pairs of molecular
function annotations from Gene Ontology. Based on an exhaustive evaluation that
includes datasets with different configurations, cross-validation and
out-of-sampling validation, the obtained results outscore the state-of-the-art
for sequence-based methods. For the final mesh model using SVM with RBF, a
consistent average AUC of 0.84 was attained.