pwc-archive/papers-with-abstracts huggingface dataset:
SOL: A Library for Scalable Online Learning Algorithms
Authors: Yue Wu, Steven C. H. Hoi, Chenghao Liu, Jing Lu, Doyen Sahoo, Nenghai Yu
Date: 2016-10-28 00:00:00
Tasks: BIG-bench Machine Learning, General Classification, Multi-class Classification

SOL is an open-source library for scalable online learning algorithms, and is
particularly suitable for learning with high-dimensional data. The library
provides a family of regular and sparse online learning algorithms for
large-scale binary and multi-class classification tasks with high efficiency,
scalability, portability, and extensibility. SOL was implemented in C++, and
provided with a collection of easy-to-use command-line tools, python wrappers
and library calls for users and developers, as well as comprehensive documents
for both beginners and advanced users. SOL is not only a practical machine
learning toolbox, but also a comprehensive experimental platform for online
learning research. Experiments demonstrate that SOL is highly efficient and
scalable for large-scale machine learning with high-dimensional data.
Missing Data Imputation for Supervised Learning
Authors: Jason Poulos, Rafael Valle
Date: 2016-10-28 00:00:00
Tasks: General Classification, Imputation

Missing data imputation can help improve the performance of prediction models
in situations where missing data hide useful information. This paper compares
methods for imputing missing categorical data for supervised classification
tasks. We experiment on two machine learning benchmark datasets with missing
categorical data, comparing classifiers trained on non-imputed (i.e., one-hot
encoded) or imputed data with different levels of additional missing-data
perturbation. We show imputation methods can increase predictive accuracy in
the presence of missing-data perturbation, which can actually improve
prediction accuracy by regularizing the classifier. We achieve the
state-of-the-art on the Adult dataset with missing-data perturbation and
k-nearest-neighbors (k-NN) imputation.
Orthogonal Random Features
Authors: Felix X. Yu, Ananda Theertha Suresh, Krzysztof Choromanski, Daniel Holtmann-Rice, Sanjiv Kumar
Date: 2016-10-28 00:00:00
Proceeding: NeurIPS 2016 12

We present an intriguing discovery related to Random Fourier Features: in
Gaussian kernel approximation, replacing the random Gaussian matrix by a
properly scaled random orthogonal matrix significantly decreases kernel
approximation error. We call this technique Orthogonal Random Features (ORF),
and provide theoretical and empirical justification for this behavior.
Motivated by this discovery, we further propose Structured Orthogonal Random
Features (SORF), which uses a class of structured discrete orthogonal matrices
to speed up the computation. The method reduces the time cost from
$\mathcal{O}(d^2)$ to $\mathcal{O}(d \log d)$, where $d$ is the data
dimensionality, with almost no compromise in kernel approximation quality
compared to ORF. Experiments on several datasets verify the effectiveness of
ORF and SORF over the existing methods. We also provide discussions on using
the same type of discrete orthogonal structure for a broader range of
applications.
A penalized likelihood method for classification with matrix-valued predictors
Authors: Aaron J. Molstad, Adam J. Rothman
Date: 2016-09-23 00:00:00
Tasks: Classification, EEG, Electroencephalogram (EEG), General Classification

We propose a penalized likelihood method to fit the linear discriminant
analysis model when the predictor is matrix valued. We simultaneously estimate
the means and the precision matrix, which we assume has a Kronecker product
decomposition. Our penalties encourage pairs of response category mean matrices
to have equal entries and also encourage zeros in the precision matrix. To
compute our estimators, we use a blockwise coordinate descent algorithm. To
update the optimization variables corresponding to response category mean
matrices, we use an alternating minimization algorithm that takes advantage of
the Kronecker structure of the precision matrix. We show that our method can
outperform relevant competitors in classification, even when our modeling
assumptions are violated. We analyze an EEG dataset to demonstrate our method's
interpretability and classification accuracy.
Symmetric and antisymmetric properties of solutions to kernel-based machine learning problems
Authors: Giorgio Gnecco
Date: 2016-06-27 00:00:00
Tasks: BIG-bench Machine Learning, Binary Classification, General Classification

A particularly interesting instance of supervised learning with kernels is
when each training example is associated with two objects, as in pairwise
classification (Brunner et al., 2012), and in supervised learning of preference
relations (Herbrich et al., 1998). In these cases, one may want to embed
additional prior knowledge into the optimization problem associated with the
training of the learning machine, modeled, respectively, by the symmetry of its
optimal solution with respect to an exchange of order between the two objects,
and by its antisymmetry. Extending the approach proposed in (Brunner et al.,
2012) (where the only symmetric case was considered), we show, focusing on
support vector binary classification, how such embedding is possible through
the choice of a suitable pairwise kernel, which takes as inputs the individual
feature vectors and also the group feature vectors associated with the two
objects. We also prove that the symmetry/antisymmetry constraints still hold
when considering the sequence of suboptimal solutions generated by one version
of the Sequential Minimal Optimization (SMO) algorithm, and we present
numerical results supporting the theoretical findings. We conclude discussing
extensions of the main results to support vector regression, to transductive
support vector machines, and to several kinds of graph kernels, including
diffusion kernels.
Hybrid CPU-GPU Framework for Network Motifs
Authors: Ryan A. Rossi, Rong Zhou
Date: 2016-08-18 00:00:00
Tasks: CPU, GPU

Massively parallel architectures such as the GPU are becoming increasingly
important due to the recent proliferation of data. In this paper, we propose a
key class of hybrid parallel graphlet algorithms that leverages multiple CPUs
and GPUs simultaneously for computing k-vertex induced subgraph statistics
(called graphlets). In addition to the hybrid multi-core CPU-GPU framework, we
also investigate single GPU methods (using multiple cores) and multi-GPU
methods that leverage all available GPUs simultaneously for computing induced
subgraph statistics. Both methods leverage GPU devices only, whereas the hybrid
multi-core CPU-GPU framework leverages all available multi-core CPUs and
multiple GPUs for computing graphlets in large networks. Compared to recent
approaches, our methods are orders of magnitude faster, while also more cost
effective enjoying superior performance per capita and per watt. In particular,
the methods are up to 300 times faster than the recent state-of-the-art method.
To the best of our knowledge, this is the first work to leverage multiple CPUs
and GPUs simultaneously for computing induced subgraph statistics.
Laplacian regularized low rank subspace clustering
Authors: Yu Song, Yiquan Wu
Date: 2016-10-24 00:00:00
Tasks: Clustering

The problem of fitting a union of subspaces to a collection of data points
drawn from multiple subspaces is considered in this paper. In the traditional
low rank representation model, the dictionary used to represent the data points
is chosen as the data points themselves and thus the dictionary is corrupted
with noise. This problem is solved in the low rank subspace clustering model
which decomposes the corrupted data matrix as the sum of a clean and
self-expressive dictionary plus a matrix of noise and gross errors. Also, the
clustering results of the low rank representation model can be enhanced by
using a graph of data similarity. This model is called Laplacian regularized
low rank representation model with a graph regularization term added to the
objective function. Inspired from the above two ideas, in this paper a
Laplacian regularized low rank subspace clustering model is proposed. This
model uses a clean dictionary to represent the data points and a graph
regularization term is also incorporated in the objective function.
Experimental results show that, compared with the traditional low rank
representation model, low rank subspace clustering model and several other
state-of-the-art subspace clustering model, the model proposed in this paper
can get better subspace clustering results with lower clustering error.
SelQA: A New Benchmark for Selection-based Question Answering
Authors: Tomasz Jurczyk, Michael Zhai, Jinho D. Choi
Date: 2016-06-27 00:00:00
Tasks: Question Answering, Sentence

This paper presents a new selection-based question answering dataset, SelQA.
The dataset consists of questions generated through crowdsourcing and sentence
length answers that are drawn from the ten most prevalent topics in the English
Wikipedia. We introduce a corpus annotation scheme that enhances the generation
of large, diverse, and challenging datasets by explicitly aiming to reduce word
co-occurrences between the question and answers. Our annotation scheme is
composed of a series of crowdsourcing tasks with a view to more effectively
utilize crowdsourcing in the creation of question answering datasets in various
domains. Several systems are compared on the tasks of answer sentence selection
and answer triggering, providing strong baseline results for future work to
improve upon.
Professor Forcing: A New Algorithm for Training Recurrent Networks
Authors: Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, Yoshua Bengio
Date: 2016-10-27 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Domain Adaptation, Handwriting generation, Image Generation, Language Modeling, Language Modelling

The Teacher Forcing algorithm trains recurrent networks by supplying observed
sequence values as inputs during training and using the network's own
one-step-ahead predictions to do multi-step sampling. We introduce the
Professor Forcing algorithm, which uses adversarial domain adaptation to
encourage the dynamics of the recurrent network to be the same when training
the network and when sampling from the network over multiple time steps. We
apply Professor Forcing to language modeling, vocal synthesis on raw waveforms,
handwriting generation, and image generation. Empirically we find that
Professor Forcing acts as a regularizer, improving test likelihood on character
level Penn Treebank and sequential MNIST. We also find that the model
qualitatively improves samples, especially when sampling for a large number of
time steps. This is supported by human evaluation of sample quality. Trade-offs
between Professor Forcing and Scheduled Sampling are discussed. We produce
T-SNEs showing that Professor Forcing successfully makes the dynamics of the
network during training and sampling more similar.
Geometric Dirichlet Means algorithm for topic inference
Authors: Mikhail Yurochkin, XuanLong Nguyen
Date: 2016-10-27 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Clustering, Variational Inference

We propose a geometric algorithm for topic learning and inference that is
built on the convex geometry of topics arising from the Latent Dirichlet
Allocation (LDA) model and its nonparametric extensions. To this end we study
the optimization of a geometric loss function, which is a surrogate to the
LDA's likelihood. Our method involves a fast optimization based weighted
clustering procedure augmented with geometric corrections, which overcomes the
computational and statistical inefficiencies encountered by other techniques
based on Gibbs sampling and variational inference, while achieving the accuracy
comparable to that of a Gibbs sampler. The topic estimates produced by our
method are shown to be statistically consistent under some conditions. The
algorithm is evaluated with extensive experiments on simulated and real data.
Icon: An Interactive Approach to Train Deep Neural Networks for Segmentation of Neuronal Structures
Authors: Felix Gonda, Verena Kaynig, Ray Thouis, Daniel Haehn, Jeff Lichtman, Toufiq Parag, Hanspeter Pfister
Date: 2016-10-27 00:00:00
Tasks: Image Segmentation, Segmentation, Semantic Segmentation

We present an interactive approach to train a deep neural network pixel
classifier for the segmentation of neuronal structures. An interactive training
scheme reduces the extremely tedious manual annotation task that is typically
required for deep networks to perform well on image segmentation problems. Our
proposed method employs a feedback loop that captures sparse annotations using
a graphical user interface, trains a deep neural network based on recent and
past annotations, and displays the prediction output to users in almost
real-time. Our implementation of the algorithm also allows multiple users to
provide annotations in parallel and receive feedback from the same classifier.
Quick feedback on classifier performance in an interactive setting enables
users to identify and label examples that are more important than others for
segmentation purposes. Our experiments show that an interactively-trained pixel
classifier produces better region segmentation results on Electron Microscopy
(EM) images than those generated by a network of the same architecture trained
offline on exhaustive ground-truth labels.
A Simple Practical Accelerated Method for Finite Sums
Authors: Aaron Defazio
Date: 2016-02-08 00:00:00
Proceeding: NeurIPS 2016 12

We describe a novel optimization method for finite sums (such as empirical
risk minimization problems) building on the recently introduced SAGA method.
Our method achieves an accelerated convergence rate on strongly convex smooth
problems. Our method has only one parameter (a step size), and is radically
simpler than other accelerated methods for finite sums. Additionally it can be
applied when the terms are non-smooth, yielding a method applicable in many
areas where operator splitting methods would traditionally be applied.
Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
Authors: Jack W. Rae, Jonathan J. Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, Timothy P. Lillicrap
Date: 2016-10-27 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Language Modeling, Language Modelling, Machine Translation, Question Answering, Translation

Neural networks augmented with external memory have the ability to learn
algorithmic solutions to complex tasks. These models appear promising for
applications such as language modeling and machine translation. However, they
scale poorly in both space and time as the amount of memory grows --- limiting
their applicability to real-world domains. Here, we present an end-to-end
differentiable memory access scheme, which we call Sparse Access Memory (SAM),
that retains the representational power of the original approaches whilst
training efficiently with very large memories. We show that SAM achieves
asymptotic lower bounds in space and time complexity, and find that an
implementation runs $1,\!000\times$ faster and with $3,\!000\times$ less
physical memory than non-sparse models. SAM learns with comparable data
efficiency to existing models on a range of synthetic tasks and one-shot
Omniglot character recognition, and can scale to tasks requiring $100,\!000$s
of time steps and memories. As well, we show how our approach can be adapted
for models that maintain temporal associations between memories, as with the
recently introduced Differentiable Neural Computer.
High-dimensional regression adjustments in randomized experiments
Authors: Stefan Wager, Wenfei Du, Jonathan Taylor, Robert Tibshirani
Date: 2016-07-22 00:00:00
Tasks: regression, valid, Vocal Bursts Intensity Prediction

We study the problem of treatment effect estimation in randomized experiments
with high-dimensional covariate information, and show that essentially any
risk-consistent regression adjustment can be used to obtain efficient estimates
of the average treatment effect. Our results considerably extend the range of
settings where high-dimensional regression adjustments are guaranteed to
provide valid inference about the population average treatment effect. We then
propose cross-estimation, a simple method for obtaining finite-sample-unbiased
treatment effect estimates that leverages high-dimensional regression
adjustments. Our method can be used when the regression model is estimated
using the lasso, the elastic net, subset selection, etc. Finally, we extend our
analysis to allow for adaptive specification search via cross-validation, and
flexible non-parametric regression adjustments with machine learning methods
such as random forests or neural networks.
Compressive Holographic Video
Authors: Zihao Wang, Leonidas Spinoulas, Kuan He, Huaijin Chen, Lei Tian, Aggelos K. Katsaggelos, Oliver Cossairt
Date: 2016-10-27 00:00:00
Tasks: 4D reconstruction, compressed sensing, Super-Resolution

Compressed sensing has been discussed separately in spatial and temporal
domains. Compressive holography has been introduced as a method that allows 3D
tomographic reconstruction at different depths from a single 2D image. Coded
exposure is a temporal compressed sensing method for high speed video
acquisition. In this work, we combine compressive holography and coded exposure
techniques and extend the discussion to 4D reconstruction in space and time
from one coded captured image. In our prototype, digital in-line holography was
used for imaging macroscopic, fast moving objects. The pixel-wise temporal
modulation was implemented by a digital micromirror device. In this paper we
demonstrate $10\times$ temporal super resolution with multiple depths recovery
from a single image. Two examples are presented for the purpose of recording
subtle vibrations and tracking small particles within 5 ms.
Cross-Modal Scene Networks
Authors: Yusuf Aytar, Lluis Castrejon, Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
Date: 2016-10-27 00:00:00
Tasks: Retrieval

People can recognize scenes across many different modalities beyond natural
images. In this paper, we investigate how to learn cross-modal scene
representations that transfer across modalities. To study this problem, we
introduce a new cross-modal scene dataset. While convolutional neural networks
can categorize scenes well, they also learn an intermediate representation not
aligned across modalities, which is undesirable for cross-modal transfer
applications. We present methods to regularize cross-modal convolutional neural
networks so that they have a shared representation that is agnostic of the
modality. Our experiments suggest that our scene representation can help
transfer representations across modalities for retrieval. Moreover, our
visualizations suggest that units emerge in the shared representation that tend
to activate on consistent concepts independently of the modality.
SoundNet: Learning Sound Representations from Unlabeled Video
Authors: Yusuf Aytar, Carl Vondrick, Antonio Torralba
Date: 2016-10-27 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: General Classification

We learn rich natural sound representations by capitalizing on large amounts
of unlabeled sound data collected in the wild. We leverage the natural
synchronization between vision and sound to learn an acoustic representation
using two-million unlabeled videos. Unlabeled video has the advantage that it
can be economically acquired at massive scales, yet contains useful signals
about natural sound. We propose a student-teacher training procedure which
transfers discriminative visual knowledge from well established visual
recognition models into the sound modality using unlabeled video as a bridge.
Our sound representation yields significant performance improvements over the
state-of-the-art results on standard benchmarks for acoustic scene/object
classification. Visualizations suggest some high-level semantics automatically
emerge in the sound network, even though it is trained without ground truth
labels.
Rapid Posterior Exploration in Bayesian Non-negative Matrix Factorization
Authors: M. Arjumand Masood, Finale Doshi-Velez
Date: 2016-10-27 00:00:00
Tasks: Variational Inference

Non-negative Matrix Factorization (NMF) is a popular tool for data
exploration. Bayesian NMF promises to also characterize uncertainty in the
factorization. Unfortunately, current inference approaches such as MCMC mix
slowly and tend to get stuck on single modes. We introduce a novel approach
using rapidly-exploring random trees (RRTs) to asymptotically cover regions of
high posterior density. These are placed in a principled Bayesian framework via
an online extension to nonparametric variational inference. On experiments on
real and synthetic data, we obtain greater coverage of the posterior and higher
ELBO values than standard NMF inference approaches.
Voice Conversion using Convolutional Neural Networks
Authors: Shariq Mobin, Joan Bruna
Date: 2016-10-27 00:00:00
Tasks: Voice Conversion

The human auditory system is able to distinguish the vocal source of
thousands of speakers, yet not much is known about what features the auditory
system uses to do this. Fourier Transforms are capable of capturing the pitch
and harmonic structure of the speaker but this alone proves insufficient at
identifying speakers uniquely. The remaining structure, often referred to as
timbre, is critical to identifying speakers but we understood little about it.
In this paper we use recent advances in neural networks in order to manipulate
the voice of one speaker into another by transforming not only the pitch of the
speaker, but the timbre. We review generative models built with neural networks
as well as architectures for creating neural networks that learn analogies. Our
preliminary results converting voices from one speaker to another are
encouraging.
Tight Complexity Bounds for Optimizing Composite Objectives
Authors: Blake Woodworth, Nathan Srebro
Date: 2016-05-25 00:00:00
Proceeding: NeurIPS 2016

We provide tight upper and lower bounds on the complexity of minimizing the
average of $m$ convex functions using gradient and prox oracles of the
component functions. We show a significant gap between the complexity of
deterministic vs randomized optimization. For smooth functions, we show that
accelerated gradient descent (AGD) and an accelerated variant of SVRG are
optimal in the deterministic and randomized settings respectively, and that a
gradient oracle is sufficient for the optimal rate. For non-smooth functions,
having access to prox oracles reduces the complexity and we present optimal
methods based on smoothing that improve over methods using just gradient
accesses.
Local Similarity-Aware Deep Feature Embedding
Authors: Chen Huang, Chen Change Loy, Xiaoou Tang
Date: 2016-10-27 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Image Retrieval, Retrieval, Transfer Learning, Zero-Shot Learning

Existing deep embedding methods in vision tasks are capable of learning a
compact Euclidean space from images, where Euclidean distances correspond to a
similarity metric. To make learning more effective and efficient, hard sample
mining is usually employed, with samples identified through computing the
Euclidean feature distance. However, the global Euclidean distance cannot
faithfully characterize the true feature similarity in a complex visual feature
space, where the intraclass distance in a high-density region may be larger
than the interclass distance in low-density regions. In this paper, we
introduce a Position-Dependent Deep Metric (PDDM) unit, which is capable of
learning a similarity metric adaptive to local feature structure. The metric
can be used to select genuinely hard samples in a local neighborhood to guide
the deep embedding learning in an online and robust manner. The new layer is
appealing in that it is pluggable to any convolutional networks and is trained
end-to-end. Our local similarity-aware feature embedding not only demonstrates
faster convergence and boosted performance on two complex image retrieval
datasets, its large margin nature also leads to superior generalization results
under the large and open set scenarios of transfer learning and zero-shot
learning on ImageNet 2010 and ImageNet-10K datasets.