pwc-archive/papers-with-abstracts huggingface dataset:
Distributed Flexible Nonlinear Tensor Factorization
Authors: Shandian Zhe, Kai Zhang, Pengyuan Wang, Kuang-Chih Lee, Zenglin Xu, Yuan Qi, Zoubin Ghahramani
Date: 2016-04-27 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Click-Through Rate Prediction, Computational Efficiency

Tensor factorization is a powerful tool to analyse multi-way data. Compared
with traditional multi-linear methods, nonlinear tensor factorization models
are capable of capturing more complex relationships in the data. However, they
are computationally expensive and may suffer severe learning bias in case of
extreme data sparsity. To overcome these limitations, in this paper we propose
a distributed, flexible nonlinear tensor factorization model. Our model can
effectively avoid the expensive computations and structural restrictions of the
Kronecker-product in existing TGP formulations, allowing an arbitrary subset of
tensorial entries to be selected to contribute to the training. At the same
time, we derive a tractable and tight variational evidence lower bound (ELBO)
that enables highly decoupled, parallel computations and high-quality
inference. Based on the new bound, we develop a distributed inference algorithm
in the MapReduce framework, which is key-value-free and can fully exploit the
memory cache mechanism in fast MapReduce systems such as SPARK. Experimental
results fully demonstrate the advantages of our method over several
state-of-the-art approaches, in terms of both predictive performance and
computational efficiency. Moreover, our approach shows a promising potential in
the application of Click-Through-Rate (CTR) prediction for online advertising.
Learning From Hidden Traits: Joint Factor Analysis and Latent Clustering
Authors: Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos
Date: 2016-05-21 00:00:00
Tasks: Clustering, Dimensionality Reduction

Dimensionality reduction techniques play an essential role in data analytics,
signal processing and machine learning. Dimensionality reduction is usually
performed in a preprocessing stage that is separate from subsequent data
analysis, such as clustering or classification. Finding reduced-dimension
representations that are well-suited for the intended task is more appealing.
This paper proposes a joint factor analysis and latent clustering framework,
which aims at learning cluster-aware low-dimensional representations of matrix
and tensor data. The proposed approach leverages matrix and tensor
factorization models that produce essentially unique latent representations of
the data to unravel latent cluster structure -- which is otherwise obscured
because of the freedom to apply an oblique transformation in latent space. At
the same time, latent cluster structure is used as prior information to enhance
the performance of factorization. Specific contributions include several
custom-built problem formulations, corresponding algorithms, and discussion of
associated convergence properties. Besides extensive simulations, real-world
datasets such as Reuters document data and MNIST image data are also employed
to showcase the effectiveness of the proposed approaches.
Chess Player by Co-Evolutionary Algorithm
Authors: Nuno Ramos, Sergio Salgado, Agostinho C. Rosa
Date: 2016-05-21 00:00:00

A co-evolutionary algorithm (CA) based chess player is presented.
Implementation details of the algorithms, namely coding, population, variation
operators are described. The alpha-beta or mini-max like behaviour of the
player is achieved through two competitive or cooperative populations. Special
attention is given to the fitness function evaluation (the heart of the
solution). Test results on algorithms vs. algorithms or human player is
provided.
Automatic Detection of Epileptiform Discharges in the EEG
Authors: Andre Rosado, Agostinho C. Rosa
Date: 2016-05-21 00:00:00
Tasks: EEG, Electroencephalogram (EEG), Specificity

The diagnosis of epilepsy generally includes a visual inspection of EEG
recorded data by the Neurologist, with the purpose of checking the occurrence
of transient waveforms called interictal epileptiform discharges. These
waveforms have short duration (less than 100 ms), so the inspection process is
usually time-consuming, particularly for ambulatory long term EEG records.
Therefore, an automatic detection system of epileptiform discharges can be a
valuable tool for a Neurology service. The proposed approach is the development
of a multi stage detection algorithm, which processes the complete EEG signals
and applies decision criteria to selected waveforms. It employs EEG analysis
techniques such as Wavelet Transform and Mimetic Analysis, complemented with a
classification based on Fuzzy Logic. In order to evaluate the algorithm's
performance, data were collected from several epileptic patients, with
epileptiform activity marked by a Neurologist. The average values obtained for
both Sensitivity and Specificity were respectively higher than 80 and 70
percent.
Fine-to-coarse Knowledge Transfer For Low-Res Image Classification
Authors: Xingchao Peng, Judy Hoffman, Stella X. Yu, Kate Saenko
Date: 2016-05-21 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification, Transfer Learning

We address the difficult problem of distinguishing fine-grained object
categories in low resolution images. Wepropose a simple an effective deep
learning approach that transfers fine-grained knowledge gained from high
resolution training data to the coarse low-resolution test scenario. Such
fine-to-coarse knowledge transfer has many real world applications, such as
identifying objects in surveillance photos or satellite images where the image
resolution at the test time is very low but plenty of high resolution photos of
similar objects are available. Our extensive experiments on two standard
benchmark datasets containing fine-grained car models and bird species
demonstrate that our approach can effectively transfer fine-detail knowledge to
coarse-detail imagery.
Efficient Document Indexing Using Pivot Tree
Authors: Gaurav Singh, Benjamin Piwowarski
Date: 2016-05-21 00:00:00
Tasks: Retrieval

We present a novel method for efficiently searching top-k neighbors for
documents represented in high dimensional space of terms based on the cosine
similarity. Mostly, documents are stored as bag-of-words tf-idf representation.
One of the most used ways of computing similarity between a pair of documents
is cosine similarity between the vector representations, but cosine similarity
is not a metric distance measure as it doesn't follow triangle inequality,
therefore most metric searching methods can not be applied directly. We propose
an efficient method for indexing documents using a pivot tree that leads to
efficient retrieval. We also study the relation between precision and
efficiency for the proposed method and compare it with a state of the art in
the area of document searching based on inner product.
Optimal Cluster Recovery in the Labeled Stochastic Block Model
Authors: Se-Young Yun, Alexandre Proutiere
Date: 2015-10-20 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Clustering, Community Detection, Stochastic Block Model

We consider the problem of community detection or clustering in the labeled
Stochastic Block Model (LSBM) with a finite number $K$ of clusters of sizes
linearly growing with the global population of items $n$. Every pair of items
is labeled independently at random, and label $\ell$ appears with probability
$p(i,j,\ell)$ between two items in clusters indexed by $i$ and $j$,
respectively. The objective is to reconstruct the clusters from the observation
of these random labels.
  Clustering under the SBM and their extensions has attracted much attention
recently. Most existing work aimed at characterizing the set of parameters such
that it is possible to infer clusters either positively correlated with the
true clusters, or with a vanishing proportion of misclassified items, or
exactly matching the true clusters. We find the set of parameters such that
there exists a clustering algorithm with at most $s$ misclassified items in
average under the general LSBM and for any $s=o(n)$, which solves one open
problem raised in \cite{abbe2015community}. We further develop an algorithm,
based on simple spectral methods, that achieves this fundamental performance
limit within $O(n \mbox{polylog}(n))$ computations and without the a-priori
knowledge of the model parameters.
Back to the Basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation
Authors: Kevin H. Wilson, Yan Karklin, Bojian Han, Chaitanya Ekanadham
Date: 2016-04-08 00:00:00
Tasks: Knowledge Tracing

Estimating student proficiency is an important task for computer based
learning systems. We compare a family of IRT-based proficiency estimation
methods to Deep Knowledge Tracing (DKT), a recently proposed recurrent neural
network model with promising initial results. We evaluate how well each model
predicts a student's future response given previous responses using two
publicly available and one proprietary data set. We find that IRT-based methods
consistently matched or outperformed DKT across all data sets at the finest
level of content granularity that was tractable for them to be trained on. A
hierarchical extension of IRT that captured item grouping structure performed
best overall. When data sets included non-trivial autocorrelations in student
response patterns, a temporal extension of IRT improved performance over
standard IRT while the RNN-based method did not. We conclude that IRT-based
models provide a simpler, better-performing alternative to existing RNN-based
models of student interaction data while also affording more interpretability
and guarantees due to their formulation as Bayesian probabilistic models.
Cross Domain Adaptation by Learning Partially Shared Classifiers and Weighting Source Data Points in the Shared Subspaces
Authors: Hongqi Wang, Anfeng Xu, Shan-Shan Wang, Sunny Chughtai
Date: 2016-05-21 00:00:00
Tasks: Domain Adaptation, Transfer Learning

Transfer learning is a problem defined over two domains. These two domains
share the same feature space and class label space, but have significantly
different distributions. One domain has sufficient labels, named as source
domain, and the other domain has few labels, named as target do- main. The
problem is to learn a effective classifier for the target domain. In this
paper, we propose a novel transfer learning method for this problem by learning
a partially shared classifier for the target domain, and weighting the source
domain data points. We learn some shared subspaces for both the data points of
the two domains, and a shared classifier in the shared subspaces. We hope that
in the shared subspaces, the distributions of two domain can match each other
well, and to match the distributions, we weight the source domain data points
with different weighting factors. Moreover, we adapt the shared classifier to
each domain by learning different adaptation functions. To learn the subspace
transformation matrices, the classifier parameters, and the adaptation
parameters, we build a objective function with weighted clas- sification
errors, parameter regularization, local reconstruction regularization, and
distribution matching. This objective function is minimized by an itera- tive
algorithm. Experiments show its effectiveness over benchmark data sets,
including travel destination review data set, face expression data set, spam
email data set, etc.
Bending the Curve: Improving the ROC Curve Through Error Redistribution
Authors: Oran Richman, Shie Mannor
Date: 2016-05-21 00:00:00
Tasks: General Classification, Meta-Learning

Classification performance is often not uniform over the data. Some areas in
the input space are easier to classify than others. Features that hold
information about the "difficulty" of the data may be non-discriminative and
are therefore disregarded in the classification process. We propose a
meta-learning approach where performance may be improved by post-processing.
This improvement is done by establishing a dynamic threshold on the
base-classifier results. Since the base-classifier is treated as a "black box"
the method presented can be used on any state of the art classifier in order to
try an improve its performance. We focus our attention on how to better control
the true-positive/false-positive trade-off known as the ROC curve. We propose
an algorithm for the derivation of optimal thresholds by redistributing the
error depending on features that hold information about difficulty. We
demonstrate the resulting benefit on both synthetic and real-life data.
SDCA without Duality, Regularization, and Individual Convexity
Authors: Shai Shalev-Shwartz
Date: 2016-02-04 00:00:00

Stochastic Dual Coordinate Ascent is a popular method for solving regularized
loss minimization for the case of convex losses. We describe variants of SDCA
that do not require explicit regularization and do not rely on duality. We
prove linear convergence rates even if individual loss functions are
non-convex, as long as the expected loss is strongly convex.
Make Workers Work Harder: Decoupled Asynchronous Proximal Stochastic Gradient Descent
Authors: Yitan Li, Linli Xu, Xiaowei Zhong, Qing Ling
Date: 2016-05-21 00:00:00

Asynchronous parallel optimization algorithms for solving large-scale machine
learning problems have drawn significant attention from academia to industry
recently. This paper proposes a novel algorithm, decoupled asynchronous
proximal stochastic gradient descent (DAP-SGD), to minimize an objective
function that is the composite of the average of multiple empirical losses and
a regularization term. Unlike the traditional asynchronous proximal stochastic
gradient descent (TAP-SGD) in which the master carries much of the computation
load, the proposed algorithm off-loads the majority of computation tasks from
the master to workers, and leaves the master to conduct simple addition
operations. This strategy yields an easy-to-parallelize algorithm, whose
performance is justified by theoretical convergence analyses. To be specific,
DAP-SGD achieves an $O(\log T/T)$ rate when the step-size is diminishing and an
ergodic $O(1/\sqrt{T})$ rate when the step-size is constant, where $T$ is the
number of total iterations.
Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping
Authors: Andrew J. R. Simpson
Date: 2015-05-22 00:00:00

Although deep neural networks (DNN) are able to scale with direct advances in
computational power (e.g., memory and processing speed), they are not well
suited to exploit the recent trends for parallel architectures. In particular,
gradient descent is a sequential process and the resulting serial dependencies
mean that DNN training cannot be parallelized effectively. Here, we show that a
DNN may be replicated over a massive parallel architecture and used to provide
a cumulative sampling of local solution space which results in rapid and robust
learning. We introduce a complimentary convolutional bootstrapping approach
that enhances performance of the parallel architecture further. Our
parallelized convolutional bootstrapping DNN out-performs an identical
fully-trained traditional DNN after only a single iteration of training.
Massively Multilingual Word Embeddings
Authors: Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, Noah A. Smith
Date: 2016-02-05 00:00:00
Tasks: Multilingual Word Embeddings, Text Categorization, Word Embeddings

We introduce new methods for estimating and evaluating embeddings of words in
more than fifty languages in a single shared embedding space. Our estimation
methods, multiCluster and multiCCA, use dictionaries and monolingual data; they
do not require parallel data. Our new evaluation method, multiQVEC-CCA, is
shown to correlate better than previous ones with two downstream tasks (text
categorization and parsing). We also describe a web portal for evaluation that
will facilitate further research in this area, along with open-source releases
of all our methods.
A Theory of Formal Synthesis via Inductive Learning
Authors: Susmit Jha, Sanjit A. Seshia
Date: 2015-05-15 00:00:00
Tasks: BIG-bench Machine Learning, Inductive Learning, Learning Theory

Formal synthesis is the process of generating a program satisfying a
high-level formal specification. In recent times, effective formal synthesis
methods have been proposed based on the use of inductive learning. We refer to
this class of methods that learn programs from examples as formal inductive
synthesis. In this paper, we present a theoretical framework for formal
inductive synthesis. We discuss how formal inductive synthesis differs from
traditional machine learning. We then describe oracle-guided inductive
synthesis (OGIS), a framework that captures a family of synthesizers that
operate by iteratively querying an oracle. An instance of OGIS that has had
much practical impact is counterexample-guided inductive synthesis (CEGIS). We
present a theoretical characterization of CEGIS for learning any program that
computes a recursive language. In particular, we analyze the relative power of
CEGIS variants where the types of counterexamples generated by the oracle
varies. We also consider the impact of bounded versus unbounded memory
available to the learning algorithm. In the special case where the universe of
candidate programs is finite, we relate the speed of convergence to the notion
of teaching dimension studied in machine learning theory. Altogether, the
results of the paper take a first step towards a theoretical foundation for the
emerging field of formal inductive synthesis.
Adaptive Algorithm and Platform Selection for Visual Detection and Tracking
Authors: Shu Zhang, Qi Zhu, Amit Roy-Chowdhury
Date: 2016-05-21 00:00:00
Tasks: Pedestrian Detection

Computer vision algorithms are known to be extremely sensitive to the
environmental conditions in which the data is captured, e.g., lighting
conditions and target density. Tuning of parameters or choosing a completely
new algorithm is often needed to achieve a certain performance level,
especially when there is a limitation of the computation source. In this paper,
we focus on this problem and propose a framework to adaptively select the
"best" algorithm-parameter combination and the computation platform under
performance and cost constraints at design time, and adapt the algorithms at
runtime based on real-time inputs. This necessitates developing a mechanism to
switch between different algorithms as the nature of the input video changes.
Our proposed algorithm calculates a similarity function between a test video
scenario and each training scenario, where the similarity calculation is based
on learning a manifold of image features that is shared by both the training
and test datasets. Similarity between training and test dataset indicates the
same algorithm can be applied to both of them and achieve similar performance.
We design a cost function with this similarity measure to find the most similar
training scenario to the test data. The "best" algorithm under a given platform
is obtained by selecting the algorithm with a specific parameter combination
that performs the best on the corresponding training data. The proposed
framework can be used first offline to choose the platform based on performance
and cost constraints, and then online whereby the "best" algorithm is selected
for each new incoming video segment for a given platform. In the experiments,
we apply our algorithm to the problems of pedestrian detection and tracking. We
show how to adaptively select platforms and algorithm-parameter combinations.
Our results provide optimal performance on 3 publicly available datasets.
DynaNewton - Accelerating Newton's Method for Machine Learning
Authors: Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann
Date: 2016-05-20 00:00:00
Tasks: BIG-bench Machine Learning

Newton's method is a fundamental technique in optimization with quadratic
convergence within a neighborhood around the optimum. However reaching this
neighborhood is often slow and dominates the computational costs. We exploit
two properties specific to empirical risk minimization problems to accelerate
Newton's method, namely, subsampling training data and increasing strong
convexity through regularization. We propose a novel continuation method, where
we define a family of objectives over increasing sample sizes and with
decreasing regularization strength. Solutions on this path are tracked such
that the minimizer of the previous objective is guaranteed to be within the
quadratic convergence region of the next objective to be optimized. Thereby
every Newton iteration is guaranteed to achieve super-linear contractions with
regard to the chosen objective, which becomes a moving target. We provide a
theoretical analysis that motivates our algorithm, called DynaNewton, and
characterizes its speed of convergence. Experiments on a wide range of data
sets and problems consistently confirm the predicted computational savings.
Functional Hashing for Compressing Neural Networks
Authors: Lei Shi, Shikun Feng, ZhifanZhu
Date: 2016-05-20 00:00:00

As the complexity of deep neural networks (DNNs) trend to grow to absorb the
increasing sizes of data, memory and energy consumption has been receiving more
and more attentions for industrial applications, especially on mobile devices.
This paper presents a novel structure based on functional hashing to compress
DNNs, namely FunHashNN. For each entry in a deep net, FunHashNN uses multiple
low-cost hash functions to fetch values in the compression space, and then
employs a small reconstruction network to recover that entry. The
reconstruction network is plugged into the whole network and trained jointly.
FunHashNN includes the recently proposed HashedNets as a degenerated case, and
benefits from larger value capacity and less reconstruction loss. We further
discuss extensions with dual space hashing and multi-hops. On several benchmark
datasets, FunHashNN demonstrates high compression ratios with little loss on
prediction accuracy.
Semismooth Newton Coordinate Descent Algorithm for Elastic-Net Penalized Huber Loss Regression and Quantile Regression
Authors: Congrui Yi, Jian Huang
Date: 2015-09-09 00:00:00
Tasks: quantile regression, regression

We propose an algorithm, semismooth Newton coordinate descent (SNCD), for the
elastic-net penalized Huber loss regression and quantile regression in high
dimensional settings. Unlike existing coordinate descent type algorithms, the
SNCD updates each regression coefficient and its corresponding subgradient
simultaneously in each iteration. It combines the strengths of the coordinate
descent and the semismooth Newton algorithm, and effectively solves the
computational challenges posed by dimensionality and nonsmoothness. We
establish the convergence properties of the algorithm. In addition, we present
an adaptive version of the "strong rule" for screening predictors to gain extra
efficiency. Through numerical experiments, we demonstrate that the proposed
algorithm is very efficient and scalable to ultra-high dimensions. We
illustrate the application via a real data example.