pwc-archive/papers-with-abstracts huggingface dataset:
Towards Ultra-High Performance and Energy Efficiency of Deep Learning Systems: An Algorithm-Hardware Co-Optimization Framework
Authors: Yanzhi Wang, Caiwen Ding, Zhe Li, Geng Yuan, Siyu Liao, Xiaolong Ma, Bo Yuan, Xuehai Qian, Jian Tang, Qinru Qiu, Xue Lin
Date: 2018-02-18 00:00:00

Hardware accelerations of deep learning systems have been extensively
investigated in industry and academia. The aim of this paper is to achieve
ultra-high energy efficiency and performance for hardware implementations of
deep neural networks (DNNs). An algorithm-hardware co-optimization framework is
developed, which is applicable to different DNN types, sizes, and application
scenarios. The algorithm part adopts the general block-circulant matrices to
achieve a fine-grained tradeoff between accuracy and compression ratio. It
applies to both fully-connected and convolutional layers and contains a
mathematically rigorous proof of the effectiveness of the method. The proposed
algorithm reduces computational complexity per layer from O($n^2$) to O($n\log
n$) and storage complexity from O($n^2$) to O($n$), both for training and
inference. The hardware part consists of highly efficient Field Programmable
Gate Array (FPGA)-based implementations using effective reconfiguration, batch
processing, deep pipelining, resource re-using, and hierarchical control.
Experimental results demonstrate that the proposed framework achieves at least
152X speedup and 71X energy efficiency gain compared with IBM TrueNorth
processor under the same test accuracy. It achieves at least 31X energy
efficiency gain compared with the reference FPGA-based work.
Visual-Only Recognition of Normal, Whispered and Silent Speech
Authors: Stavros Petridis, Jie Shen, Doruk Cetin, Maja Pantic
Date: 2018-02-18 00:00:00
Tasks: Silent Speech Recognition, speech-recognition, Speech Recognition, Visual Speech Recognition

Silent speech interfaces have been recently proposed as a way to enable
communication when the acoustic signal is not available. This introduces the
need to build visual speech recognition systems for silent and whispered
speech. However, almost all the recently proposed systems have been trained on
vocalised data only. This is in contrast with evidence in the literature which
suggests that lip movements change depending on the speech mode. In this work,
we introduce a new audiovisual database which is publicly available and
contains normal, whispered and silent speech. To the best of our knowledge,
this is the first study which investigates the differences between the three
speech modes using the visual modality only. We show that an absolute decrease
in classification rate of up to 3.7% is observed when training and testing on
normal and whispered, respectively, and vice versa. An even higher decrease of
up to 8.5% is reported when the models are tested on silent speech. This
reveals that there are indeed visual differences between the 3 speech modes and
the common assumption that vocalized training data can be used directly to
train a silent speech recognition system may not be true.
HybridSVD: When Collaborative Information is Not Enough
Authors: Evgeny Frolov, Ivan Oseledets
Date: 2018-02-18 00:00:00
Tasks: Collaborative Filtering, Model Selection

We propose a new hybrid algorithm that allows incorporating both user and item side information within the standard collaborative filtering technique. One of its key features is that it naturally extends a simple PureSVD approach and inherits its unique advantages, such as highly efficient Lanczos-based optimization procedure, simplified hyper-parameter tuning and a quick folding-in computation for generating recommendations instantly even in highly dynamic online environments. The algorithm utilizes a generalized formulation of the singular value decomposition, which adds flexibility to the solution and allows imposing the desired structure on its latent space. Conveniently, the resulting model also admits an efficient and straightforward solution for the cold start scenario. We evaluate our approach on a diverse set of datasets and show its superiority over similar classes of hybrid models.
Training Big Random Forests with Little Resources
Authors: Fabian Gieseke, Christian Igel
Date: 2018-02-18 00:00:00

Without access to large compute clusters, building random forests on large
datasets is still a challenging problem. This is, in particular, the case if
fully-grown trees are desired. We propose a simple yet effective framework that
allows to efficiently construct ensembles of huge trees for hundreds of
millions or even billions of training instances using a cheap desktop computer
with commodity hardware. The basic idea is to consider a multi-level
construction scheme, which builds top trees for small random subsets of the
available data and which subsequently distributes all training instances to the
top trees' leaves for further processing. While being conceptually simple, the
overall efficiency crucially depends on the particular implementation of the
different phases. The practical merits of our approach are demonstrated using
dense datasets with hundreds of millions of training instances.
Leveraging the Exact Likelihood of Deep Latent Variable Models
Authors: Pierre-Alexandre Mattei, Jes Frellsen
Date: 2018-02-13 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Imputation

Deep latent variable models (DLVMs) combine the approximation abilities of
deep neural networks and the statistical foundations of generative models.
Variational methods are commonly used for inference; however, the exact
likelihood of these models has been largely overlooked. The purpose of this
work is to study the general properties of this quantity and to show how they
can be leveraged in practice. We focus on important inferential problems that
rely on the likelihood: estimation and missing data imputation. First, we
investigate maximum likelihood estimation for DLVMs: in particular, we show
that most unconstrained models used for continuous data have an unbounded
likelihood function. This problematic behaviour is demonstrated to be a source
of mode collapse. We also show how to ensure the existence of maximum
likelihood estimates, and draw useful connections with nonparametric mixture
models. Finally, we describe an algorithm for missing data imputation using the
exact conditional likelihood of a deep latent variable model. On several data
sets, our algorithm consistently and significantly outperforms the usual
imputation scheme used for DLVMs.
Tensor Methods and Recommender Systems
Authors: Evgeny Frolov, Ivan Oseledets
Date: 2016-03-19 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems

A substantial progress in development of new and efficient tensor
factorization techniques has led to an extensive research of their
applicability in recommender systems field. Tensor-based recommender models
push the boundaries of traditional collaborative filtering techniques by taking
into account a multifaceted nature of real environments, which allows to
produce more accurate, situational (e.g. context-aware, criteria-driven)
recommendations. Despite the promising results, tensor-based methods are poorly
covered in existing recommender systems surveys. This survey aims to complement
previous works and provide a comprehensive overview on the subject. To the best
of our knowledge, this is the first attempt to consolidate studies from various
application domains in an easily readable, digestible format, which helps to
get a notion of the current state of the field. We also provide a high level
discussion of the future perspectives and directions for further improvement of
tensor-based recommendation systems.
Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentation
Authors: Florian Wenzel, Theo Galy-Fajou, Christan Donner, Marius Kloft, Manfred Opper
Date: 2018-02-18 00:00:00
Tasks: Classification, Data Augmentation, General Classification

We propose a scalable stochastic variational approach to GP classification
building on Polya-Gamma data augmentation and inducing points. Unlike former
approaches, we obtain closed-form updates based on natural gradients that lead
to efficient optimization. We evaluate the algorithm on real-world datasets
containing up to 11 million data points and demonstrate that it is up to two
orders of magnitude faster than the state-of-the-art while being competitive in
terms of prediction performance.
Node Centralities and Classification Performance for Characterizing Node Embedding Algorithms
Authors: Kento Nozawa, Masanari Kimura, Atsunori Kanemura
Date: 2018-02-18 00:00:00
Tasks: BIG-bench Machine Learning, Classification, General Classification

Embedding graph nodes into a vector space can allow the use of machine
learning to e.g. predict node classes, but the study of node embedding
algorithms is immature compared to the natural language processing field
because of a diverse nature of graphs. We examine the performance of node
embedding algorithms with respect to graph centrality measures that
characterize diverse graphs, through systematic experiments with four node
embedding algorithms, four or five graph centralities, and six datasets.
Experimental results give insights into the properties of node embedding
algorithms, which can be a basis for further research on this topic.
Efficient Sparse-Winograd Convolutional Neural Networks
Authors: Xingyu Liu, Jeff Pool, Song Han, William J. Dally
Date: 2018-02-18 00:00:00
Proceeding: ICLR 2018 1
Tasks: Network Pruning

Convolutional Neural Networks (CNNs) are computationally intensive, which
limits their application on mobile devices. Their energy is dominated by the
number of multiplies needed to perform the convolutions. Winograd's minimal
filtering algorithm (Lavin, 2015) and network pruning (Han et al., 2015) can
reduce the operation count, but these two methods cannot be directly combined
$-$ applying the Winograd transform fills in the sparsity in both the weights
and the activations. We propose two modifications to Winograd-based CNNs to
enable these methods to exploit sparsity. First, we move the ReLU operation
into the Winograd domain to increase the sparsity of the transformed
activations. Second, we prune the weights in the Winograd domain to exploit
static weight sparsity. For models on CIFAR-10, CIFAR-100 and ImageNet
datasets, our method reduces the number of multiplications by $10.4\times$,
$6.8\times$ and $10.8\times$ respectively with loss of accuracy less than
$0.1\%$, outperforming previous baselines by $2.0\times$-$3.0\times$. We also
show that moving ReLU to the Winograd domain allows more aggressive pruning.
A Large Dimensional Study of Regularized Discriminant Analysis Classifiers
Authors: Khalil Elkhalil, Abla Kammoun, Romain Couillet, Tareq Y. Al-Naffouri, Mohamed-Slim Alouini
Date: 2017-11-01 00:00:00
Tasks: valid

This article carries out a large dimensional analysis of standard regularized discriminant analysis classifiers designed on the assumption that data arise from a Gaussian mixture model with different means and covariances. The analysis relies on fundamental results from random matrix theory (RMT) when both the number of features and the cardinality of the training data within each class grow large at the same pace. Under mild assumptions, we show that the asymptotic classification error approaches a deterministic quantity that depends only on the means and covariances associated with each class as well as the problem dimensions. Such a result permits a better understanding of the performance of regularized discriminant analsysis, in practical large but finite dimensions, and can be used to determine and pre-estimate the optimal regularization parameter that minimizes the misclassification error probability. Despite being theoretically valid only for Gaussian data, our findings are shown to yield a high accuracy in predicting the performances achieved with real data sets drawn from the popular USPS data base, thereby making an interesting connection between theory and practice.
Anomaly Detection using One-Class Neural Networks
Authors: Raghavendra Chalapathy, Aditya Krishna Menon, Sanjay Chawla
Date: 2018-02-18 00:00:00
Tasks: Anomaly Detection

We propose a one-class neural network (OC-NN) model to detect anomalies in
complex data sets. OC-NN combines the ability of deep networks to extract a
progressively rich representation of data with the one-class objective of
creating a tight envelope around normal data. The OC-NN approach breaks new
ground for the following crucial reason: data representation in the hidden
layer is driven by the OC-NN objective and is thus customized for anomaly
detection. This is a departure from other approaches which use a hybrid
approach of learning deep features using an autoencoder and then feeding the
features into a separate anomaly detection method like one-class SVM (OC-SVM).
The hybrid OC-SVM approach is sub-optimal because it is unable to influence
representational learning in the hidden layers. A comprehensive set of
experiments demonstrate that on complex data sets (like CIFAR and GTSRB), OC-NN
performs on par with state-of-the-art methods and outperformed conventional
shallow methods in some scenarios.
Convergence of Online Mirror Descent
Authors: Yunwen Lei, Ding-Xuan Zhou
Date: 2018-02-18 00:00:00

In this paper we consider online mirror descent (OMD) algorithms, a class of scalable online learning algorithms exploiting data geometric structures through mirror maps. Necessary and sufficient conditions are presented in terms of the step size sequence $\{\eta_t\}_{t}$ for the convergence of an OMD algorithm with respect to the expected Bregman distance induced by the mirror map. The condition is $\lim_{t\to\infty}\eta_t=0, \sum_{t=1}^{\infty}\eta_t=\infty$ in the case of positive variances. It is reduced to $\sum_{t=1}^{\infty}\eta_t=\infty$ in the case of zero variances for which the linear convergence may be achieved by taking a constant step size sequence. A sufficient condition on the almost sure convergence is also given. We establish tight error bounds under mild conditions on the mirror map, the loss function, and the regularizer. Our results are achieved by some novel analysis on the one-step progress of the OMD algorithm using smoothness and strong convexity of the mirror map and the loss function.
Stochastic Chebyshev Gradient Descent for Spectral Optimization
Authors: Insu Han, Haim Avron, Jinwoo Shin
Date: 2018-02-18 00:00:00
Proceeding: NeurIPS 2018 12

A large class of machine learning techniques requires the solution of
optimization problems involving spectral functions of parametric matrices, e.g.
log-determinant and nuclear norm. Unfortunately, computing the gradient of a
spectral function is generally of cubic complexity, as such gradient descent
methods are rather expensive for optimizing objectives involving the spectral
function. Thus, one naturally turns to stochastic gradient methods in hope that
they will provide a way to reduce or altogether avoid the computation of full
gradients. However, here a new challenge appears: there is no straightforward
way to compute unbiased stochastic gradients for spectral functions. In this
paper, we develop unbiased stochastic gradients for spectral-sums, an important
subclass of spectral functions. Our unbiased stochastic gradients are based on
combining randomized trace estimators with stochastic truncation of the
Chebyshev expansions. A careful design of the truncation distribution allows us
to offer distributions that are variance-optimal, which is crucial for fast and
stable convergence of stochastic gradient methods. We further leverage our
proposed stochastic gradients to devise stochastic methods for objective
functions involving spectral-sums, and rigorously analyze their convergence
rate. The utility of our methods is demonstrated in numerical experiments.
Deep Learning-Based Food Calorie Estimation Method in Dietary Assessment
Authors: Yanchao Liang, Jianhua Li
Date: 2017-06-10 00:00:00
Tasks: Deep Learning, Object

Obesity treatment requires obese patients to record all food intakes per day.
Computer vision has been introduced to estimate calories from food images. In
order to increase accuracy of detection and reduce the error of volume
estimation in food calorie estimation, we present our calorie estimation method
in this paper. To estimate calorie of food, a top view and side view is needed.
Faster R-CNN is used to detect the food and calibration object. GrabCut
algorithm is used to get each food's contour. Then the volume is estimated with
the food and corresponding object. Finally we estimate each food's calorie. And
the experiment results show our estimation method is effective.
Deep Neural Networks for Bot Detection
Authors: Sneha Kudugunta, Emilio Ferrara
Date: 2018-02-12 00:00:00
Tasks: General Classification, Sentiment Analysis

The problem of detecting bots, automated social media accounts governed by
software but disguising as human users, has strong implications. For example,
bots have been used to sway political elections by distorting online discourse,
to manipulate the stock market, or to push anti-vaccine conspiracy theories
that caused health epidemics. Most techniques proposed to date detect bots at
the account level, by processing large amount of social media posts, and
leveraging information from network structure, temporal dynamics, sentiment
analysis, etc.
  In this paper, we propose a deep neural network based on contextual long
short-term memory (LSTM) architecture that exploits both content and metadata
to detect bots at the tweet level: contextual features are extracted from user
metadata and fed as auxiliary input to LSTM deep nets processing the tweet
text.
  Another contribution that we make is proposing a technique based on synthetic
minority oversampling to generate a large labeled dataset, suitable for deep
nets training, from a minimal amount of labeled data (roughly 3,000 examples of
sophisticated Twitter bots). We demonstrate that, from just one single tweet,
our architecture can achieve high classification accuracy (AUC > 96%) in
separating bots from humans.
  We apply the same architecture to account-level bot detection, achieving
nearly perfect classification accuracy (AUC > 99%). Our system outperforms
previous state of the art while leveraging a small and interpretable set of
features yet requiring minimal training data.
Zeroth-Order Online Alternating Direction Method of Multipliers: Convergence Analysis and Applications
Authors: Sijia Liu, Jie Chen, Pin-Yu Chen, Alfred O. Hero
Date: 2017-10-21 00:00:00

In this paper, we design and analyze a new zeroth-order online algorithm,
namely, the zeroth-order online alternating direction method of multipliers
(ZOO-ADMM), which enjoys dual advantages of being gradient-free operation and
employing the ADMM to accommodate complex structured regularizers. Compared to
the first-order gradient-based online algorithm, we show that ZOO-ADMM requires
$\sqrt{m}$ times more iterations, leading to a convergence rate of
$O(\sqrt{m}/\sqrt{T})$, where $m$ is the number of optimization variables, and
$T$ is the number of iterations. To accelerate ZOO-ADMM, we propose two
minibatch strategies: gradient sample averaging and observation averaging,
resulting in an improved convergence rate of $O(\sqrt{1+q^{-1}m}/\sqrt{T})$,
where $q$ is the minibatch size. In addition to convergence analysis, we also
demonstrate ZOO-ADMM to applications in signal processing, statistics, and
machine learning.
Large Neighborhood-Based Metaheuristic and Branch-and-Price for the Pickup and Delivery Problem with Split Loads
Authors: Matheus Nohra Haddad, Rafael Martinelli, Thibaut Vidal, Luiz Satoru Ochi, Simone Martins, Marcone Jamilson Freitas Souza, Richard Hartl
Date: 2018-02-18 00:00:00

We consider the multi-vehicle one-to-one pickup and delivery problem with
split loads, a NP-hard problem linked with a variety of applications for bulk
product transportation, bike-sharing systems and inventory re-balancing. This
problem is notoriously difficult due to the interaction of two challenging
vehicle routing attributes, "pickups and deliveries" and "split deliveries".
This possibly leads to optimal solutions of a size that grows exponentially
with the instance size, containing multiple visits per customer pair, even in
the same route. To solve this problem, we propose an iterated local search
metaheuristic as well as a branch-and-price algorithm. The core of the
metaheuristic consists of a new large neighborhood search, which reduces the
problem of finding the best insertion combination of a pickup and delivery pair
into a route (with possible splits) to a resource-constrained shortest path and
knapsack problem. Similarly, the branch-and-price algorithm uses sophisticated
labeling techniques, route relaxations, pre-processing and branching rules for
an efficient resolution. Our computational experiments on classical
single-vehicle instances demonstrate the excellent performance of the
metaheuristic, which produces new best known solutions for 92 out of 93 test
instances, and outperforms all previous algorithms. Experimental results on new
multi-vehicle instances with distance constraints are also reported. The
branch-and-price algorithm produces optimal solutions for instances with up to
20 pickup-and-delivery pairs, and very accurate solutions are found by the
metaheuristic.
Isointense Infant Brain Segmentation with a Hyper-dense Connected Convolutional Neural Network
Authors: Jose Dolz, Ismail Ben Ayed, Jing Yuan, Christian Desrosiers
Date: 2017-10-16 00:00:00
Tasks: Brain Segmentation, Infant Brain Mri Segmentation, MRI segmentation, Segmentation

Neonatal brain segmentation in magnetic resonance (MR) is a challenging
problem due to poor image quality and low contrast between white and gray
matter regions. Most existing approaches for this problem are based on
multi-atlas label fusion strategies, which are time-consuming and sensitive to
registration errors. As alternative to these methods, we propose a
hyper-densely connected 3D convolutional neural network that employs MR-T1 and
T2 images as input, which are processed independently in two separated paths.
An important difference with previous densely connected networks is the use of
direct connections between layers from the same and different paths. Adopting
such dense connectivity helps the learning process by including deep
supervision and improving gradient flow. We evaluated our approach on data from
the MICCAI Grand Challenge on 6-month infant Brain MRI Segmentation (iSEG),
obtaining very competitive results. Among 21 teams, our approach ranked first
or second in most metrics, translating into a state-of-the-art performance.
Unorganized Malicious Attacks Detection
Authors: Ming Pang, Wei Gao, Min Tao, Zhi-Hua Zhou
Date: 2016-10-13 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Matrix Completion, Recommendation Systems

Recommender system has attracted much attention during the past decade. Many
attack detection algorithms have been developed for better recommendations,
mostly focusing on shilling attacks, where an attack organizer produces a large
number of user profiles by the same strategy to promote or demote an item. This
work considers a different attack style: unorganized malicious attacks, where
attackers individually utilize a small number of user profiles to attack
different items without any organizer. This attack style occurs in many real
applications, yet relevant study remains open. We first formulate the
unorganized malicious attacks detection as a matrix completion problem, and
propose the Unorganized Malicious Attacks detection (UMA) approach, a proximal
alternating splitting augmented Lagrangian method. We verify, both
theoretically and empirically, the effectiveness of our proposed approach.