pwc-archive/papers-with-abstracts huggingface dataset:
A Deep Network Model for Paraphrase Detection in Short Text Messages
Authors: Basant Agarwal, Heri Ramampiaro, Helge Langseth, Massimiliano Ruocco
Date: 2017-12-07 00:00:00
Tasks: Question Answering, Sentence, Text Summarization

This paper is concerned with paraphrase detection. The ability to detect
similar sentences written in natural language is crucial for several
applications, such as text mining, text summarization, plagiarism detection,
authorship authentication and question answering. Given two sentences, the
objective is to detect whether they are semantically identical. An important
insight from this work is that existing paraphrase systems perform well when
applied on clean texts, but they do not necessarily deliver good performance
against noisy texts. Challenges with paraphrase detection on user generated
short texts, such as Twitter, include language irregularity and noise. To cope
with these challenges, we propose a novel deep neural network-based approach
that relies on coarse-grained sentence modeling using a convolutional neural
network and a long short-term memory model, combined with a specific
fine-grained word-level similarity matching model. Our experimental results
show that the proposed approach outperforms existing state-of-the-art
approaches on user-generated noisy social media data, such as Twitter texts,
and achieves highly competitive performance on a cleaner corpus.
Improved Bayesian Compression
Authors: Marco Federici, Karen Ullrich, Max Welling
Date: 2017-11-17 00:00:00
Tasks: Model Compression

Compression of Neural Networks (NN) has become a highly studied topic in
recent years. The main reason for this is the demand for industrial scale usage
of NNs such as deploying them on mobile devices, storing them efficiently,
transmitting them via band-limited channels and most importantly doing
inference at scale. In this work, we propose to join the Soft-Weight Sharing
and Variational Dropout approaches that show strong results to define a new
state-of-the-art in terms of model compression.
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Authors: Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun
Date: 2017-07-04 00:00:00
Proceeding: CVPR 2018 6
Tasks: General Classification, Image Classification, Object Detection, Person Re-Identification

We introduce an extremely computation-efficient CNN architecture named
ShuffleNet, which is designed specially for mobile devices with very limited
computing power (e.g., 10-150 MFLOPs). The new architecture utilizes two new
operations, pointwise group convolution and channel shuffle, to greatly reduce
computation cost while maintaining accuracy. Experiments on ImageNet
classification and MS COCO object detection demonstrate the superior
performance of ShuffleNet over other structures, e.g. lower top-1 error
(absolute 7.8%) than recent MobileNet on ImageNet classification task, under
the computation budget of 40 MFLOPs. On an ARM-based mobile device, ShuffleNet
achieves ~13x actual speedup over AlexNet while maintaining comparable
accuracy.
End-to-end Learning of Deterministic Decision Trees
Authors: Thomas Hehn, Fred A. Hamprecht
Date: 2017-12-07 00:00:00

Conventional decision trees have a number of favorable properties, including
interpretability, a small computational footprint and the ability to learn from
little training data. However, they lack a key quality that has helped fuel the
deep learning revolution: that of being end-to-end trainable, and to learn from
scratch those features that best allow to solve a given supervised learning
problem. Recent work (Kontschieder 2015) has addressed this deficit, but at the
cost of losing a main attractive trait of decision trees: the fact that each
sample is routed along a small subset of tree nodes only. We here propose a
model and Expectation-Maximization training scheme for decision trees that are
fully probabilistic at train time, but after a deterministic annealing process
become deterministic at test time. We also analyze the learned oblique split
parameters on image datasets and show that Neural Networks can be trained at
each split node. In summary, we present the first end-to-end learning scheme
for deterministic decision trees and present results on par with or superior to
published standard oblique decision tree algorithms.
Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing
Authors: Syed Shakib Sarwar, Aayush Ankit, Kaushik Roy
Date: 2017-12-07 00:00:00
Tasks: General Classification, image-classification, Image Classification, Incremental Learning, Transfer Learning

Deep convolutional neural network (DCNN) based supervised learning is a
widely practiced approach for large-scale image classification. However,
retraining these large networks to accommodate new, previously unseen data
demands high computational time and energy requirements. Also, previously seen
training samples may not be available at the time of retraining. We propose an
efficient training methodology and incrementally growing DCNN to learn new
tasks while sharing part of the base network. Our proposed methodology is
inspired by transfer learning techniques, although it does not forget
previously learned tasks. An updated network for learning new set of classes is
formed using previously learned convolutional layers (shared from initial part
of base network) with addition of few newly added convolutional kernels
included in the later layers of the network. We employed a `clone-and-branch'
technique which allows the network to learn new tasks one after another without
any performance loss in old tasks. We evaluated the proposed scheme on several
recognition applications. The classification accuracy achieved by our approach
is comparable to the regular incremental learning approach (where networks are
updated with new training samples only, without any network sharing), while
achieving energy efficiency, reduction in storage requirements, memory access
and training time.
Connectivity Learning in Multi-Branch Networks
Authors: Karim Ahmed, Lorenzo Torresani
Date: 2017-09-27 00:00:00
Proceeding: ICLR 2018 1
Tasks: image-classification, Image Classification

While much of the work in the design of convolutional networks over the last
five years has revolved around the empirical investigation of the importance of
depth, filter sizes, and number of feature channels, recent studies have shown
that branching, i.e., splitting the computation along parallel but distinct
threads and then aggregating their outputs, represents a new promising
dimension for significant improvements in performance. To combat the complexity
of design choices in multi-branch architectures, prior work has adopted simple
strategies, such as a fixed branching factor, the same input being fed to all
parallel branches, and an additive combination of the outputs produced by all
branches at aggregation points.
  In this work we remove these predefined choices and propose an algorithm to
learn the connections between branches in the network. Instead of being chosen
a priori by the human designer, the multi-branch connectivity is learned
simultaneously with the weights of the network by optimizing a single loss
function defined with respect to the end task. We demonstrate our approach on
the problem of multi-class image classification using three different datasets
where it yields consistently higher accuracy compared to the state-of-the-art
"ResNeXt" multi-branch network given the same learning capacity.
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
Authors: Chia-Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei zhang, Kailash Gopalakrishnan
Date: 2017-12-07 00:00:00
Tasks: Quantization

Highly distributed training of Deep Neural Networks (DNNs) on future compute
platforms (offering 100 of TeraOps/s of computational capacity) is expected to
be severely communication constrained. To overcome this limitation, new
gradient compression techniques are needed that are computationally friendly,
applicable to a wide variety of layers seen in Deep Neural Networks and
adaptable to variations in network architectures as well as their
hyper-parameters. In this paper we introduce a novel technique - the Adaptive
Residual Gradient Compression (AdaComp) scheme. AdaComp is based on localized
selection of gradient residues and automatically tunes the compression rate
depending on local activity. We show excellent results on a wide spectrum of
state of the art Deep Learning models in multiple domains (vision, speech,
language), datasets (MNIST, CIFAR10, ImageNet, BN50, Shakespeare), optimizers
(SGD with momentum, Adam) and network parameters (number of learners,
minibatch-size etc.). Exploiting both sparsity and quantization, we demonstrate
end-to-end compression rates of ~200X for fully-connected and recurrent layers,
and ~40X for convolutional layers, without any noticeable degradation in model
accuracies.
Forecasting day-ahead electricity prices in Europe: the importance of considering market integration
Authors: Jesus Lago, Fjo De Ridder, Peter Vrancx, Bart De Schutter
Date: 2017-08-01 00:00:00
Tasks: Bayesian Optimization, feature selection

Motivated by the increasing integration among electricity markets, in this
paper we propose two different methods to incorporate market integration in
electricity price forecasting and to improve the predictive performance. First,
we propose a deep neural network that considers features from connected markets
to improve the predictive accuracy in a local market. To measure the importance
of these features, we propose a novel feature selection algorithm that, by
using Bayesian optimization and functional analysis of variance, evaluates the
effect of the features on the algorithm performance. In addition, using market
integration, we propose a second model that, by simultaneously predicting
prices from two markets, improves the forecasting accuracy even further. As a
case study, we consider the electricity market in Belgium and the improvements
in forecasting accuracy when using various French electricity features. We show
that the two proposed models lead to improvements that are statistically
significant. Particularly, due to market integration, the predictive accuracy
is improved from 15.7% to 12.5% sMAPE (symmetric mean absolute percentage
error). In addition, we show that the proposed feature selection algorithm is
able to perform a correct assessment, i.e. to discard the irrelevant features.
Robust 3D Action Recognition through Sampling Local Appearances and Global Distributions
Authors: Mengyuan Liu, Hong Liu, Chen Chen
Date: 2017-12-04 00:00:00
Tasks: 3D Action Recognition, Action Recognition, Temporal Action Localization

3D action recognition has broad applications in human-computer interaction
and intelligent surveillance. However, recognizing similar actions remains
challenging since previous literature fails to capture motion and shape cues
effectively from noisy depth data. In this paper, we propose a novel two-layer
Bag-of-Visual-Words (BoVW) model, which suppresses the noise disturbances and
jointly encodes both motion and shape cues. First, background clutter is
removed by a background modeling method that is designed for depth data. Then,
motion and shape cues are jointly used to generate robust and distinctive
spatial-temporal interest points (STIPs): motion-based STIPs and shape-based
STIPs. In the first layer of our model, a multi-scale 3D local steering kernel
(M3DLSK) descriptor is proposed to describe local appearances of cuboids around
motion-based STIPs. In the second layer, a spatial-temporal vector (STV)
descriptor is proposed to describe the spatial-temporal distributions of
shape-based STIPs. Using the Bag-of-Visual-Words (BoVW) model, motion and shape
cues are combined to form a fused action representation. Our model performs
favorably compared with common STIP detection and description methods. Thorough
experiments verify that our model is effective in distinguishing similar
actions and robust to background clutter, partial occlusions and pepper noise.
BDgraph: An R Package for Bayesian Structure Learning in Graphical Models
Authors: Reza Mohammadi, Ernst C. Wit
Date: 2015-01-21 00:00:00

Graphical models provide powerful tools to uncover complicated patterns in multivariate data and are commonly used in Bayesian statistics and machine learning. In this paper, we introduce the R package BDgraph which performs Bayesian structure learning for general undirected graphical models (decomposable and non-decomposable) with continuous, discrete, and mixed variables. The package efficiently implements recent improvements in the Bayesian literature, including that of Mohammadi and Wit (2015) and Dobra and Mohammadi (2018). To speed up computations, the computationally intensive tasks have been implemented in C++ and interfaced with R, and the package has parallel computing capabilities. In addition, the package contains several functions for simulation and visualization, as well as several multivariate datasets taken from the literature and used to describe the package capabilities. The paper includes a brief overview of the statistical methods which have been implemented in the package. The main part of the paper explains how to use the package. Furthermore, we illustrate the package's functionality in both real and artificial examples.
Using SVDD in SimpleMKL for 3D-Shapes Filtering
Authors: Gaëlle Loosli, Hattoibe Aboubacar
Date: 2017-12-07 00:00:00

This paper proposes the adaptation of Support Vector Data Description (SVDD)
to the multiple kernel case (MK-SVDD), based on SimpleMKL. It also introduces a
variant called Slim-MK-SVDD that is able to produce a tighter frontier around
the data. For the sake of comparison, the equivalent methods are also developed
for One-Class SVM, known to be very similar to SVDD for certain shapes of
kernels.
  Those algorithms are illustrated in the context of 3D-shapes filtering and
outliers detection. For the 3D-shapes problem, the objective is to be able to
select a sub-category of 3D-shapes, each sub-category being learned with our
algorithm in order to create a filter. For outliers detection, we apply the
proposed algorithms for unsupervised outliers detection as well as for the
supervised case.
Predicting Demographics, Moral Foundations, and Human Values from Digital Behaviors
Authors: Kyriaki Kalimeri, Mariano G. Beiro, Matteo Delfino, Robert Raleigh, Ciro Cattuto
Date: 2017-12-05 00:00:00
Tasks: Attribute

Personal electronic devices including smartphones give access to behavioural
signals that can be used to learn about the characteristics and preferences of
individuals. In this study, we explore the connection between demographic and
psychological attributes and the digital behavioural records, for a cohort of
7,633 people, closely representative of the US population with respect to
gender, age, geographical distribution, education, and income. Along with the
demographic data, we collected self-reported assessments on validated
psychometric questionnaires for moral traits and basic human values and
combined this information with passively collected multi-modal digital data
from web browsing behaviour and smartphone usage. A machine learning framework
was then designed to infer both the demographic and psychological attributes
from the behavioural data. In a cross-validated setting, our models predicted
demographic attributes with good accuracy as measured by the weighted AUROC
score (Area Under the Receiver Operating Characteristic), but were less
performant for the moral traits and human values. These results call for
further investigation since they are still far from unveiling individuals'
psychological fabric. This connection, along with the most predictive features
that we provide for each attribute, might prove useful for designing
personalised services, communication strategies, and interventions, and can be
used to sketch a portrait of people with a similar worldview.
Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision
Authors: Mostafa Dehghani, Aliaksei Severyn, Sascha Rothe, Jaap Kamps
Date: 2017-11-01 00:00:00
Tasks: Document Ranking, Sentiment Analysis, Sentiment Classification

Training deep neural networks requires massive amounts of training data, but
for many tasks only limited labeled data is available. This makes weak
supervision attractive, using weak or noisy signals like the output of
heuristic methods or user click-through data for training. In a semi-supervised
setting, we can use a large set of data with weak labels to pretrain a neural
network and then fine-tune the parameters with a small amount of data with true
labels. This feels intuitively sub-optimal as these two independent stages
leave the model unaware about the varying label quality. What if we could
somehow inform the model about the label quality? In this paper, we propose a
semi-supervised learning method where we train two neural networks in a
multi-task fashion: a "target network" and a "confidence network". The target
network is optimized to perform a given task and is trained using a large set
of unlabeled data that are weakly annotated. We propose to weight the gradient
updates to the target network using the scores provided by the second
confidence network, which is trained on a small amount of supervised data. Thus
we avoid that the weight updates computed from noisy labels harm the quality of
the target network model. We evaluate our learning strategy on two different
tasks: document ranking and sentiment classification. The results demonstrate
that our approach not only enhances the performance compared to the baselines
but also speeds up the learning process from weak labels.
High-dimensional robust regression and outliers detection with SLOPE
Authors: Alain Virouleau, Agathe Guilloux, Stéphane Gaïffas, Malgorzata Bogdan
Date: 2017-12-07 00:00:00
Tasks: regression, Vocal Bursts Intensity Prediction

The problems of outliers detection and robust regression in a
high-dimensional setting are fundamental in statistics, and have numerous
applications. Following a recent set of works providing methods for
simultaneous robust regression and outliers detection, we consider in this
paper a model of linear regression with individual intercepts, in a
high-dimensional setting. We introduce a new procedure for simultaneous
estimation of the linear regression coefficients and intercepts, using two
dedicated sorted-$\ell_1$ penalizations, also called SLOPE. We develop a
complete theory for this problem: first, we provide sharp upper bounds on the
statistical estimation error of both the vector of individual intercepts and
regression coefficients. Second, we give an asymptotic control on the False
Discovery Rate (FDR) and statistical power for support selection of the
individual intercepts. As a consequence, this paper is the first to introduce a
procedure with guaranteed FDR and statistical power control for outliers
detection under the mean-shift model. Numerical illustrations, with a
comparison to recent alternative approaches, are provided on both simulated and
several real-world datasets. Experiments are conducted using an open-source
software written in Python and C++.
A trans-disciplinary review of deep learning research for water resources scientists
Authors: Chaopeng Shen
Date: 2017-12-06 00:00:00

Deep learning (DL), a new-generation of artificial neural network research,
has transformed industries, daily lives and various scientific disciplines in
recent years. DL represents significant progress in the ability of neural
networks to automatically engineer problem-relevant features and capture highly
complex data distributions. I argue that DL can help address several major new
and old challenges facing research in water sciences such as
inter-disciplinarity, data discoverability, hydrologic scaling, equifinality,
and needs for parameter regionalization. This review paper is intended to
provide water resources scientists and hydrologists in particular with a simple
technical overview, trans-disciplinary progress update, and a source of
inspiration about the relevance of DL to water. The review reveals that various
physical and geoscientific disciplines have utilized DL to address data
challenges, improve efficiency, and gain scientific insights. DL is especially
suited for information extraction from image-like data and sequential data.
Techniques and experiences presented in other disciplines are of high relevance
to water research. Meanwhile, less noticed is that DL may also serve as a
scientific exploratory tool. A new area termed 'AI neuroscience,' where
scientists interpret the decision process of deep networks and derive insights,
has been born. This budding sub-discipline has demonstrated methods including
correlation-based analysis, inversion of network-extracted features,
reduced-order approximations by interpretable models, and attribution of
network decisions to inputs. Moreover, DL can also use data to condition
neurons that mimic problem-specific fundamental organizing units, thus
revealing emergent behaviors of these units. Vast opportunities exist for DL to
propel advances in water sciences.
Solving internal covariate shift in deep learning with linked neurons
Authors: Carles Roger Riera Molina, Oriol Pujol Vila
Date: 2017-12-07 00:00:00
Tasks: Deep Learning

This work proposes a novel solution to the problem of internal covariate
shift and dying neurons using the concept of linked neurons. We define the
neuron linkage in terms of two constraints: first, all neuron activations in
the linkage must have the same operating point. That is to say, all of them
share input weights. Secondly, a set of neurons is linked if and only if there
is at least one member of the linkage that has a non-zero gradient in regard to
the input of the activation function. This means that for any input in the
activation function, there is at least one member of the linkage that operates
in a non-flat and non-zero area. This simple change has profound implications
in the network learning dynamics. In this article we explore the consequences
of this proposal and show that by using this kind of units, internal covariate
shift is implicitly solved. As a result of this, the use of linked neurons
allows to train arbitrarily large networks without any architectural or
algorithmic trick, effectively removing the need of using re-normalization
schemes such as Batch Normalization, which leads to halving the required
training time. It also solves the problem of the need for standarized input
data. Results show that the units using the linkage not only do effectively
solve the aforementioned problems, but are also a competitive alternative with
respect to state-of-the-art with very promising results.