pwc-archive/papers-with-abstracts huggingface dataset:
Parallel training of linear models without compromising convergence
Authors: Nikolas Ioannou, Celestine Dünner, Kornilios Kourtis, Thomas Parnell
Date: 2018-11-05 00:00:00
Tasks: CPU

In this paper we analyze, evaluate, and improve the performance of training
generalized linear models on modern CPUs. We start with a state-of-the-art
asynchronous parallel training algorithm, identify system-level performance
bottlenecks, and apply optimizations that improve data parallelism, cache line
locality, and cache line prefetching of the algorithm. These modifications
reduce the per-epoch run-time significantly, but take a toll on algorithm
convergence in terms of the required number of epochs. To alleviate these
shortcomings of our systems-optimized version, we propose a novel, dynamic data
partitioning scheme across threads which allows us to approach the convergence
of the sequential version. The combined set of optimizations result in a
consistent bottom line speedup in convergence of up to 12x compared to the
initial asynchronous parallel training algorithm and up to 42x, compared to
state of the art implementations (scikit-learn and h2o) on a range of
multi-core CPU architectures.
Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
Authors: Qianxiao Li, Cheng Tai, Weinan E
Date: 2018-11-05 00:00:00

We develop the mathematical foundations of the stochastic modified equations
(SME) framework for analyzing the dynamics of stochastic gradient algorithms,
where the latter is approximated by a class of stochastic differential
equations with small noise parameters. We prove that this approximation can be
understood mathematically as an weak approximation, which leads to a number of
precise and useful results on the approximations of stochastic gradient descent
(SGD), momentum SGD and stochastic Nesterov's accelerated gradient method in
the general setting of stochastic objectives. We also demonstrate through
explicit calculations that this continuous-time approach can uncover important
analytical insights into the stochastic gradient algorithms under consideration
that may not be easy to obtain in a purely discrete-time setting.
Representation Learning by Reconstructing Neighborhoods
Authors: Chin-Chia Michael Yeh, Yan Zhu, Evangelos E. Papalexakis, Abdullah Mueen, Eamonn Keogh
Date: 2018-11-05 00:00:00
Tasks: Clustering, Decoder, Dimensionality Reduction, Information Retrieval, Representation Learning, Retrieval, Time Series Analysis

Since its introduction, unsupervised representation learning has attracted a
lot of attention from the research community, as it is demonstrated to be
highly effective and easy-to-apply in tasks such as dimension reduction,
clustering, visualization, information retrieval, and semi-supervised learning.
In this work, we propose a novel unsupervised representation learning framework
called neighbor-encoder, in which domain knowledge can be easily incorporated
into the learning process without modifying the general encoder-decoder
architecture of the classic autoencoder.In contrast to autoencoder, which
reconstructs the input data itself, neighbor-encoder reconstructs the input
data's neighbors. As the proposed representation learning problem is
essentially a neighbor reconstruction problem, domain knowledge can be easily
incorporated in the form of an appropriate definition of similarity between
objects. Based on that observation, our framework can leverage any
off-the-shelf similarity search algorithms or side information to find the
neighbor of an input object. Applications of other algorithms (e.g.,
association rule mining) in our framework are also possible, given that the
appropriate definition of neighbor can vary in different contexts. We have
demonstrated the effectiveness of our framework in many diverse domains,
including images, text, and time series, and for various data mining tasks
including classification, clustering, and visualization. Experimental results
show that neighbor-encoder not only outperforms autoencoder in most of the
scenarios we consider, but also achieves the state-of-the-art performance on
text document clustering.
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition
Authors: Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Li-Min Wang, Shilei Wen
Date: 2018-11-05 00:00:00
Tasks: Action Recognition, Temporal Action Localization

Despite the success of deep learning for static image understanding, it
remains unclear what are the most effective network architectures for the
spatial-temporal modeling in videos. In this paper, in contrast to the existing
CNN+RNN or pure 3D convolution based approaches, we explore a novel spatial
temporal network (StNet) architecture for both local and global
spatial-temporal modeling in videos. Particularly, StNet stacks N successive
video frames into a \emph{super-image} which has 3N channels and applies 2D
convolution on super-images to capture local spatial-temporal relationship. To
model global spatial-temporal relationship, we apply temporal convolution on
the local spatial-temporal feature maps. Specifically, a novel temporal
Xception block is proposed in StNet. It employs a separate channel-wise and
temporal-wise convolution over the feature sequence of video. Extensive
experiments on the Kinetics dataset demonstrate that our framework outperforms
several state-of-the-art approaches in action recognition and can strike a
satisfying trade-off between recognition accuracy and model complexity. We
further demonstrate the generalization performance of the leaned video
representations on the UCF101 dataset.
PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks
Authors: P. Guo, K. Wang, X. L. Zhou
Date: 2018-11-05 00:00:00
Tasks: Decoder

In this work, a non-gradient descent learning (NGDL) scheme was proposed for deep feedforward neural networks (DNN). It is known that an autoencoder can be used as the building blocks of the multi-layer perceptron (MLP) DNN, the MLP is taken as an example to illustrate the proposed scheme of pseudoinverse learning algorithm for autoencoder (PILAE) in this paper. The PILAE with low rank approximation is a NGDL algorithm, and the encoder weight matrix is set to be the low rank approximation of the pseudoinverse of the input matrix, while the decoder weight matrix is calculated by the pseudoinverse learning algorithm. It is worth to note that only very few network structure hyper-parameters need to be tuned compared with classical gradient descent learning algorithm. Hence, the proposed algorithm could be regarded as a quasi-automated training algorithm which could be utilized in automated machine learning field. The experimental results show that the proposed learning scheme for DNN could achieve better performance on considering the tradeoff between training efficiency and classification accuracy.
Redefining Ultrasound Compounding: Computational Sonography
Authors: Rüdiger Göbl, Diana Mateus, Christoph Hennersperger, Maximilian Baust, Nassir Navab
Date: 2018-11-05 00:00:00

Freehand three-dimensional ultrasound (3D-US) has gained considerable
interest in research, but even today suffers from its high inter-operator
variability in clinical practice. The high variability mainly arises from
tracking inaccuracies as well as the directionality of the ultrasound data,
being neglected in most of today's reconstruction methods. By providing a novel
paradigm for the acquisition and reconstruction of tracked freehand 3D
ultrasound, this work presents the concept of Computational Sonography (CS) to
model the directionality of ultrasound information. CS preserves the
directionality of the acquired data, and allows for its exploitation by
computational algorithms. In this regard, we propose a set of mathematical
models to represent 3D-US data, inspired by the physics of ultrasound imaging.
We compare different models of Computational Sonography to classical scalar
compounding for freehand acquisitions, providing both an improved preservation
of US directionality as well as improved image quality in 3D. The novel concept
is evaluated for a set of phantom datasets, as well as for in-vivo acquisitions
of muscoloskeletal and vascular applications.
Transfer learning for time series classification
Authors: Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, Pierre-Alain Muller
Date: 2018-11-05 00:00:00
Tasks: Classification, Dynamic Time Warping, General Classification, Object Localization, Time Series, Time Series Analysis, Time Series Classification, Transfer Learning

Transfer learning for deep neural networks is the process of first training a
base network on a source dataset, and then transferring the learned features
(the network's weights) to a second network to be trained on a target dataset.
This idea has been shown to improve deep neural network's generalization
capabilities in many computer vision tasks such as image recognition and object
localization. Apart from these applications, deep Convolutional Neural Networks
(CNNs) have also recently gained popularity in the Time Series Classification
(TSC) community. However, unlike for image recognition problems, transfer
learning techniques have not yet been investigated thoroughly for the TSC task.
This is surprising as the accuracy of deep learning models for TSC could
potentially be improved if the model is fine-tuned from a pre-trained neural
network instead of training it from scratch. In this paper, we fill this gap by
investigating how to transfer deep CNNs for the TSC task. To evaluate the
potential of transfer learning, we performed extensive experiments using the
UCR archive which is the largest publicly available TSC benchmark containing 85
datasets. For each dataset in the archive, we pre-trained a model and then
fine-tuned it on the other datasets resulting in 7140 different deep neural
networks. These experiments revealed that transfer learning can improve or
degrade the model's predictions depending on the dataset used for transfer.
Therefore, in an effort to predict the best source dataset for a given target
dataset, we propose a new method relying on Dynamic Time Warping to measure
inter-datasets similarities. We describe how our method can guide the transfer
to choose the best source dataset leading to an improvement in accuracy on 71
out of 85 datasets.
Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures using Spatial Information
Authors: Efthymios Tzinis, Shrikant Venkataramani, Paris Smaragdis
Date: 2018-11-05 00:00:00
Tasks: Clustering, Deep Clustering, Multi-Speaker Source Separation, Speech Separation

We present a monophonic source separation system that is trained by only
observing mixtures with no ground truth separation information. We use a deep
clustering approach which trains on multi-channel mixtures and learns to
project spectrogram bins to source clusters that correlate with various spatial
features. We show that using such a training process we can obtain separation
performance that is as good as making use of ground truth separation
information. Once trained, this system is capable of performing sound
separation on monophonic inputs, despite having learned how to do so using
multi-channel recordings.
Unsupervised RGBD Video Object Segmentation Using GANs
Authors: Maryam Sultana, Arif Mahmood, Sajid Javed, Soon Ki Jung
Date: 2018-11-05 00:00:00
Tasks: Object, Segmentation, Semantic Segmentation, Video Object Segmentation, Video Semantic Segmentation

Video object segmentation is a fundamental step in many advanced vision
applications. Most existing algorithms are based on handcrafted features such
as HOG, super-pixel segmentation or texture-based techniques, while recently
deep features have been found to be more efficient. Existing algorithms observe
performance degradation in the presence of challenges such as illumination
variations, shadows, and color camouflage. To handle these challenges we
propose a fusion based moving object segmentation algorithm which exploits
color as well as depth information using GAN to achieve more accuracy. Our goal
is to segment moving objects in the presence of challenging background scenes,
in real environments. To address this problem, GAN is trained in an
unsupervised manner on color and depth information independently with
challenging video sequences. During testing, the trained GAN generates
backgrounds similar to that in the test sample. The generated background
samples are then compared with the test sample to segment moving objects. The
final result is computed by fusion of object boundaries in both modalities, RGB
and the depth. The comparison of our proposed algorithm with five
state-of-the-art methods on publicly available dataset has shown the strength
of our algorithm for moving object segmentation in videos in the presence of
challenging real scenarios.
Session-based Recommendation with Graph Neural Networks
Authors: Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, Tieniu Tan
Date: 2018-11-01 00:00:00
Tasks: Recommendation Systems, Session-Based Recommendations

The problem of session-based recommendation aims to predict user actions
based on anonymous sessions. Previous methods model a session as a sequence and
estimate user representations besides item representations to make
recommendations. Though achieved promising results, they are insufficient to
obtain accurate user vectors in sessions and neglect complex transitions of
items. To obtain accurate item embedding and take complex transitions of items
into account, we propose a novel method, i.e. Session-based Recommendation with
Graph Neural Networks, SR-GNN for brevity. In the proposed method, session
sequences are modeled as graph-structured data. Based on the session graph, GNN
can capture complex transitions of items, which are difficult to be revealed by
previous conventional sequential methods. Each session is then represented as
the composition of the global preference and the current interest of that
session using an attention network. Extensive experiments conducted on two real
datasets show that SR-GNN evidently outperforms the state-of-the-art
session-based recommendation methods consistently.
Theoretical and Experimental Analysis on the Generalizability of Distribution Regression Network
Authors: Connie Kou, Hwee Kuan Lee, Jorge Sanz, Teck Khim Ng
Date: 2018-11-05 00:00:00
Tasks: regression, Time Series Analysis

There is emerging interest in performing regression between distributions. In contrast to prediction on single instances, these machine learning methods can be useful for population-based studies or on problems that are inherently statistical in nature. The recently proposed distribution regression network (DRN) has shown superior performance for the distribution-to-distribution regression task compared to conventional neural networks. However, in Kou et al. (2018) and some other works on distribution regression, there is a lack of comprehensive comparative study on both theoretical basis and generalization abilities of the methods. We derive some mathematical properties of DRN and qualitatively compare it to conventional neural networks. We also perform comprehensive experiments to study the generalizability of distribution regression models, by studying their robustness to limited training data, data sampling noise and task difficulty. DRN consistently outperforms conventional neural networks, requiring fewer training data and maintaining robust performance with noise. Furthermore, the theoretical properties of DRN can be used to provide some explanation on the ability of DRN to achieve better generalization performance than conventional neural networks.
Deep Multiple Description Coding by Learning Scalar Quantization
Authors: Lijun Zhao, Huihui Bai, Anhong Wang, Yao Zhao
Date: 2018-11-05 00:00:00
Tasks: Decoder, Quantization

In this paper, we propose a deep multiple description coding framework, whose
quantizers are adaptively learned via the minimization of multiple description
compressive loss. Firstly, our framework is built upon auto-encoder networks,
which have multiple description multi-scale dilated encoder network and
multiple description decoder networks. Secondly, two entropy estimation
networks are learned to estimate the informative amounts of the quantized
tensors, which can further supervise the learning of multiple description
encoder network to represent the input image delicately. Thirdly, a pair of
scalar quantizers accompanied by two importance-indicator maps is automatically
learned in an end-to-end self-supervised way. Finally, multiple description
structural dissimilarity distance loss is imposed on multiple description
decoded images in pixel domain for diversified multiple description generations
rather than on feature tensors in feature domain, in addition to multiple
description reconstruction loss. Through testing on two commonly used datasets,
it is verified that our method is beyond several state-of-the-art multiple
description coding approaches in terms of coding efficiency.
Lifted Proximal Operator Machines
Authors: Jia Li, Cong Fang, Zhouchen Lin
Date: 2018-11-05 00:00:00

We propose a new optimization method for training feed-forward neural
networks. By rewriting the activation function as an equivalent proximal
operator, we approximate a feed-forward neural network by adding the proximal
operators to the objective function as penalties, hence we call the lifted
proximal operator machine (LPOM). LPOM is block multi-convex in all layer-wise
weights and activations. This allows us to use block coordinate descent to
update the layer-wise weights and activations in parallel. Most notably, we
only use the mapping of the activation function itself, rather than its
derivatives, thus avoiding the gradient vanishing or blow-up issues in gradient
based training methods. So our method is applicable to various non-decreasing
Lipschitz continuous activation functions, which can be saturating and
non-differentiable. LPOM does not require more auxiliary variables than the
layer-wise activations, thus using roughly the same amount of memory as
stochastic gradient descent (SGD) does. We further prove the convergence of
updating the layer-wise weights and activations. Experiments on MNIST and
CIFAR-10 datasets testify to the advantages of LPOM.
Underwater Fish Detection using Deep Learning for Water Power Applications
Authors: Wenwei Xu, Shari Matzner
Date: 2018-11-05 00:00:00
Tasks: Deep Learning, Fish Detection

Clean energy from oceans and rivers is becoming a reality with the
development of new technologies like tidal and instream turbines that generate
electricity from naturally flowing water. These new technologies are being
monitored for effects on fish and other wildlife using underwater video.
Methods for automated analysis of underwater video are needed to lower the
costs of analysis and improve accuracy. A deep learning model, YOLO, was
trained to recognize fish in underwater video using three very different
datasets recorded at real-world water power sites. Training and testing with
examples from all three datasets resulted in a mean average precision (mAP)
score of 0.5392. To test how well a model could generalize to new datasets, the
model was trained using examples from only two of the datasets and then tested
on examples from all three datasets. The resulting model could not recognize
fish in the dataset that was not part of the training set. The mAP scores on
the other two datasets that were included in the training set were higher than
the scores achieved by the model trained on all three datasets. These results
indicate that different methods are needed in order to produce a trained model
that can generalize to new data sets such as those encountered in real world
applications.
Contingency-Aware Exploration in Reinforcement Learning
Authors: Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, Honglak Lee
Date: 2018-11-05 00:00:00
Proceeding: ICLR 2019 5
Tasks: Atari Games, Montezuma's Revenge, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

This paper investigates whether learning contingency-awareness and
controllable aspects of an environment can lead to better exploration in
reinforcement learning. To investigate this question, we consider an
instantiation of this hypothesis evaluated on the Arcade Learning Element
(ALE). In this study, we develop an attentive dynamics model (ADM) that
discovers controllable elements of the observations, which are often associated
with the location of the character in Atari games. The ADM is trained in a
self-supervised fashion to predict the actions taken by the agent. The learned
contingency information is used as a part of the state representation for
exploration purposes. We demonstrate that combining actor-critic algorithm with
count-based exploration using our representation achieves impressive results on
a set of notoriously challenging Atari games due to sparse rewards. For
example, we report a state-of-the-art score of >11,000 points on Montezuma's
Revenge without using expert demonstrations, explicit high-level information
(e.g., RAM states), or supervisory data. Our experiments confirm that
contingency-awareness is indeed an extremely powerful concept for tackling
exploration problems in reinforcement learning and opens up interesting
research questions for further investigations.
FairMod - Making Predictive Models Discrimination Aware
Authors: Jixue Liu, Jiuyong Li, Lin Liu, Thuc Duy Le, Feiyue Ye, Gefei Li
Date: 2018-11-05 00:00:00
Tasks: General Classification

Predictive models such as decision trees and neural networks may produce
discrimination in their predictions. This paper proposes a method to
post-process the predictions of a predictive model to make the processed
predictions non-discriminatory. The method considers multiple protected
variables together. Multiple protected variables make the problem more
challenging than a simple protected variable. The method uses a well-cited
discrimination metric and adapts it to allow the specification of explanatory
variables, such as position, profession, education, that describe the contexts
of the applications. It models the post-processing of predictions problem as a
nonlinear optimization problem to find best adjustments to the predictions so
that the discrimination constraints of all protected variables are all met at
the same time. The proposed method is independent of classification methods. It
can handle the cases that existing methods cannot handle: satisfying multiple
protected attributes at the same time, allowing multiple explanatory
attributes, and being independent of classification model types. An evaluation
using four real world data sets shows that the proposed method is as
effectively as existing methods, in addition to its extra power.