pwc-archive/papers-with-abstracts huggingface dataset:
HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array
Authors: Linghao Song, Jiachen Mao, Youwei Zhuo, Xuehai Qian, Hai Li, Yiran Chen
Date: 2019-01-07 00:00:00
Tasks: Deep Learning

With the rise of artificial intelligence in recent years, Deep Neural Networks (DNNs) have been widely used in many domains. To achieve high performance and energy efficiency, hardware acceleration (especially inference) of DNNs is intensively studied both in academia and industry. However, we still face two challenges: large DNN models and datasets, which incur frequent off-chip memory accesses; and the training of DNNs, which is not well-explored in recent accelerator designs. To truly provide high throughput and energy efficient acceleration for the training of deep and large models, we inevitably need to use multiple accelerators to explore the coarse-grain parallelism, compared to the fine-grain parallelism inside a layer considered in most of the existing architectures. It poses the key research question to seek the best organization of computation and dataflow among accelerators. In this paper, inspired by recent work in machine learning systems, we propose a solution HyPar to determine layer-wise parallelism for deep neural network training with an array of DNN accelerators. HyPar partitions the feature map tensors (input and output), the kernel tensors, the gradient tensors, and the error tensors for the DNN accelerators. A partition constitutes the choice of parallelism for weighted layers. The optimization target is to search a partition that minimizes the total communication during training a complete DNN. To solve this problem, we propose a communication model to explain the source and amount of communications. Then, we use a hierarchical layer-wise dynamic programming method to search for the partition for each layer.
Multi-Source Transfer Learning for Non-Stationary Environments
Authors: Honghui Du, Leandro L. Minku, Huiyu Zhou
Date: 2019-01-07 00:00:00
Tasks: Transfer Learning

In data stream mining, predictive models typically suffer drops in predictive
performance due to concept drift. As enough data representing the new concept
must be collected for the new concept to be well learnt, the predictive
performance of existing models usually takes some time to recover from concept
drift. To speed up recovery from concept drift and improve predictive
performance in data stream mining, this work proposes a novel approach called
Multi-sourcE onLine TrAnsfer learning for Non-statIonary Environments
(Melanie). Melanie is the first approach able to transfer knowledge between
multiple data streaming sources in non-stationary environments. It creates
several sub-classifiers to learn different aspects from different source and
target concepts over time. The sub-classifiers that match the current target
concept well are identified, and used to compose an ensemble for predicting
examples from the target concept. We evaluate Melanie on several synthetic data
streams containing different types of concept drift and on real world data
streams. The results indicate that Melanie can deal with a variety drifts and
improve predictive performance over existing data stream learning algorithms by
making use of multiple sources.
DPPNet: Approximating Determinantal Point Processes with Deep Networks
Authors: Zelda Mariet, Yaniv Ovadia, Jasper Snoek
Date: 2019-01-07 00:00:00
Proceeding: ICLR 2019 5
Tasks: Diversity, Point Processes

Determinantal Point Processes (DPPs) provide an elegant and versatile way to
sample sets of items that balance the point-wise quality with the set-wise
diversity of selected items. For this reason, they have gained prominence in
many machine learning applications that rely on subset selection. However,
sampling from a DPP over a ground set of size $N$ is a costly operation,
requiring in general an $O(N^3)$ preprocessing cost and an $O(Nk^3)$ sampling
cost for subsets of size $k$. We approach this problem by introducing DPPNets:
generative deep models that produce DPP-like samples for arbitrary ground sets.
We develop an inhibitive attention mechanism based on transformer networks that
captures a notion of dissimilarity between feature vectors. We show
theoretically that such an approximation is sensible as it maintains the
guarantees of inhibition or dissimilarity that makes DPPs so powerful and
unique. Empirically, we demonstrate that samples from our model receive high
likelihood under the more expensive DPP alternative.
Sinusoidal wave generating network based on adversarial learning and its application: synthesizing frog sounds for data augmentation
Authors: Sangwook Park, David K. Han, Hanseok Ko
Date: 2019-01-07 00:00:00
Tasks: Data Augmentation, Sound Classification

Simulators that generate observations based on theoretical models can be
important tools for development, prediction, and assessment of signal
processing algorithms. In order to design these simulators, painstaking effort
is required to construct mathematical models according to their application.
Complex models are sometimes necessary to represent a variety of real
phenomena. In contrast, obtaining synthetic observations from generative models
developed from real observations often require much less effort. This paper
proposes a generative model based on adversarial learning. Given that
observations are typically signals composed of a linear combination of
sinusoidal waves and random noises, sinusoidal wave generating networks are
first designed based on an adversarial network. Audio waveform generation can
then be performed using the proposed network. Several approaches to designing
the objective function of the proposed network using adversarial learning are
investigated experimentally. In addition, amphibian sound classification is
performed using a convolutional neural network trained with real and synthetic
sounds. Both qualitative and quantitative results show that the proposed
generative model makes realistic signals and is very helpful for data
augmentation and data analysis.
A New Perspective on Machine Learning: How to do Perfect Supervised Learning
Authors: Hui Jiang
Date: 2019-01-07 00:00:00
Tasks: BIG-bench Machine Learning, Generalization Bounds

In this work, we introduce the concept of bandlimiting into the theory of
machine learning because all physical processes are bandlimited by nature,
including real-world machine learning tasks. After the bandlimiting constraint
is taken into account, our theoretical analysis has shown that all practical
machine learning tasks are asymptotically solvable in a perfect sense.
Furthermore, the key towards this solvability almost solely relies on two
factors: i) a sufficiently large amount of training samples beyond a threshold
determined by a difficulty measurement of the underlying task; ii) a
sufficiently complex and bandlimited model. Moreover, for some special cases,
we have derived new error bounds for perfect learning, which can quantify the
difficulty of learning. These generalization bounds are not only asymptotically
convergent but also irrelevant to model complexity. Our new results on
generalization have provided a new perspective to explain the recent successes
of large-scale supervised learning using complex models like neural networks.
Semi-parametric dynamic contextual pricing
Authors: Virag Shah, Jose Blanchet, Ramesh Johari
Date: 2019-01-07 00:00:00
Proceeding: NeurIPS 2019 12

Motivated by the application of real-time pricing in e-commerce platforms, we consider the problem of revenue-maximization in a setting where the seller can leverage contextual information describing the customer's history and the product's type to predict her valuation of the product. However, her true valuation is unobservable to the seller, only binary outcome in the form of success-failure of a transaction is observed. Unlike in usual contextual bandit settings, the optimal price/arm given a covariate in our setting is sensitive to the detailed characteristics of the residual uncertainty distribution. We develop a semi-parametric model in which the residual distribution is non-parametric and provide the first algorithm which learns both regression parameters and residual distribution with $\tilde O(\sqrt{n})$ regret. We empirically test a scalable implementation of our algorithm and observe good performance.
Reproducibility Evaluation of SLANT Whole Brain Segmentation Across Clinical Magnetic Resonance Imaging Protocols
Authors: Yunxi Xiong, Yuankai Huo, Jiachen Wang, L. Taylor Davis, Maureen McHugo, Bennett A. Landman
Date: 2019-01-07 00:00:00
Tasks: Brain Segmentation, Computational Efficiency, Segmentation

Whole brain segmentation on structural magnetic resonance imaging (MRI) is
essential for understanding neuroanatomical-functional relationships.
Traditionally, multi-atlas segmentation has been regarded as the standard
method for whole brain segmentation. In past few years, deep convolutional
neural network (DCNN) segmentation methods have demonstrated their advantages
in both accuracy and computational efficiency. Recently, we proposed the
spatially localized atlas network tiles (SLANT) method, which is able to
segment a 3D MRI brain scan into 132 anatomical regions. Commonly, DCNN
segmentation methods yield inferior performance under external validations,
especially when the testing patterns were not presented in the training
cohorts. Recently, we obtained a clinically acquired, multi-sequence MRI brain
cohort with 1480 clinically acquired, de-identified brain MRI scans on 395
patients using seven different MRI protocols. Moreover, each subject has at
least two scans from different MRI protocols. Herein, we assess the SLANT
method's intra- and inter-protocol reproducibility. SLANT achieved less than
0.05 coefficient of variation (CV) for intra-protocol experiments and less than
0.15 CV for inter-protocol experiments. The results show that the SLANT method
achieved high intra- and inter- protocol reproducibility.
Spherical CNNs on Unstructured Grids
Authors: Chiyu "Max" Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, Matthias Niessner
Date: 2019-01-07 00:00:00
Proceeding: ICLR 2019 5
Tasks: Semantic Segmentation

We present an efficient convolution kernel for Convolutional Neural Networks
(CNNs) on unstructured grids using parameterized differential operators while
focusing on spherical signals such as panorama images or planetary signals. To
this end, we replace conventional convolution kernels with linear combinations
of differential operators that are weighted by learnable parameters.
Differential operators can be efficiently estimated on unstructured grids using
one-ring neighbors, and learnable parameters can be optimized through standard
back-propagation. As a result, we obtain extremely efficient neural networks
that match or outperform state-of-the-art network architectures in terms of
performance but with a significantly lower number of network parameters. We
evaluate our algorithm in an extensive series of experiments on a variety of
computer vision and climate science tasks, including shape classification,
climate pattern segmentation, and omnidirectional image semantic segmentation.
Overall, we present (1) a novel CNN approach on unstructured grids using
parameterized differential operators for spherical signals, and (2) we show
that our unique kernel parameterization allows our model to achieve the same or
higher accuracy with significantly fewer network parameters.
Towards a Decentralized, Autonomous Multiagent Framework for Mitigating Crop Loss
Authors: Roi Ceren, Shannon Quinn, Glen Raines
Date: 2019-01-07 00:00:00
Tasks: Clustering, Reinforcement Learning

We propose a generalized decision-theoretic system for a heterogeneous team
of autonomous agents who are tasked with online identification of
phenotypically expressed stress in crop fields.. This system employs four
distinct types of agents, specific to four available sensor modalities:
satellites (Layer 3), uninhabited aerial vehicles (L2), uninhabited ground
vehicles (L1), and static ground-level sensors (L0). Layers 3, 2, and 1 are
tasked with performing image processing at the available resolution of the
sensor modality and, along with data generated by layer 0 sensors, identify
erroneous differences that arise over time. Our goal is to limit the use of the
more computationally and temporally expensive subsequent layers. Therefore,
from layer 3 to 1, each layer only investigates areas that previous layers have
identified as potentially afflicted by stress. We introduce a reinforcement
learning technique based on Perkins' Monte Carlo Exploring Starts for a
generalized Markovian model for each layer's decision problem, and label the
system the Agricultural Distributed Decision Framework (ADDF). As our domain is
real-world and online, we illustrate implementations of the two major
components of our system: a clustering-based image processing methodology and a
two-layer POMDP implementation.
Self-Supervised Learning from Web Data for Multimodal Retrieval
Authors: Raul Gomez, Lluis Gomez, Jaume Gibert, Dimosthenis Karatzas
Date: 2019-01-07 00:00:00
Tasks: Image Retrieval, Retrieval, Self-Supervised Learning

Self-Supervised learning from multimodal image and text data allows deep
neural networks to learn powerful features with no need of human annotated
data. Web and Social Media platforms provide a virtually unlimited amount of
this multimodal data. In this work we propose to exploit this free available
data to learn a multimodal image and text embedding, aiming to leverage the
semantic knowledge learnt in the text domain and transfer it to a visual model
for semantic image retrieval. We demonstrate that the proposed pipeline can
learn from images with associated textwithout supervision and analyze the
semantic structure of the learnt joint image and text embedding space. We
perform a thorough analysis and performance comparison of five different state
of the art text embeddings in three different benchmarks. We show that the
embeddings learnt with Web and Social Media data have competitive performances
over supervised methods in the text based image retrieval task, and we clearly
outperform state of the art in the MIRFlickr dataset when training in the
target data. Further, we demonstrate how semantic multimodal image retrieval
can be performed using the learnt embeddings, going beyond classical
instance-level retrieval problems. Finally, we present a new dataset,
InstaCities1M, composed by Instagram images and their associated texts that can
be used for fair comparison of image-text embeddings.
Learning the optimal state-feedback via supervised imitation learning
Authors: Dharmesh Tailor, Dario Izzo
Date: 2019-01-07 00:00:00
Tasks: Imitation Learning

Imitation learning is a control design paradigm that seeks to learn a control
policy reproducing demonstrations from expert agents. By substituting expert
demonstrations for optimal behaviours, the same paradigm leads to the design of
control policies closely approximating the optimal state-feedback. This
approach requires training a machine learning algorithm (in our case deep
neural networks) directly on state-control pairs originating from optimal
trajectories. We have shown in previous work that, when restricted to
low-dimensional state and control spaces, this approach is very successful in
several deterministic, non-linear problems in continuous-time. In this work, we
refine our previous studies using as a test case a simple quadcopter model with
quadratic and time-optimal objective functions. We describe in detail the best
learning pipeline we have developed, that is able to approximate via deep
neural networks the state-feedback map to a very high accuracy. We introduce
the use of the softplus activation function in the hidden units of neural
networks showing that it results in a smoother control profile whilst retaining
the benefits of rectifiers. We show how to evaluate the optimality of the
trained state-feedback, and find that already with two layers the objective
function reached and its optimal value differ by less than one percent. We
later consider also an additional metric linked to the system asymptotic
behaviour - time taken to converge to the policy's fixed point. With respect to
these metrics, we show that improvements in the mean absolute error do not
necessarily correspond to better policies.
Analogy-Based Preference Learning with Kernels
Authors: Mohsen Ahmadi Fahandar, Eyke Hüllermeier
Date: 2019-01-07 00:00:00

Building on a specific formalization of analogical relationships of the form
"A relates to B as C relates to D", we establish a connection between two
important subfields of artificial intelligence, namely analogical reasoning and
kernel-based machine learning. More specifically, we show that so-called
analogical proportions are closely connected to kernel functions on pairs of
objects. Based on this result, we introduce the analogy kernel, which can be
seen as a measure of how strongly four objects are in analogical relationship.
As an application, we consider the problem of object ranking in the realm of
preference learning, for which we develop a new method based on support vector
machines trained with the analogy kernel. Our first experimental results for
data sets from different domains (sports, education, tourism, etc.) are
promising and suggest that our approach is competitive to state-of-the-art
algorithms in terms of predictive accuracy.
Forecasting People Trajectories and Head Poses by Jointly Reasoning on Tracklets and Vislets
Authors: Irtiza Hasan, Francesco Setti, Theodore Tsesmelis, Vasileios Belagiannis, Sikandar Amin, Alessio Del Bue, Marco Cristani, Fabio Galasso
Date: 2019-01-07 00:00:00
Tasks: Trajectory Forecasting

In this work, we explore the correlation between people trajectories and their head orientations. We argue that people trajectory and head pose forecasting can be modelled as a joint problem. Recent approaches on trajectory forecasting leverage short-term trajectories (aka tracklets) of pedestrians to predict their future paths. In addition, sociological cues, such as expected destination or pedestrian interaction, are often combined with tracklets. In this paper, we propose MiXing-LSTM (MX-LSTM) to capture the interplay between positions and head orientations (vislets) thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. We additionally exploit the head orientations as a proxy for the visual attention, when modeling social interactions. MX-LSTM predicts future pedestrians location and head pose, increasing the standard capabilities of the current approaches on long-term trajectory forecasting. Compared to the state-of-the-art, our approach shows better performances on an extensive set of public benchmarks. MX-LSTM is particularly effective when people move slowly, i.e. the most challenging scenario for all other models. The proposed approach also allows for accurate predictions on a longer time horizon.
Truncated nuclear norm regularization for low-rank tensor completion
Authors: Shengke Xue, Wenyuan Qiu, Fan Liu, Xinyu Jin
Date: 2019-01-07 00:00:00

Recently, low-rank tensor completion has become increasingly attractive in
recovering incomplete visual data. Considering a color image or video as a
three-dimensional (3D) tensor, existing studies have put forward several
definitions of tensor nuclear norm. However, they are limited and may not
accurately approximate the real rank of a tensor, and they do not explicitly
use the low-rank property in optimization. It is proved that the recently
proposed truncated nuclear norm (TNN) can replace the traditional nuclear norm,
as an improved approximation to the rank of a matrix. In this paper, we propose
a new method called the tensor truncated nuclear norm (T-TNN), which suggests a
new definition of tensor nuclear norm. The truncated nuclear norm is
generalized from the matrix case to the tensor case. With the help of the low
rankness of TNN, our approach improves the efficacy of tensor completion. We
adopt the definition of the previously proposed tensor singular value
decomposition, the alternating direction method of multipliers, and the
accelerated proximal gradient line search method in our algorithm. Substantial
experiments on real-world videos and images illustrate that the performance of
our approach is better than those of previous methods.
Compressive-Sensing Data Reconstruction for Structural Health Monitoring: A Machine-Learning Approach
Authors: Yuequan Bao, Zhiyi Tang, Hui Li
Date: 2019-01-07 00:00:00
Tasks: BIG-bench Machine Learning, Compressive Sensing, Structural Health Monitoring

Compressive sensing (CS) has been studied and applied in structural health
monitoring for wireless data acquisition and transmission, structural modal
identification, and spare damage identification. The key issue in CS is finding
the optimal solution for sparse optimization. In the past years, many
algorithms have been proposed in the field of applied mathematics. In this
paper, we propose a machine-learning-based approach to solve the CS
data-reconstruction problem. By treating a computation process as a data flow,
the process of CS-based data reconstruction is formalized into a standard
supervised-learning task. The prior knowledge, i.e., the basis matrix and the
CS-sampled signals, are used as the input and the target of the network; the
basis coefficient matrix is embedded as the parameters of a certain layer; the
objective function of conventional compressive sensing is set as the loss
function of the network. Regularized by l1-norm, these basis coefficients are
optimized to reduce the error between the original CS-sampled signals and the
masked reconstructed signals with a common optimization algorithm. Also, the
proposed network can handle complex bases, such as a Fourier basis. Benefiting
from the nature of a multi-neuron layer, multiple signal channels can be
reconstructed simultaneously. Meanwhile, the disassembled use of a large-scale
basis makes the method memory-efficient. A numerical example of multiple
sinusoidal waves and an example of field-test wireless data from a suspension
bridge are carried out to illustrate the data-reconstruction ability of the
proposed approach. The results show that high reconstruction accuracy can be
obtained by the machine learning-based approach. Also, the parameters of the
network have clear meanings; the inference of the mapping between input and
output is fully transparent, making the CS data reconstruction neural network
interpretable.
Recurrent Control Nets for Deep Reinforcement Learning
Authors: Vincent Liu, Ademi Adeniji, Nathaniel Lee, Jason Zhao, Mario Srouji
Date: 2019-01-06 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Central Pattern Generators (CPGs) are biological neural circuits capable of
producing coordinated rhythmic outputs in the absence of rhythmic input. As a
result, they are responsible for most rhythmic motion in living organisms. This
rhythmic control is broadly applicable to fields such as locomotive robotics
and medical devices. In this paper, we explore the possibility of creating a
self-sustaining CPG network for reinforcement learning that learns rhythmic
motion more efficiently and across more general environments than the current
multilayer perceptron (MLP) baseline models. Recent work introduces the
Structured Control Net (SCN), which maintains linear and nonlinear modules for
local and global control, respectively. Here, we show that time-sequence
architectures such as Recurrent Neural Networks (RNNs) model CPGs effectively.
Combining previous work with RNNs and SCNs, we introduce the Recurrent Control
Net (RCN), which adds a linear component to the, RCNs match and exceed the
performance of baseline MLPs and SCNs across all environment tasks. Our
findings confirm existing intuitions for RNNs on reinforcement learning tasks,
and demonstrate promise of SCN-like structures in reinforcement learning.