pwc-archive/papers-with-abstracts huggingface dataset:
Toward Understanding the Impact of Staleness in Distributed Machine Learning
Authors: Wei Dai, Yi Zhou, Nanqing Dong, Hao Zhang, Eric P. Xing
Date: 2018-10-08 00:00:00
Proceeding: ICLR 2019 5
Tasks: BIG-bench Machine Learning, Diversity

Many distributed machine learning (ML) systems adopt the non-synchronous
execution in order to alleviate the network communication bottleneck, resulting
in stale parameters that do not reflect the latest updates. Despite much
development in large-scale ML, the effects of staleness on learning are
inconclusive as it is challenging to directly monitor or control staleness in
complex distributed environments. In this work, we study the convergence
behaviors of a wide array of ML models and algorithms under delayed updates.
Our extensive experiments reveal the rich diversity of the effects of staleness
on the convergence of ML algorithms and offer insights into seemingly
contradictory reports in the literature. The empirical findings also inspire a
new convergence analysis of stochastic gradient descent in non-convex
optimization under staleness, matching the best-known convergence rate of
O(1/\sqrt{T}).
Deep Diffeomorphic Normalizing Flows
Authors: Hadi Salman, Payman Yadollahpour, Tom Fletcher, Kayhan Batmanghelich
Date: 2018-10-08 00:00:00
Tasks: Density Estimation, Variational Inference

The Normalizing Flow (NF) models a general probability density by estimating
an invertible transformation applied on samples drawn from a known
distribution. We introduce a new type of NF, called Deep Diffeomorphic
Normalizing Flow (DDNF). A diffeomorphic flow is an invertible function where
both the function and its inverse are smooth. We construct the flow using an
ordinary differential equation (ODE) governed by a time-varying smooth vector
field. We use a neural network to parametrize the smooth vector field and a
recursive neural network (RNN) for approximating the solution of the ODE. Each
cell in the RNN is a residual network implementing one Euler integration step.
The architecture of our flow enables efficient likelihood evaluation,
straightforward flow inversion, and results in highly flexible density
estimation. An end-to-end trained DDNF achieves competitive results with
state-of-the-art methods on a suite of density estimation and variational
inference tasks. Finally, our method brings concepts from Riemannian geometry
that, we believe, can open a new research direction for neural density
estimation.
Triple Attention Mixed Link Network for Single Image Super Resolution
Authors: Xi Cheng, Xiang Li, Jian Yang
Date: 2018-10-08 00:00:00
Tasks: Image Super-Resolution, Super-Resolution

Single image super resolution is of great importance as a low-level computer
vision task. Recent approaches with deep convolutional neural networks have
achieved im-pressive performance. However, existing architectures have
limitations due to the less sophisticated structure along with less strong
representational power. In this work, to significantly enhance the feature
representation, we proposed Triple Attention mixed link Network (TAN) which
consists of 1) three different aspects (i.e., kernel, spatial and channel) of
attention mechanisms and 2) fu-sion of both powerful residual and dense
connections (i.e., mixed link). Specifically, the network with multi kernel
learns multi hierarchical representations under different receptive fields. The
output features are recalibrated by the effective kernel and channel attentions
and feed into next layer partly residual and partly dense, which filters the
information and enable the network to learn more powerful representations. The
features finally pass through the spatial attention in the reconstruction
network which generates a fusion of local and global information, let the
network restore more details and improves the quality of reconstructed images.
Thanks to the diverse feature recalibrations and the advanced information flow
topology, our proposed model is strong enough to per-form against the
state-of-the-art methods on the bench-mark evaluations.
Arc-support Line Segments Revisited: An Efficient and High-quality Ellipse Detection
Authors: Changsheng Lu, Siyu Xia, Ming Shao, Yun Fu
Date: 2018-10-08 00:00:00
Tasks: ARC, Clustering, valid

Over the years many ellipse detection algorithms spring up and are studied broadly, while the critical issue of detecting ellipses accurately and efficiently in real-world images remains a challenge. In this paper, we propose a valuable industry-oriented ellipse detector by arc-support line segments, which simultaneously reaches high detection accuracy and efficiency. To simplify the complicated curves in an image while retaining the general properties including convexity and polarity, the arc-support line segments are extracted, which grounds the successful detection of ellipses. The arc-support groups are formed by iteratively and robustly linking the arc-support line segments that latently belong to a common ellipse. Afterward, two complementary approaches, namely, locally selecting the arc-support group with higher saliency and globally searching all the valid paired groups, are adopted to fit the initial ellipses in a fast way. Then, the ellipse candidate set can be formulated by hierarchical clustering of 5D parameter space of initial ellipses. Finally, the salient ellipse candidates are selected and refined as detections subject to the stringent and effective verification. Extensive experiments on three public datasets are implemented and our method achieves the best F-measure scores compared to the state-of-the-art methods. The source code is available at https://github.com/AlanLuSun/High-quality-ellipse-detection.
Diagnosing Convolutional Neural Networks using their Spectral Response
Authors: Victor Stamatescu, Mark D. McDonnell
Date: 2018-10-08 00:00:00
Tasks: Diagnostic, General Classification, image-classification, Image Classification

Convolutional Neural Networks (CNNs) are a class of artificial neural
networks whose computational blocks use convolution, together with other linear
and non-linear operations, to perform classification or regression. This paper
explores the spectral response of CNNs and its potential use in diagnosing
problems with their training. We measure the gain of CNNs trained for image
classification on ImageNet and observe that the best models are also the most
sensitive to perturbations of their input. Further, we perform experiments on
MNIST and CIFAR-10 to find that the gain rises as the network learns and then
saturates as the network converges. Moreover, we find that strong gain
fluctuations can point to overfitting and learning problems caused by a poor
choice of learning rate. We argue that the gain of CNNs can act as a diagnostic
tool and potential replacement for the validation loss when hold-out validation
data are not available.
Task-Embedded Control Networks for Few-Shot Imitation Learning
Authors: Stephen James, Michael Bloesch, Andrew J. Davison
Date: 2018-10-08 00:00:00
Tasks: Few-Shot Imitation Learning, Few-Shot Learning, Imitation Learning, Meta-Learning, Metric Learning

Much like humans, robots should have the ability to leverage knowledge from
previously learned tasks in order to learn new tasks quickly in new and
unfamiliar environments. Despite this, most robot learning approaches have
focused on learning a single task, from scratch, with a limited notion of
generalisation, and no way of leveraging the knowledge to learn other tasks
more efficiently. One possible solution is meta-learning, but many of the
related approaches are limited in their ability to scale to a large number of
tasks and to learn further tasks without forgetting previously learned ones.
With this in mind, we introduce Task-Embedded Control Networks, which employ
ideas from metric learning in order to create a task embedding that can be used
by a robot to learn new tasks from one or more demonstrations. In the area of
visually-guided manipulation, we present simulation results in which we surpass
the performance of a state-of-the-art method when using only visual information
from each demonstration. Additionally, we demonstrate that our approach can
also be used in conjunction with domain randomisation to train our few-shot
learning ability in simulation and then deploy in the real world without any
additional training. Once deployed, the robot can learn new tasks from a single
real-world demonstration.
Exposition and Interpretation of the Topology of Neural Networks
Authors: Rickard Brüel Gabrielsson, Gunnar Carlsson
Date: 2018-10-08 00:00:00
Tasks: Topological Data Analysis

Convolutional neural networks (CNN's) are powerful and widely used tools. However, their interpretability is far from ideal. One such shortcoming is the difficulty of deducing a network's ability to generalize to unseen data. We use topological data analysis to show that the information encoded in the weights of a CNN can be organized in terms of a topological data model and demonstrate how such information can be interpreted and utilized. We show that the weights of convolutional layers at depths from 1 through 13 learn simple global structures. We also demonstrate the change of the simple structures over the course of training. In particular, we define and analyze the spaces of spatial filters of convolutional layers and show the recurrence, among all networks, depths, and during training, of a simple circle consisting of rotating edges, as well as a less recurring unanticipated complex circle that combines lines, edges, and non-linear patterns. We also demonstrate that topological structure correlates with a network's ability to generalize to unseen data and that topological information can be used to improve a network's performance. We train over a thousand CNN's on MNIST, CIFAR-10, SVHN, and ImageNet.
Towards Gradient Free and Projection Free Stochastic Optimization
Authors: Anit Kumar Sahu, Manzil Zaheer, Soummya Kar
Date: 2018-10-08 00:00:00
Tasks: Stochastic Optimization

This paper focuses on the problem of \emph{constrained} \emph{stochastic}
optimization. A zeroth order Frank-Wolfe algorithm is proposed, which in
addition to the projection-free nature of the vanilla Frank-Wolfe algorithm
makes it gradient free. Under convexity and smoothness assumption, we show that
the proposed algorithm converges to the optimal objective function at a rate
$O\left(1/T^{1/3}\right)$, where $T$ denotes the iteration count. In
particular, the primal sub-optimality gap is shown to have a dimension
dependence of $O\left(d^{1/3}\right)$, which is the best known dimension
dependence among all zeroth order optimization algorithms with one directional
derivative per iteration. For non-convex functions, we obtain the
\emph{Frank-Wolfe} gap to be $O\left(d^{1/3}T^{-1/4}\right)$. Experiments on
black-box optimization setups demonstrate the efficacy of the proposed
algorithm.
Rethinking Recurrent Latent Variable Model for Music Composition
Authors: Eunjeong Stella Koh, Shlomo Dubnov, Dustin Wright
Date: 2018-10-07 00:00:00
Tasks: Decoder, model

We present a model for capturing musical features and creating novel
sequences of music, called the Convolutional Variational Recurrent Neural
Network. To generate sequential data, the model uses an encoder-decoder
architecture with latent probabilistic connections to capture the hidden
structure of music. Using the sequence-to-sequence model, our generative model
can exploit samples from a prior distribution and generate a longer sequence of
music. We compare the performance of our proposed model with other types of
Neural Networks using the criteria of Information Rate that is implemented by
Variable Markov Oracle, a method that allows statistical characterization of
musical information dynamics and detection of motifs in a song. Our results
suggest that the proposed model has a better statistical resemblance to the
musical structure of the training data, which improves the creation of new
sequences of music in the style of the originals.
Recovering Quantized Data with Missing Information Using Bilinear Factorization and Augmented Lagrangian Method
Authors: Ashkan Esmaeili, Kayhan Behdin, Sina Al-E-Mohammad, Farokh Marvasti
Date: 2018-10-07 00:00:00

In this paper, we propose a novel approach in order to recover a quantized
matrix with missing information. We propose a regularized convex cost function
composed of a log-likelihood term and a Trace norm term. The Bi-factorization
approach and the Augmented Lagrangian Method (ALM) are applied to find the
global minimizer of the cost function in order to recover the genuine data. We
provide mathematical convergence analysis for our proposed algorithm. In the
Numerical Experiments Section, we show the superiority of our method in
accuracy and also its robustness in computational complexity compared to the
state-of-the-art literature methods.
Principled Deep Neural Network Training through Linear Programming
Authors: Daniel Bienstock, Gonzalo Muñoz, Sebastian Pokutta
Date: 2018-10-07 00:00:00
Proceeding: ICLR 2019 5

Deep learning has received much attention lately due to the impressive empirical performance achieved by training algorithms. Consequently, a need for a better theoretical understanding of these problems has become more evident in recent years. In this work, using a unified framework, we show that there exists a polyhedron which encodes simultaneously all possible deep neural network training problems that can arise from a given architecture, activation functions, loss function, and sample-size. Notably, the size of the polyhedral representation depends only linearly on the sample-size, and a better dependency on several other network parameters is unlikely (assuming $P\neq NP$). Additionally, we use our polyhedral representation to obtain new and better computational complexity results for training problems of well-known neural network architectures. Our results provide a new perspective on training problems through the lens of polyhedral theory and reveal a strong structure arising from these problems.
Image Completion on CIFAR-10
Authors: Mason Swofford
Date: 2018-10-07 00:00:00
Tasks: Decoder

This project performed image completion on CIFAR-10, a dataset of 60,000
32x32 RGB images, using three different neural network architectures: fully
convolutional networks, convolutional networks with fully connected layers, and
encoder-decoder convolutional networks. The highest performing model was a deep
fully convolutional network, which was able to achieve a mean squared error of
.015 when comparing the original image pixel values with the predicted pixel
values. As well, this network was able to output in-painted images which
appeared real to the human eye.
Pre-Synaptic Pool Modification (PSPM): A Supervised Learning Procedure for Spiking Neural Networks
Authors: Bryce Bagley, Blake Bordelon, Benjamin Moseley, Ralf Wessel
Date: 2018-10-07 00:00:00

Learning synaptic weights of spiking neural network (SNN) models that can reproduce target spike trains from provided neural firing data is a central problem in computational neuroscience and spike-based computing. The discovery of the optimal weight values can be posed as a supervised learning task wherein the weights of the model network are chosen to maximize the similarity between the target spike trains and the model outputs. It is still largely unknown whether optimizing spike train similarity of highly recurrent SNNs produces weight matrices similar to those of the ground truth model. To this end, we propose flexible heuristic supervised learning rules, termed Pre-Synaptic Pool Modification (PSPM), that rely on stochastic weight updates in order to produce spikes within a short window of the desired times and eliminate spikes outside of this window. PSPM improves spike train similarity for all-to-all SNNs and makes no assumption about the post-synaptic potential of the neurons or the structure of the network since no gradients are required. We test whether optimizing for spike train similarity entails the discovery of accurate weights and explore the relative contributions of local and homeostatic weight updates. Although PSPM improves similarity between spike trains, the learned weights often differ from the weights of the ground truth model, implying that connectome inference from spike data may require additional constraints on connectivity statistics. We also find that spike train similarity is sensitive to local updates, but other measures of network activity such as avalanche distributions, can be learned through synaptic homeostasis.
Reinforcement Evolutionary Learning Method for self-learning
Authors: Kumarjit Pathak, Jitin Kapila
Date: 2018-10-07 00:00:00
Tasks: Incremental Learning, Marketing, Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Self-Learning

In statistical modelling the biggest threat is concept drift which makes the
model gradually showing deteriorating performance over time. There are state of
the art methodologies to detect the impact of concept drift, however general
strategy considered to overcome the issue in performance is to rebuild or
re-calibrate the model periodically as the variable patterns for the model
changes significantly due to market change or consumer behavior change etc.
Quantitative research is the most widely spread application of data science in
Marketing or financial domain where applicability of state of the art
reinforcement learning for auto-learning is less explored paradigm.
Reinforcement learning is heavily dependent on having a simulated environment
which is majorly available for gaming or online systems, to learn from the live
feedback. However, there are some research happened on the area of online
advertisement, pricing etc where due to the nature of the online learning
environment scope of reinforcement learning is explored. Our proposed solution
is a reinforcement learning based, true self-learning algorithm which can adapt
to the data change or concept drift and auto learn and self-calibrate for the
new patterns of the data solving the problem of concept drift.
  Keywords - Reinforcement learning, Genetic Algorithm, Q-learning,
Classification modelling, CMA-ES, NES, Multi objective optimization, Concept
drift, Population stability index, Incremental learning, F1-measure, Predictive
Modelling, Self-learning, MCTS, AlphaGo, AlphaZero
Recycled ADMM: Improve Privacy and Accuracy with Less Computation in Distributed Algorithms
Authors: Xueru Zhang, Mohammad Mahdi Khalili, Mingyan Liu
Date: 2018-10-07 00:00:00

Alternating direction method of multiplier (ADMM) is a powerful method to
solve decentralized convex optimization problems. In distributed settings, each
node performs computation with its local data and the local results are
exchanged among neighboring nodes in an iterative fashion. During this
iterative process the leakage of data privacy arises and can accumulate
significantly over many iterations, making it difficult to balance the
privacy-utility tradeoff. In this study we propose Recycled ADMM (R-ADMM),
where a linear approximation is applied to every even iteration, its solution
directly calculated using only results from the previous, odd iteration. It
turns out that under such a scheme, half of the updates incur no privacy loss
and require much less computation compared to the conventional ADMM. We obtain
a sufficient condition for the convergence of R-ADMM and provide the privacy
analysis based on objective perturbation.
Phonology-Augmented Statistical Framework for Machine Transliteration using Limited Linguistic Resources
Authors: Gia H. Ngo, Minh Nguyen, Nancy F. Chen
Date: 2018-10-07 00:00:00
Tasks: Transliteration

Transliteration converts words in a source language (e.g., English) into
words in a target language (e.g., Vietnamese). This conversion considers the
phonological structure of the target language, as the transliterated output
needs to be pronounceable in the target language. For example, a word in
Vietnamese that begins with a consonant cluster is phonologically invalid and
thus would be an incorrect output of a transliteration system. Most statistical
transliteration approaches, albeit being widely adopted, do not explicitly
model the target language's phonology, which often results in invalid outputs.
The problem is compounded by the limited linguistic resources available when
converting foreign words to transliterated words in the target language. In
this work, we present a phonology-augmented statistical framework suitable for
transliteration, especially when only limited linguistic resources are
available. We propose the concept of pseudo-syllables as structures
representing how segments of a foreign word are organized according to the
syllables of the target language's phonology. We performed transliteration
experiments on Vietnamese and Cantonese. We show that the proposed framework
outperforms the statistical baseline by up to 44.68% relative, when there are
limited training examples (587 entries).
Fast and Robust Small Infrared Target Detection Using Absolute Directional Mean Difference Algorithm
Authors: Saed Moradi, Payman Moallem, Mohamad Farzan Sabahi
Date: 2018-10-07 00:00:00

Infrared small target detection in an infrared search and track (IRST) system is a challenging task. This situation becomes more complicated when high gray-intensity structural backgrounds appear in the field of view (FoV) of the infrared seeker. While the majority of the infrared small target detection algorithms neglect directional information, in this paper, a directional approach is presented to suppress structural backgrounds and develop a more effective detection algorithm. To this end, a similar concept to the average absolute gray difference (AAGD) is utilized to construct a novel directional small target detection algorithm called absolute directional mean difference (ADMD). Also, an efficient implementation procedure is presented for the proposed algorithm. The proposed algorithm effectively enhances the target area and eliminates background clutter. Simulation results on real infrared images prove the significant effectiveness of the proposed algorithm.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Authors: Hui Su, Xiaoyu Shen, Wenjie Li, Dietrich Klakow
Date: 2018-09-27 00:00:00
Proceeding: EMNLP 2018 10
Tasks: Dialogue Generation

Sequence-to-Sequence (seq2seq) models have become overwhelmingly popular in
building end-to-end trainable dialogue systems. Though highly efficient in
learning the backbone of human-computer communications, they suffer from the
problem of strongly favoring short generic responses. In this paper, we argue
that a good response should smoothly connect both the preceding dialogue
history and the following conversations. We strengthen this connection through
mutual information maximization. To sidestep the non-differentiability of
discrete natural language tokens, we introduce an auxiliary continuous code
space and map such code space to a learnable prior distribution for generation
purpose. Experiments on two dialogue datasets validate the effectiveness of our
model, where the generated responses are closely related to the dialogue
context and lead to more interactive conversations.