pwc-archive/papers-with-abstracts huggingface dataset:
A Many-Objective Evolutionary Algorithm With Two Interacting Processes: Cascade Clustering and Reference Point Incremental Learning
Authors: Hongwei Ge, Mingde Zhao, Liang Sun, Zhen Wang, Guozhen Tan, Qiang Zhang, C. L. Philip Chen
Date: 2018-03-03 00:00:00
Tasks: Clustering, Diversity, Incremental Learning

Researches have shown difficulties in obtaining proximity while maintaining diversity for many-objective optimization problems. Complexities of the true Pareto front pose challenges for the reference vector-based algorithms for their insufficient adaptability to the diverse characteristics with no priori. This paper proposes a many-objective optimization algorithm with two interacting processes: cascade clustering and reference point incremental learning (CLIA). In the population selection process based on cascade clustering (CC), using the reference vectors provided by the process based on incremental learning, the nondominated and the dominated individuals are clustered and sorted with different manners in a cascade style and are selected by round-robin for better proximity and diversity. In the reference vector adaptation process based on reference point incremental learning, using the feedbacks from the process based on CC, proper distribution of reference points is gradually obtained by incremental learning. Experimental studies on several benchmark problems show that CLIA is competitive compared with the state-of-the-art algorithms and has impressive efficiency and versatility using only the interactions between the two processes without incurring extra evaluations.
Inception Score, Label Smoothing, Gradient Vanishing and -log(D(x)) Alternative
Authors: Zhiming Zhou, Wei-Nan Zhang, Jun Wang
Date: 2017-08-05 00:00:00

In this article, we mathematically study several GAN related topics,
including Inception score, label smoothing, gradient vanishing and the
-log(D(x)) alternative.
  ---
  An advanced version is included in arXiv:1703.02000 "Activation Maximization
Generative Adversarial Nets".
  Please refer Section 6 in 1703.02000 for detailed analysis on Inception
Score, and refer its appendix for the discussions on Label Smoothing, Gradient
Vanishing and -log(D(x)) Alternative.
Analyzing Business Process Anomalies Using Autoencoders
Authors: Timo Nolle, Stefan Luettgen, Alexander Seeliger, Max Mühlhäuser
Date: 2018-03-03 00:00:00
Tasks: Anomaly Detection

Businesses are naturally interested in detecting anomalies in their internal
processes, because these can be indicators for fraud and inefficiencies. Within
the domain of business intelligence, classic anomaly detection is not very
frequently researched. In this paper, we propose a method, using autoencoders,
for detecting and analyzing anomalies occurring in the execution of a business
process. Our method does not rely on any prior knowledge about the process and
can be trained on a noisy dataset already containing the anomalies. We
demonstrate its effectiveness by evaluating it on 700 different datasets and
testing its performance against three state-of-the-art anomaly detection
methods. This paper is an extension of our previous work from 2016 [30].
Compared to the original publication we have further refined the approach in
terms of performance and conducted an elaborate evaluation on more
sophisticated datasets including real-life event logs from the Business Process
Intelligence Challenges of 2012 and 2017. In our experiments our approach
reached an F1 score of 0.87, whereas the best unaltered state-of-the-art
approach reached an F1 score of 0.72. Furthermore, our approach can be used to
analyze the detected anomalies in terms of which event within one execution of
the process causes the anomaly.
On Modular Training of Neural Acoustics-to-Word Model for LVCSR
Authors: Zhehuai Chen, Qi Liu, Hao Li, Kai Yu
Date: 2018-03-03 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Modeling, Language Modelling, speech-recognition, Speech Recognition

End-to-end (E2E) automatic speech recognition (ASR) systems directly map
acoustics to words using a unified model. Previous works mostly focus on E2E
training a single model which integrates acoustic and language model into a
whole. Although E2E training benefits from sequence modeling and simplified
decoding pipelines, large amount of transcribed acoustic data is usually
required, and traditional acoustic and language modelling techniques cannot be
utilized. In this paper, a novel modular training framework of E2E ASR is
proposed to separately train neural acoustic and language models during
training stage, while still performing end-to-end inference in decoding stage.
Here, an acoustics-to-phoneme model (A2P) and a phoneme-to-word model (P2W) are
trained using acoustic data and text data respectively. A phone synchronous
decoding (PSD) module is inserted between A2P and P2W to reduce sequence
lengths without precision loss. Finally, modules are integrated into an
acousticsto-word model (A2W) and jointly optimized using acoustic data to
retain the advantage of sequence modeling. Experiments on a 300- hour
Switchboard task show significant improvement over the direct A2W model. The
efficiency in both training and decoding also benefits from the proposed
method.
GPU Accelerated Sub-Sampled Newton's Method
Authors: Sudhir B. Kylasa, Farbod Roosta-Khorasani, Michael W. Mahoney, Ananth Grama
Date: 2018-02-26 00:00:00
Tasks: GPU, Second-order methods

First order methods, which solely rely on gradient information, are commonly
used in diverse machine learning (ML) and data analysis (DA) applications. This
is attributed to the simplicity of their implementations, as well as low
per-iteration computational/storage costs. However, they suffer from
significant disadvantages; most notably, their performance degrades with
increasing problem ill-conditioning. Furthermore, they often involve a large
number of hyper-parameters, and are notoriously sensitive to parameters such as
the step-size. By incorporating additional information from the Hessian,
second-order methods, have been shown to be resilient to many such adversarial
effects. However, these advantages of using curvature information come at the
cost of higher per-iteration costs, which in \enquote{big data} regimes, can be
computationally prohibitive.
  In this paper, we show that, contrary to conventional belief, second-order
methods, when implemented appropriately, can be more efficient than first-order
alternatives in many large-scale ML/ DA applications. In particular, in convex
settings, we consider variants of classical Newton\textsf{'}s method in which
the Hessian and/or the gradient are randomly sub-sampled. We show that by
effectively leveraging the power of GPUs, such randomized Newton-type
algorithms can be significantly accelerated, and can easily outperform state of
the art implementations of existing techniques in popular ML/ DA software
packages such as TensorFlow. Additionally these randomized methods incur a
small memory overhead compared to first-order methods. In particular, we show
that for million-dimensional problems, our GPU accelerated sub-sampled
Newton\textsf{'}s method achieves a higher test accuracy in milliseconds as
compared with tens of seconds for first order alternatives.
Practical Contextual Bandits with Regression Oracles
Authors: Dylan J. Foster, Alekh Agarwal, Miroslav Dudík, Haipeng Luo, Robert E. Schapire
Date: 2018-03-03 00:00:00
Proceeding: ICML 2018 7
Tasks: General Classification, Multi-Armed Bandits, regression

A major challenge in contextual bandits is to design general-purpose
algorithms that are both practically useful and theoretically well-founded. We
present a new technique that has the empirical and computational advantages of
realizability-based approaches combined with the flexibility of agnostic
methods. Our algorithms leverage the availability of a regression oracle for
the value-function class, a more realistic and reasonable oracle than the
classification oracles over policies typically assumed by agnostic methods. Our
approach generalizes both UCB and LinUCB to far more expressive possible model
classes and achieves low regret under certain distributional assumptions. In an
extensive empirical evaluation, compared to both realizability-based and
agnostic baselines, we find that our approach typically gives comparable or
superior results.
Deep Neural Networks as Gaussian Processes
Authors: Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
Date: 2017-11-01 00:00:00
Proceeding: ICLR 2018 1
Tasks: Bayesian Inference, Gaussian Processes

It has long been known that a single-layer fully-connected neural network
with an i.i.d. prior over its parameters is equivalent to a Gaussian process
(GP), in the limit of infinite network width. This correspondence enables exact
Bayesian inference for infinite width neural networks on regression tasks by
means of evaluating the corresponding GP. Recently, kernel functions which
mimic multi-layer random neural networks have been developed, but only outside
of a Bayesian framework. As such, previous work has not identified that these
kernels can be used as covariance functions for GPs and allow fully Bayesian
prediction with a deep neural network.
  In this work, we derive the exact equivalence between infinitely wide deep
networks and GPs. We further develop a computationally efficient pipeline to
compute the covariance function for these GPs. We then use the resulting GPs to
perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10.
We observe that trained neural network accuracy approaches that of the
corresponding GP with increasing layer width, and that the GP uncertainty is
strongly correlated with trained network prediction error. We further find that
test performance increases as finite-width trained networks are made wider and
more similar to a GP, and thus that GP predictions typically outperform those
of finite-width networks. Finally we connect the performance of these GPs to
the recent theory of signal propagation in random neural networks.
High-Dynamic-Range Imaging for Cloud Segmentation
Authors: Soumyabrata Dev, Florian M. Savoy, Yee Hui Lee, Stefan Winkler
Date: 2018-03-02 00:00:00
Tasks: Benchmarking, Image Generation, Segmentation, Vocal Bursts Intensity Prediction

Sky/cloud images obtained from ground-based sky-cameras are usually captured
using a fish-eye lens with a wide field of view. However, the sky exhibits a
large dynamic range in terms of luminance, more than a conventional camera can
capture. It is thus difficult to capture the details of an entire scene with a
regular camera in a single shot. In most cases, the circumsolar region is
over-exposed, and the regions near the horizon are under-exposed. This renders
cloud segmentation for such images difficult. In this paper, we propose
HDRCloudSeg -- an effective method for cloud segmentation using
High-Dynamic-Range (HDR) imaging based on multi-exposure fusion. We describe
the HDR image generation process and release a new database to the community
for benchmarking. Our proposed approach is the first using HDR radiance maps
for cloud segmentation and achieves very good results.
Specialized Interior Point Algorithm for Stable Nonlinear System Identification
Authors: Jack Umenberger, Ian R. Manchester
Date: 2018-03-02 00:00:00

Estimation of nonlinear dynamic models from data poses many challenges,
including model instability and non-convexity of long-term simulation fidelity.
Recently Lagrangian relaxation has been proposed as a method to approximate
simulation fidelity and guarantee stability via semidefinite programming (SDP),
however the resulting SDPs have large dimension, limiting their utility in
practical problems. In this paper we develop a path-following interior point
algorithm that takes advantage of special structure in the problem and reduces
computational complexity from cubic to linear growth with the length of the
data set. The new algorithm enables empirical comparisons to established
methods including Nonlinear ARX, and we demonstrate superior generalization to
new data. We also explore the "regularizing" effect of stability constraints as
an alternative to regressor subset selection.
Multi-task Learning with Gradient Guided Policy Specialization
Authors: Wenhao Yu, C. Karen Liu, Greg Turk
Date: 2017-09-23 00:00:00
Tasks: Multi-Task Learning

We present a method for efficient learning of control policies for multiple
related robotic motor skills. Our approach consists of two stages, joint
training and specialization training. During the joint training stage, a neural
network policy is trained with minimal information to disambiguate the motor
skills. This forces the policy to learn a common representation of the
different tasks. Then, during the specialization training stage we selectively
split the weights of the policy based on a per-weight metric that measures the
disagreement among the multiple tasks. By splitting part of the control policy,
it can be further trained to specialize to each task. To update the control
policy during learning, we use Trust Region Policy Optimization with
Generalized Advantage Function (TRPOGAE). We propose a modification to the
gradient update stage of TRPO to better accommodate multi-task learning
scenarios. We evaluate our approach on three continuous motor skill learning
problems in simulation: 1) a locomotion task where three single legged robots
with considerable difference in shape and size are trained to hop forward, 2) a
manipulation task where three robot manipulators with different sizes and joint
types are trained to reach different locations in 3D space, and 3) locomotion
of a two-legged robot, whose range of motion of one leg is constrained in
different ways. We compare our training method to three baselines. The first
baseline uses only joint training for the policy, the second trains independent
policies for each task, and the last randomly selects weights to split. We show
that our approach learns more efficiently than each of the baseline methods.
Multi-Agent Imitation Learning for Driving Simulation
Authors: Raunak P. Bhattacharyya, Derek J. Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, Mykel J. Kochenderfer
Date: 2018-03-02 00:00:00
Tasks: Autonomous Vehicles, Imitation Learning

Simulation is an appealing option for validating the safety of autonomous
vehicles. Generative Adversarial Imitation Learning (GAIL) has recently been
shown to learn representative human driver models. These human driver models
were learned through training in single-agent environments, but they have
difficulty in generalizing to multi-agent driving scenarios. We argue these
difficulties arise because observations at training and test time are sampled
from different distributions. This difference makes such models unsuitable for
the simulation of driving scenes, where multiple agents must interact
realistically over long time horizons. We extend GAIL to address these
shortcomings through a parameter-sharing approach grounded in curriculum
learning. Compared with single-agent GAIL policies, policies generated by our
PS-GAIL method prove superior at interacting stably in a multi-agent setting
and capturing the emergent behavior of human drivers.
Building a Telescope to Look Into High-Dimensional Image Spaces
Authors: Mitch Hill, Erik Nijkamp, Song-Chun Zhu
Date: 2018-03-02 00:00:00
Tasks: Vocal Bursts Intensity Prediction

An image pattern can be represented by a probability distribution whose
density is concentrated on different low-dimensional subspaces in the
high-dimensional image space. Such probability densities have an astronomical
number of local modes corresponding to typical pattern appearances. Related
groups of modes can join to form macroscopic image basins that represent
pattern concepts. Recent works use neural networks that capture high-order
image statistics to learn Gibbs models capable of synthesizing realistic images
of many patterns. However, characterizing a learned probability density to
uncover the Hopfield memories of the model, encoded by the structure of the
local modes, remains an open challenge. In this work, we present novel
computational experiments that map and visualize the local mode structure of
Gibbs densities. Efficient mapping requires identifying the global basins
without enumerating the countless modes. Inspired by Grenander's jump-diffusion
method, we propose a new MCMC tool called Attraction-Diffusion (AD) that can
capture the macroscopic structure of highly non-convex densities by measuring
metastability of local modes. AD involves altering the target density with a
magnetization potential penalizing distance from a known mode and running an
MCMC sample of the altered density to measure the stability of the initial
chain state. Using a low-dimensional generator network to facilitate
exploration, we map image spaces with up to 12,288 dimensions (64 $\times$ 64
pixels in RGB). Our work shows: (1) AD can efficiently map highly non-convex
probability densities, (2) metastable regions of pattern probability densities
contain coherent groups of images, and (3) the perceptibility of differences
between training images influences the metastability of image basins.
Generalizing Hamiltonian Monte Carlo with Neural Networks
Authors: Daniel Levy, Matthew D. Hoffman, Jascha Sohl-Dickstein
Date: 2017-11-25 00:00:00
Proceeding: ICLR 2018 1

We present a general-purpose method to train Markov chain Monte Carlo
kernels, parameterized by deep neural networks, that converge and mix quickly
to their target distribution. Our method generalizes Hamiltonian Monte Carlo
and is trained to maximize expected squared jumped distance, a proxy for mixing
speed. We demonstrate large empirical gains on a collection of simple but
challenging distributions, for instance achieving a 106x improvement in
effective sample size in one case, and mixing when standard HMC makes no
measurable progress in a second. Finally, we show quantitative and qualitative
gains on a real-world task: latent-variable generative modeling. We release an
open source TensorFlow implementation of the algorithm.
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Authors: Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, William W. Cohen
Date: 2017-11-10 00:00:00
Proceeding: ICLR 2018 1
Tasks: Language Modeling, Language Modelling, Vocal Bursts Intensity Prediction, Word Embeddings

We formulate language modeling as a matrix factorization problem, and show
that the expressiveness of Softmax-based models (including the majority of
neural language models) is limited by a Softmax bottleneck. Given that natural
language is highly context-dependent, this further implies that in practice
Softmax with distributed word embeddings does not have enough capacity to model
natural language. We propose a simple and effective method to address this
issue, and improve the state-of-the-art perplexities on Penn Treebank and
WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on
the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points
in perplexity.
PRESISTANT: Learning based assistant for data pre-processing
Authors: Besim Bilalli, Alberto Abelló, Tomàs Aluja-Banet, Robert Wrembel
Date: 2018-03-02 00:00:00
Tasks: General Classification

Data pre-processing is one of the most time consuming and relevant steps in a
data analysis process (e.g., classification task). A given data pre-processing
operator (e.g., transformation) can have positive, negative or zero impact on
the final result of the analysis. Expert users have the required knowledge to
find the right pre-processing operators. However, when it comes to non-experts,
they are overwhelmed by the amount of pre-processing operators and it is
challenging for them to find operators that would positively impact their
analysis (e.g., increase the predictive accuracy of a classifier). Existing
solutions either assume that users have expert knowledge, or they recommend
pre-processing operators that are only "syntactically" applicable to a dataset,
without taking into account their impact on the final analysis. In this work,
we aim at providing assistance to non-expert users by recommending data
pre-processing operators that are ranked according to their impact on the final
analysis. We developed a tool PRESISTANT, that uses Random Forests to learn the
impact of pre-processing operators on the performance (e.g., predictive
accuracy) of 5 different classification algorithms, such as J48, Naive Bayes,
PART, Logistic Regression, and Nearest Neighbor. Extensive evaluations on the
recommendations provided by our tool, show that PRESISTANT can effectively help
non-experts in order to achieve improved results in their analytical tasks.
Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning
Authors: Teng Li, Zhiyuan Xu, Jian Tang, Yanzhi Wang
Date: 2018-03-02 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Scheduling

In this paper, we focus on general-purpose Distributed Stream Data Processing
Systems (DSDPSs), which deal with processing of unbounded streams of continuous
data at scale distributedly in real or near-real time. A fundamental problem in
a DSDPS is the scheduling problem with the objective of minimizing average
end-to-end tuple processing time. A widely-used solution is to distribute
workload evenly over machines in the cluster in a round-robin manner, which is
obviously not efficient due to lack of consideration for communication delay.
Model-based approaches do not work well either due to the high complexity of
the system environment. We aim to develop a novel model-free approach that can
learn to well control a DSDPS from its experience rather than accurate and
mathematically solvable system models, just as a human learns a skill (such as
cooking, driving, swimming, etc). Specifically, we, for the first time, propose
to leverage emerging Deep Reinforcement Learning (DRL) for enabling model-free
control in DSDPSs; and present design, implementation and evaluation of a novel
and highly effective DRL-based control framework, which minimizes average
end-to-end tuple processing time by jointly learning the system environment via
collecting very limited runtime statistics data and making decisions under the
guidance of powerful Deep Neural Networks. To validate and evaluate the
proposed framework, we implemented it based on a widely-used DSDPS, Apache
Storm, and tested it with three representative applications. Extensive
experimental results show 1) Compared to Storm's default scheduler and the
state-of-the-art model-based method, the proposed framework reduces average
tuple processing by 33.5% and 14.0% respectively on average. 2) The proposed
framework can quickly reach a good scheduling solution during online learning,
which justifies its practicability for online control in DSDPSs.