pwc-archive/papers-with-abstracts huggingface dataset:
Generating Long Sequences with Sparse Transformers
Authors: Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever
Date: 2019-04-23 00:00:00
Proceeding: Preprint 2019 4
Tasks: Diversity, Image Generation, Language Modelling, Open-Domain Question Answering, Question Answering

Transformers are powerful sequence models, but require time and memory that
grows quadratically with the sequence length. In this paper we introduce sparse
factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We
also introduce a) a variation on architecture and initialization to train
deeper networks, b) the recomputation of attention matrices to save memory, and
c) fast attention kernels for training. We call networks with these changes
Sparse Transformers, and show they can model sequences tens of thousands of
timesteps long using hundreds of layers. We use the same architecture to model
images, audio, and text from raw bytes, setting a new state of the art for
density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate
unconditional samples that demonstrate global coherence and great diversity,
and show it is possible in principle to use self-attention to model sequences
of length one million or more.
多等级电动汽车充电站的选址与算法
Authors: 张国亮
Date: 2011-11-01 00:00:00
Proceeding: journal 2011 11

Abstract: The charging station is part of the crucial infrastructure of the electric; vehicle, and the location and scale of
the charging stations are significant for the development of the electric; vehicle industry. A location and solving algorithm
of multi-level electric; vehicle charging stations was proposed.  Concerning the user' distributions, based on the objective
program thought, the multi-level electric; site model was developed to minimize the initial c;onstruc;tion cost and the total
users’charging service cost. An improved tabu search algorithm was presented to solve the model. The new c;ombina-
tion of tabu coding and the generating of initial solutions were designed to c;harac;terize the two decision variables of the
multi-level sites and the allocation to the users, using the strategy of 2}pt search neighbor, locating sites, station levels
and the allocation of the demand points to the stations were simultaneously determined.  Finally，a numerical example
demonstrated that the algorithm had better global searching performance and c;onvergenc;e property and the proposed
model was also effective.
Low-Rank Tucker Approximation of a Tensor From Streaming Data
Authors: Yiming Sun, Yang Guo, Charlene Luo, Joel Tropp, Madeleine Udell
Date: 2019-04-24 00:00:00

This paper describes a new algorithm for computing a low-Tucker-rank approximation of a tensor. The method applies a randomized linear map to the tensor to obtain a sketch that captures the important directions within each mode, as well as the interactions among the modes. The sketch can be extracted from streaming or distributed data or with a single pass over the tensor, and it uses storage proportional to the degrees of freedom in the output Tucker approximation. The algorithm does not require a second pass over the tensor, although it can exploit another view to compute a superior approximation. The paper provides a rigorous theoretical guarantee on the approximation error. Extensive numerical experiments show that that the algorithm produces useful results that improve on the state of the art for streaming Tucker decomposition.
Detailed Human Shape Estimation from a Single Image by Hierarchical Mesh Deformation
Authors: Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, Ruigang Yang
Date: 2019-04-24 00:00:00
Proceeding: CVPR 2019 6

This paper presents a novel framework to recover detailed human body shapes from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric based template that lacks the surface details. As such the resulting body shape appears to be without clothing. In this paper, we propose a novel learning-based framework that combines the robustness of parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. We are able to restore detailed human body shapes beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance. The code is available in https://github.com/zhuhao-nju/hmd.git
On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrieval
Authors: Ankita Pasad, Bowen Shi, Herman Kamper, Karen Livescu
Date: 2019-04-24 00:00:00
Tasks: Retrieval, Visual Grounding

Recent work has shown that speech paired with images can be used to learn semantically meaningful speech representations even without any textual supervision. In real-world low-resource settings, however, we often have access to some transcribed speech. We study whether and how visual grounding is useful in the presence of varying amounts of textual supervision. In particular, we consider the task of semantic speech retrieval in a low-resource setting. We use a previously studied data set and task, where models are trained on images with spoken captions and evaluated on human judgments of semantic relevance. We propose a multitask learning approach to leverage both visual and textual modalities, with visual supervision in the form of keyword probabilities from an external tagger. We find that visual grounding is helpful even in the presence of textual supervision, and we analyze this effect over a range of sizes of transcribed data sets. With ~5 hours of transcribed speech, we obtain 23% higher average precision when also using visual supervision.
Target-Based Temporal Difference Learning
Authors: Donghwan Lee, Niao He
Date: 2019-04-24 00:00:00
Tasks: Q-Learning, Reinforcement Learning

The use of target networks has been a popular and key component of recent deep Q-learning algorithms for reinforcement learning, yet little is known from the theory side. In this work, we introduce a new family of target-based temporal difference (TD) learning algorithms and provide theoretical analysis on their convergences. In contrast to the standard TD-learning, target-based TD algorithms maintain two separate learning parameters-the target variable and online variable. Particularly, we introduce three members in the family, called the averaging TD, double TD, and periodic TD, where the target variable is updated through an averaging, symmetric, or periodic fashion, mirroring those techniques used in deep Q-learning practice. We establish asymptotic convergence analyses for both averaging TD and double TD and a finite sample analysis for periodic TD. In addition, we also provide some simulation results showing potentially superior convergence of these target-based TD algorithms compared to the standard TD-learning. While this work focuses on linear function approximation and policy evaluation setting, we consider this as a meaningful step towards the theoretical understanding of deep Q-learning variants with target networks.
Tactile Mapping and Localization from High-Resolution Tactile Imprints
Authors: Maria Bauza, Oleguer Canal, Alberto Rodriguez
Date: 2019-04-24 00:00:00
Tasks: Object, Object Localization, Vocal Bursts Intensity Prediction

This work studies the problem of shape reconstruction and object localization using a vision-based tactile sensor, GelSlim. The main contributions are the recovery of local shapes from contact, an approach to reconstruct the tactile shape of objects from tactile imprints, and an accurate method for object localization of previously reconstructed objects. The algorithms can be applied to a large variety of 3D objects and provide accurate tactile feedback for in-hand manipulation. Results show that by exploiting the dense tactile information we can reconstruct the shape of objects with high accuracy and do on-line object identification and localization, opening the door to reactive manipulation guided by tactile sensing. We provide videos and supplemental information in the project's website http://web.mit.edu/mcube/research/tactile_localization.html.
Horseshoe Regularization for Machine Learning in Complex and Deep Models
Authors: Anindya Bhadra, Jyotishka Datta, Yunfan Li, Nicholas G. Polson
Date: 2019-04-24 00:00:00
Tasks: BIG-bench Machine Learning, regression

Since the advent of the horseshoe priors for regularization, global-local shrinkage methods have proved to be a fertile ground for the development of Bayesian methodology in machine learning, specifically for high-dimensional regression and classification problems. They have achieved remarkable success in computation, and enjoy strong theoretical support. Most of the existing literature has focused on the linear Gaussian case; see Bhadra et al. (2019b) for a systematic survey. The purpose of the current article is to demonstrate that the horseshoe regularization is useful far more broadly, by reviewing both methodological and computational developments in complex models that are more relevant to machine learning applications. Specifically, we focus on methodological challenges in horseshoe regularization in nonlinear and non-Gaussian models; multivariate models; and deep neural networks. We also outline the recent computational developments in horseshoe shrinkage for complex models along with a list of available software implementations that allows one to venture out beyond the comfort zone of the canonical linear regression problems.
Generated Loss and Augmented Training of MNIST VAE
Authors: Jason Chou
Date: 2019-04-24 00:00:00

The variational autoencoder (VAE) framework is a popular option for training
unsupervised generative models, featuring ease of training and latent
representation of data. The objective function of VAE does not guarantee to
achieve the latter, however, and failure to do so leads to a frequent failure
mode called posterior collapse. Even in successful cases, VAEs often result in
low-precision reconstructions and generated samples. The introduction of the
KL-divergence weight $\beta$ can help steer the model clear of posterior
collapse, but its tuning is often a trial-and-error process with no guiding
metrics. Here we test the idea of using the total VAE loss of generated samples
(generated loss) as the proxy metric for generation quality, the related
hypothesis that VAE reconstruction from the mean latent vector tends to be a
more typical example of its class than the original, and the idea of exploiting
this property by augmenting training data with generated variants (augmented
training). The results are mixed, but repeated encoding and decoding indeed
result in qualitatively and quantitatively more typical examples from both
convolutional and fully-connected MNIST VAEs, suggesting that it may be an
inherent property of the VAE framework.
Evolving Neural Networks in Reinforcement Learning by means of UMDAc
Authors: Mikel Malagon, Josu Ceberio
Date: 2019-04-24 00:00:00
Tasks: OpenAI Gym, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Neural networks are gaining popularity in the reinforcement learning field
due to the vast number of successfully solved complex benchmark problems. In
fact, artificial intelligence algorithms are, in some cases, able to overcome
human professionals. Usually, neural networks have more than a couple of hidden
layers, and thus, they involve a large quantity of parameters that need to be
optimized. Commonly, numeric approaches are used to optimize the inner
parameters of neural networks, such as the stochastic gradient descent.
However, these techniques tend to be computationally very expensive, and for
some tasks, where effectiveness is crucial, high computational costs are not
acceptable. Along these research lines, in this paper we propose to optimize
the parameters of neural networks by means of estimation of distribution
algorithms. More precisely, the univariate marginal distribution algorithm is
used for evolving neural networks in various reinforcement learning tasks. For
the sake of validating our idea, we run the proposed algorithm on four OpenAI
Gym benchmark problems. In addition, the obtained results were compared with a
standard genetic algorithm. Revealing, that optimizing with UMDAc provides
better results than the genetic algorithm in most of the cases.
Prediction of Progression to Alzheimer's disease with Deep InfoMax
Authors: Alex Fedorov, R. Devon Hjelm, Anees Abrol, Zening Fu, Yuhui Du, Sergey Plis, Vince D. Calhoun
Date: 2019-04-24 00:00:00
Tasks: General Classification

Arguably, unsupervised learning plays a crucial role in the majority of
algorithms for processing brain imaging. A recently introduced unsupervised
approach Deep InfoMax (DIM) is a promising tool for exploring brain structure
in a flexible non-linear way. In this paper, we investigate the use of variants
of DIM in a setting of progression to Alzheimer's disease in comparison with
supervised AlexNet and ResNet inspired convolutional neural networks. As a
benchmark, we use a classification task between four groups: patients with
stable, and progressive mild cognitive impairment (MCI), with Alzheimer's
disease, and healthy controls. Our dataset is comprised of 828 subjects from
the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. Our
experiments highlight encouraging evidence of the high potential utility of DIM
in future neuroimaging studies.
The Scientific Method in the Science of Machine Learning
Authors: Jessica Zosa Forde, Michela Paganini
Date: 2019-04-24 00:00:00
Tasks: BIG-bench Machine Learning

In the quest to align deep learning with the sciences to address calls for
rigor, safety, and interpretability in machine learning systems, this
contribution identifies key missing pieces: the stages of hypothesis
formulation and testing, as well as statistical and systematic uncertainty
estimation -- core tenets of the scientific method. This position paper
discusses the ways in which contemporary science is conducted in other domains
and identifies potentially useful practices. We present a case study from
physics and describe how this field has promoted rigor through specific
methodological practices, and provide recommendations on how machine learning
researchers can adopt these practices into the research ecosystem. We argue
that both domain-driven experiments and application-agnostic questions of the
inner workings of fundamental building blocks of machine learning models ought
to be examined with the tools of the scientific method, to ensure we not only
understand effect, but also begin to understand cause, which is the raison
d'\^{e}tre of science.
Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks
Authors: Jaedeok Kim, Chiyoun Park, Hyun-Joo Jung, Yoonsuck Choe
Date: 2019-04-24 00:00:00
Tasks: Efficient Neural Network, image-classification, Image Classification, Machine Translation, Multiple-choice, Optical Flow Estimation, Style Transfer, Translation

Architecture optimization, which is a technique for finding an efficient neural network that meets certain requirements, generally reduces to a set of multiple-choice selection problems among alternative sub-structures or parameters. The discrete nature of the selection problem, however, makes this optimization difficult. To tackle this problem we introduce a novel concept of a trainable gate function. The trainable gate function, which confers a differentiable property to discretevalued variables, allows us to directly optimize loss functions that include non-differentiable discrete values such as 0-1 selection. The proposed trainable gate can be applied to pruning. Pruning can be carried out simply by appending the proposed trainable gate functions to each intermediate output tensor followed by fine-tuning the overall model, using any gradient-based training methods. So the proposed method can jointly optimize the selection of the pruned channels while fine-tuning the weights of the pruned model at the same time. Our experimental results demonstrate that the proposed method efficiently optimizes arbitrary neural networks in various tasks such as image classification, style transfer, optical flow estimation, and neural machine translation.
The iterative convolution-thresholding method (ICTM) for image segmentation
Authors: Dong Wang, Xiao-Ping Wang
Date: 2019-04-24 00:00:00
Tasks: Image Segmentation, Segmentation, Semantic Segmentation

In this paper, we propose a novel iterative convolution-thresholding method
(ICTM) that is applicable to a range of variational models for image
segmentation. A variational model usually minimizes an energy functional
consisting of a fidelity term and a regularization term. In the ICTM, the
interface between two different segment domains is implicitly represented by
their characteristic functions. The fidelity term is then usually written as a
linear functional of the characteristic functions and the regularized term is
approximated by a functional of characteristic functions in terms of heat
kernel convolution. This allows us to design an iterative
convolution-thresholding method to minimize the approximate energy. The method
is simple, efficient and enjoys the energy-decaying property. Numerical
experiments show that the method is easy to implement, robust and applicable to
various image segmentation models.
Applying machine learning to improve simulations of a chaotic dynamical system using empirical error correction
Authors: Peter A. G. Watson
Date: 2019-04-24 00:00:00
Tasks: BIG-bench Machine Learning

Dynamical weather and climate prediction models underpin many studies of the
Earth system and hold the promise of being able to make robust projections of
future climate change based on physical laws. However, simulations from these
models still show many differences compared with observations. Machine learning
has been applied to solve certain prediction problems with great success, and
recently it's been proposed that this could replace the role of
physically-derived dynamical weather and climate models to give better quality
simulations. Here, instead, a framework using machine learning together with
physically-derived models is tested, in which it is learnt how to correct the
errors of the latter from timestep to timestep. This maintains the physical
understanding built into the models, whilst allowing performance improvements,
and also requires much simpler algorithms and less training data. This is
tested in the context of simulating the chaotic Lorenz '96 system, and it is
shown that the approach yields models that are stable and that give both
improved skill in initialised predictions and better long-term climate
statistics. Improvements in long-term statistics are smaller than for single
time-step tendencies, however, indicating that it would be valuable to develop
methods that target improvements on longer time scales. Future strategies for
the development of this approach and possible applications to making progress
on important scientific problems are discussed.
Learning big Gaussian Bayesian networks: partition, estimation, and fusion
Authors: Jiaying Gu, Qing Zhou
Date: 2019-04-24 00:00:00
Tasks: Clustering

Structure learning of Bayesian networks has always been a challenging
problem. Nowadays, massive-size networks with thousands or more of nodes but
fewer samples frequently appear in many areas. We develop a divide-and-conquer
framework, called partition-estimation-fusion (PEF), for structure learning of
such big networks. The proposed method first partitions nodes into clusters,
then learns a subgraph on each cluster of nodes, and finally fuses all learned
subgraphs into one Bayesian network. The PEF method is designed in a flexible
way so that any structure learning method may be used in the second step to
learn a subgraph structure as either a DAG or a CPDAG. In the clustering step,
we adapt the hierarchical clustering method to automatically choose a proper
number of clusters. In the fusion step, we propose a novel hybrid method that
sequentially add edges between subgraphs. Extensive numerical experiments
demonstrate the competitive performance of our PEF method, in terms of both
speed and accuracy compared to existing methods. Our method can improve the
accuracy of structure learning by 20% or more, while reducing running time up
to two orders-of-magnitude.
ViDeNN: Deep Blind Video Denoising
Authors: Michele Claus, Jan van Gemert
Date: 2019-04-24 00:00:00
Tasks: Denoising, Video Denoising

We propose ViDeNN: a CNN for Video Denoising without prior knowledge on the
noise distribution (blind denoising). The CNN architecture uses a combination
of spatial and temporal filtering, learning to spatially denoise the frames
first and at the same time how to combine their temporal information, handling
objects motion, brightness changes, low-light conditions and temporal
inconsistencies. We demonstrate the importance of the data used for CNNs
training, creating for this purpose a specific dataset for low-light
conditions. We test ViDeNN on common benchmarks and on self-collected data,
achieving good results comparable with the state-of-the-art.
Listening between the Lines: Learning Personal Attributes from Conversations
Authors: Anna Tigunova, Andrew Yates, Paramita Mirza, Gerhard Weikum
Date: 2019-04-24 00:00:00
Tasks: Articles, Attribute

Open-domain dialogue agents must be able to converse about many topics while
incorporating knowledge about the user into the conversation. In this work we
address the acquisition of such knowledge, for personalization in downstream
Web applications, by extracting personal attributes from conversations. This
problem is more challenging than the established task of information extraction
from scientific publications or Wikipedia articles, because dialogues often
give merely implicit cues about the speaker. We propose methods for inferring
personal attributes, such as profession, age or family status, from
conversations using deep learning. Specifically, we propose several Hidden
Attribute Models, which are neural networks leveraging attention mechanisms and
embeddings. Our methods are trained on a per-predicate basis to output rankings
of object values for a given subject-predicate combination (e.g., ranking the
doctor and nurse professions high when speakers talk about patients, emergency
rooms, etc). Experiments with various conversational texts including Reddit
discussions, movie scripts and a collection of crowdsourced personal dialogues
demonstrate the viability of our methods and their superior performance
compared to state-of-the-art baselines.
Integrating Social Media into a Pan-European Flood Awareness System: A Multilingual Approach
Authors: V. Lorini, C. Castillo, F. Dottori, M. Kalas, D. Nappo, P. Salamon
Date: 2019-04-24 00:00:00
Tasks: Word Embeddings

This paper describes a prototype system that integrates social media analysis
into the European Flood Awareness System (EFAS). This integration allows the
collection of social media data to be automatically triggered by flood risk
warnings determined by a hydro-meteorological model. Then, we adopt a
multi-lingual approach to find flood-related messages by employing two
state-of-the-art methodologies: language-agnostic word embeddings and
language-aligned word embeddings. Both approaches can be used to bootstrap a
classifier of social media messages for a new language with little or no
labeled data. Finally, we describe a method for selecting relevant and
representative messages and displaying them back in the interface of EFAS.