pwc-archive/papers-with-abstracts huggingface dataset:
Disentangled GANs for Controllable Generation of High-Resolution Images
Authors: Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debhath, Anjul Patney, Ankit B. Patel, Anima Anandkumar
Date: 2019-09-25 00:00:00
Tasks: Disentanglement, Vocal Bursts Intensity Prediction

Generative adversarial networks (GANs) have achieved great success at generating realistic samples. However, achieving disentangled and controllable generation still remains challenging for GANs, especially in the high-resolution image domain. Motivated by this, we introduce AC-StyleGAN, a combination of AC-GAN and StyleGAN, for demonstrating that the controllable generation of high-resolution images is possible with sufficient supervision. More importantly, only using 5% of the labelled data significantly improves the disentanglement quality. Inspired by the observed separation of fine and coarse styles in StyleGAN, we then extend AC-StyleGAN to a new image-to-image model called FC-StyleGAN for semantic manipulation of fine-grained factors in a high-resolution image. In experiments, we show that FC-StyleGAN performs well in only controlling fine-grained factors, with the use of instance normalization, and also demonstrate its good generalization ability to unseen images. Finally, we create two new datasets -- Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, as compared to existing disentanglement datasets.
UNIVERSAL MODAL EMBEDDING OF DYNAMICS IN VIDEOS AND ITS APPLICATIONS
Authors: Israr Ul Haq, Yoshinobu Kawahara
Date: 2019-09-25 00:00:00
Tasks: Decoder, Time Series, Time Series Analysis, Video Classification

Extracting underlying dynamics of objects in image sequences is one of the challenging problems in computer vision. On the other hand, dynamic mode decomposition (DMD) has recently attracted attention as a way of obtaining modal representations of nonlinear dynamics from (general multivariate time-series) data without explicit prior knowledge about the dynamics. In this paper, we propose a convolutional autoencoder based DMD (CAE-DMD) that is an extended DMD (EDMD) approach, to extract underlying dynamics in videos. To this end, we develop a modified CAE model by incorporating DMD on the encoder, which gives a more meaningful compressed representation of input image sequences. On the reconstruction side, a decoder is used to minimize the reconstruction error after applying the DMD, which in result gives an accurate reconstruction of inputs. We empirically investigated the performance of CAE-DMD in two applications: background/foreground extraction and video classification, on publicly available datasets.
MLModelScope: A Distributed Platform for ML Model Evaluation and Benchmarking at Scale
Authors: Cheng Li, Abdul Dakkak, JinJun Xiong, Wen-mei Hwu
Date: 2019-09-25 00:00:00
Tasks: Benchmarking

Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major "pain point" for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking.  We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.
Statistical Adaptive Stochastic Optimization
Authors: Pengchuan Zhang, Hunter Lang, Qiang Liu, Lin Xiao
Date: 2019-09-25 00:00:00
Tasks: Scheduling, Stochastic Optimization

We investigate statistical methods for automatically scheduling the learning rate (step size) in stochastic optimization. First, we consider a broad family of stochastic optimization methods with constant hyperparameters (including the learning rate and various forms of momentum) and derive a general necessary condition for the resulting dynamics to be stationary. Based on this condition, we develop a simple online statistical test to detect (non-)stationarity and use it to automatically drop the learning rate by a constant factor whenever stationarity is detected. Unlike in prior work, our stationarity condition and our statistical test applies to different algorithms without modification. Finally, we propose a smoothed stochastic line-search method that can be used to warm up the optimization process before the statistical test can be applied effectively. This removes the expensive trial and error for setting a good initial learning rate. The combined method is highly autonomous and it attains state-of-the-art training and testing performance in our experiments on several deep learning tasks.
Multichannel Generative Language Models
Authors: Harris Chan, Jamie Kiros, William Chan
Date: 2019-09-25 00:00:00
Tasks: Machine Translation

A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning but through two separate channels corresponding to their languages. In this work, we present Multichannel Generative Language Models (MGLM), which models the joint distribution over multiple channels, and all its decompositions using a single neural network. MGLM can be trained by feeding it k way parallel-data, bilingual data, or monolingual data across pre-determined channels. MGLM is capable of both conditional generation and unconditional sampling. For conditional generation, the model is given a fully observed channel, and generates the k-1 channels in parallel. In the case of machine translation, this is akin to giving it one source, and the model generates k-1 targets. MGLM can also do partial conditional sampling, where the channels are seeded with prespecified words, and the model is asked to infill the rest. Finally, we can sample from MGLM unconditionally over all k channels. Our experiments on the Multi30K dataset containing English, French, Czech, and German languages suggest that the multitask training with the joint objective leads to improvements in bilingual translations. We provide a quantitative analysis of the quality-diversity trade-offs for different variants of the multichannel model for conditional generation, and a measurement of self-consistency during unconditional generation. We provide qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.
HIPPOCAMPAL NEURONAL REPRESENTATIONS IN CONTINUAL LEARNING
Authors: Samia Mohinta, Rui Ponte Costa, Stephane Ciocchi
Date: 2019-09-25 00:00:00
Tasks: Continual Learning, Deep Reinforcement Learning, Hippocampus, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The hippocampus has long been associated with spatial memory and goal-directed
spatial navigation. However, the region’s independent role in continual learning of
navigational strategies has seldom been investigated. Here we analyse populationlevel
activity of hippocampal CA1 neurons in the context of continual learning of
two different spatial navigation strategies. Demixed Principal Component Analysis
(dPCA) is applied on neuronal recordings from 612 hippocampal CA1 neurons
of rodents learning to perform allocentric and egocentric spatial tasks. The components
uncovered using dPCA from the firing activity reveal that hippocampal
neurons encode relevant task variables such decisions, navigational strategies and
reward location. We compare this hippocampal features with standard reinforcement
learning algorithms, highlighting similarities and differences. Finally, we
demonstrate that a standard deep reinforcement learning model achieves similar
average performance when compared to animal learning, but fails to mimic animals
during task switching. Overall, our results gives insights into how the hippocampus
solves reinforced spatial continual learning, and puts forward a framework
to explicitly compare biological and machine learning during spatial continual
learning.
Augmenting Transformers with KNN-Based Composite Memory
Authors: Angela Fan, Claire Gardent, Chloe Braud, Antoine Bordes
Date: 2019-09-25 00:00:00

Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augmenting Transformer neural networks with KNN-based Information Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules to generative dialogue modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge from Wikipedia, images, and human-written dialogue utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.
Winning Privately: The Differentially Private Lottery Ticket Mechanism
Authors: Lovedeep Gondara, Ke Wang, Ricardo Silva Carvalho
Date: 2019-09-25 00:00:00

We propose the differentially private lottery ticket mechanism (DPLTM). An end-to-end differentially private training paradigm based on the lottery ticket hypothesis. Using ``high-quality winners", selected via our custom score function, DPLTM significantly outperforms state-of-the-art. We show that DPLTM converges faster, allowing for early stopping with reduced privacy budget consumption. We further show that the tickets from DPLTM are transferable across datasets, domains, and architectures. Our extensive evaluation on several public datasets provides evidence to our claims.
Localizing and Amortizing: Efficient Inference for Gaussian Processes
Authors: Linfeng Liu, LiPing Liu
Date: 2019-09-25 00:00:00
Tasks: Gaussian Processes, Stochastic Optimization

The inference of Gaussian Processes concerns the distribution of the underlying function given observed data points. GP inference based on local ranges of data points is able to capture fine-scale correlations and allow fine-grained decomposition of the computation. Following this direction, we propose a new inference model that considers the correlations and observations of the K nearest neighbors for the inference at a data point. Compared with previous works, we also eliminate the data ordering prerequisite to simplify the inference process. Additionally, the inference task is decomposed to small subtasks with several technique innovations, making our model well suits the stochastic optimization. Since the decomposed small subtasks have the same structure, we further speed up the inference procedure with amortized inference. Our model runs efficiently and achieves good performances on several benchmark tasks.
Entropy Penalty: Towards Generalization Beyond the IID Assumption
Authors: Devansh Arpit, Caiming Xiong, Richard Socher
Date: 2019-09-25 00:00:00

It has been shown that instead of learning actual object features, deep networks tend to exploit non-robust (spurious) discriminative features that are shared between training and test sets. Therefore, while they achieve state of the art performance on such test sets, they achieve poor generalization on out of distribution (OOD) samples where the IID (independent, identical distribution) assumption breaks and the distribution of non-robust features shifts. Through theoretical and empirical analysis, we show that this happens because maximum likelihood training (without appropriate regularization) leads the model to depend on all the correlations (including spurious ones) present between inputs and targets in the dataset. We then show evidence that the information bottleneck (IB) principle can address this problem. To do so, we propose a regularization approach based on IB called Entropy Penalty, that reduces the model's dependence on spurious features-- features corresponding to such spurious correlations. This allows deep networks trained with Entropy Penalty to generalize well even under distribution shift of spurious features. As a controlled test-bed for evaluating our claim, we train deep networks with Entropy Penalty on a colored MNIST (C-MNIST) dataset and show that it is able to generalize well on vanilla MNIST, MNIST-M and SVHN datasets in addition to an OOD version of C-MNIST itself. The baseline regularization methods we compare against fail to generalize on this test-bed.
Model-free Learning Control of Nonlinear Stochastic Systems with Stability Guarantee
Authors: Minghao Han, Yuan Tian, Lixian Zhang, Jun Wang, Wei Pan
Date: 2019-09-25 00:00:00
Tasks: continuous-control, Continuous Control, Open-Ended Question Answering, Reinforcement Learning (RL)

Reinforcement learning (RL) offers a principled way to achieve the optimal cumulative performance index in discrete-time nonlinear stochastic systems, which are modeled as Markov decision processes. Its integration with deep learning techniques has promoted the field of deep RL with an impressive performance in complicated continuous control tasks. However, from a control-theoretic perspective, the first and most important property of a system to be guaranteed is stability. Unfortunately, stability is rarely assured in RL and remains an open question. In this paper, we propose a stability guaranteed RL framework which simultaneously learns a Lyapunov function along with the controller or policy, both of which are parameterized by deep neural networks, by borrowing the concept of Lyapunov function from control theory. Our framework can not only offer comparable or superior control performance over state-of-the-art RL algorithms, but also construct a Lyapunov function to validate the closed-loop stability. In the simulated experiments, our approach is evaluated on several well-known examples including classic CartPole balancing, 3-dimensional robot control and control of synthetic biology gene regulatory networks. Compared with RL algorithms without stability guarantee, our approach can enable the system to recover to the operating point when interfered by uncertainties such as unseen disturbances and system parametric variations to a certain extent.
On the Pareto Efficiency of Quantized CNN
Authors: Ting-Wu Chin, Pierce I-Jen Chuang, Vikas Chandra, Diana Marculescu
Date: 2019-09-25 00:00:00
Tasks: Gesture Recognition, Quantization, Scene Understanding, Semantic Segmentation

Weight Quantization for deep convolutional neural networks (CNNs) has shown promising results in compressing and accelerating CNN-powered applications such as semantic segmentation, gesture recognition, and scene understanding. Prior art has shown that different datasets, tasks, and network architectures admit different iso-accurate precision values, which increase the complexity of efficient quantized neural network implementations from both hardware and software perspectives. In this work, we show that when the number of channels is allowed to vary in an iso-model size scenario, lower precision values Pareto dominate higher precision ones (in accuracy vs. model size) for networks with standard convolutions. Relying on comprehensive empirical analyses, we find that the Pareto optimal precision value of a convolution layer depends on the number of input channels per output filters and provide theoretical insights for it. To this end, we develop a simple algorithm to select the precision values for CNNs that outperforms corresponding 8-bit quantized networks by 0.9% and 2.2% in top-1 accuracy on ImageNet for ResNet50 and MobileNetV2, respectively.
Recurrent Neural Networks are Universal Filters
Authors: Wenjie Xu, Xiuqiong Chen, Stephen S.-T. Yau
Date: 2019-09-25 00:00:00
Tasks: Time Series Analysis

Recurrent neural networks (RNN) are powerful time series modeling tools in ma-
chine learning. It has been successfully applied in a variety of fields such as natural
language processing (Mikolov et al. (2010), Graves et al. (2013), Du et al. (2015)),
control (Fei & Lu (2017)) and traffic forecasting (Ma et al. (2015)), etc. In those
application scenarios, RNN can be viewed as implicitly modelling a stochastic dy-
namic system. Another type of popular neural network, deep (feed-forward) neural
network has also been successfully applied in different engineering disciplines,
whose approximation capability has been well characterized by universal approxi-
mation theorem (Hornik et al. (1989), Park & Sandberg (1991), Lu et al. (2017)).
However, the underlying approximation capability of RNN has not been fully
understood in a quantitative way. In our paper, we consider a stochastic dynamic
system with noisy observations and analyze the approximation capability of RNN
in synthesizing the optimal state estimator, namely optimal filter. We unify the
recurrent neural network into Bayesian filtering framework and show that recurrent
neural network is a universal approximator of optimal finite dimensional filters
under some mild conditions. That is to say, for any stochastic dynamic systems
with noisy sequential observations that satisfy some mild conditions, we show that
(informal)
∀ > 0, ∃ RNN-based filter, s.t. lim sup x̂ k|k − E[x k |Y k ] < ,
k→∞
where x̂ k|k is RNN-based filter’s estimate of state x k at step k conditioned on
the observation history and E[x k |Y k ] is the conditional mean of x k , known as the
optimal estimate of the state in minimum mean square error sense. As an interesting
special case, the widely used Kalman filter (KF) can be synthesized by RNN.
Soft Token Matching for Interpretable Low-Resource Classification
Authors: Federico Errica, Fabrizio Silvestri, Bora Edizel, Sebastian Riedel, Ludovic Denoyer, Vassilis Plachouras
Date: 2019-09-25 00:00:00
Tasks: Classification, text-classification, Text Classification

We propose a model to tackle classification tasks in the presence of very little training data. To this aim, we introduce a novel matching mechanism to focus on elements of the input by using vectors that represent semantically meaningful concepts for the task at hand.
By leveraging highlighted portions of the training data, a simple, yet effective, error boosting technique guides the learning process. In practice, it increases the error associated to relevant parts of the input by a given factor. Results on text classification tasks confirm the benefits of the proposed approach in both balanced and unbalanced cases, thus being of practical use when labeling new examples is expensive. In addition, the model is interpretable, as it allows for human inspection of the learned weights.
A Mention-Pair Model of Annotation with Nonparametric User Communities
Authors: Silviu Paun, Juntao Yu, Jon Chamberlain, Udo Kruschwitz, Massimo Poesio
Date: 2019-09-25 00:00:00

The availability of large datasets is essential for progress in coreference and other areas of NLP. Crowdsourcing has proven a viable alternative to expert annotation, offering similar quality for better scalability. However, crowdsourcing require adjudication, and most models of annotation focus on classification tasks where the set of classes is predetermined. This restriction does not apply to anaphoric annotation, where coders relate markables to coreference chains whose number cannot be predefined. This gap was recently covered with the introduction of a mention pair model of anaphoric annotation (MPA). In this work we extend MPA to alleviate the effects of sparsity inherent in some crowdsourcing environments. Specifically, we use a nonparametric partially pooled structure (based on a stick breaking process), fitting jointly with the ability of the annotators hierarchical community profiles. The individual estimates can thus be improved using information about the community when the data is scarce. We show, using a recently published large-scale crowdsourced anaphora dataset, that the proposed model performs better than its unpooled counterpart in conditions of sparsity, and on par when enough observations are available. The model is thus more resilient to different crowdsourcing setups, and, further provides insights into the community of workers. The model is also flexible enough to be used in standard annotation tasks for classification where it registers on par performance with the state of the art.
iWGAN: an Autoencoder WGAN for Inference
Authors: Yao Chen, Qingyi Gao, Xiao Wang
Date: 2019-09-25 00:00:00

Generative Adversarial Networks (GANs) have been impactful on many problems and applications but suffer from unstable training. Wasserstein GAN (WGAN) leverages the Wasserstein distance to avoid the caveats in the minmax two-player training of GANs but has other defects such as mode collapse and lack of metric to detect the convergence. We introduce a novel inference WGAN (iWGAN) model, which is a principled framework to fuse auto-encoders and WGANs. The iWGAN jointly learns an encoder network and a generative network using an iterative primal dual optimization process. We establish the generalization error bound of iWGANs. We further provide a rigorous probabilistic interpretation of our model under the framework of maximum likelihood estimation. The iWGAN, with a clear stopping criteria, has many advantages over other autoencoder GANs. The empirical experiments show that our model greatly mitigates the symptom of mode collapse, speeds up the convergence, and is able to provide a measurement of quality check for each individual sample. We illustrate the ability of iWGANs by obtaining a competitive and stable performance with state-of-the-art for benchmark datasets.
Characterize and Transfer Attention in Graph Neural Networks
Authors: Mufei Li, Hao Zhang, Xingjian Shi, Minjie Wang, Yixing Guan, Zheng Zhang
Date: 2019-09-25 00:00:00
Tasks: Transductive Learning

Does attention matter and, if so, when and how? Our study on both inductive and transductive learning suggests that datasets have a strong influence on the effects of attention in graph neural networks. Independent of learning setting, task and attention variant, attention mostly degenerate to simple averaging for all three citation networks, whereas they behave strikingly different in the protein-protein interaction networks and molecular graphs: nodes attend to different neighbors per head and get more focused in deeper layers. Consequently, attention distributions become telltale features of the datasets themselves. We further explore the possibility of transferring attention for graph sparsification and show that, when applicable, attention-based sparsification retains enough information to obtain good performance while reducing computational and storage costs. Finally, we point out several possible directions for further study and transfer of attention.