pwc-archive/papers-with-abstracts huggingface dataset:
Meta-Continual Learning Via Dynamic Programming
Authors: Krishnan Raghavan, Prasanna Balaprakash
Date: 2021-01-01 00:00:00
Tasks: Continual Learning

Meta continual learning algorithms seek to train a model when faced with similar tasks observed in a sequential manner. Despite promising methodological advancements, there is a lack of theoretical frameworks that enable analysis of learning challenges such as generalization and catastrophic forgetting. To that end, we develop a new theoretical approach for meta continual learning~(MCL) where we mathematically model the learning dynamics using dynamic programming, and we establish conditions of optimality for the MCL problem.  Moreover, using the theoretical framework, we derive a new dynamic-programming-based MCL method that adopts stochastic-gradient-driven alternating optimization to balance generalization and catastrophic forgetting. We show that, on MCL benchmark data sets, our theoretically grounded method achieves accuracy better than or comparable to that of existing state-of-the-art methods.
Physics-Informed Deep Learning of Incompressible Fluid Dynamics
Authors: Nils Wandel, Michael Weinmann, Reinhard Klein
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Deep Learning

Fast and stable fluid simulations are an essential prerequisite for applications ranging from computer generated imagery to computer aided design in research and development. Solving the partial differential equations governed by the dynamics of incompressible fluids, however, is a challenging task and traditional numerical approximation schemes come at high computational costs. Recent deep learning based approaches promise vast speed-ups but either do not generalize to new domain geometries or rely on complex pipelines that outsource major parts of the fluid simulation to traditional methods.

In this work, we propose a novel unsupervised training framework that allows powerful convolutional neural networks to learn the entire update step of mapping a fluid state from time-point t to a subsequent state at time t+dt. For this purpose, we introduce a physics informed loss function that penalizes residuals of the incompressible Navier Stokes equations on a staggered grid. This greatly simplifies the pipeline to train and evaluate neural fluid models. After training, the framework yields models that are capable of fast fluid simulations and can handle various fluid phenomena including the Magnus effect and Kármán vortex streets. We present an interactive real-time demo and show that trained models can generalize to new domain geometries unseen during training. Moreover, the trained neural networks offer a differentiable update step to advance the fluid simulation in time and, thus, can be used as efficient differentiable fluid solvers. This can be exploited for optimal control tasks as we demonstrate in a proof-of-concept experiment. Our models significantly outperform a recent differentiable fluid solver in terms of computational speed and accuracy.
Wasserstein Distributionally Robust Optimization: A Three-Player Game Framework
Authors: Zhuozhuo Tu, Shan You, Tao Huang, DaCheng Tao
Date: 2021-01-01 00:00:00

Wasserstein distributionally robust optimization (DRO) has recently received significant attention in machine learning due to its connection to generalization, robustness and regularization. Existing methods only consider a limited class of loss functions or apply to small values of robustness. In this paper, we present a three-player game framework for solving Wasserstein DRO problem with arbitrary level of robustness, which can handle general loss functions. Specifically, we formulate a min-max game between three players who optimize over probability measures, model parameters and Lagrange multipliers. We also propose new algorithms for finding an equilibrium of the game in convex and non-convex settings which both enjoy provable convergence guarantees. Furthermore, we prove an excess risk bound for the proposed algorithms which shows that the solution returned by the algorithms closely achieves the optimal minimax risk.
Convergent Adaptive Gradient Methods in Decentralized Optimization
Authors: Xiangyi Chen, Belhal Karimi, Weijie Zhao, Ping Li
Date: 2021-01-01 00:00:00
Tasks: Distributed Optimization

Adaptive gradient methods including Adam, AdaGrad, and their variants have been very successful for training deep learning models, such as neural networks, in the past few years. Meanwhile, given the need for distributed training procedures, distributed optimization algorithms are at the center of attention. With the growth of computing power and the need for using machine learning models on mobile devices, the communication cost of distributed training algorithms needs careful consideration. In that regard, more and more attention is shifted from the traditional parameter server training paradigm to the decentralized one, which usually requires lower communication costs. In this paper, we rigorously incorporate adaptive gradient methods into decentralized training procedures and introduce novel convergent decentralized adaptive gradient methods. Specifically, we propose a general algorithmic framework that can convert existing adaptive gradient methods to their decentralized counterparts. In addition, we thoroughly analyze the convergence behavior of the proposed algorithmic framework and show that if a given adaptive gradient method converges, under some specific conditions, then its decentralized counterpart is also convergent.
Efficient randomized smoothing by denoising with learned score function
Authors: Kyungmin Lee, Seyoon Oh
Date: 2021-01-01 00:00:00
Tasks: Denoising, Image Denoising

The randomized smoothing with various noise distributions is a promising approach to protect classifiers from $\ell_p$ adversarial attacks. However, it requires an ensemble of classifiers trained with different noise types and magnitudes, which is computationally expensive. In this work, we present an efficient method for randomized smoothing that does not require any re-training of classifiers. We built upon denoised smoothing, which prepends denoiser to the pre-trained classifier. We investigate two approaches to the image denoising problem for randomized smoothing and show that using the score function suits for both. Moreover, we present an efficient algorithm that can scale to randomized smoothing and can be applied regardless of noise types or levels. To validate, we demonstrate the effectiveness of our methods through extensive experiments on CIFAR-10 and ImageNet, under various $\ell_p$ adversaries.
Using Synthetic Data to Improve the Long-range Forecasting of Time Series Data
Authors: Shiyu Liu, Mehul Motani
Date: 2021-01-01 00:00:00
Tasks: Clustering, Time Series, Time Series Analysis

Effective long-range forecasting of time series data remains an unsolved and open problem. One possible approach is to use generative models to improve long-range forecasting, but the challenge then is how to generate high-quality synthetic data. In this paper, we propose a conditional Wasserstein GAN with Gradient and Error Penalty (cWGAN-GEP), aiming to generate accurate synthetic data that preserves the temporal dynamics between the conditioning input and generated data. By using such synthetic data, we develop a long-range forecasting method called Generative Forecasting (GenF). GenF consists of three key components: (i) a cWGAN-GEP based generator, to generate synthetic data for next few time steps. (ii) a predictor which makes long-range predictions based on generated and observed data. (iii) an information theoretic clustering (ITC) algorithm to better train the cWGAN-GEP based generator and the predictor. Our experimental results on three public datasets demonstrate that GenF significantly outperforms a diverse range of state-of-the-art benchmarks and classical approaches. In most cases, we find an improvement of at least 10% over all studied methods. Lastly, we conduct an ablation study to demonstrate the effectiveness of the cWGAN-GEP and the ITC algorithm.
Task-Agnostic and Adaptive-Size BERT Compression
Authors: Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Li Jian, Tao Qin, Tie-Yan Liu
Date: 2021-01-01 00:00:00
Tasks: Language Modelling, Model Compression, Neural Architecture Search

While pre-trained language models such as BERT and RoBERTa have achieved impressive results on various natural language processing tasks, they have huge numbers of parameters and suffer from huge computational and memory costs, which make them difficult for real-world deployment. Hence, model compression should be performed in order to reduce the computation and memory cost of pre-trained models. In this work, we aim to compress BERT and address the following two challenging practical issues: (1) The compression algorithm should be able to output multiple compressed models with different sizes and latencies, so as to support devices with different kinds of memory and latency limitations; (2) the algorithm should be downstream task agnostic, so that the compressed models are generally applicable for different downstream tasks.  We leverage techniques in neural architecture search (NAS) and propose NAS-BERT, an efficient method for BERT compression. NAS-BERT trains a big supernet on a carefully designed search space containing various architectures and outputs multiple compressed models with adaptive sizes and latency. Furthermore, the training of NAS-BERT is conducted on standard self-supervised pre-training tasks (e.g., masked language model) and does not depend on specific downstream tasks. Thus, the models it produces can be used across various downstream tasks. The technical challenge of NAS-BERT is that training a big supernet on the pre-training task is extremely costly. We employ several techniques including block-wise search, search space pruning, and performance approximation to improve search efficiency and accuracy. Extensive experiments on GLUE benchmark datasets demonstrate that NAS-BERT can find lightweight models with better accuracy than previous approaches, and can be directly applied to different downstream tasks with adaptive model sizes for different requirements of memory or latency.
Recurrently Controlling a Recurrent Network with Recurrent Networks Controlled by More Recurrent Networks
Authors: Yi Tay, Yikang Shen, Alvin Chan, Aston Zhang, Shuai Zhang
Date: 2021-01-01 00:00:00
Tasks: Code Generation, Inductive Bias, Machine Translation, Music Modeling, Semantic Parsing, Translation

This paper explores an intriguing idea of recursively parameterizing recurrent nets. Simply speaking, this refers to recurrently controlling a recurrent network with recurrent networks controlled by recurrent networks. The proposed architecture recursively parameterizes its gating functions whereby gating mechanisms of X-RNN are controlled by instances of itself, which are repeatedly called in a recursive fashion. We postulate that our proposed inductive bias provides modeling benefits pertaining to learning with inherently hierarchically-structured sequence data. To this end, we conduct extensive experiments on recursive logic tasks (sorting, tree traversal, logical inference), sequential pixel-by-pixel classification, semantic parsing, code generation, machine translation and polyphonic music modeling, demonstrating the widespread utility of the proposed approach, i.e., achieving optimistic and competitive results on all tasks.
Predictive Uncertainty in Deep Object Detectors: Estimation and Evaluation
Authors: Ali Harakeh, Steven L. Waslander
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Object, object-detection, Object Detection, regression, scoring rule

Predictive uncertainty estimation is an essential next step for the reliable deployment of deep object detectors in safety-critical tasks. In this work, we focus on estimating predictive distributions for bounding box regression output with variance networks. We show that in the context of object detection, training variance networks with negative log likelihood (NLL) can lead to high entropy predictive distributions regardless of the correctness of the output mean. We propose to use the energy score as a non-local proper scoring rule and find that when used for training, the energy score leads to better calibrated and lower entropy predictive distributions than NLL. We also address the widespread use of non-proper scoring metrics for evaluating predictive distributions from deep object detectors by proposing an alternate evaluation approach founded on proper scoring rules. Using the proposed evaluation tools, we show that although variance networks can be used to produce high quality predictive distributions, ad-hoc approaches used by seminal object detectors for choosing regression targets during training do not provide wide enough data support for reliable variance learning. We hope that our work helps shift evaluation in probabilistic object detection to better align with predictive uncertainty evaluation in other machine learning domains.
Learning Efficient Planning-based Rewards for Imitation Learning
Authors: Xingrui Yu, Yueming Lyu, Ivor Tsang
Date: 2021-01-01 00:00:00
Tasks: Atari Games, continuous-control, Continuous Control, Imitation Learning, Reinforcement Learning (RL)

Imitation learning from limited demonstrations is challenging. Most inverse reinforcement learning (IRL) methods are unable to perform as good as the demonstrator, especially in high-dimensional environment, e.g, the Atari domain. To address this challenge, we propose a novel reward learning method, which streamlines a differential planning module with dynamics modeling. Our method learns useful planning computations with a meaningful reward function that focuses on the resulting region of an agent executing an action. Such planning-based reward function leads to policies that generalize well to new tasks. Empirical results with multiple network architectures and reward instances show that our method can outperform state-of-the-art IRL methods on multiple Atari games and continuous control tasks. Our method achieves performance that is averagely 1,139.1% of the demonstration.
An Adversarial Attack via Feature Contributive Regions
Authors: Yaguan Qian, Jiamin Wang, Xiang Ling, Zhaoquan Gu, Bin Wang, Chunming Wu
Date: 2021-01-01 00:00:00
Tasks: Adversarial Attack

Recently, to deal with the vulnerability to generate examples of CNNs, there are many advanced algorithms that have been proposed. These algorithms focus on modifying global pixels directly with small perturbations, and some work involves modifying local pixels. However, the global attacks have the problem of perturbations’ redundancy and the local attacks are not effective. To overcome this challenge, we achieve a trade-off between the perturbation power and the number of perturbed pixels in this paper. The key idea is to find the feature contributive regions (FCRs) of the images. Furthermore, in order to create an adversarial example similar to the corresponding clean image as much as possible, we redefine a loss function as the objective function of the optimization in this paper and then using gradient descent optimization algorithm to find the efficient perturbations. Various experiments have been carried out on CIFAR-10 and ILSVRC2012 datasets, which show the excellence of this method, and in addition, the FCRs attack shows strong attack ability in both white-box and black-box settings.
On the Stability of Multi-branch Network
Authors: Huishuai Zhang, Da Yu, Wei Chen, Tie-Yan Liu
Date: 2021-01-01 00:00:00

Multi-branch architectures are widely used  in state-of-the-art neural networks. Their empirical success relies on some design wisdom, like adding normalization layers or/and scaling down the initialization.  In this paper, we investigate the multi-branch architecture from the stability perspective. Specifically, we establish the forward/backward stability of multi-branch network, which leads to several new findings. Our analysis shows that only scaling down the initialization may not be enough for training multi-branch network successfully because of the uncontrollable backward process. We also unveil a new role of the normalization layer in terms of stabilizing the multi-branch architectures. More importantly, we propose a new design ``STAM aggregation" that can guarantee to STAbilize the forward/backward process of Multi-branch networks irrespective of the number of branches. We demonstrate that with STAM aggregation, the same training strategy is applicable to models with different numbers of branches, which can reduce the hyper-parameter tuning burden. Our experiments verify our theoretical findings and also demonstrate that the STAM aggregation can improve the performance of multi-branch networks considerably.
More or Less: When and How to Build Neural Network Ensembles
Authors: Abdul Wasay, Stratos Idreos
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Navigate

Neural network models are applied to many tasks with rising complexity and data sizes. Researchers and practitioners seek to scale the representational power of these models by adding more parameters. Increasing parameters, though, requires additional critical resources such as memory and compute time leading to increased training and inference cost. Thus a consistent challenge is to obtain as high as possible accuracy within a parameter budget. As neural network designers navigate this complex landscape, they are guided by conventional wisdom. This comes from empirical studies of the design space. We identify a critical part of this design space that is not well-understood: That is how to decide between the alternatives of expanding a single network model or increasing the number of networks and using them together in an ensemble. We study this question in detail through extensive experimentation across various network architectures and data sets. Crucially, we provide a robust assessment, missing in previous studies, by fixing the number of parameters. We also consider a holistic set of metrics such as training time, inference time, and memory usage. Contrary to conventional wisdom, we show that when we perform a holistic and robust assessment, we uncover a wide design space, where ensembles not only provide better accuracy but also train faster, and deploy at a speed comparable to single convolutional networks with the same total number of parameters.
A Simple Unified Information Regularization Framework for Multi-Source Domain Adaptation
Authors: Geon Yeong Park, Sang Wan Lee
Date: 2021-01-01 00:00:00
Tasks: Computational Efficiency, Domain Adaptation, Unsupervised Domain Adaptation

Adversarial learning strategy has demonstrated remarkable performance in dealing with single-source unsupervised Domain Adaptation (DA) problems, and it has recently been applied to multi-source DA problems. A potential pitfall of the existing DA methods that use multiple domain discriminators for each source domain is that domain-discriminative information is inevitably distributed across multiple discriminators. Despite this issue, the effect of using multiple discriminators on the quality of latent space representations has been poorly understood. To fully address this issue, we situate adversarial DA in the context of information regularization. First, we present a unified information regularization framework for multi-source DA. Our framework shows that the information shared across domains cannot be gleaned with multiple discriminators. It further provides a theoretical justification for using a single and unified domain discriminator to encourage the synergistic integration of the information gleaned from each domain. Second, this motivates us to implement a novel neural architecture called a Multi-source Information-regularized Adaptation Networks (MIAN). The proposed model significantly reduces the variance of stochastic gradients and increases computational-efficiency. Large-scale simulations on various multi-source DA scenarios demonstrate that MIAN, despite its structural simplicity, reliably outperforms other state-of-the-art methods by a large margin. Our work offers the possibility of garnering fundamental insights from multiple domains into the development of highly generalizable algorithms.
Generalizing Tree Models for Improving Prediction Accuracy
Authors: Jaemin Yoo, Lee Sael
Date: 2021-01-01 00:00:00
Tasks: Prediction

Can we generalize and improve the representation power of tree models? Tree models are often favored over deep neural networks due to their interpretable structures in problems where the interpretability is required, such as in the classification of feature-based data where each feature is meaningful. However, most tree models have low accuracies and easily overfit to training data. In this work, we propose Decision Transformer Network (DTN), our highly accurate and interpretable tree model based on our generalized framework of tree models, decision transformers. Decision transformers allow us to describe tree models in the context of deep learning. Our DTN is proposed based on improving the generalizable components of the decision transformer, which increases the representation power of tree models while preserving the inherent interpretability of the tree structure. Our extensive experiments on 121 feature-based datasets show that DTN outperforms the state-of-the-art tree models and even deep neural networks.
Goal-Driven Imitation Learning from Observation by Inferring Goal Proximity
Authors: Andrew Szot, Youngwoon Lee, Shao-Hua Sun, Joseph J Lim
Date: 2021-01-01 00:00:00
Tasks: Imitation Learning

Humans can effectively learn to estimate how close they are to completing a desired task simply by watching others fulfill the task. To solve the task, they can then take actions towards states with higher estimated proximity to the goal. From this intuition, we propose a simple yet effective method for imitation learning that learns a goal proximity function from expert demonstrations and online agent experience, and then uses the learned proximity to provide a dense reward signal for training a policy to solve the task. By predicting task progress as the temporal distance to the goal, the goal proximity function improves generalization to unseen states over methods that aim to directly imitate expert behaviors. We demonstrate that our proposed method efficiently learns a set of goal-driven tasks from state-only demonstrations in navigation, robotic arm manipulation, and locomotion tasks.
Bypassing the Random Input Mixing in Mixup
Authors: Hongyu Guo
Date: 2021-01-01 00:00:00
Tasks: Descriptive

Mixup and its variants have promoted a surge of interest due to their capability of boosting the accuracy of deep  models. For a random sample pair, such approaches generate a set of synthetic samples through interpolating both the  inputs and their corresponding one-hot labels. Current methods either interpolate random features from an input pair or learn to mix salient features  from the  pair. Nevertheless, the former methods can create misleading synthetic    samples or remove important features from the given inputs, and the latter strategies incur significant computation cost for selecting descriptive input regions. In this paper, we show that the effort needed for the input mixing can be bypassed. For a given sample pair, averaging the features from the two inputs and then assigning it with a set of  soft labels can  effectively regularize the training. We empirically show that the proposed  approach performs on par with state-of-the-art strategies in terms of predictive accuracy.