pwc-archive/papers-with-abstracts huggingface dataset:
Deconstructing the Regularization of BatchNorm
Authors: Yann Dauphin, Ekin Dogus Cubuk
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

Batch normalization (BatchNorm) has become a standard technique in deep learning. Its popularity is in no small part due to its often positive effect on generalization. Despite this success, the regularization effect of the technique is still poorly understood. This study aims to decompose BatchNorm into separate mechanisms that are much simpler. We identify three effects of BatchNorm and assess their impact directly with ablations and interventions. We find that preventing explosive growth at the final layer at initialization and during training explains most of BatchNorm's generalization boost. This regularization mechanism can lift accuracy by $2.9\%$ for Resnet-50 on Imagenet without BatchNorm. We show it is linked to other methods like Dropout and recent initializations like Fixup. Surprisingly, this simple mechanism matches the improvement of $0.8\%$ of the more complex Dropout regularization for the state-of-the-art Efficientnet-B8 model on Imagenet. This demonstrates the underrated effectiveness of simple regularizations and sheds light on directions to further improve generalization for deep nets.
Amending Mistakes Post-hoc in Deep Networks by Leveraging Class Hierarchies
Authors: Shyamgopal Karthik, Ameya Prabhu, Puneet K. Dokania, Vineet Gandhi
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

There has been increasing interest in building deep hierarchy-aware classifiers, aiming to quantify and reduce the severity of mistakes and not just count the number of errors. The idea is to exploit the label hierarchy (e.g., WordNet ontology) and consider graph distances as a proxy for mistake severity. Surprisingly, on examining mistake-severity distributions of the top-1 prediction, we find that current state-of-the-art hierarchy-aware deep classifiers do not show practical improvement in making better mistakes than the standard cross-entropy baseline. In fact, they reduce the average mistake-severity metric by largely making additional low-severity or easily avoidable mistakes. This might explain the noticeable accuracy drop. 
To this end, we resort to the classical Conditional Risk Minimization (CRM) framework for hierarchy aware classification. Given a cost matrix and a reliable estimate of likelihoods (obtained from a trained network), CRM simply amends mistakes at inference time; it needs no extra parameters; it requires adding just one line of code to the standard cross-entropy baseline. It significantly outperforms the state-of-the-art and consistently obtains large reductions in the average hierarchical distance of top-k predictions across datasets, with very little loss in accuracy. Since CRM does not require retraining or fine-tuning of any hyperparameter, it can be used with any off-the-shelf cross-entropy trained model.
Learning a Latent Search Space for Routing Problems using Variational Autoencoders
Authors: André Hottung, Bhanu Bhandari, Kevin Tierney
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

Methods for automatically learning to solve routing problems are rapidly improving in performance. While most of these methods excel at generating solutions quickly, they are unable to effectively utilize longer run times because they lack a sophisticated search component. We present a learning-based optimization approach that allows a guided search in the distribution of high-quality solutions for a problem instance. More precisely, our method uses a conditional variational autoencoder that learns to map points in a continuous (latent) search space to high-quality, instance-specific routing problem solutions. The learned space can then be searched by any unconstrained continuous optimization method. We show that even using a standard differential evolution search strategy our approach is able to outperform existing purely machine learning based approaches.
Energy-based Out-of-distribution Detection for Multi-label Classification
Authors: Haoran Wang, Weitang Liu, Alex Bocchieri, Yixuan Li
Date: 2021-01-01 00:00:00
Tasks: Classification, General Classification, Multi-class Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Out-of-Distribution Detection, Out of Distribution (OOD) Detection

Out-of-distribution (OOD) detection is essential to prevent anomalous inputs from causing a model to fail during deployment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose SumEnergy, a simple and effective method, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. We show that SumEnergy can be mathematically interpreted from a joint likelihood perspective. Our results show consistent improvement over previous methods that are based on the maximum-valued scores, which fail to capture joint information from multiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MS-COCO, PASCAL-VOC, and NUS-WIDE. We show that SumEnergy reduces the FPR95 by up to 10.05% compared to the previous best baseline, establishing state-of-the-art performance.
Approximating Pareto Frontier through Bayesian-optimization-directed Robust Multi-objective Reinforcement Learning
Authors: Xiangkun He, Jianye Hao, Dong Li, Bin Wang, Wulong Liu
Date: 2021-01-01 00:00:00
Tasks: Bayesian Optimization, Diversity, Multi-Objective Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Many real-word decision or control problems involve multiple conflicting objectives and uncertainties, which requires learned policies are not only Pareto optimal but also robust. In this paper, we proposed a novel algorithm to approximate a representation for robust Pareto frontier through Bayesian-optimization-directed robust multi-objective reinforcement learning (BRMORL). Firstly, environmental uncertainty is modeled as an adversarial agent over the entire space of preferences by incorporating zero-sum game into multi-objective reinforcement learning (MORL). Secondly, a comprehensive metric based on hypervolume and information entropy is presented to evaluate convergence, diversity and evenness of the distribution for Pareto solutions. Thirdly, the agent’s learning process is regarded as a black-box, and the comprehensive metric we proposed is computed after each episode of training, then a Bayesian optimization (BO) algorithm is adopted to guide the agent to evolve towards improving the quality of the approximated Pareto frontier. Finally, we demonstrate the effectiveness of proposed approach on challenging high-dimensional multi-objective tasks across several environments, and show our scheme can produce robust policies under environmental uncertainty.
Studying relationship between geometry of decision boundaries with network complexity for robustness analysis
Authors: Anonymous
Date: 2021-01-01 00:00:00

Deep Neural networks are susceptible to adversarial attacks: if inputs are perturbed in a specific manner, it can result in misclassification. However, recent studies have shown that the robustness of the network has certain connection with the complexity of the architecture. In this paper, we investigate the distinctive influence of additional convolutional layers at the decision boundaries of several DNN architectures. To generate adversarial examples, we have utilized two different attack mechanisms, Fast Gradient Method (FGM) and One Step Spectral Attack (OSSA) using common datasets like MNIST, Fashion MNIST and Cifar 10. We have investigated the aftermaths of adding layers to the robustness of the architecture. For reasoning, we have analyzed separation width from linear class partitions and local geometry (curvature) near the decision boundary. The result shows that model complexity has significant roles in changing distances, as well as, the local features of decision boundary, which impacts the robustness of the network.
Extracting Strong Policies for Robotics Tasks from zero-order trajectory optimizers
Authors: Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Georg Martius
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Model-based Reinforcement Learning

Solving high-dimensional, continuous robotic tasks is a challenging optimization problem. Model-based methods that rely on zero-order optimizers like the cross-entropy method (CEM) have so far shown strong performance and are considered state-of-the-art in the model-based reinforcement learning community. However, this success comes at the cost of high computational complexity, being therefore not suitable for real-time control. In this paper, we propose a technique to jointly optimize the trajectory and distill a policy, which is essential for fast execution in real robotic systems. Our method builds upon standard approaches, like guidance cost and dataset aggregation, and introduces a novel adaptive factor which prevents the optimizer from collapsing to the learner's behavior at the beginning of the training. The extracted policies reach unprecedented performance on challenging tasks as making a humanoid stand up and opening a door without reward shaping
Quantifying Task Complexity Through Generalized Information Measures
Authors: Aditya Chattopadhyay, Benjamin David Haeffele, Donald Geman, Rene Vidal
Date: 2021-01-01 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification

How can we measure the “complexity” of a learning task so that we can compare one task to another? From classical information theory, we know that entropy is a useful measure of the complexity of a random variable and provides a lower bound on the minimum expected number of bits needed for transmitting its state. In this paper, we propose to measure the complexity of a learning task by the minimum expected number of questions that need to be answered to solve the task. For example, the minimum expected number of patches that need to be observed to classify FashionMNIST images. We prove several properties of the proposed complexity measure, including connections with classical entropy and sub-additivity for multiple tasks. As the computation of the minimum expected number of questions is generally intractable, we propose a greedy procedure called “information pursuit” (IP), which selects one question at a time depending on previous questions and their answers. This requires learning a probabilistic generative model relating data and questions to the task, for which we employ variational autoencoders and normalizing flows. We illustrate the usefulness of the proposed measure on various binary image classification tasks using image patches as the query set. Our results indicate that the complexity of a classification task increases as signal-to-noise ratio decreases, and that classification of the KMNIST dataset is more complex than classification of the FashionMNIST dataset. As a byproduct of choosing patches as queries, our approach also provides a principled way of determining which pixels in an image are most informative for a task.
Block Skim Transformer for Efficient Question Answering
Authors: Yue Guan, Jingwen Leng, Yuhao Zhu, Minyi Guo
Date: 2021-01-01 00:00:00
Tasks: Language Modeling, Language Modelling, Model Compression, Question Answering

  Transformer based encoder models have achieved promising results on natural language processing (NLP) task including question answering (QA).  Different from sequence classification or language modeling tasks, hidden states at all position are used for the final classification in QA.  However, we do not always need all the context to answer the raised question.  Following this idea, we proposed Block Skim Transformer (BST) to improve and accelerate the processing of transformer QA models.  The key idea of BST is to identify the context that must be further processed and the blocks that could be safely rejected early on during inference. Critically, we learn such information from self attention weights.  As a result, the model hidden states are pruned at sequence dimension, achieving significant inference speedup.  We also show that such extra training optimization objection also improves model performance.  As an plugin to the transformer based QA models, BST is compatible to other model compression methods without changing existing network architectures.  BST improves QA models performance on different datasets and achieves $1.6\times$ speedup on $\text{BERT}_{\text{large}}$ model.
Fewmatch: Dynamic Prototype Refinement for Semi-Supervised Few-Shot Learning
Authors: Xu Lan, Steven McDonagh, Shaogang Gong, Jiali Wang, Zhenguo Li, Sarah Parisot
Date: 2021-01-01 00:00:00
Tasks: Few-Shot Learning, Pseudo Label

Semi-Supervised Few-shot Learning (SS-FSL) investigates the benefit of incorporating unlabelled data in few-shot settings. Recent work has relied on the popular Semi-Supervised Learning (SSL) concept of iterative pseudo-labelling, yet often yield models that are susceptible to error propagation and are sensitive to initialisation. Alternative work utilises the concept of consistency regularisation (CR), a popular SSL state of the art technique where a student model is trained to consistently agree with teacher predictions under different input perturbations, without pseudo-label requirements. However, applications of CR to the SS-FSL set-up struggle to outperform pseudo-labelling approaches; limited available training data yields unreliable early stage predictions and requires fast convergence that is not amenable for, typically slower to converge, CR approaches. In this paper, we introduce a prototype-based approach for SS-FSL that exploits model consistency in a robust manner. Our Dynamic Prototype Refinement (DPR) approach is a novel training paradigm for few-shot model adaptation to new unseen classes, combining concepts from metric and meta-gradient based FSL methods. New class prototypes are alternatively refined 1) explicitly, using labelled and unlabelled data with high confidence class predictions and 2) implicitly, by model fine-tuning using a data selective CR loss. DPR affords CR convergence, with the explicit refinement providing an increasingly stronger initialisation. We demonstrate method efficacy and report extensive experiments on two competitive benchmarks; miniImageNet and tieredImageNet. The ability to effectively utilise and combine information from both labelled base-class and auxiliary unlabelled novel-class data results in significant accuracy improvements.
Learning Self-Similarity in Space and Time as a Generalized Motion for Action Recognition
Authors: Heeseung Kwon, Manjin Kim, Suha Kwak, Minsu Cho
Date: 2021-01-01 00:00:00
Tasks: Action Recognition, Video Understanding

Spatio-temporal convolution often fails to learn motion dynamics in videos and thus an effective motion representation is required for video understanding in the wild. In this paper, we propose a rich and robust motion representation method based on spatio-temporal self-similarity (STSS). Given a sequence of frames, STSS represents each local region as similarities to its neighbors in space and time. By converting appearance features into relational values, it enables the learner to better recognize structural patterns in space and time. We leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it. 
The proposed method is implemented as a neural block, dubbed SELFY, that can be easily inserted into neural architectures and learned end-to-end without additional supervision. With a sufficient volume of the neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition.
Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolution. On the standard action recognition benchmarks, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results.
A New Variant of Stochastic Heavy ball Optimization Method for Deep Learning
Authors: Zhou Shao, Tong Lin
Date: 2021-01-01 00:00:00
Tasks: Deep Learning

Stochastic momentum optimization methods, also known as stochastic heavy ball (SHB) methods, are one of the most popular optimization methods for deep learning. These methods can help accelerate stochastic gradient descent and dampen oscillations. In this paper we provide a new variant of the stochastic heavy ball method, called stochastic Euler’s heavy ball (SEHB). The proposed SEHB method modifies the steepest descent direction to achieve acceleration, and combines Euler‘s method to adaptively adjust learning rates as well. A convergence analysis of the regret bound is discussed under the online convex optimization framework. Furthermore, we conduct experiments on various popular datasets and deep learning models. Empirical results demonstrate that our SEHB method shows comparable or even better generalization performance than state-of-the-art optimization methods such as SGD and Adam.
A Probabilistic Approach to Constrained Deep Clustering
Authors: Laura Manduchi, Kieran Chin-Cheong, Holger Michel, Sven Wellmann, Julia E Vogt
Date: 2021-01-01 00:00:00
Tasks: Clustering, Constrained Clustering, Deep Clustering, Image Generation

Clustering with constraints has gained significant attention in the field of semi-supervised machine learning as it can leverage partial prior information on a growing amount of unlabelled data. Following recent advances in deep generative models, we derive a novel probabilistic approach to constrained clustering that can be trained efficiently in the framework of stochastic gradient variational Bayes. In contrast to existing approaches, our model (CVaDE) uncovers the underlying distribution of the data conditioned on prior clustering preferences, expressed as pairwise constraints. The inclusion of such constraints allows the user to guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong to the same class. We provide extensive experiments to demonstrate that CVaDE shows superior clustering performances and robustness compared to state-of-the-art deep constrained clustering methods in a variety of data sets. We further demonstrate the usefulness of our approach on challenging real-world medical applications and face image generation.
Bayesian Metric Learning for Robust Training of Deep Models under Noisy Labels
Authors: Toan Tran, Hieu Vu, Gustavo Carneiro, Hung Bui
Date: 2021-01-01 00:00:00
Tasks: General Classification, Metric Learning, Triplet, Variational Inference

Label noise is a natural event of data collection and annotation and has been shown to have significant impact on the performance of deep learning models regarding accuracy reduction and sample complexity increase. This paper aims to develop a novel theoretically sound Bayesian deep metric learning that is robust against noisy labels. 
Our proposed approach is inspired by a linear Bayesian large margin nearest neighbor classification, and is a combination of Bayesian learning, triplet loss-based deep metric learning and variational inference frameworks. We theoretically show the robustness under label noise of our proposed method. The experimental results on benchmark data sets that contain both simulated and realistic label noise show a considerable improvement in the classification accuracy of our method compared to the linear Bayesian metric learning and the point estimate deep metric learning.
Incremental Policy Gradients for Online Reinforcement Learning Control
Authors: Kristopher De Asis, Alan Chan, Yi Wan, Richard S. Sutton
Date: 2021-01-01 00:00:00
Tasks: Policy Gradient Methods, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Policy gradient methods are built on the policy gradient theorem, which involves a term representing the complete sum of rewards into the future: the return. Due to this, one usually either waits until the end of an episode before performing updates, or learns an estimate of this return--a so-called critic. Our emphasis is on the first approach in this work, detailing an incremental policy gradient update which neither waits until the end of the episode, nor relies on learning estimates of the return. We provide on-policy and off-policy variants of our algorithm, for both the discounted return and average reward settings. Theoretically, we draw a connection between the traces our methods use and the stationary distributions of the discounted and average reward settings. We conclude with an experimental evaluation of our methods on both simple-to-understand and complex domains.
Ask Question with Double Hints: Visual Question Generation with Answer-awareness and Region-reference
Authors: Shen Kai, Lingfei Wu, Siliang Tang, Fangli Xu, Zhu Zhang, Yu Qiang, Yueting Zhuang
Date: 2021-01-01 00:00:00
Tasks: Graph-to-Sequence, Question Generation, Question-Generation

The task of visual question generation~(VQG) aims to generate human-like questions from an image and potentially other side information (e.g. answer type or the answer itself). Despite promising results have been achieved, previous works on VQG either i) suffer from one image to many questions mapping problem rendering the failure of generating referential and meaningful questions from an image, or ii) ignore rich correlations among the visual objects in an image and potential interactions between the side information and image. To address these limitations, we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. In particular, we aim to ask the right visual questions with \emph{Double Hints - textual answers and visual regions of interests}, effectively mitigating the existing one-to-many mapping issue. To this end, we develop a simple methodology to self-learn the visual hints without introducing any additional human annotations. Furthermore, to capture these sophisticated relationships, we propose a new double-hints guided Graph-to-Sequence learning framework that first models them as a dynamic graph and learns the implicit topology end-to-end, and then utilize a graph-to-sequence model to generate the questions with double hints. Our experiments on VQA2.0 and COCO-QA datasets demonstrate that our proposed model on this new setting can significantly outperform existing state-of-the-art baselines by a large margin.
Faster Training of Word Embeddings
Authors: Eliza Wszola, Martin Jaggi, Markus Püschel
Date: 2021-01-01 00:00:00
Tasks: Word Embeddings

Word embeddings have gained increasing popularity in the recent years due to the Word2vec library and its extension fastText that uses subword information. In this paper, we aim at improving the execution speed of fastText training on homogeneous multi- and manycore CPUs while maintaining accuracy. We present a novel open-source implementation that flexibly incorporates various algorithmic variants including negative sample sharing, batched updates, and a byte-pair encoding-based alternative for subword units. We build these novel variants over a fastText implementation that we carefully optimized for the architecture, memory hierarchy, and parallelism of current manycore CPUs. Our experiments on three languages demonstrate 3-20x speed-up in training time at competitive semantic and syntactic accuracy.