pwc-archive/papers-with-abstracts huggingface dataset:
Uncertainty for deep image classifiers on out of distribution data.
Authors: Tiago Salvador, Alexander Iannantuono, Adam M Oberman
Date: 2021-01-01 00:00:00

In addition to  achieving high accuracy, in many applications, it is important to  estimate the probability that a model prediction is correct. Predictive uncertainty is particularly important on out of distribution (OOD) data where accuracy degrades.  However, models are typically overconfident, and model calibration on OOD data remains a challenge. In this paper we propose a simple post hoc calibration method that significantly  improves on benchmark results [Ovadia et al 2019] on a wide range of corrupted data.  Our method uses outlier exposure to properly calibrate the model probabilities.
On the Importance of Looking at the Manifold
Authors: Nil Adell Mill, Jannis Born, Nathaniel Park, James Hedrick, María Rodríguez Martínez, Matteo Manica
Date: 2021-01-01 00:00:00
Tasks: Representation Learning

Data rarely lies on uniquely Euclidean spaces. Even data typically represented in regular domains, such as images, can have a higher level of relational information, either between data samples or even relations within samples, e.g., how the objects in an image are linked. With this perspective our data points can be enriched by explicitly accounting for this connectivity and analyzing them as a graph. Herein, we analyze various approaches for unsupervised representation learning and investigate the importance of considering topological information and its impact when learning representations. We explore a spectrum of models, ranging from uniquely learning representations based on the isolated features of the nodes (focusing on Variational Autoencoders), to uniquely learning representations based on the topology (using node2vec) passing through models that integrate both node features and topological information in a hybrid fashion. For the latter we use Graph Neural Networks, precisely Deep Graph Infomax (DGI), and an extension of the typical formulation of the VAE where the topological structure is accounted for via an explicit regularization of the loss (Graph-Regularized VAEs, introduced in this work). To extensively investigate these methodologies, we consider a wide variety of data types: synthetic data point clouds, MNIST, citation networks, and chemical reactions. We show that each of the representations learned by these models may have critical importance for further downstream tasks, and that accounting for the topological features can greatly improve the modeling capabilities for certain problems. We further provide a framework to analyze these, and future models under different scenarios and types of data.
Pretrain-to-Finetune Adversarial Training via Sample-wise Randomized Smoothing
Authors: Lei Wang, Runtian Zhai, Di He, LiWei Wang, Li Jian
Date: 2021-01-01 00:00:00

Developing certified models that can provably defense adversarial perturbations is important in machine learning security. Recently, randomized smoothing, combined with other techniques (Cohen et al., 2019; Salman et al., 2019),  has been shown to be an effective method to certify models under $l_2$ perturbations.  Existing work for certifying $l_2$ perturbations added the same level of Gaussian noise to each sample. The noise level determines the trade-off between the test accuracy and the average certified robust radius. We propose to further improve the defense via sample-wise randomized smoothing, which assigns different noise levels to different samples. Specifically, we propose a pretrain-to-finetune framework that first pretrains a model and then adjusts the noise levels for higher performance based on the model’s outputs. For certification, we carefully allocate specific robust regions for each test sample. We perform extensive experiments on CIFAR-10 and MNIST datasets and the experimental results demonstrate that our method can achieve better accuracy-robustness trade-off in the transductive setting.
Practical Phase Retrieval: Low-Photon Holography with Untrained Priors
Authors: Hannah Lawrence, David Barmherzig, Henry Li, Michael Eickenberg, Marylou Gabrié
Date: 2021-01-01 00:00:00
Tasks: Retrieval

Phase retrieval is the inverse problem of recovering a signal from magnitude-only Fourier measurements, and underlies numerous imaging modalities, such as Coherent Diffraction Imaging (CDI). A variant of this setup, known as holography, includes a reference object that is placed adjacent to the specimen of interest before measurements are collected. The resulting inverse problem, known as holographic phase retrieval, is well-known to have improved problem conditioning relative to the original. This innovation, i.e. Holographic CDI, becomes crucial at the nanoscale, where imaging specimens such as viruses, proteins, and crystals require low-photon measurements. This data is highly corrupted by Poisson shot noise, and often lacks low-frequency content as well. In this work, we introduce a dataset-free deep learning framework for holographic phase retrieval adapted to these challenges. The key ingredients of our approach are the explicit and flexible incorporation of the physical forward model into the automatic differentiation procedure, the Poisson log-likelihood objective function, and an optional untrained deep image prior. We perform extensive evaluation under realistic conditions. Compared to competing classical methods, our method recovers signal from higher noise levels and is more resilient to suboptimal reference design, as well as to large missing regions of low-frequencies  in the observations. To the best of our knowledge, this is the first work to consider a dataset-free machine learning approach for holographic phase retrieval.
Addressing Extrapolation Error in Deep Offline Reinforcement Learning
Authors: Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, Nando de Freitas
Date: 2021-01-01 00:00:00
Tasks: Offline RL, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Reinforcement learning (RL) encompasses both online and offline regimes.  Unlike its online counterpart, offline RL agents are trained using logged-data only, without interaction with the environment.  Therefore, offline RL is a promising direction for real-world applications, such as healthcare, where repeated interaction with environments is prohibitive.  However, since offline RL losses often involve evaluating state-action pairs not well-covered by training data, they can suffer due to the errors introduced when the function approximator attempts to extrapolate those pairs' value. These errors can be compounded by bootstrapping when the function approximator overestimates, leading the value function to *grow unbounded*, thereby crippling learning. In this paper, we introduce a three-part solution to combat extrapolation errors: (i) behavior value estimation, (ii) ranking regularization, and (iii) reparametrization of the value function.  We provide ample empirical evidence on the effectiveness of our method, showing state of the art performance on the RL Unplugged (RLU) ATARI dataset. Furthermore, we introduce new datasets for bsuite as well as partially observable DeepMind Lab environments, on which our method outperforms state of the art offline RL algorithms.
Certified robustness against physically-realizable patch attack via randomized cropping
Authors: Wan-Yi Lin, Fatemeh Sheikholeslami, Jinghao Shi, Leslie Rice, J Zico Kolter
Date: 2021-01-01 00:00:00
Tasks: Classification, Crop Classification, General Classification, image-classification, Image Classification

This paper studies a certifiable defense against adversarial patch attacks on image classification. Our approach classifies random crops from the original image independently and the original image is classified as the vote over these crops.  This process minimizes changes to the training process, as only the crop classification model needs to be trained, and can be trained in a standard manner without explicit adversarial training.  Leveraging the fact that a patch attack can only influence some pixels of the image, we derive certified robustness bounds on the resulting classification. Our method is particularly effective when realistic physical transformations are applied to the adversarial patch, such as affine transformations. Such transformations occur naturally when an adversarial patch is physically introduced to a scene. Our method improves upon the current state of the art in defending against patch attacks on CIFAR10 and ImageNet, both in terms of certified accuracy and inference time.
Interpretable Reinforcement Learning With Neural Symbolic Logic
Authors: Zhihao Ma, Yuzheng Zhuang, Paul Weng, Dong Li, Kun Shao, Wulong Liu, Hankz Hankui Zhuo, Jianye Hao
Date: 2021-01-01 00:00:00
Tasks: Deep Reinforcement Learning, Hierarchical Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Relational Reasoning

Recent progress in deep reinforcement learning (DRL) can be largely attributed to the use of neural networks.  However, this black-box approach fails to explain the learned policy in a human understandable way.  To address this challenge and improve the transparency, we introduce symbolic logic into DRL and propose a Neural Symbolic Reinforcement Learning framework, in which states and actions are represented in an interpretable way using first-order logic.  This framework features a relational reasoning module, which performs on task-level in Hierarchical Reinforcement Learning, enabling end-to-end learning with prior symbolic knowledge.  Moreover, interpretability is enabled by extracting the logical rules learned by the reasoning module in a symbolic rule space, providing explainability on task level. Experimental results demonstrate better interpretability of subtasks, along with competing performance compared with existing approaches.
Leveraging affinity cycle consistency to isolate factors of variation in learned representations
Authors: Kieran A Murphy, Varun Jampani, Srikumar Ramalingam, Ameesh Makadia
Date: 2021-01-01 00:00:00
Tasks: Pose Transfer, Representation Learning

Identifying the dominant factors of variation across a dataset is a central goal of representation learning.  Generative approaches lead to descriptions that are rich enough to recreate the data, but often only a partial description is needed to complete downstream tasks or to gain insights about the dataset. In this work, we operate in the setting where limited information is known about the data in the form of groupings, or set membership, and the task is to learn representations which isolate the factors of variation that are common across the groupings.  Our key insight is the use of affinity cycle consistency (ACC) between the learned embeddings of images belonging to different sets. In contrast to prior work, we demonstrate that we can work with significantly fewer constraints on the factors of variation, and in some cases, with almost no constraints on the factors of variation for part of the data. By curating datasets from Shapes3D, we quantify the effectiveness of ACC through mutual information between the learned representations and the known image generative factors.  In addition, we demonstrate the applicability of ACC to the tasks of digit style isolation and synthetic-to-real object pose transfer.
Transformer protein language models are unsupervised structure learners
Authors: Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, Alexander Rives
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Language Modeling, Language Modelling

Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model.
Quickest change detection for multi-task problems under unknown parameters
Authors: Firas Jarboui, Vianney Perchet
Date: 2021-01-01 00:00:00
Tasks: Change Detection, Two-sample testing

We consider the quickest change detection problem where both the parameters of pre- and post- change distributions are unknown, which prevent the use of classical simple hypothesis testing. Without additional assumptions, optimal solutions are not tractable as they rely on some minimax and robust variant of the objective. As a consequence, change points might be detected too late for practical applications (in economics, health care or maintenance for instance). 
Other approaches solve a relaxed version of the problem through the use of particular probability distributions or the use of domain knowledge. 

We tackle this problem in the more complex Markovian case and we provide a new scalable approximate algorithm with near optimal performance that runs in $\mathcal{O}(1)$.
Implicit Acceleration of Gradient Flow in Overparameterized Linear Models
Authors: Salma Tarmoun, Guilherme França, Benjamin David Haeffele, Rene Vidal
Date: 2021-01-01 00:00:00

We study the implicit acceleration of gradient flow in over-parameterized two-layer linear models. We show that implicit acceleration emerges from a conservation law that constrains the dynamics to follow certain trajectories. More precisely, gradient flow preserves the difference of the Gramian~matrices of the input and output weights and we show that the amount of acceleration depends on both the magnitude of that difference (which is fixed at initialization) and the spectrum of the data. In addition, and generalizing prior work, we prove our results without assuming small, balanced or spectral initialization for the weights, and establish interesting connections between the matrix factorization problem and Riccati type differential equations.
Dual-Tree Wavelet Packet CNNs for Image Classification
Authors: Hubert Leterme, Kévin Polisano, Valérie Perrier, Karteek Alahari
Date: 2021-01-01 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification

In this paper, we target an important issue of deep convolutional neural networks (CNNs) — the lack of a mathematical understanding of their properties. We present an explicit formalism that is motivated by the similarities between trained CNN kernels and oriented Gabor filters for addressing this problem. The core idea is to constrain the behavior of convolutional layers by splitting them into a succession of wavelet packet decompositions, which are modulated by freely-trained mixture weights. We evaluate our approach with three variants of wavelet decompositions with the AlexNet architecture for image classification as an example. The first variant relies on the separable wavelet packet transform while the other two implement the 2D dual-tree real and complex wavelet packet transforms, taking advantage of their feature extraction properties such as directional selectivity and shift invariance. Our experiments show that we achieve the accuracy rate of standard AlexNet, but with a significantly lower number of parameters, and an interpretation of the network that is grounded in mathematical theory.
Learning to Balance with Incremental Learning
Authors: Joel Jang, Yoonjeon Kim, Jaewoo Kang
Date: 2021-01-01 00:00:00
Tasks: Incremental Learning

Classification tasks require balanced distribution of data in order to ensure the learner to be trained to generalize over all classes. In realistic settings, however, the number of instances vary substantially among classes. This typically leads to a learner that promotes bias towards the majority group due to its dominating property. Therefore, methods to handle imbalanced data is crucial for alleviating distributional skews and fully utilizing the under-represented data. We propose a novel training method, Sequential Targeting, that forces an incremental learning setting by splitting the data into mutually exclusive subset and adaptively balancing the data distribution as tasks develop. To address problems that arise within incremental learning, we apply dropout and elastic weight consolidation with our method. It is demonstrated in a variety of experiments on both text and image dataset (IMDB, CIFAR-10, MNIST) and has proven its superiority over traditional methods such as oversampling and under-sampling.
The role of Disentanglement in Generalisation
Authors: Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, Jeffrey Bowers
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Disentanglement, Out-of-Distribution Generalization, Systematic Generalization

Combinatorial generalization — the ability to understand and produce novel combinations of familiar elements — is considered a core capacity of humans that modern AI systems currently struggle at. Recently, it has been suggested that learning disentangled representations may help address this problem. It is claimed that such representations should be able to capture the compositional structure of the world which can then be combined to produce novel representations. That is, compositional representations are thought to support combinatorial generalisation. In this study, we systematically tested how the degree of disentanglement affects various forms of generalisation, including two forms of combinatorial generalisation that varied in difficulty. We trained variational autoencoders (VAEs) with different levels of disentanglement on an unsupervised task by excluding combinations of generative factors during training. At test time we ask the models to reconstruct the missing combinations in order to measure generalisation performance. Irrespective of the degree of disentanglement, we found that the models supported only weak combinatorial generalisation. Next, we tested our model in a more complex task which explicitly required independent generative factors to be controlled. Although this added pressure improved the level of disentanglement compared to normal VAEs, generalisation did not improve. While learning disentangled representations does improve interpretability and sample efficiency in downstream tasks, our results suggest that they are not sufficient for supporting more difficult forms of generalisation.
Learning to live with Dale's principle: ANNs with separate excitatory and inhibitory units
Authors: Jonathan Cornford, Damjan Kalajdzievski, Marco Leite, Amélie Lamarquette, Dimitri Michael Kullmann, Blake Aaron Richards
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

 The units in artificial neural networks (ANNs) can be thought of as abstractions of biological neurons, and ANNs are increasingly used in neuroscience research. However, there are many important differences between ANN units and real neurons. One of the most notable is the absence of Dale's principle, which ensures that biological neurons are either exclusively excitatory or inhibitory. Dale's principle is typically left out of ANNs because its inclusion impairs learning. This is problematic, because one of the great advantages of ANNs for neuroscience research is their ability to learn complicated, realistic tasks. Here, by taking inspiration from feedforward inhibitory interneurons in the brain we show that we can develop ANNs with separate populations of excitatory and inhibitory units that learn just as well as standard ANNs. We call these networks Dale's ANNs (DANNs). We present two insights that enable DANNs to learn well: (1) DANNs are related to normalization schemes, and can be initialized such that the inhibition centres and standardizes the excitatory activity, (2) updates to inhibitory neuron parameters should be scaled using corrections based on the Fisher Information matrix. These results demonstrate how ANNs that respect Dale's principle can be built without sacrificing learning performance, which is important for future work using ANNs as models of the brain. The results also may have interesting implications for how inhibitory plasticity in the real brain operates.
A Simple Approach To Define Curricula For Training Neural Networks
Authors: Vinu Sankar Sadasivan, Anirban Dasgupta
Date: 2021-01-01 00:00:00

In practice, sequence of mini-batches generated by uniform sampling of examples from the entire data is used for training neural networks. Curriculum learning is a training strategy that sorts the training examples by their difficulty and gradually exposes them to the learner. In this work, we propose two novel curriculum learning algorithms and empirically show their improvements in performance with convolutional and fully-connected neural networks on multiple datasets. Our dynamic curriculum learning algorithm tries to reduce the distance between the network weight and the optimal weight at any training step by greedily sampling examples with gradients that are directed towards the optimal weight. It achieves a speedup of $\sim 45\%$ in our experiments. We also introduce a new task-specific curriculum learning strategy  that uses statistical measures such as standard deviation and entropy values to score the difficulty of data points in natural image datasets. We show that this new approach yields a mean speedup of $\sim 43\%$ in the experiments we perform. Further, we also use our algorithms to learn why curriculum learning works. Based on our study, we argue that curriculum learning removes noisy examples from the initial phases of training, and gradually exposes them to the learner acting like a regularizer that helps in improving the generalization ability of the learner.
Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks
Authors: Christian H.X. Ali Mehmeti-Göpel, David Hartmann, Michael Wand
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

In this paper, we apply harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain. Each nonlinear layer creates higher-frequency harmonics, which we call "blueshift", whose magnitude increases with network depth, thereby increasing the “roughness” of the output landscape. Unlike differential models (such as vanishing gradients, sharpness), this provides a global view of how network architectures behave across larger areas of their parameter domain. For example, the model predicts that residual connections are able to counter the effect by dampening corresponding higher frequency modes. We empirically verify the connection between blueshift and architectural choices, and provide evidence for a connection with trainability.
A generalized probability kernel on discrete distributions and its application in two-sample test
Authors: Le Niu
Date: 2021-01-01 00:00:00

We propose a generalized probability kernel on discrete distributions with finite support. This probability kernel, defined as kernel between distributions instead of samples, generalizes a variety of existing discrepancy statistics including maximum mean discrepancy(MMD) as well as kernelized Stein discrepancy(KSD), and extends to more general cases. For both existing and newly proposed statistics, we estimate them through empirical frequency and illustrate the strategy to search for the unbiased ones. We further analyze several unbiased plugin(empirical) estimators for the task of two-sample test. Our work connects the fields of discrete distribution-property estimation and kernel-based hypothesis test, which might shed light on more new possibilities.