pwc-archive/papers-with-abstracts huggingface dataset:
Learning Visual Representation from Human Interactions
Authors: Kiana Ehsani, Daniel Gordon, Thomas Hai Dang Nguyen, Roozbeh Mottaghi, Ali Farhadi
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Action Recognition, Depth Estimation, Representation Learning, Scene Classification

Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision. Most representation learning approaches rely solely on visual data such as images or videos. In this paper, we explore a novel approach, where we use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. For this study, we collect a dataset of human interactions capturing body part movements and gaze in their daily lives. Our experiments show that our self-supervised representation that encodes interaction and attention cues outperforms a visual-only state-of-the-art method MoCo (He et al.,2020), on a variety of target tasks: scene classification (semantic), action recognition (temporal), depth estimation (geometric), dynamics prediction (physics) and walkable surface estimation (affordance).
Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Sparse Neural Networks
Authors: Shuai Zhang, Meng Wang, Sijia Liu, Pin-Yu Chen, JinJun Xiong
Date: 2021-01-01 00:00:00
Proceeding: NeurIPS 2021 12

The $\textit{lottery ticket hypothesis}$ (LTH) states that learning on a properly pruned network (the $\textit{winning ticket}$) has improved test accuracy over the originally unpruned network. Although LTH has been justified empirically in a broad range of deep neural network (DNN) involved applications like computer vision and natural language processing, the theoretical validation of the improved generalization of a winning ticket remains elusive. To the best of our knowledge, our work, for the first time, characterizes the performance of training a sparse neural network by analyzing the geometric structure of the objective function and the sample complexity to achieve zero generalization error. We show that the convex region near a desirable model with guaranteed generalization enlarges as the neural network model is pruned, indicating the structural importance of a winning ticket. Moreover, as the algorithm for training a sparse neural network is specified as (accelerated) stochastic gradient descent algorithm, we theoretically show that the number of samples required for achieving zero generalization error is proportional to the number of the non-pruned model weights in the hidden layer. With a fixed number of samples, training a pruned neural network enjoys a faster convergence rate to the desirable model than training the original unpruned one, providing a formal justification of the improved generalization of the winning ticket. Our theoretical results are acquired from learning a sparse neural network of one hidden layer, while experimental results are further provided to justify the implications in pruning multi-layer neural networks.
Hyperrealistic neural decoding: Reconstruction of face stimuli from fMRI measurements via the GAN latent space
Authors: Thirza Dado, Yağmur Güçlütürk, Luca Ambrogioni, Gabrielle Ras, Sander E. Bosch, Marcel van Gerven, Umut Güçlü
Date: 2021-01-01 00:00:00

We introduce a new framework for hyperrealistic reconstruction of perceived naturalistic stimuli from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring functional magnetic resonance imaging data as subjects perceived face images created by the generator network of a GAN. Subsequently, we used a decoding approach to predict the latent state of the GAN from brain data. Hence, latent representations for stimulus (re-)generation are obtained, leading to state-of-the-art image reconstructions. Altogether, we have developed a highly promising approach for decoding sensory perception from brain activity and systematically analyzing neural information processing in the human brain.
Approximate Birkhoff-von-Neumann decomposition: a differentiable approach
Authors: Andrés Hoyos-Idrobo
Date: 2021-01-01 00:00:00
Tasks: Fairness, Riemannian optimization

The Birkhoff-von-Neumann (BvN) decomposition is a standard tool used to draw permutation matrices from a doubly stochastic (DS) matrix. The BvN decomposition represents such a DS matrix as a convex combination of several permutation matrices. Currently, most algorithms to compute the BvN decomposition employ either greedy strategies or custom-made heuristics. In this paper, we present a novel differentiable approach to approximate the BvN decomposition. Our algorithm builds upon recent advances in Riemannian optimization on Birkhoff polytopes. We offer an empirical evaluation of this approach in the fairness of exposure in rankings, where we show that the outcome of our method behaves similarly to greedy algorithms. Our approach is an excellent addition to existing methods for sampling from DS matrices, such as sampling from a Gumbel-Sinkhorn distribution. However, our approach is better suited for applications where the latency in prediction time is a constraint. Indeed, we can generally precompute an approximated BvN decomposition offline. Then, we select a permutation matrix at random with probability proportional to its coefficient. Finally, we provide an implementation of our method.
Adaptive N-step Bootstrapping with Off-policy Data
Authors: Guan Wang, Dong Yan, Hang Su, Jun Zhu
Date: 2021-01-01 00:00:00
Tasks: Atari Games, MuJoCo

The  definition  of  the  update  target  is  a  crucial  design  choice  in  reinforcement learning.  Due to the low computation cost and empirical high performance,  n-step returns with off-policy data is a widely used update target to bootstrap from scratch. A critical issue of applying n-step returns is to identify the optimal value of n.  In practice, n is often set to a fixed value, which is either determined by an empirical guess or by some hyper-parameter search.  In this work,  we point out that the optimal value of n actually differs on each data point, while the fixed value n is a rough average of them. The estimation error can be decomposed into two sources, off-policy bias and approximation error, and the fixed value of n is a trade-off between them.  Based on that observation, we introduce a new metric, policy age, to quantify the off-policyness of each data point.  We propose the Adaptive N-step Bootstrapping, which calculates the value of n for each data point by its policy age instead of the empirical guess. We conduct experiments on both MuJoCo and Atari games.  The results show that adaptive n-step bootstrap-ping achieves state-of-the-art performance in terms of both final reward and data efficiency.
Estimation of Number of Communities in Assortative Sparse Networks
Authors: Neil Hwang, Jiarui Xu, Shirshendu Chatterjee, Sharmodeep Bhattacharyya
Date: 2021-01-01 00:00:00
Tasks: Community Detection, Computational Efficiency

Most community detection algorithms assume the number of communities, K, to be known a priori. Of the various approaches that have been proposed to estimate K, the non-parametric method based on the spectral properties of the Bethe Hessian matrix has garnered much popularity for its simplicity, computational efficiency, and robustness to sparsity of data. Recently, its consistency for networks in semi-dense regimes with the expected degree greater than log(N) has been shown (N being the number of nodes in the network). In this paper, we show that the spectral method based on the Bethe Hessian matrix is in fact also consistent in sparse regimes with expected degree lesser than log(N) at any rate, thus establishing it as a method which is robust to a wide range of problem settings, regardless of the sparsity of networks.
Knowledge distillation via softmax regression representation learning
Authors: Jing Yang, Brais Martinez, Adrian Bulat, Georgios Tzimiropoulos
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Knowledge Distillation, Model Compression, regression, Representation Learning

This paper addresses the problem of model compression via knowledge distillation. We advocate for a method that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning. Previous distillation methods which typically impose direct feature matching between the student and the teacher do not take into account the classification problem at hand. On the contrary, our distillation method decouples representation learning and classification and utilizes the teacher's pre-trained classifier to train the student's penultimate layer feature. In particular, for the same input image, we wish the teacher's and student's feature to produce the same output when passed through the teacher's classifier which is achieved with a simple $L_2$ loss. Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-art methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains.
Are Graph Convolutional Networks Fully Exploiting the Graph Structure?
Authors: Davide Buffelli, Fabio Vandin
Date: 2021-01-01 00:00:00

Graph Convolutional Networks (GCNs) represent the state-of-the-art for many graph related tasks. At every layer, GCNs rely on the graph structure to define an aggregation strategy where each node updates its representation by combining information from its neighbours. A known limitation of GCNs is their inability to infer long-range dependencies. In fact, as the number of layers increases, information gets smoothed and node embeddings become indistinguishable, negatively affecting performance. In this paper we formalize four levels of injection of graph structural information, and use them to analyze the importance of long-range dependencies. We then propose a novel regularization technique based on random walks with restart, called RWRReg, which encourages the network to encode long-range information into node embeddings. RWRReg does not require additional operations at inference time, is model-agnostic, and is further supported by our theoretical analysis connecting it to the Weisfeiler-Leman algorithm. Our experimental analysis, on both transductive and inductive tasks, shows that the lack of long-range structural information greatly affects the performance of state-of-the-art models, and that the long-range information exploited by RWRReg leads to an average accuracy improvement of more than $5\%$ on all considered tasks.
Bractivate: Dendritic Branching in Segmentation Neural Architecture Search
Authors: Leila Abdelrahman
Date: 2021-01-01 00:00:00
Tasks: Neural Architecture Search

Researchers manually compose most neural networks through painstaking experimentation.
This process is taxing and explores only a limited subset of possible
architecture. Researchers design architectures to address objectives ranging
from low space complexity to high accuracy through hours of experimentation.
Neural architecture search (NAS) is a thriving field for automatically discovering
architectures achieving these same objectives. Addressing these ever-increasing
challenges in computing, we take inspiration from the brain because it has the
most efficient neuronal wiring of any complex structure; its physiology inspires
us to propose Bractivate, a NAS algorithm inspired by neural dendritic branching.
Neurons generate new input connections to active neurons, propagating salient information
through the network. We apply our methods to lung x-ray, cell nuclei
microscopy, and electron microscopy segmentation tasks to highlight Bractivate’s
robustness. Moreover, our ablation studies emphasize dendritic branching’s necessity:
ablating these connections leads to significantly lower model performance.
We finally compare our discovered architecture with other state-of-the-art UNet
models, highlighting how efficient skip connections allow Bractivate to achieve
comparable results with substantially lower space and time complexity, proving
how Bractivate balances efficiency with performance. We invite you to work with
our code here: https://tinyurl.com/bractivate.
Scalable Bayesian Inverse Reinforcement Learning by Auto-Encoding Reward
Authors: Alex James Chan, Mihaela van der Schaar
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Bayesian Inference, Imitation Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Bayesian inference over the reward presents an ideal solution to the ill-posed nature of the inverse reinforcement learning problem. Unfortunately current methods generally do not scale well beyond the small tabular setting due to the need for an inner-loop MDP solver, and even non-Bayesian methods that do themselves scale often require extensive interaction with the environment to perform well, being inappropriate for high stakes or costly applications such as healthcare. In this paper we introduce our method, Auto-encoded Variational Reward Imitation Learning (AVRIL), that addresses both of these issues by jointly learning an approximate posterior distribution over the reward that scales to arbitrarily complicated state spaces alongside an appropriate policy in a completely offline manner through a variational approach to said latent reward. Applying our method to real medical data alongside classic control simulations, we demonstrate Bayesian reward inference in environments beyond the scope of current methods, as well as task performance competitive with focused offline imitation learning algorithms.
Federated learning using mixture of experts
Authors: Edvin Listo Zec, John Martinsson, Olof Mogren, Leon René Sütfeld, Daniel Gillblad
Date: 2021-01-01 00:00:00
Tasks: Federated Learning, Mixture-of-Experts

Federated learning has received attention for its efficiency and privacy benefits,in settings where data is distributed among devices.   Although federated learning shows significant promise as a key approach when data cannot be shared or centralized, current incarnations show limited privacy properties and have short-comings  when  applied  to  common  real-world  scenarios.   One  such  scenario is heterogeneous data among devices, where data may come from different generating distributions. In this paper, we propose a federated learning framework using a mixture of experts to balance the specialist nature of a locally trained model with the generalist knowledge of a global model in a federated learning setting.  Our results show that the mixture of experts model is better suited as a personalized model for devices when data is heterogeneous, outperforming both global and lo-cal models.  Furthermore, our framework gives strict privacy guarantees, which allows clients to select parts of their data that may be excluded from the federation.   The evaluation shows that the proposed solution is robust to the setting where some users require a strict privacy setting and do not disclose their models to a central server at all, opting out from the federation partially or entirely.  The proposed framework is general enough to include any kind of machine learning models, and can even use combinations of different kinds.
Just How Toxic is Data Poisoning? A Benchmark for Backdoor and Data Poisoning Attacks
Authors: Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, Tom Goldstein
Date: 2021-01-01 00:00:00
Tasks: Data Poisoning, Experimental Design

Data poisoning and backdoor attacks manipulate training data in order to cause models to fail during inference.  A recent survey of industry practitioners found that data poisoning is the number one concern among threats ranging from model stealing to adversarial attacks. However, we find that the impressive performance evaluations from data poisoning attacks are, in large part, artifacts of inconsistent experimental design. Moreover, we find that existing poisoning methods have been tested in contrived scenarios, and many fail in more realistic settings. In order to promote fair comparison in future work, we develop standardized benchmarks for data poisoning and backdoor attacks.
Diffeomorphic Spatial Transformer Networks
Authors: Tycho F.A. van der Ouderaa, Ivana Isgum, Wouter B. Veldhuis, Bob D. de Vos, Pim Moeskops
Date: 2021-01-01 00:00:00

In this paper we propose a spatial transformer network where the spatial transformations are limited to the group of diffeomorphisms. Diffeomorphic transformations are a kind of homeomorphism, which by definition preserve topology, a compelling property in certain applications.
We apply this diffemorphic spatial transformer to model the output of a neural network as a topology preserving mapping of a prior shape. By carefully choosing the prior shape we can enforce properties on the output of the network without requiring any changes to the loss function, such as smooth boundaries and a hard constraint on the number of connected components.
The diffeomorphic transformer networks outperform their non-diffeomorphic precursors when applied to learn data invariances in classification tasks. On a breast tissue segmentation task, we show that the approach is robust and flexible enough to deform simple artificial priors, such as Gaussian-shaped prior energies, into high-quality predictive probability densities. In addition to desirable topological properties, the segmentation maps have competitive quantitative fidelity compared to those obtained by direct estimation (i.e. plain U-Net).
"Hey, that's not an ODE'": Faster ODE Adjoints with 12 Lines of Code
Authors: Patrick Kidger, Ricky T. Q. Chen, Terry Lyons
Date: 2021-01-01 00:00:00
Tasks: Time Series, Time Series Analysis

Neural differential equations may be trained by backpropagating gradients via the adjoint method, which is another differential equation typically solved using an adaptive-step-size numerical differential equation solver. A proposed step is accepted if its error, \emph{relative to some norm}, is sufficiently small; else it is rejected, the step is shrunk, and the process is repeated. Here, we demonstrate that the particular structure of the adjoint equations makes the usual choices of norm (such as $L^2$) unnecessarily stringent. By replacing it with a more appropriate (semi)norm, fewer steps are unnecessarily rejected and the backpropagation is made faster. This requires only minor code modifications. Experiments on a wide range of tasks---including time series, generative modeling, and physical control---demonstrate a median improvement of 40% fewer function evaluations. On some problems we see as much as 62% fewer function evaluations, so that the overall training time is roughly halved.
Reviving Autoencoder Pretraining
Authors: You Xie, Nils Thuerey
Date: 2021-01-01 00:00:00

The pressing need for pretraining algorithms has been diminished by numerous advances in terms of regularization, architectures, and optimizers. Despite this trend, we re-visit the classic idea of unsupervised autoencoder pretraining and propose a modified variant that relies on a full reverse pass trained in conjunction with a given training task. We establish links between SVD and pretraining and show how it can be leveraged for gaining insights about the learned structures. Most importantly, we demonstrate that our approach yields an improved performance for a wide variety of relevant learning and transfer tasks ranging from fully connected networks over ResNets to GANs. Our results demonstrate that unsupervised pretraining has not lost its practical relevance in today's deep learning environment.
Neurally Guided Genetic Programming for Turing Complete Programming by Example
Authors: Alexander Newton Wild, Barry Porter
Date: 2021-01-01 00:00:00

The ability to synthesise source code from input/output examples allows nonexperts to generate programs, and experts to abstract away a wide range of simple programming tasks. Current research in this area has explored neural synthesis, SMT solvers, and genetic programming; each of these approaches is limited, however, often using highly specialised target languages for synthesis. In this paper we present a novel hybrid approach using neural networks to guide genetic programming (GP), which allows us to successfully synthesise code from just ten I/O examples in a generalised Turing complete target language, up to and including a sorting algorithm. We show that GP by itself is able to synthesise a set of simple programs, and show which hints (suggested lines of code for inclusion) are of most utility to GP in solving harder problems. Using a form of unstructured curriculum learning, we then demonstrate that neural networks can be used to determine when to make use of these high-utility hints for specific I/O problems and thus enable complex functions to be successfully synthesised. We apply our approach to two different problem sets: common array-to-array programs (including sorting), and a canvas drawing problem set inspired by So & Oh (2018).
Exploring single-path Architecture Search ranking correlations
Authors: Kevin Alexander Laube, Andreas Zell
Date: 2021-01-01 00:00:00
Tasks: Neural Architecture Search

Recently presented benchmarks for Neural Architecture Search (NAS) provide the results of training thousands of different architectures in a specific search space, thus enabling the fair and rapid comparison of different methods.
Based on these results, we quantify the ranking correlations of single-path architecture search methods
in different search space subsets and under several training variations;
studying their impact on the expected search results.
The experiments support the few-shot approach and Linear Transformers,
provide evidence against disabling cell topology sharing during the training phase or using strong regularization in the NAS-Bench-201 search space,
and show the necessity of further research regarding super-network size and path sampling strategies.
Do Transformers Understand Polynomial Simplification?
Authors: Vishesh Agarwal, Somak Aditya, Navin Goyal
Date: 2021-01-01 00:00:00
Tasks: Dataset Generation, Form

Recently researchers have demonstrated that Transformers can be trained to learn symbolic tasks such as solving integration and differential equations in an end-to-end fashion. In these setups, for an input symbolic expression, the Transformer predicts the final solution in a single step. Since such tasks may consist of a sequence of logical steps, question remains whether such networks have understood and learnt individual steps to reach the solution. To take a deeper look, we consider the task of polynomial simplification. Polynomials can be written in a simple normal form as a sum of monomials which are ordered in a lexicographic order. For a polynomial which is not necessarily in this normal form, a sequence of simplification steps is applied to reach the fully simplified (i.e., in the normal form) polynomial. For this task, we describe a synthetic Polynomial dataset generation algorithm which generates polynomials with unique proof steps. Then, we conduct an extensive analysis of the Transformer’s abilities to learn the polynomial simplification task along different dimensions.