pwc-archive/papers-with-abstracts huggingface dataset:
A Block Minifloat Representation for Training Deep Neural Networks
Authors: Sean Fox, Seyedramin Rasoulinezhad, Julian Faraone, David Boland, Philip Leong
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

Training Deep Neural Networks (DNN) with high efficiency can be difficult to achieve with native floating point representations and commercially available hardware. Specialized arithmetic with custom acceleration offers perhaps the most promising alternative. Ongoing research is trending towards narrow floating point representations, called minifloats, that pack more operations for a given silicon area and consume less power. In this paper, we introduce Block Minifloat (BM), a new spectrum of minifloat formats capable of training DNNs end-to-end with only 4-8 bit weight, activation and gradient tensors. While standard floating point representations have two degrees of freedom, via the exponent and mantissa, BM exposes the exponent bias as an additional field for optimization. Crucially, this enables training with fewer exponent bits, yielding dense integer-like hardware for fused multiply-add (FMA) operations. For ResNet trained on ImageNet, 6-bit BM achieves almost no degradation in floating point accuracy with FMA units that are $4.1\times(23.9\times)$ smaller and consume $2.3\times(16.1\times)$ less energy than FP8 (FP32). Furthermore, our 8-bit BM format matches floating-point accuracy while delivering a higher computational density and faster expected training times.
On the Role of Pre-training for Meta Few-Shot Learning
Authors: Chia-You Chen, Hsuan-Tien Lin, Gang Niu, Masashi Sugiyama
Date: 2021-01-01 00:00:00
Tasks: Disentanglement, Few-Shot Learning

Few-shot learning aims to classify unknown classes of examples with a few new examples per class. There are two key routes for few-shot learning. One is to (pre-)train a classifier with examples from known classes, and then transfer the pre-trained classifier to unknown classes using the new examples. The other, called meta few-shot learning, is to couple pre-training with episodic training, which contains episodes of few-shot learning tasks simulated from the known classes. Pre-training is known to play a crucial role for the transfer route, but the role of pre-training for the episodic route is less clear. In this work, we study the role of pre-training for the episodic route. We find that pre-training serves a major role of disentangling representations of known classes, which makes the resulting learning tasks easier for episodic training. The finding allows us to shift the huge simulation burden of episodic learning to a simpler pre-training stage. We justify such a benefit of shift by designing a new disentanglement-based pre-training model, which helps episodic learning achieve competitive performance more efficiently.
Gated Relational Graph Attention Networks
Authors: Denis Lukovnikov, Asja Fischer
Date: 2021-01-01 00:00:00
Tasks: Graph Attention, Long-range modeling

Relational Graph Neural Networks (GNN) are a class of GNN that are capable of handling multi-relational graphs. Like all GNNs, they suffer from a drop in performance when training deeper networks, which may be caused by vanishing gradients, over-parameterization, and oversmoothing. Previous works have investigated methods that improve the training of deeper GNNs, which include normalization techniques and various types of skip connection within a node. However, learning long-range patterns in multi-relational graphs using GNNs remains an under-explored topic. In this work, we propose a novel GNN architecture based on the Graph Attention Network (GAT) that uses gated skip connections to improve long-range modeling between nodes and uses a more scalable vector-based approach for parameterizing relations. We perform an extensive experimental analysis on synthetic and real data, focusing explicitly on learning long-range patterns. The results indicate that the proposed method significantly outperforms several commonly used relational GNN variants when used in deeper configurations and stays competitive to existing architectures in a shallow setup.
Universal Sentence Representations Learning with Conditional Masked Language Model
Authors: ZiYi Yang, Yinfei Yang, Daniel M Cer, Jax Law, Eric Darve
Date: 2021-01-01 00:00:00
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Natural Language Inference, Representation Learning, Retrieval, Sentence

This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-of-the-art performance on SentEval, even outperforming models learned using (semi-)supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval~(BR) and natural language inference~(NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin. We explore the same language bias of the learned representations, and propose a principle component based approach to remove the language identifying information from the representation while still retaining sentence semantics.
An Euler-based GAN for time series
Authors: Carl Remlinger, Joseph Mickael, Romuald Elie
Date: 2021-01-01 00:00:00
Tasks: Time Series, Time Series Analysis, Transfer Learning

A new model of generative adversarial networks for time series based on Euler scheme and Wasserstein distances including Sinkhorn divergence is proposed. Euler scheme improves stability of learning, provides meaningful learning parameters such as drift and volatility while allowing the representation of a large class of processes. We test our Euler GAN generations with usual Monte Carlo simulations in one-dimension and in a multi-dimensional case. We show how the proposed methodology can be combined with transfer learning to include the latest historical dataset features. The approach is tested on financial indicators computation on S\&P500 and on an option hedging problem.
GLUECode: A Benchmark for Source Code Machine Learning Models
Authors: Anjan Karmakar, Julian Aron Prenner, Miltiadis Allamanis, Romain Robbes
Date: 2021-01-01 00:00:00
Tasks: BIG-bench Machine Learning

A multitude of machine learning models for source code have been proposed in the recent years capturing various aspects of the inherent rich structure and semantics of code. However, these models are commonly designed to perform well on a single task, failing to capture code's  multifaceted nature. To address this, we present GLUECode, Global and Local Understanding Evaluation of Code, a benchmark of diverse tasks to evaluate machine learning models of source code.  

Crucially, GLUECode accounts for the distinct characteristics of source code: (1) source code is highly structured and (2) source code is often composed of multiple interacting entities. Existing tasks incentivize researchers to create models and code representations that perform well on a single task - commonly focusing on local reasoning. GLUECode aims to allow researchers to experiment with multiple local and global source code representations, and evaluate these models on their ability to capture the diverse characteristics of source code, thus driving the community towards building robust source code models incorporating global reasoning. 

We present results for several baselines. The GLUECode tasks are challenging for the evaluated baselines; no model achieves convincing performance across all tasks. This indicates that there is ample room for progress on GLUECode.
ROMUL: Scale Adaptative Population Based Training
Authors: Daniel Haziza, Jérémy Rapin, Gabriel Synnaeve
Date: 2021-01-01 00:00:00
Tasks: Data Augmentation, image-classification, Image Classification, Language Modeling, Language Modelling

In most pragmatic settings, data augmentation and regularization are essential, and require hyperparameter search.
Population based training (PBT) is an effective tool for efficiently finding them as well as schedules over hyperparameters.
In this paper, we compare existing PBT algorithms and contribute a new one: ROMUL, for RObust MULtistep search, which adapts its stepsize over the course of training.
We report competitive results with standard models on CIFAR (image classification) as well as Penn Tree Bank (language modeling), which both depend on heavy regularization.
We also open-source hoptim, a PBT library agnostic to the training framework, which is simple to use, reentrant, and provides good defaults with ROMUL.
Rethinking Graph Neural Networks for Graph Coloring
Authors: Wei Li, Ruxuan Li, Yuzhe ma, Siu On Chan, Bei Yu
Date: 2021-01-01 00:00:00

The development of graph neural networks (GNNs) stimulated the interest in GNNs for NP-hard problems, while most works apply GNNs for NP-hard problems by empirical intuition and experimental trials and improve the results with the help of some heuristic designs. We start from a simple contradiction that a graph coloring problem requires two connected and symmetric nodes to be assigned different colors while an expressively powerful GNN always maps the two nodes to the same node embeddings. To characterize the power of GNNs for the graph coloring problem, we first formalize the discrimination power of GNNs as the capability to assign nodes different colors. We then study a popular class of GNNs, called AC-GNN, and identify the node pairs that AC-GNNs fail to discriminate and provide corresponding solutions to discriminate them. We show that any AC-GNN is a local coloring method and any local coloring method is non-optimal. Moreover, we discuss the color equivalence of graphs and give a condition for the color equivalent AC-GNN theoretically. Following the approaches which prove to enhance the discriminative power, we develop a simple architecture for the graph coloring problem without heuristic pre-processing and post-processing procedures. We empirically validate our theoretical findings and demonstrate that our model is discriminatively powerful and even comparable with state-of-the-art heuristic algorithms.
Hindsight Curriculum Generation Based Multi-Goal Experience Replay
Authors: Xiaoyun Feng
Date: 2021-01-01 00:00:00
Tasks: Reinforcement Learning (RL)

In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown it possible for off-policy reinforcement learning (RL) to make use of failed experiences. However, the replayed experiences may not lead to well-explored state-action pairs, especially for a pseudo goal, which instead results in a poor estimate of the value function. To tackle the problem, we propose to resample hindsight experiences based on their likelihood under the current policy and the overall distribution. Based on the hindsight strategy, we introduce a novel multi-goal experience replay method that automatically generates a training curriculum, namely Hindsight Curriculum Generation (HCG). As the range of experiences expands, the generated curriculum strikes a dynamic balance between exploiting and exploring. We implement HCG with the vanilla Deep Deterministic Policy Gradient(DDPG), and experiments on several tasks with sparse binary rewards demonstrate that HCG improves sample efficiency of the state of the art.
Uncertainty-aware Active Learning for Optimal Bayesian Classifier
Authors: Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, Xiaoning Qian
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Active Learning, Classification, General Classification

For pool-based active learning, in each iteration a candidate training sample is chosen for labeling by optimizing an acquisition function. Expected Loss Reduction~(ELR) methods maximize the expected reduction in the classification error given a new labeled candidate based on a one-step-look-ahead strategy. ELR is the optimal strategy with a single query; however, since such myopic strategies cannot identify the long-term effect of a query on the classification error, ELR may get stuck before reaching the optimal classifier. To improve convergence in the context of optimal Bayesian classification, we utilize an one-step-look-ahead acquisition function based on a weighted form of the mean objective cost of uncertainty (MOCU) that focuses on the uncertainty related to the classification error. The weight scheme is designed to guarantee the Weighted-MOCU algorithm converging to the optimal classifier of the true model. We demonstrate its performance with both synthetic and real-world datasets.
ResNet After All: Neural ODEs and Their Numerical Solution
Authors: Katharina Ott, Prateek Katiyar, Philipp Hennig, Michael Tiemann
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: All, valid

A key appeal of the recently proposed Neural Ordinary Differential Equation (ODE) framework is that it seems to provide a continuous-time extension of discrete residual neural networks. 
As we show herein, though, trained Neural ODE models actually depend on the specific numerical method used during training.
If the trained model is supposed to be a flow generated from an ODE, it should be possible to choose another numerical solver with equal or smaller numerical error without loss of performance.
We observe that if training relies on a solver with overly coarse discretization, then testing with another solver of equal or smaller numerical error results in a sharp drop in accuracy. 
In such cases, the combination of vector field and numerical method cannot be interpreted as a flow generated from an ODE, which arguably poses a fatal breakdown of the Neural ODE concept.
We observe, however, that there exists a critical step size beyond which the training yields a valid ODE vector field. 
We propose a method that monitors the behavior of the ODE solver during training to adapt its step size, aiming to ensure a valid ODE without unnecessarily increasing computational cost.
We verify this adaption algorithm on a common bench mark dataset as well as a synthetic dataset.
An Automated Domain Understanding Technique for Knowledge Graph Generation
Authors: Dimitrios Christofidellis, Matteo Manica, Leonidas Georgopoulos, Hans Vandierendonck
Date: 2021-01-01 00:00:00
Tasks: Graph Generation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Relation Extraction

Domain-specific Knowledge Graph (KG) generation is a labor intensive task usually orchestrated and supervised by subject matter experts. Herein, we propose a strategy to automate the generation process following a two-step approach. Initially, the structure of the domain of interest is inferred from the corpus in the form of a metagraph. Afterwards, once the domain structure has been discovered, named entity recognition (NER) and relation extraction (RE) models can be used to generate a domain-specific KG.  We argue why the automated definition of the domain's structure as a first step is beneficial both in terms of  construction time and quality of the generated graph. Furthermore, we present a machine learning approach, based on Transformers, to infer the structure of a corpus's domain. The proposed method is extensively validated on three public datasets (WebNLG, NYT and DocRED) by comparing it with two reference methods using CNNs and RNNs. Lastly, we demonstrate how this work lays the foundation for fully automated and unsupervised KG generation.
Deep Ecological Inference
Authors: Nic Fishman, Colin McAuliffe
Date: 2021-01-01 00:00:00

We introduce an efficient approximation to the loss function for the ecological inference problem, where individual labels are predicted from aggregates. This allows us to construct ecological versions of linear models, deep neural networks, and Bayesian neural networks. Using these models we infer probabilities of vote choice for candidates in the Maryland 2018 midterm elections for 2,322,277 voters in 2055 precincts. We show that increased network depth and joint learning of multiple races within an election improves the accuracy of ecological inference when compared to benchmark data from polling. Additionally we leverage data on the joint distribution of ballots (available from ballot images which are public for election administration purposes) to show that joint learning leads to significantly improved recovery of the covariance structure for multi-task ecological inference. Our approach also allows learning latent representations of voters, which we show outperform raw covariates for leave-one-out prediction.
NODE-SELECT: A FLEXIBLE GRAPH NEURAL NETWORK BASED ON REALISTIC PROPAGATION SCHEME
Authors: Steph-Yves Louis, Alireza Nasiri, Fatima Christina Rolland, Cameron Mitro, Jianjun Hu
Date: 2021-01-01 00:00:00
Tasks: Graph Neural Network, Node Classification

While there exists a wide variety of graph neural networks (GNN) for node classification, only a minority of them adopt effective mechanisms to propagate the nodes' information with respect to these nodes' global importance. Additionally, two very important challenges that still significantly affect graph neural networks are the over-fitting and over-smoothing issues. Essentially, both issues cause poor generalization of the model and much poorer node classification performance. In this paper we propose the NODE-SELECT graph neural network (NSGNN): a novel and flexible graph neural network that uses subsetting filters to learn the contribution from the nodes selected to share their information. For the selected nodes, the way their learned information propagates resembles that of actual networks of the real world; where only a subset of nodes simultaneously share information. With the ability to manipulate the message passing operations through the use of numerous ensembled filters, our NODE-SELECT graph neural network is able to address the over-fitting problem and by-pass the over-smoothing challenge for graph neural networks. Furthermore, we also propose an efficient and informative measure named MICS to quantify the over-smoothing problem. Our NODE-SELECT achieved or matched state-of-the art results in a number of transductive experiments over different benchmark datasets.
Grey-box Extraction of Natural Language Models
Authors: Santiago Zanella-Beguelin, Shruti Tople, Andrew Paverd, Boris Köpf
Date: 2021-01-01 00:00:00
Tasks: Model extraction

Model extraction attacks attempt to replicate a target machine learning model from predictions obtained by querying its inference API. Most existing attacks on Deep Neural Networks achieve this by supervised training of the copy using the victim's predictions. An emerging class of attacks exploit algebraic properties of DNNs to obtain high-fidelity copies using orders of magnitude fewer queries than the prior state-of-the-art. So far, such powerful attacks have been limited to networks with few hidden layers and ReLU activations.

In this paper we present algebraic attacks on large-scale natural language models in a grey-box setting, targeting models with a pre-trained (public) encoder followed by a single (private) classification layer. Our key observation is that a small set of arbitrary embedding vectors is likely to form a basis of the classification layer's input space, which a grey-box adversary can compute. We show how to use this information to solve an equation system that determines the classification layer from the corresponding probability outputs.

We evaluate the effectiveness of our attacks on different sizes of transformer models and downstream tasks. Our key findings are that (i) with frozen base layers, high-fidelity extraction is possible with a number of queries that is as small as twice the input dimension of the last layer. This is true even for queries that are entirely in-distribution, making extraction attacks indistinguishable from legitimate use; (ii) with fine-tuned base layers, the effectiveness of algebraic attacks decreases with the learning rate, showing that fine-tuning is not only beneficial for accuracy but also indispensable for model confidentiality.
Cortico-cerebellar networks as decoupled neural interfaces
Authors: Joseph Pemberton, Ellen Boven, Richard Apps, Rui Ponte Costa
Date: 2021-01-01 00:00:00
Tasks: Caption Generation

The brain solves the credit assignment problem remarkably well. For credit to be correctly assigned across multiple cortical areas a given area should, in principle, wait for others to finish their computation. How the brain deals with this locking problem has remained unclear. Deep learning methods suffer from similar locking constraints both on the forward and backward phase. Recently, decoupled neural interfaces (DNI) were introduced as a solution to the forward and backward locking problems.
Here we propose that a specialised brain region, the cerebellum, helps the cerebral cortex solve the locking problem closely matching the computations and architecture of DNI. In particular, we propose that classical cerebellar forward and inverse models are equivalent to solving the backward and forward locking problems, respectively. We tested the cortico-cerebellar-DNI (CC-DNI) model in a range of sensorimotor and cognitive tasks that have been shown to be cerebellar-dependent. First, we show that the CC-DNI unlocking mechanisms facilitate learning in a simple target reaching task. Next, by building on the sequential MNIST task we demonstrate that these results generalise to more complex sensorimotor tasks. Our cortico-cerebellar model readily applies to a wider range of modalities, to demonstrate this we tested the model in a cognitive task, caption generation. Models without the cerebellar-DNI component exhibit deficits similar to what has been observed in cerebellar patients in both motor and cognitive tasks. Finally, we introduce a CC-DNI model with highly sparse connectivity as observed in the cerebellum, which substantially reduces the number of parameters while improving learning through decorrelation.
Overall, our work offers a novel perspective on the cerebellum as a brain-wide decoupling machine for efficient credit assignment and opens a new avenue of research between deep learning and neuroscience.
Evidence against implicitly recurrent computations in residual neural networks
Authors: Samuel Lippl, Benjamin Peters, Nikolaus Kriegeskorte
Date: 2021-01-01 00:00:00

Recent work on residual neural networks (ResNets) has suggested that a ResNet's deep feedforward computation may be characterized as implicitly recurrent in that it iteratively refines the same representation like a recurrent network. To test this hypothesis, we manipulate the degree of weight sharing across layers in ResNets using soft gradient coupling. This new method, which provides a form of recurrence regularization, can interpolate smoothly between an ordinary ResNet and a ``"recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). We define three indices of recurrent iterative computation and show that a higher degree of gradient coupling promotes iterative convergent computation in ResNets. To measure the degree of weight sharing, we quantify the effective number of parameters of models along the continuum between nonrecurrent and recurrent. For a given effective number of parameters, recurrence regularization does not improve classification accuracy on three visual recognition tasks (MNIST, CIFAR-10, Digitclutter). ResNets, thus, may not benefit from a more similar sets of weights across layers, suggesting that their power does not derive from implicitly recurrent computation.