pwc-archive/papers-with-abstracts huggingface dataset:
GINN: Fast GPU-TEE Based Integrity for Neural Network Training
Authors: Aref Asvadishirehjini, Murat Kantarcioglu, Bradley A. Malin
Date: 2021-01-01 00:00:00
Tasks: GPU, Self-Driving Cars

Machine learning models based on Deep Neural Networks (DNNs) are increasingly being deployed in a wide range of applications ranging from self-driving cars to Covid-19 diagnostics. The computational power necessary to learn a DNN is non-trivial. So, as a result, cloud environments with dedicated hardware support emerged as important infrastructure. However, outsourcing computation to the cloud raises security, privacy, and integrity challenges. To address these challenges, previous works tried to leverage homomorphic encryption, secure multi-party computation, and trusted execution environments (TEE). Yet, none of these approaches can scale up to support realistic DNN model training workloads with deep architectures and millions of training examples without sustaining a significant performance hit. In this work, we focus on the setting where the integrity of the outsourced Deep Learning (DL) model training is ensured by TEE. We choose the TEE based approach because it has been shown to be more efficient compared to the pure cryptographic solutions, and the availability of TEEs on cloud environments. To mitigate the loss in performance, we combine random verification of selected computation steps with careful adjustments of DNN  used for training. Our experimental results show that the proposed approach may achieve 2X to 20X performance improvement compared to the pure TEE based solution while guaranteeing the integrity of the computation with high probability (e.g., 0.999) against the state-of-the-art DNN backdoor attacks.
Learning to Disentangle Textual Representations and Attributes via Mutual Information
Authors: Pierre Colombo, Chloé Clavel, Pablo Piantanida
Date: 2021-01-01 00:00:00
Tasks: Attribute, Disentanglement, Sentence, Style Transfer

Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification (\textit{e.g.}  building classifiers whose decisions cannot disproportionately hurt or benefit specific groups identified by sensitive attributes), style transfer and sentence generation, among others. The existent dominant approaches in the context of text data have been  based on training an adversary (discriminator or teacher) that aims at making  attribute values difficult to be   inferred from the latent code. Although these approaches are remarkably simple and even though the adversary seems to be performing perfectly  during the training phase, after training is completed a fair amount of sensitive information to infer the attribute still remains. This paper investigates learning to disentangle representations by minimizing a novel variational (upper) bound of the mutual information between an identified attribute and the latent code of a deep neural network encoder. We demonstrate that our surrogate leads to better disentangled representations on both fair classification and sentence generation tasks while not suffering from the degeneracy of adversarial losses in multi-class scenarios. Furthermore, by optimizing the trade-off between the level of disentanglement and quality of the generated sentences for polarity transfer and sentence generation tasks, we provide some lights to the well-known debate on whether or not \textit{``disentangled representations may be helpful for polarity transfer and sentence generation purposes''}.
A spectral perspective on GCNs
Authors: Nathan Grinsztajn, Philippe Preux, Edouard Oyallon
Date: 2021-01-01 00:00:00

In this work, we study the behavior of standard GCNs under spectral manipulations. We relate the numerical performances of a GCN through bandpass filtering: we empirically show that most of the necessary and used information for nodes classification is contained in the low-frequency domain, through various ablation experiments. In particular, it is possible to recover accuracies competitive with the state of the art with simple MLPs and only a few low frequencies. We further derive a simple procedure to adjust the trade-off between smoothing and high-frequency propagation for deeper GCNs, without diagonalizing the Laplacian. We conduct our experiments on standard and challenging benchmarks.
Attention-driven Robotic Manipulation
Authors: Stephen James, Andrew Davison
Date: 2021-01-01 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Despite the success of reinforcement learning methods, they have yet to have their breakthrough moment when applied to a broad range of robotic manipulation tasks. This is partly due to the fact that reinforcement learning algorithms are notoriously difficult and time consuming to train, which is exacerbated when training from images rather than full-state inputs. As humans perform manipulation tasks, our eyes closely monitor every step of the process with our gaze focusing sequentially on the objects being manipulated. With this in mind, we present our Attention-driven Robotic Manipulation (ARM) algorithm, which is a general manipulation algorithm that can be applied to a range of real-world sparse-rewarded tasks without any prior task knowledge. ARM splits the complex task of manipulation into a 3 stage pipeline: (1) a Q-attention agent extracts interesting pixel locations from RGB and point cloud inputs, (2) a next-best pose agent that accepts crops from the Q-attention agent and outputs poses, and (3) a control agent that takes the goal pose and outputs joint actions. We show that current state-of-the-art reinforcement learning algorithms catastrophically fail on a range of RLBench tasks, whilst ARM is successful within a few hours.
Learning to Dynamically Select Between Reward Shaping Signals
Authors: Alexander Politowicz, Bing Liu
Date: 2021-01-01 00:00:00
Tasks: Reinforcement Learning (RL)

Reinforcement learning (RL) algorithms often have the limitation of sample complexity. Previous research has shown that the reliance on large amounts of experience can be mitigated through the presence of additional feedback. Automatic reward shaping is one approach to solving this problem, using automatic identification and modulation of shaping reward signals that are more informative about how agents should behave in any given scenario to learn and adapt faster. However, automatic reward shaping is still very challenging. To better study it, we break it down into two separate sub-problems: learning shaping reward signals in an application and learning how the signals can be adaptively used to provide a single reward feedback in the RL learning process. This paper focuses on the latter sub-problem. Unlike existing research, which tries to learn one shaping reward function from shaping signals, the proposed method learns to dynamically select the right reward signal to apply at each state, which is considerably more flexible. We further show that using an online strategy that seeks to match the learned shaping feedback with optimal value differences can lead to effective reward shaping and accelerated learning. The proposed ideas are verified through experiments in a variety of environments using different shaping reward paradigms.
Fair Empirical Risk Minimization via Exponential Rényi Mutual Information
Authors: Rakesh Pavan, Andrew Lowy, Sina Baharlouei, Meisam Razaviyayn, Ahmad Beirami
Date: 2021-01-01 00:00:00
Tasks: Attribute, Fairness, Stochastic Optimization

Several notions of fairness, such as demographic parity and equal opportunity, are defined based on statistical independence between a predicted target and a sensitive attribute. In machine learning applications, however, the data distribution is unknown to the learner and statistical independence is not verifiable. Hence, the learner could only resort to empirical evaluation of the degree of fairness violation. Many fairness violation notions are defined as a divergence/distance  between the joint distribution of the target and  sensitive attributes and the Kronecker product of their marginals, such as Rényi correlation,  mutual information, $L_1$ distance, to name a few. In this paper, we propose  another notion of fairness violation, called Exponential Rényi Mutual Information (ERMI) between sensitive attributes and the predicted target. We show that ERMI is a strong fairness violation notion in the sense that it provides an upper bound guarantee on all of the aforementioned notions of fairness violation.  We also propose the Fair Empirical Risk Minimization via ERMI regularization framework, called FERMI. We provide scalable batch and stochastic optimization methods to solve FERMI with theoretical convergence guarantees. Our experiments show that FERMI achieves the most favorable tradeoffs between fairness violation and performance on test data across different problem setups, even when fairness violation is measured in notions other than ERMI.
Navigating the Trade-Off between Learning Efficacy and Processing Efficiency in Deep Neural Networks
Authors: Anonymous
Date: 2021-01-01 00:00:00
Tasks: Meta-Learning

A number of training protocols in machine learning seek to enhance learning efficacy by training a single agent on multiple tasks in sequence. Sequential acquisition exploits the discovery of common structure between tasks in the form of shared representations to improve learning speed and generalization. The learning of shared representations, however, is known to impair the execution of multiple tasks in parallel. The parallel execution of tasks results in higher efficiency of processing and is promoted by separating representations between tasks to avoid processing interference. Here, we build on previous work involving shallow networks and simple task settings suggesting that there is a trade-off between learning efficacy and processing efficiency, mediated by the use of shared versus separated representations. We show that the same tension arises in deep networks and discuss a meta-learning algorithm for an agent to manage this trade-off in an unfamiliar environment. We display through different experiments that the agent is able to successfully optimize its training strategy as a function of the environment.
A Robust Fuel Optimization Strategy For Hybrid Electric Vehicles: A Deep Reinforcement Learning Based Continuous Time Design Approach
Authors: Nilanjan Mukherjee, Sudeshna Sarkar
Date: 2021-01-01 00:00:00
Tasks: Deep Reinforcement Learning, Management, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

This paper deals with the fuel optimization problem for hybrid electric vehicles in deep reinforcement learning framework. Firstly, considering the hybrid electric vehicle as an uncertain non-linear system with
unknown dynamics in continuous time frame, we solve an open-loop deterministic trajectory optimization problem without explicitly considering
the system model. This is followed by the design of a deep reinforcement
learning based optimal control law for the non-linear system (i.e., hybrid
electric vehicles) such that the actual states and the control policy remain
close to the optimal trajectory and optimal policy even in the presence of
external disturbances, modeling errors, uncertainties and noise. The low
value of the H-infinity ($H_{\infty})$ performance index (i.e., the ratio of the disturbance to the control energy) of the RL based optimization technique in
comparison with the traditional methods addresses the robustness issue.
The control strategy will also autonomously learn the optimal policy and
adapt itself to the different conditions which is in sharp contrast to the
conventional techniques that mostly depend on a set of pre-defined rules
and provide sub-optimal solutions to the fuel management problem. The
controller thus designed is compared with the traditional fuel optimization strategies for hybrid electric vehicles to illustate the efficacy of the
proposed method.
On Sparse Critical Paths of Neural Response
Authors: Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Seong Tae Kim, Nassir Navab
Date: 2021-01-01 00:00:00

Is critical input information encoded in specific sparse paths within the network? The pruning objective --- finding a subset of neurons for which the response remains unchanged --- has been used to discover such paths. However, we show that paths obtained from this objective do not necessarily encode the input features and also encompass (dead) neurons that were not originally contributing to the response. We investigate selecting paths based on neurons' contributions to the response to ensure that the paths envelop the critical segments of the encoded input information. We show that these paths have the property of being provably locally linear in an $\ell_{2}$-ball of the input, thus having stable gradients. This property is leveraged for proposing a feature attribution paradigm that is guided by neurons, therefore inherently taking interactions between input features into account. We evaluate the attribution methodology quantitatively in mainstream benchmarks.
Synthesizer: Rethinking Self-Attention for Transformer Models
Authors: Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, Che Zheng
Date: 2021-01-01 00:00:00
Tasks: Language Modeling, Language Modelling, Machine Translation, Text Generation, Translation

The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.
A Flexible Framework for Discovering Novel Categories with Contrastive Learning
Authors: Xuhui Jia, Kai Han, Yukun Zhu, Bradley Green
Date: 2021-01-01 00:00:00
Tasks: Contrastive Learning, Representation Learning

This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we proposed using category discrimination on labelled data and cross-modal discrimination on multi-modal data to augment instance discrimination used in conventional contrastive learning approaches. We further introduce Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data to better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet, obtaining state-of-the-art results.
K-PLUG: KNOWLEDGE-INJECTED PRE-TRAINED LANGUAGE MODEL FOR NATURAL LANGUAGE UNDERSTANDING AND GENERATION
Authors: Song Xu, Haoran Li, Peng Yuan, Yujia Wang, Youzheng Wu, Xiaodong He, Ying Liu, BoWen Zhou
Date: 2021-01-01 00:00:00
Tasks: Chatbot, Decoder, Knowledge Base Completion, Language Modeling, Language Modelling, Natural Language Understanding, Self-Supervised Learning, Text Summarization

Existing pre-trained language models (PLMs) have demonstrated the effectiveness of self-supervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. In this paper, we propose K-PLUG, a knowledge-injected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks. We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge. Specifically, we propose five knowledge-aware self-supervised pre-training objectives to formulate the learning of domain-specific knowledge, including e-commerce domain-specific knowledge-bases, aspects of product entities,  categories of product entities, and unique selling propositions of product entities. K-PLUG achieves new state-of-the-art results on a suite of domain-specific NLP tasks, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue, significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks. The code,  data, and models will be publicly available.
Reinforcement Learning Based Asymmetrical DNN Modularization for Optimal Loading
Authors: Brijraj Singh, Yash Jain, Mayukh Das, Praveen Doreswamy Naidu
Date: 2021-01-01 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Latency of DNN (Deep Neural Network) based prediction is the summation of model loading latency and inference latency. Model loading latency affects the first response from the applications, whereas inference latency affects the subsequent responses. As model loading latency is directly proportional to the model size, this work aims at improving the response time of an intelligent app by reducing the loading latency. The speedup is gained by asymmetrically modularizing the given DNN model among several small child models and loading them in parallel. The decision about number of feasible child models and their corresponding split positions are taken care by reinforcement learning unit (RLU). RLU takes into account the available hardware resources on-device and provides the best splitting index $k$ and their positions  $\vec{p}$ specific to the DNN model and device, where  $\vec{p}=(p_1, p_2, ..., p_k)$ and $p_i$  is the end position of $i^{th}$  child: $M_i$. The proposed method has shown significant loading improvement (up to 7X) on popular DNNs, used for camera use-case. The proposed method can be used to speed up the app response. Along with that RLU driven approach facilitates for On-device personalization by separating one module only with trainable layers and loading that particular module while training on-device.
Adversarial Privacy Preservation in MRI Scans of the Brain
Authors: Lennart Alexander Van der Goten, Tobias Hepp, Zeynep Akata, Kevin Smith
Date: 2021-01-01 00:00:00
Tasks: De-identification

De-identification of magnetic resonance imagery (MRI) is intrinsically difficult since, even with all metadata removed, a person's face can easily be rendered and matched against a database. Existing de-identification methods tackle this task by obfuscating or removing parts of the face, but they either fail to reliably hide the patient's identity or they remove so much information that they adversely affect further analyses in the 3D space surrounding the face. In this work, we describe a new class of MRI de-identification techniques that remodel privacy-sensitive facial features as opposed to removing them.To accomplish this, we propose a conditional, multi-scale, 3D GAN architecture that takes a patient's MRI scan as input and generates a 3D volume in which the brain is not modified but the face has been de-identified. Compared to the classical removal-based techniques, our deep learning framework preserves privacy more reliably without adversely affecting downstream medical analyses on the brain, including segmentation and age prediction.
Dynamic Probabilistic Pruning: Training sparse networks based on stochastic and dynamic masking
Authors: Lizeth Gonzalez Carabarin, Iris A.M. Huijben, Bastiaan S. Veeling, Alexandre Schmid, Ruud Van Sloun
Date: 2021-01-01 00:00:00
Tasks: Model Compression

Deep Learning (DL) models are known to be heavily over-parametrized, resulting in a large memory footprint and power consumption. This hampers the use of such models in hardware-constrained edge technologies such as wearables and mobile devices. Model compression during training can be achieved by promoting sparse network structures both through weight regularization and by leveraging dynamic pruning methods. State-of-the-art pruning methods are however mostly magnitude-based which impedes their use in e.g. binary settings. Importantly, most of the pruning methods do not provide a structural sparsity, resulting in an inefficient memory allocation and access for hardware implementations.  In this paper, we propose a novel dynamic pruning solution that we term Dynamic Probabilistic Pruning (DPP). DPP leverages Gumbel top-K sampling to select subsets of weights during training, which enables exploring which weights are most relevant. Our approach allows for setting an explicit per-neuron layer-wise sparsity level and structural pruning across weights and feature maps, without relying on weight magnitude heuristics. Relevantly, our method generates a hardware-oriented structural sparsity for fully-connected and convolutional layers that facilitates memory allocation and access, in contrast with conventional unstructured pruning.
We show that DPP achieves competitive sparsity levels and classification accuracy on MNIST and CIFAR-10, CIFAR-100 datasets compared to a state-of-the-art baseline for various DL architectures, while respecting per-neuron sparsity constraints.
Differentiable End-to-End Program Executor for Sample and Computationally Efficient VQA
Authors: Karan Samel, Zelin Zhao, Kuan Wang, Robin Luo, Binghong Chen, Le Song
Date: 2021-01-01 00:00:00
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

We present a differentiable end-to-end program executor (DePe), which addresses Visual Question Answering (VQA) in a sample and computationally efficient manner. DePe parses the question into probabilistic programs and softly executes them to acquire the final answer. These functional programs adopt soft-logic functions to enable approximate probabilistic logic reasoning. In addition to the language, DePe also jointly learns visual object-centric representations in an end-to-end manner. We demonstrate through extensive experiments that DePe is more sample and computationally efficient than other VQA methodologies while retaining state-of-the-art performance.