pwc-archive/papers-with-abstracts huggingface dataset:
DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks
Authors: Vasimuddin Md, Sanchit Misra, Guixiang Ma, Ramanarayan Mohanty, Evangelos Georganas, Alexander Heinecke, Dhiraj Kalamkar, Nesreen K. Ahmed, Sasikanth Avancha
Date: 2021-04-14 00:00:00
Tasks: CPU, graph partitioning

Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. It is challenging due to large memory capacity and bandwidth requirements on a single compute node and high communication volumes across multiple nodes. In this paper, we present DistGNN that optimizes the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters via an efficient shared memory implementation, communication reduction using a minimum vertex-cut graph partitioning algorithm and communication avoidance using a family of delayed-update algorithms. Our results on four common GNN benchmark datasets: Reddit, OGB-Products, OGB-Papers and Proteins, show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets, respectively, over baseline DGL implementations running on a single CPU socket
BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning
Authors: Heng Zhu, Qing Ling
Date: 2021-04-14 00:00:00
Tasks: Federated Learning

Communication between workers and the master node to collect local stochastic gradients is a key bottleneck in a large-scale federated learning system. Various recent works have proposed to compress the local stochastic gradients to mitigate the communication overhead. However, robustness to malicious attacks is rarely considered in such a setting. In this work, we investigate the problem of Byzantine-robust compressed federated learning, where the attacks from Byzantine workers can be arbitrarily malicious. We theoretically point out that different to the attacks-free compressed stochastic gradient descent (SGD), its vanilla combination with geometric median-based robust aggregation seriously suffers from the compression noise in the presence of Byzantine attacks. In light of this observation, we propose to reduce the compression noise with gradient difference compression so as to improve the Byzantine-robustness. We also observe the impact of the intrinsic stochastic noise caused by selecting random samples, and adopt the stochastic average gradient algorithm (SAGA) to gradually eliminate the inner variations of regular workers. We theoretically prove that the proposed algorithm reaches a neighborhood of the optimal solution at a linear convergence rate, and the asymptotic learning error is in the same order as that of the state-of-the-art uncompressed method. Finally, numerical experiments demonstrate the effectiveness of the proposed method.
Multi-Party Dual Learning
Authors: Maoguo Gong, Yuan Gao, Yu Xie, A. K. Qin, Ke Pan, Yew-Soon Ong
Date: 2021-04-14 00:00:00
Tasks: BIG-bench Machine Learning, Self-Learning

The performance of machine learning algorithms heavily relies on the availability of a large amount of training data. However, in reality, data usually reside in distributed parties such as different institutions and may not be directly gathered and integrated due to various data policy constraints. As a result, some parties may suffer from insufficient data available for training machine learning models. In this paper, we propose a multi-party dual learning (MPDL) framework to alleviate the problem of limited data with poor quality in an isolated party. Since the knowledge sharing processes for multiple parties always emerge in dual forms, we show that dual learning is naturally suitable to handle the challenge of missing data, and explicitly exploits the probabilistic correlation and structural relationship between dual tasks to regularize the training process. We introduce a feature-oriented differential privacy with mathematical proof, in order to avoid possible privacy leakage of raw features in the dual inference process. The approach requires minimal modifications to the existing multi-party learning structure, and each party can build flexible and powerful models separately, whose accuracy is no less than non-distributed self-learning approaches. The MPDL framework achieves significant improvement compared with state-of-the-art multi-party learning methods, as we demonstrated through simulations on real-world datasets.
End-to-end Keyword Spotting using Neural Architecture Search and Quantization
Authors: David Peter, Wolfgang Roth, Franz Pernkopf
Date: 2021-04-14 00:00:00
Tasks: Keyword Spotting, Neural Architecture Search, Quantization

This paper introduces neural architecture search (NAS) for the automatic discovery of end-to-end keyword spotting (KWS) models in limited resource environments. We employ a differentiable NAS approach to optimize the structure of convolutional neural networks (CNNs) operating on raw audio waveforms. After a suitable KWS model is found with NAS, we conduct quantization of weights and activations to reduce the memory footprint. We conduct extensive experiments on the Google speech commands dataset. In particular, we compare our end-to-end approach to mel-frequency cepstral coefficient (MFCC) based systems. For quantization, we compare fixed bit-width quantization and trained bit-width quantization. Using NAS only, we were able to obtain a highly efficient model with an accuracy of 95.55% using 75.7k parameters and 13.6M operations. Using trained bit-width quantization, the same model achieves a test accuracy of 93.76% while using on average only 2.91 bits per activation and 2.51 bits per weight.
A Novel Malware Detection Mechanism based on Features Extracted from Converted Malware Binary Images
Authors: Abhijitt Dhavlle, Sanket Shukla
Date: 2021-04-14 00:00:00
Tasks: Malware Detection

Our computer systems for decades have been threatened by various types of hardware and software attacks of which Malwares have been one of them. This malware has the ability to steal, destroy, contaminate, gain unintended access, or even disrupt the entire system. There have been techniques to detect malware by performing static and dynamic analysis of malware files, but, stealthy malware has circumvented the static analysis method and for dynamic analysis, there have been previous works that propose different methods to detect malware but, in this work we propose a novel technique to detect malware. We use malware binary images and then extract different features from the same and then employ different ML-classifiers on the dataset thus obtained. We show that this technique is successful in differentiating classes of malware based on the features extracted.
Root-finding Approaches for Computing Conformal Prediction Set
Authors: Eugene Ndiaye, Ichiro Takeuchi
Date: 2021-04-14 00:00:00
Tasks: Conformal Prediction, Prediction

Conformal prediction constructs a confidence set for an unobserved response of a feature vector based on previous identically distributed and exchangeable observations of responses and features. It has a coverage guarantee at any nominal level without additional assumptions on their distribution. Its computation deplorably requires a refitting procedure for all replacement candidates of the target response. In regression settings, this corresponds to an infinite number of model fits. Apart from relatively simple estimators that can be written as pieces of linear function of the response, efficiently computing such sets is difficult, and is still considered as an open problem. We exploit the fact that, \emph{often}, conformal prediction sets are intervals whose boundaries can be efficiently approximated by classical root-finding algorithms. We investigate how this approach can overcome many limitations of formerly used strategies; we discuss its complexity and drawbacks.
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
Authors: Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela
Date: 2021-04-14 00:00:00
Proceeding: EMNLP 2021 11
Tasks: Language Modeling, Language Modelling, Masked Language Modeling

A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks -- including on tasks specifically designed to be challenging for models that ignore word order. Our models perform surprisingly well according to some parametric syntactic probes, indicating possible deficiencies in how we test representations for syntactic information. Overall, our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.
Generative Causal Explanations for Graph Neural Networks
Authors: WanYu Lin, Hao Lan, Baochun Li
Date: 2021-04-14 00:00:00
Tasks: Graph Learning

This paper presents Gem, a model-agnostic approach for providing interpretable explanations for any GNNs on various graph learning tasks. Specifically, we formulate the problem of providing explanations for the decisions of GNNs as a causal learning task. Then we train a causal explanation model equipped with a loss function based on Granger causality. Different from existing explainers for GNNs, Gem explains GNNs on graph-structured data from a causal perspective. It has better generalization ability as it has no requirements on the internal structure of the GNNs or prior knowledge on the graph learning tasks. In addition, Gem, once trained, can be used to explain the target GNN very quickly. Our theoretical analysis shows that several recent explainers fall into a unified framework of additive feature attribution methods. Experimental results on synthetic and real-world datasets show that Gem achieves a relative increase of the explanation accuracy by up to $30\%$ and speeds up the explanation process by up to $110\times$ as compared to its state-of-the-art alternatives.
A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals
Authors: Connor van Rossum, Candice Feinberg, Adam Abu Shumays, Kyle Baxter, Benedek Bartha
Date: 2021-04-14 00:00:00
Tasks: Generative Adversarial Network, reinforcement-learning, Reinforcement Learning (RL)

Two key challenges within Reinforcement Learning involve improving (a) agent learning within environments with sparse extrinsic rewards and (b) the explainability of agent actions. We describe a curious subgoal focused agent to address both these challenges. We use a novel method for curiosity produced from a Generative Adversarial Network (GAN) based model of environment transitions that is robust to stochastic environment transitions. Additionally, we use a subgoal generating network to guide navigation. The explainability of the agent's behavior is increased by decomposing complex tasks into a sequence of interpretable subgoals that do not require any manual design. We show that this method also enables the agent to solve challenging procedurally-generated tasks that contain stochastic transitions above other state-of-the-art methods.
Detection of Dataset Shifts in Learning-Enabled Cyber-Physical Systems using Variational Autoencoder for Regression
Authors: Feiyang Cai, Ali I. Ozdagli, Xenofon Koutsoukos
Date: 2021-04-14 00:00:00
Tasks: Anomaly Detection, regression, Self-Driving Cars

Cyber-physical systems (CPSs) use learning-enabled components (LECs) extensively to cope with various complex tasks under high-uncertainty environments. However, the dataset shifts between the training and testing phase may lead the LECs to become ineffective to make large-error predictions, and further, compromise the safety of the overall system. In our paper, we first provide the formal definitions for different types of dataset shifts in learning-enabled CPS. Then, we propose an approach to detect the dataset shifts effectively for regression problems. Our approach is based on the inductive conformal anomaly detection and utilizes a variational autoencoder for regression model which enables the approach to take into consideration both LEC input and output for detecting dataset shifts. Additionally, in order to improve the robustness of detection, layer-wise relevance propagation (LRP) is incorporated into our approach. We demonstrate our approach by using an advanced emergency braking system implemented in an open-source simulator for self-driving cars. The evaluation results show that our approach can detect different types of dataset shifts with a small number of false alarms while the execution time is smaller than the sampling period of the system.
Search to aggregate neighborhood for graph neural network
Authors: Huan Zhao, Quanming Yao, WeiWei Tu
Date: 2021-04-14 00:00:00
Tasks: AutoML, Graph Neural Network, Neural Architecture Search

Recent years have witnessed the popularity and success of graph neural networks (GNN) in various scenarios. To obtain data-specific GNN architectures, researchers turn to neural architecture search (NAS), which has made impressive success in discovering effective architectures in convolutional neural networks. However, it is non-trivial to apply NAS approaches to GNN due to challenges in search space design and the expensive searching cost of existing NAS methods. In this work, to obtain the data-specific GNN architectures and address the computational challenges facing by NAS approaches, we propose a framework, which tries to Search to Aggregate NEighborhood (SANE), to automatically design data-specific GNN architectures. By designing a novel and expressive search space, we propose a differentiable search algorithm, which is more efficient than previous reinforcement learning based methods. Experimental results on four tasks and seven real-world datasets demonstrate the superiority of SANE compared to existing GNN models and NAS approaches in terms of effectiveness and efficiency. (Code is available at: https://github.com/AutoML-4Paradigm/SANE).
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Authors: Guanghui Qin, Jason Eisner
Date: 2021-04-14 00:00:00
Proceeding: NAACL 2021 4
Tasks: Language Modeling, Language Modelling

Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks, using a fill-in-the-blank paradigm (Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al., 2020). For example, language models retain factual knowledge from their training corpora that can be extracted by asking them to "fill in the blank" in a sentential prompt. However, where does this prompt come from? We explore the idea of learning prompts by gradient descent -- either fine-tuning prompts taken from previous work, or starting from random initialization. Our prompts consist of "soft words," i.e., continuous vectors that are not necessarily word type embeddings from the language model. Furthermore, for each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them. Across multiple English LMs and tasks, our approach hugely outperforms previous methods, showing that the implicit factual knowledge in language models was previously underestimated. Moreover, this knowledge is cheap to elicit: random initialization is nearly as good as informed initialization.
Learning Regularization Parameters of Inverse Problems via Deep Neural Networks
Authors: Babak Maboudi Afkham, Julianne Chung, Matthias Chung
Date: 2021-04-14 00:00:00
Tasks: Bilevel Optimization, Experimental Design

In this work, we describe a new approach that uses deep neural networks (DNN) to obtain regularization parameters for solving inverse problems. We consider a supervised learning approach, where a network is trained to approximate the mapping from observation data to regularization parameters. Once the network is trained, regularization parameters for newly obtained data can be computed by efficient forward propagation of the DNN. We show that a wide variety of regularization functionals, forward models, and noise models may be considered. The network-obtained regularization parameters can be computed more efficiently and may even lead to more accurate solutions compared to existing regularization parameter selection methods. We emphasize that the key advantage of using DNNs for learning regularization parameters, compared to previous works on learning via optimal experimental design or empirical Bayes risk minimization, is greater generalizability. That is, rather than computing one set of parameters that is optimal with respect to one particular design objective, DNN-computed regularization parameters are tailored to the specific features or properties of the newly observed data. Thus, our approach may better handle cases where the observation is not a close representation of the training set. Furthermore, we avoid the need for expensive and challenging bilevel optimization methods as utilized in other existing training approaches. Numerical results demonstrate the potential of using DNNs to learn regularization parameters.
Five Degree-of-Freedom Property Interpolation of Arbitrary Grain Boundaries via Voronoi Fundamental Zone Octonion Framework
Authors: Sterling G. Baird, Eric R. Homer, David T. Fullwood, Oliver K. Johnson
Date: 2021-04-14 00:00:00
Tasks: CPU, GPR

We introduce the Voronoi fundamental zone octonion interpolation framework for grain boundary (GB) structure-property models and surrogates. The VFZO framework offers an advantage over other five degree-of-freedom based property interpolation methods because it is constructed as a point set in a manifold. This means that directly computed Euclidean distances approximate the original octonion distance with significantly reduced computation runtime (~7 CPU minutes vs. 153 CPU days for a 50000x50000 pairwise-distance matrix). This increased efficiency facilitates lower interpolation error through the use of significantly more input data. We demonstrate grain boundary energy interpolation results for a non-smooth validation function and simulated bi-crystal datasets for Fe and Ni using four interpolation methods: barycentric interpolation, Gaussian process regression (GPR), inverse-distance weighting, and nearest-neighbor interpolation. These are evaluated for 50000 random input GBs and 10 000 random prediction GBs. The best performance was achieved with GPR, which resulted in a reduction of the root mean square error (RMSE) by 83.0% relative to RMSE of a constant, average model. Likewise, interpolation on a large, noisy, molecular statics Fe simulation dataset improves performance by 34.4% compared to 21.2% in prior work. Interpolation on a small, low-noise MS Ni simulation dataset is similar to interpolation results for the original octonion metric (57.6% vs. 56.4%). A vectorized, parallelized, MATLAB interpolation function (interp5DOF.m) and related routines are available in our VFZO repository (github.com/sgbaird-5dof/interp) which can be applied to other crystallographic point groups. The VFZO framework offers advantages for computing distances between GBs, estimating property values for arbitrary GBs, and modeling surrogates of computationally expensive 5DOF functions and simulations.
Joint Negative and Positive Learning for Noisy Labels
Authors: Youngdong Kim, Juseung Yun, Hyounguk Shon, Junmo Kim
Date: 2021-04-14 00:00:00
Proceeding: CVPR 2021 1

Training of Convolutional Neural Networks (CNNs) with data with noisy labels is known to be a challenge. Based on the fact that directly providing the label to the data (Positive Learning; PL) has a risk of allowing CNNs to memorize the contaminated labels for the case of noisy data, the indirect learning approach that uses complementary labels (Negative Learning for Noisy Labels; NLNL) has proven to be highly effective in preventing overfitting to noisy data as it reduces the risk of providing faulty target. NLNL further employs a three-stage pipeline to improve convergence. As a result, filtering noisy data through the NLNL pipeline is cumbersome, increasing the training cost. In this study, we propose a novel improvement of NLNL, named Joint Negative and Positive Learning (JNPL), that unifies the filtering pipeline into a single stage. JNPL trains CNN via two losses, NL+ and PL+, which are improved upon NL and PL loss functions, respectively. We analyze the fundamental issue of NL loss function and develop new NL+ loss function producing gradient that enhances the convergence of noisy data. Furthermore, PL+ loss function is designed to enable faster convergence to expected-to-be-clean data. We show that the NL+ and PL+ train CNN simultaneously, significantly simplifying the pipeline, allowing greater ease of practical use compared to NLNL. With a simple semi-supervised training technique, our method achieves state-of-the-art accuracy for noisy data classification based on the superior filtering ability.
Identity Inference on Blockchain using Graph Neural Network
Authors: Jie Shen, Jiajun Zhou, Yunyi Xie, Shanqing Yu, Qi Xuan
Date: 2021-04-14 00:00:00
Tasks: Graph Classification, Graph Mining, Graph Neural Network

The anonymity of blockchain has accelerated the growth of illegal activities and criminal behaviors on cryptocurrency platforms. Although decentralization is one of the typical characteristics of blockchain, we urgently call for effective regulation to detect these illegal behaviors to ensure the safety and stability of user transactions. Identity inference, which aims to make a preliminary inference about account identity, plays a significant role in blockchain security. As a common tool, graph mining technique can effectively represent the interactive information between accounts and be used for identity inference. However, existing methods cannot balance scalability and end-to-end architecture, resulting high computational consumption and weak feature representation. In this paper, we present a novel approach to analyze user's behavior from the perspective of the transaction subgraph, which naturally transforms the identity inference task into a graph classification pattern and effectively avoids computation in large-scale graph. Furthermore, we propose a generic end-to-end graph neural network model, named $\text{I}^2 \text{BGNN}$, which can accept subgraph as input and learn a function mapping the transaction subgraph pattern to account identity, achieving de-anonymization. Extensive experiments on EOSG and ETHG datasets demonstrate that the proposed method achieve the state-of-the-art performance in identity inference.
Situational Confidence Assistance for Lifelong Shared Autonomy
Authors: Matthew Zurek, Andreea Bobu, Daniel S. Brown, Anca D. Dragan
Date: 2021-04-14 00:00:00
Tasks: Lifelong learning

Shared autonomy enables robots to infer user intent and assist in accomplishing it. But when the user wants to do a new task that the robot does not know about, shared autonomy will hinder their performance by attempting to assist them with something that is not their intent. Our key idea is that the robot can detect when its repertoire of intents is insufficient to explain the user's input, and give them back control. This then enables the robot to observe unhindered task execution, learn the new intent behind it, and add it to this repertoire. We demonstrate with both a case study and a user study that our proposed method maintains good performance when the human's intent is in the robot's repertoire, outperforms prior shared autonomy approaches when it isn't, and successfully learns new skills, enabling efficient lifelong learning for confidence-based shared autonomy.