pwc-archive/papers-with-abstracts huggingface dataset:
Graph Neural Networks in TensorFlow and Keras with Spektral
Authors: Daniele Grattarola, Cesare Alippi
Date: 2020-06-22 00:00:00
Tasks: Deep Learning, General Classification, Graph Classification, Graph Regression, Node Classification, regression

In this paper we present Spektral, an open-source Python library for building graph neural networks with TensorFlow and the Keras application programming interface. Spektral implements a large set of methods for deep learning on graphs, including message-passing and pooling operators, as well as utilities for processing graphs and loading popular benchmark datasets. The purpose of this library is to provide the essential building blocks for creating graph neural networks, focusing on the guiding principles of user-friendliness and quick prototyping on which Keras is based. Spektral is, therefore, suitable for absolute beginners and expert deep learning practitioners alike. In this work, we present an overview of Spektral's features and report the performance of the methods implemented by the library in scenarios of node classification, graph classification, and graph regression.
Safe Reinforcement Learning via Curriculum Induction
Authors: Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, Alekh Agarwal
Date: 2020-06-22 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Autonomous Driving, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Safe Reinforcement Learning

In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. In such settings, the agent needs to behave safely not only after but also while learning. To achieve this, existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations during exploration with high probability, but both the probabilistic guarantees and the smoothness assumptions inherent in the priors are not viable in many scenarios of interest such as autonomous driving. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor that saves the agent from violating constraints during learning. In this model, we introduce the monitor that neither needs to know how to do well at the task the agent is learning nor needs to know how the environment works. Instead, it has a library of reset controllers that it activates when the agent starts behaving dangerously, preventing it from doing damage. Crucially, the choices of which reset controller to apply in which situation affect the speed of agent learning. Based on observing agents' progress, the teacher itself learns a policy for choosing the reset controllers, a curriculum, to optimize the agent's final policy reward. Our experiments use this framework in two environments to induce curricula for safe and efficient learning.
Learning to Generate Noise for Multi-Attack Robustness
Authors: Divyam Madaan, Jinwoo Shin, Sung Ju Hwang
Date: 2020-06-22 00:00:00
Tasks: Meta-Learning

Adversarial learning has emerged as one of the successful techniques to circumvent the susceptibility of existing methods against adversarial perturbations. However, the majority of existing defense methods are tailored to defend against a single category of adversarial perturbation (e.g. $\ell_\infty$-attack). In safety-critical applications, this makes these methods extraneous as the attacker can adopt diverse adversaries to deceive the system. Moreover, training on multiple perturbations simultaneously significantly increases the computational overhead during training. To address these challenges, we propose a novel meta-learning framework that explicitly learns to generate noise to improve the model's robustness against multiple types of attacks. Its key component is Meta Noise Generator (MNG) that outputs optimal noise to stochastically perturb a given sample, such that it helps lower the error on diverse adversarial perturbations. By utilizing samples generated by MNG, we train a model by enforcing the label consistency across multiple perturbations. We validate the robustness of models trained by our scheme on various datasets and against a wide variety of perturbations, demonstrating that it significantly outperforms the baselines across multiple perturbations with a marginal computational cost.
Organising a Successful AI Online Conference: Lessons from SoCS 2020
Authors: Daniel Harabor, Mauro Vallati
Date: 2020-06-22 00:00:00
Tasks: Scheduling

The 13th Symposium on Combinatorial Search (SoCS) was held May 26-28, 2020. Originally scheduled to take place in Vienna, Austria, the symposium pivoted toward a fully online technical program in early March. As an in-person event SoCS offers participants a diverse array of scholarly activities including technical talks (long and short), poster sessions, plenary sessions, a community meeting and, new for 2020, a Master Class tutorial program. This paper describes challenges, approaches and opportunities associated with adapting these many different activities to the online setting. We consider issues such as scheduling, dissemination, attendee interaction and community engagement before, during and after the event. We report on the approaches taken by SoCS in each case, we give a post-hoc analysis of their their effectiveness and we discuss how these decisions continue to impact the SoCS community in the days after SoCS 2020.
Supervised dimensionality reduction by a Linear Discriminant Analysis on pre-trained CNN features
Authors: Francisco J. H. Heras, Gonzalo G. de Polavieja
Date: 2020-06-22 00:00:00
Tasks: Dimensionality Reduction, Supervised dimensionality reduction

We explore the application of linear discriminant analysis (LDA) to the features obtained in different layers of pretrained deep convolutional neural networks (CNNs). The advantage of LDA compared to other techniques in dimensionality reduction is that it reduces dimensions while preserving the global structure of data, so distances in the low-dimensional structure found are meaningful. The LDA applied to the CNN features finds that the centroids of classes corresponding to the similar data lay closer than classes corresponding to different data. We applied the method to a modification of the MNIST dataset with ten additional classes, each new class with half of the images from one of the standard ten classes. The method finds the new classes close to the corresponding standard classes we took the data form. We also applied the method to a dataset of images of butterflies to find that related subspecies are found to be close. For both datasets, we find a performance similar to state-of-the-art methods.
Self-Supervised Representations Improve End-to-End Speech Translation
Authors: Anne Wu, Changhan Wang, Juan Pino, Jiatao Gu
Date: 2020-06-22 00:00:00
Tasks: Cross-Lingual Transfer, speech-recognition, Speech Recognition, Speech-to-Text, Speech-to-Text Translation, Translation

End-to-end speech-to-text translation can provide a simpler and smaller system but is facing the challenge of data scarcity. Pre-training methods can leverage unlabeled data and have been shown to be effective on data-scarce settings. In this work, we explore whether self-supervised pre-trained speech representations can benefit the speech translation task in both high- and low-resource settings, whether they can transfer well to other languages, and whether they can be effectively combined with other common methods that help improve low-resource end-to-end speech translation such as using a pre-trained high-resource speech recognition system. We demonstrate that self-supervised pre-trained features can consistently improve the translation performance, and cross-lingual transfer allows to extend to a variety of languages without or with little tuning.
Learning with AMIGo: Adversarially Motivated Intrinsic Goals
Authors: Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B. Tenenbaum, Tim Rocktäschel, Edward Grefenstette
Date: 2020-06-22 00:00:00
Proceeding: ICLR 2021 1
Tasks: Meta-Learning, Reinforcement Learning (RL)

A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose AMIGo, a novel agent incorporating -- as form of meta-learning -- a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals to train a goal-conditioned "student" policy in the absence of (or alongside) environment reward. Specifically, through a simple but effective "constructively adversarial" objective, the teacher learns to propose increasingly challenging -- yet achievable -- goals that allow the student to learn general skills for acting in a new environment, independent of the task to be solved. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks where other forms of intrinsic motivation and state-of-the-art RL methods fail.
Automated machine vision enabled detection of movement disorders from hand drawn spirals
Authors: Nabeel Seedat, Vered Aharonson, Ilana Schlesinger
Date: 2020-06-22 00:00:00

A widely used test for the diagnosis of Parkinson's disease (PD) and Essential tremor (ET) is hand-drawn shapes,where the analysis is observationally performed by the examining neurologist. This method is subjective and is prone to bias amongst different physicians. Due to the similarities in the symptoms of the two diseases, they are often misdiagnosed.Studies which attempt to automate the process typically use digitized input, where the tablet or specialized equipment are not affordable in many clinical settings. This study uses a dataset of scanned pen and paper drawings and a convolutional neural network (CNN) to perform classification between PD, ET and control subjects. The discrimination accuracy of PD from controls was 98.2%. The discrimination accuracy of PD from ET and from controls was 92%. An ablation study was conducted and indicated that correct hyper parameter optimization can increases the accuracy up to 4.33%. Finally, the study indicates the viability of using a CNN-enabled machine vision system to provide robust and accurate detection of movement disorders from hand drawn spirals.
The color out of space: learning self-supervised representations for Earth Observation imagery
Authors: Stefano Vincenzi, Angelo Porrello, Pietro Buzzega, Marco Cipriano, Pietro Fronte, Roberto Cuccu, Carla Ippoliti, Annamaria Conte, Simone Calderara
Date: 2020-06-22 00:00:00
Tasks: Colorization, Disease Prediction, Earth Observation, Image Classification, Land Cover Classification, Remote Sensing Image Classification, Self-Supervised Learning

The recent growth in the number of satellite images fosters the development of effective deep-learning techniques for Remote Sensing (RS). However, their full potential is untapped due to the lack of large annotated datasets. Such a problem is usually countered by fine-tuning a feature extractor that is previously trained on the ImageNet dataset. Unfortunately, the domain of natural images differs from the RS one, which hinders the final performance. In this work, we propose to learn meaningful representations from satellite imagery, leveraging its high-dimensionality spectral bands to reconstruct the visible colors. We conduct experiments on land cover classification (BigEarthNet) and West Nile Virus detection, showing that colorization is a solid pretext task for training a feature extractor. Furthermore, we qualitatively observe that guesses based on natural images and colorization rely on different parts of the input. This paves the way to an ensemble model that eventually outperforms both the above-mentioned techniques.
Machine Learning Pipelines: Provenance, Reproducibility and FAIR Data Principles
Authors: Sheeba Samuel, Frank Löffler, Birgitta König-Ries
Date: 2020-06-22 00:00:00
Tasks: BIG-bench Machine Learning, Decision Making

Machine learning (ML) is an increasingly important scientific tool supporting decision making and knowledge generation in numerous fields. With this, it also becomes more and more important that the results of ML experiments are reproducible. Unfortunately, that often is not the case. Rather, ML, similar to many other disciplines, faces a reproducibility crisis. In this paper, we describe our goals and initial steps in supporting the end-to-end reproducibility of ML pipelines. We investigate which factors beyond the availability of source code and datasets influence reproducibility of ML experiments. We propose ways to apply FAIR data practices to ML workflows. We present our preliminary results on the role of our tool, ProvBook, in capturing and comparing provenance of ML experiments and their reproducibility using Jupyter Notebooks.
Continual Learning in Recurrent Neural Networks
Authors: Benjamin Ehret, Christian Henning, Maria R. Cervera, Alexander Meulemans, Johannes von Oswald, Benjamin F. Grewe
Date: 2020-06-22 00:00:00
Proceeding: ICLR 2021 1
Tasks: Continual Learning

While a diverse collection of continual learning (CL) methods has been proposed to prevent catastrophic forgetting, a thorough investigation of their effectiveness for processing sequential data with recurrent neural networks (RNNs) is lacking. Here, we provide the first comprehensive evaluation of established CL methods on a variety of sequential data benchmarks. Specifically, we shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs. In contrast to feedforward networks, RNNs iteratively reuse a shared set of weights and require working memory to process input samples. We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements, which lead to an increased need for stability at the cost of decreased plasticity for learning subsequent tasks. We additionally provide theoretical arguments supporting this interpretation by studying linear RNNs. Our study shows that established CL methods can be successfully ported to the recurrent case, and that a recent regularization approach based on hypernetworks outperforms weight-importance methods, thus emerging as a promising candidate for CL in RNNs. Overall, we provide insights on the differences between CL in feedforward networks and RNNs, while guiding towards effective solutions to tackle CL on sequential data.
Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series
Authors: Hermanni Hälvä, Aapo Hyvärinen
Date: 2020-06-22 00:00:00
Tasks: Disentanglement, Time Series, Time Series Analysis

Recent advances in nonlinear Independent Component Analysis (ICA) provide a principled framework for unsupervised feature learning and disentanglement. The central idea in such works is that the latent components are assumed to be independent conditional on some observed auxiliary variables, such as the time-segment index. This requires manual segmentation of data into non-stationary segments which is computationally expensive, inaccurate and often impossible. These models are thus not fully unsupervised. We remedy these limitations by combining nonlinear ICA with a Hidden Markov Model, resulting in a model where a latent state acts in place of the observed segment-index. We prove identifiability of the proposed model for a general mixing nonlinearity, such as a neural network. We also show how maximum likelihood estimation of the model can be done using the expectation-maximization algorithm. Thus, we achieve a new nonlinear ICA framework which is unsupervised, more efficient, as well as able to model underlying temporal dynamics.
Exploiting Non-Taxonomic Relations for Measuring Semantic Similarity and Relatedness in WordNet
Authors: Mohannad AlMousa, Rachid Benlamri, Richard Khoury
Date: 2020-06-22 00:00:00
Tasks: Clustering, Information Retrieval, Machine Translation, Relation, Retrieval, Semantic Similarity, Semantic Textual Similarity, text-classification, Text Classification, Translation, Word Sense Disambiguation

Various applications in the areas of computational linguistics and artificial intelligence employ semantic similarity to solve challenging tasks, such as word sense disambiguation, text classification, information retrieval, machine translation, and document clustering. Previous work on semantic similarity followed a mono-relational approach using mostly the taxonomic relation "ISA". This paper explores the benefits of using all types of non-taxonomic relations in large linked data, such as WordNet knowledge graph, to enhance existing semantic similarity and relatedness measures. We propose a holistic poly-relational approach based on a new relation-based information content and non-taxonomic-based weighted paths to devise a comprehensive semantic similarity and relatedness measure. To demonstrate the benefits of exploiting non-taxonomic relations in a knowledge graph, we used three strategies to deploy non-taxonomic relations at different granularity levels. We conducted experiments on four well-known gold standard datasets, and the results demonstrated the robustness and scalability of the proposed semantic similarity and relatedness measure, which significantly improves existing similarity measures.
P3GM: Private High-Dimensional Data Release via Privacy Preserving Phased Generative Model
Authors: Shun Takagi, Tsubasa Takahashi, Yang Cao, Masatoshi Yoshikawa
Date: 2020-06-22 00:00:00
Tasks: Privacy Preserving

How can we release a massive volume of sensitive data while mitigating privacy risks? Privacy-preserving data synthesis enables the data holder to outsource analytical tasks to an untrusted third party. The state-of-the-art approach for this problem is to build a generative model under differential privacy, which offers a rigorous privacy guarantee. However, the existing method cannot adequately handle high dimensional data. In particular, when the input dataset contains a large number of features, the existing techniques require injecting a prohibitive amount of noise to satisfy differential privacy, which results in the outsourced data analysis meaningless. To address the above issue, this paper proposes privacy-preserving phased generative model (P3GM), which is a differentially private generative model for releasing such sensitive data. P3GM employs the two-phase learning process to make it robust against the noise, and to increase learning efficiency (e.g., easy to converge). We give theoretical analyses about the learning complexity and privacy loss in P3GM. We further experimentally evaluate our proposed method and demonstrate that P3GM significantly outperforms existing solutions. Compared with the state-of-the-art methods, our generated samples look fewer noises and closer to the original data in terms of data diversity. Besides, in several data mining tasks with synthesized data, our model outperforms the competitors in terms of accuracy.
A Self-Attention Network based Node Embedding Model
Authors: Dai Quoc Nguyen, Tu Dinh Nguyen, Dinh Phung
Date: 2020-06-22 00:00:00
Tasks: General Classification, Link Prediction, model, Node Classification

Despite several signs of progress have been made recently, limited research has been conducted for an inductive setting where embeddings are required for newly unseen nodes -- a setting encountered commonly in practical applications of deep learning for graph networks. This significantly affects the performances of downstream tasks such as node classification, link prediction or community extraction. To this end, we propose SANNE -- a novel unsupervised embedding model -- whose central idea is to employ a transformer self-attention network to iteratively aggregate vector representations of nodes in random walks. Our SANNE aims to produce plausible embeddings not only for present nodes, but also for newly unseen nodes. Experimental results show that the proposed SANNE obtains state-of-the-art results for the node classification task on well-known benchmark datasets.
Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning
Authors: Wonyong Jeong, Jaehong Yoon, Eunho Yang, Sung Ju Hwang
Date: 2020-06-22 00:00:00
Proceeding: ICLR 2021 1
Tasks: Federated Learning

While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning (FSSL). In this work, we study two essential scenarios of FSSL based on the location of the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server). We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning. The code is available at https://github.com/wyjeong/FedMatch.
Machine learning discrimination of Parkinson's Disease stages from walker-mounted sensors data
Authors: Nabeel Seedat, Vered Aharonson
Date: 2020-06-22 00:00:00
Tasks: Attribute, BIG-bench Machine Learning, feature selection

Clinical methods that assess gait in Parkinson's Disease (PD) are mostly qualitative. Quantitative methods necessitate costly instrumentation or cumbersome wearable devices, which limits their usability. Only few of these methods can discriminate different stages in PD progression. This study applies machine learning methods to discriminate six stages of PD. The data was acquired by low cost walker-mounted sensors in an experiment at a movement disorders clinic and the PD stages were clinically labeled. A large set of features, some unique to this study are extracted and three feature selection methods are compared using a multi-class Random Forest (RF) classifier. The feature subset selected by the Analysis of Variance (ANOVA) method provided performance similar to the full feature set: 93% accuracy and had significantly shorter computation time. Compared to PCA, this method also enabled clinical interpretability of the selected features, an essential attribute to healthcare applications. All selected-feature sets are dominated by information theoretic features and statistical features and offer insights into the characteristics of gait deterioration in PD. The results indicate a feasibility of machine learning to accurately classify PD severity stages from kinematic signals acquired by low-cost, walker-mounted sensors and implies a potential to aid medical practitioners in the quantitative assessment of PD progression. The study presents a solution to the small and noisy data problem, which is common in most sensor-based healthcare assessments.