pwc-archive/papers-with-abstracts huggingface dataset:
Collective Robustness Certificates
Authors: Jan Schuchardt, Aleksandar Bojchevski, Johannes Klicpera, Stephan Günnemann
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Adversarial Robustness, Image Segmentation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Node Classification, Semantic Segmentation

In tasks like node classification, image segmentation, and named-entity recognition we have a classifier that simultaneously outputs multiple predictions (a vector of labels) based on a single input, i.e. a single graph, image, or document respectively. Existing adversarial robustness certificates consider each prediction independently and are thus overly pessimistic for such tasks. They implicitly assume that an adversary can use different perturbed inputs to attack different predictions, ignoring the fact that  we have a single shared input.  We propose the first collective robustness certificate which computes the number of predictions which are simultaneously guaranteed to remain stable under perturbation, i.e. cannot be attacked. We focus on Graph Neural Networks and leverage their locality property - perturbations only affect the predictions in a close neighborhood - to  fuse multiple single-node certificates into a drastically stronger collective certificate. For example, on the Citeseer dataset our collective certificate for node classification increases the number of average certifiable feature perturbations from $7$ to $351$.
Modelling Drug-Target Binding Affinity using a BERT based Graph Neural network
Authors: Mark Lennox, Neil M. Robertson, Barry Devereux
Date: 2021-01-01 00:00:00
Tasks: Graph Neural Network

Understanding the interactions between novel drugs and target proteins is fundamentally important in disease research as discovering drug-protein interactions can be an exceptionally time-consuming and expensive process. Alternatively, this process can be simulated using modern deep learning methods that have the potential of utilising vast quantities of data to reduce the cost and time required to provide accurate predictions. In this paper, we seek to leverage a set of BERT-style models that have been pre-trained on vast quantities of both protein and drug data. The encodings produced by each model are then utilised as node representations for a graph convolutional neural network, which in turn models the interactions without the need to simultaneously fine-tune both protein and drug BERT models to the task. We evaluate the performance of our approach on two drug-target interaction datasets that were previously used as benchmarks in recent work. Our results significantly improve upon a vanilla BERT baseline approach as well as the former state-of-the-art methods for each task dataset. Our approach builds upon past work in two key areas; firstly, we take full advantage of two large pre-trained BERT models that provide improved representations of task-relevant properties of both drugs and proteins. Secondly, inspired by work in natural language processing that investigates how linguistic structure is represented in such models, we perform interpretability analyses that allow us to locate functionally-relevant areas of interest within each drug and protein. By modelling the drug-target interactions as a graph as opposed to a set of isolated interactions, we demonstrate the benefits of combining large pre-trained models and a graph neural network to make state-of-the-art predictions on drug-target binding affinity.
Zero-shot Fairness with Invisible Demographics
Authors: Thomas Kehrenberg, Viktoriia Sharmanska, Myles Scott Bartlett, Novi Quadrianto
Date: 2021-01-01 00:00:00
Tasks: Disentanglement, Fairness

In a statistical notion of algorithmic fairness, we partition individuals into groups based on some key demographic factors such as race and gender, and require that some statistics of a classifier be approximately equalized across those groups. Current approaches require complete annotations for demographic factors, or focus on an abstract worst-off group rather than demographic groups. In this paper, we consider the setting where the demographic factors are only partially available. For example, we have training examples for white-skinned and dark-skinned males, and white-skinned females, but we have zero examples for dark-skinned females. We could also have zero examples for females regardless of their skin colors. Without additional knowledge, it is impossible to directly control the discrepancy of the classifier's statistics for those invisible groups. We develop a disentanglement algorithm that splits a representation of data into a component that captures the demographic factors and another component that is invariant to them based on a context dataset. The context dataset is much like the deployment dataset, it is unlabeled but it contains individuals from all demographics including the invisible. We cluster the context set, equalize the cluster size to form a "perfect batch", and use it as a supervision signal for the disentanglement. We propose a new discriminator loss based on a learnable attention mechanism to distinguish a perfect batch from a non-perfect one. We evaluate our approach on standard classification benchmarks and show that it is indeed possible to protect invisible demographics.
Rotograd: Dynamic Gradient Homogenization for Multitask Learning
Authors: Adrián Javaloy, Isabel Valera
Date: 2021-01-01 00:00:00

GradNorm (Chen et al., 2018) is a broadly used gradient-based approach for training multitask networks, where different tasks share, and thus compete during learning, for the network parameters. GradNorm eases the fitting of all individual tasks by dynamically equalizing the contribution of each task to the overall gradient magnitude. However, it does not prevent the individual tasks’ gradients from conflicting, i.e., pointing towards opposite directions, and thus resulting in a poor multitask performance. In this work we propose Rotograd, an extension to GradNorm that addresses this problem by dynamically homogenizing not only the gradient magnitudes but also their directions across tasks. For this purpose,Rotograd adds a layer of task-specific rotation matrices that aligns all the task gradients. Importantly, we then analyze Rotograd (and its predecessor) through the lens of game theory, providing theoretical guarantees on the algorithm stability and convergence. Finally, our experiments on several real-world datasets and network architectures show that Rotograd outperforms previous approaches for multitask learning.
A General Framework for Unsupervised Anomaly Detection
Authors: Nima Rafiee, Rahil Gholamipoor, Markus Kollmann
Date: 2021-01-01 00:00:00
Tasks: Anomaly Detection, Out-of-Distribution Detection, Unsupervised Anomaly Detection

In this paper we present GenAD, a simple and generic framework for detecting examples that lie out-of-distribution for a given training set. The approach is based on first learning a semantic neighbourhood for each example in the training set and then training a binary discriminator to classify whether two examples come from the same or different semantic neighbourhoods. The framework is applicable to a wide range of anomaly detection problems, including visual, audio, and textual data. We are able to outperform previous approaches for anomaly, novelty, or out-of-distribution detection in the visual domain by a large margin. In particular we obtain AUROC $>0.999$ for the challenging task of detecting examples from SVHN as out-of-distribution given CIFAR-10 as in-distribution, without making use of label information.
Legendre Deep Neural Network (LDNN) and its application for approximation of nonlinear Volterra–Fredholm–Hammerstein integral equations
Authors: Kourosh Parand, Zeinab Hajimohammadi, Ali Ghodsi
Date: 2021-01-01 00:00:00

Various phenomena in biology, physics, and engineering are modeled by differential equations. These differential equations including partial differential equations and ordinary differential equations can be converted and represented as integral equations. In particular, Volterra–Fredholm–Hammerstein integral equations are the main type of these integral equations and researchers are interested in investigating and solving these equations.  In this paper, we propose Legendre Deep Neural Network (LDNN) for solving nonlinear Volterra–Fredholm–Hammerstein integral equations (V-F-H-IEs). LDNN utilizes Legendre orthogonal polynomials as activation functions of the Deep structure. We present how LDNN can be used to solve nonlinear V-F-H-IEs. We show using the Gaussian quadrature collocation method in combination with LDNN results in a novel numerical solution for nonlinear V-F-H-IEs. Several examples are given to verify the performance and accuracy of LDNN.
Rethinking Parameter Counting: Effective Dimensionality Revisited
Authors: Gregory Benton, Wesley Maddox, Andrew Gordon Wilson
Date: 2021-01-01 00:00:00
Tasks: Diversity, Model Selection

Neural networks appear to have mysterious generalization properties when using parameter counting as a proxy for complexity. Indeed, neural networks often have many more parameters than there are data points, yet still provide good generalization performance. Moreover, when we measure generalization as a function of parameters, we see double descent behaviour, where the test error decreases, increases, and then again decreases. We show that many of these properties become understandable when viewed through the lens of effective dimensionality, which measures the dimensionality of the parameter space determined by the data. We relate effective dimensionality to posterior contraction in Bayesian deep learning, model selection, width-depth tradeoffs, double descent, and functional diversity in loss surfaces,  leading to a richer understanding of the interplay between parameters and functions in deep models. We also show that effective dimensionality compares favourably to alternative norm- and flatness- based generalization measures.
Regression from Upper One-side Labeled Data
Authors: Takayuki Katsuki
Date: 2021-01-01 00:00:00
Tasks: regression

We address a regression problem from weakly labeled data that are correctly labeled only above a regression line, i.e., upper one-side labeled data.
The label values of the data are the results of sensing the magnitude of some phenomenon.
In this case, the labels often contain missing or incomplete observations whose values are lower than those of correct observations and are also usually lower than the regression line. It follows that data labeled with lower values than the estimations of a regression function (lower-side data) are mixed with data that should originally be labeled above the regression line (upper-side data).
When such missing label observations are observed in a non-negligible amount, we thus should assume our lower-side data to be unlabeled data that are a mix of original upper- and lower-side data.
We formulate a regression problem from these upper-side labeled and lower-side unlabeled data. We then derive a learning algorithm in an unbiased and consistent manner to ordinary regression that is learned from data labeled correctly in both upper- and lower-side cases. Our key idea is that we can derive a gradient that requires only upper-side data and unlabeled data as the equivalent expression of that for ordinary regression. We additionally found that a specific class of losses enables us to learn unbiased solutions practically. In numerical experiments on synthetic and real-world datasets, we demonstrate the advantages of our algorithm.
Inhibition-augmented ConvNets
Authors: Nicola Strisciuglio, George Azzopardi, Nicolai Petkov
Date: 2021-01-01 00:00:00

Convolutional  Networks  (ConvNets) suffer from insufficient  robustness to common corruptions and perturbations of the input, unseen during training. We address this problem by including a form of response inhibition in the processing of the early layers of existing ConvNets and assess the resulting representation power and generalization on corrupted inputs. The considered inhibition mechanism  consists of a non-linear computation that is inspired  by the  push-pull  inhibition exhibited by some neurons in the visual system of the brain. In practice, each convolutional filter (push) in the early layers of conventional ConvNets is coupled with another filter (pull), which responds to the preferred pattern of the corresponding push filter but of opposite contrast. The rectified responses of the push and pull filter pairs are then combined by a linear function. This results in a representation that suppresses responses to noisy patterns (e.g. texture, Gaussian, shot, distortion, and others) and accentuates responses to preferred patterns. 

We deploy the layer into existing architectures, (Wide-)ResNet  and  DenseNet, and propose new residual and dense push-pull layers. We demonstrate that ConvNets that embed this inhibition into the initial layers are able to learn representations that are robust to several types of input corruptions.  We validated the approach on the ImageNet  and  CIFAR  data  sets and their corrupted and perturbed versions, ImageNet-C/P and CIFAR-C/P. It turns out that the push-pull inhibition enhances the overall robustness and generalization of ConvNets to corrupted and perturbed input data. Besides the improvement in generalization, notable is the fact that ConvNets with push-pull inhibition have a sparser representation than conventional ones without inhibition.  The code and trained models will be made available.
On the Effectiveness of Deep Ensembles for Small Data Tasks
Authors: Lorenzo Brigato, Luca Iocchi
Date: 2021-01-01 00:00:00
Tasks: image-classification, Image Classification

Deep neural networks represent the gold standard for image classification and many other tasks.
However, they usually need large amounts of data to reach superior performance.
In this work, we focus on classification problems with a few labeled examples per class and improve sample efficiency in the low data regime by using an ensemble of relatively small deep networks. 
For the first time, our work broadly studies the existing concept of neural ensembling in small data domains.
We compare different ensemble configurations by varying the complexity of base members given a total fixed computational budget.
We found that the state of the art is improved by keeping low the complexity of single models and increasing the ensemble dimension.
Furthermore, we investigate the effectiveness of different losses and show that their choice should be made considering different factors.
Finally, according to our results, our proposed ensemble resulted to be the most robust configuration considering variation in dataset type and size.
Our findings are empirically validated on popular datasets after an extensive number of experiments.
AT-GAN: An Adversarial Generative Model for Non-constrained Adversarial Examples
Authors: Xiaosen Wang, Kun He, Chuanbiao Song, LiWei Wang, John E. Hopcroft
Date: 2021-01-01 00:00:00
Tasks: Adversarial Attack, Transfer Learning

With the rapid development of adversarial machine learning, numerous adversarial attack methods have been proposed. Typical attacks are based on a search in the neighborhood of input image to generate a perturbed adversarial example. Since 2017, generative models are adopted for adversarial attacks, and most of them focus on generating adversarial perturbations from input noise or input image. Thus the output is restricted by input for these works. A recent work targets unrestricted adversarial example using generative model but their method is based on a search in the neighborhood of input noise, so actually their output is still constrained by input. In this work, we propose AT-GAN (Adversarial Transfer on Generative Adversarial Net) to train an adversarial generative model that can directly produce adversarial examples. Different from  previous works, we aim to learn the distribution of adversarial examples so as to generate semantically meaningful adversaries. AT-GAN achieves this goal by first learning a generative model for real data, followed by transfer learning to obtain the desired generative model. Once trained and transferred, AT-GAN could generate adversarial examples directly for any input noise, denoted as non-constrained adversarial examples. Extensive experiments and visualizations show that AT-GAN can efficiently generate diverse adversarial examples that are realistic to human perception, and yields higher attack success rates against adversarially trained models.
A neural method for symbolically solving partial differential equations
Authors: Anonymous
Date: 2021-01-01 00:00:00

We describe a neural-based method for generating exact or approximate solutions to differential equations in the form of mathematical expressions. Unlike other neural methods, our system returns symbolic expressions that can be interpreted directly. Our method uses a neural architecture for learning mathematical expressions to optimize a customizable objective, and is scalable, compact, and easily adaptable for a variety of tasks and configurations. The system has been shown to effectively find exact or approximate symbolic solutions to various differential equations with applications in natural sciences. In this work, we highlight how our method applies to partial differential equations over multiple variables and more complex boundary and initial value conditions.
Understanding Mental Representations Of Objects Through Verbs Applied To Them
Authors: Ka Chun Lam, Francisco Pereira, Maryam Vaziri-Pashkam, Kristin Woodard, Emalie McMahon
Date: 2021-01-01 00:00:00
Tasks: Object

In order to interact with objects in our environment, we rely on an understanding of the actions that can be performed on them, and the extent to which they rely or have an effect on the properties of the object. This knowledge is called the object "affordance". We propose an approach for creating an embedding of objects in an affordance space, in which each dimension corresponds to an aspect of meaning shared by many actions, using text corpora. This embedding makes it possible to predict which verbs will be applicable to a given object, as captured in human judgments of affordance, better than a variety of alternative approaches. Furthermore, we show that the dimensions learned are interpretable, and that they correspond to typical patterns of interaction with objects. Finally, we show that the dimensions can be used to predict a state-of-the-art mental representation of objects, derived purely from human judgements of object similarity.
Balancing training time vs. performance with Bayesian Early Pruning
Authors: Mohit Rajpal, Yehong Zhang, Bryan Kian Hsiang Low
Date: 2021-01-01 00:00:00
Tasks: Computational Efficiency

Pruning is an approach to alleviate overparameterization of deep neural networks (DNN) by zeroing out or pruning DNN elements with little to no efficacy at a given task. In contrast to related works that do pruning before or after training, this paper presents a novel method to perform early pruning of DNN elements (e.g., neurons or convolutional filters) during the training process while preserving performance upon convergence. To achieve this, we model the future efficacy of DNN elements in a Bayesian manner conditioned upon efficacy data collected during the training and prune DNN elements which are predicted to have low efficacy after training completion. Empirical evaluations show that the proposed Bayesian early pruning improves the computational efficiency of DNN training with small sacrifices in performance. Using our approach we are able to achieve a $48.6\%$ faster training time for ResNet-$50$ on ImageNet to achieve a validation accuracy of $72.5\%$.
Attention Based Joint Learning for Supervised Premature Ventricular Contraction Differentiation with Unsupervised Abnormal Beat Segmentation
Authors: Xinrong Hu, long wen, shushui wang, Dongpo Liang, Jian Zhuang, Yiyu Shi
Date: 2021-01-01 00:00:00
Tasks: Classification, General Classification

Deep learning has shown great promise in arrhythmia classification in electrocar-diogram (ECG). Existing works, when classifying an ECG segment with multiplebeats, do not identify the locations of the anomalies, which reduces clinical inter-pretability.  On the other hand, segmenting abnormal beats by deep learning usu-ally requires annotation for a large number of regular and irregular beats, whichcan be laborious, sometimes even challenging, with strong inter-observer variabil-ity between experts. In this work, using Premature Ventricular Contraction (PVC)differentiation as an example of arrhythmia classification, we propose a methodcapable of not only differentiating the origin of PVC but also segmenting the as-sociated abnormal beats.  Only the PVC origin labels are used in the training andno segmentation labels are needed. Imitating human’s perception of an ECG sig-nal, the framework consists of a segmenter and classifier.  The segmenter outputsan attention map, which aims to highlight the abnormal sections in the ECG byelement-wise modulation. Afterwards, the signals are sent to a classifier for PVCorigin differentiation.  Though the training data is only labeled to supervise theclassifier, the segmenter and the classifier are trained in an end-to-end manner sothat optimizing classification performance also adjusts how the abnormal beats aresegmented.  Validation of our method is conducted on a dataset with 508 12-leadECG records of PVC patients.  We observe that involving the unsupervised seg-mentation in fact boosts the classification performance. Meanwhile, a grade studyperformed by experts suggests that the segmenter also achieves satisfactory qual-ity in identifying abnormal beats, which significantly enhances the interpretabilityof the classification results.
Multi-Faceted Trust Based Recommendation System
Authors: Ramamoorthy Srinath, Shruthi Gopinath, Vaani Sundaresh, Varsha Venkatasubramanian
Date: 2021-01-01 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems

Recommendation systems play a decisive role in the choices we make on the internet. They seek to tailor decisions to a user. This makes trust a very important factor in recommendation systems, since it is believed that users are similar to the people that they trust, and hence will make similar choices to those users. Trust and its effects on the choices people make have been widely studied in the context of collaborative recommendation systems. It is understood that trust is not a single-faceted entity but can vary contextually. Recent research in the domain of trust based recommendation systems has shown that taking into account the facets of trust greatly improves the quality of recommendations (Mauro et al., 2019; Fang et al., 2015). We propose a recommendation system that takes multiple facets of trust into account while looking at how suitable a product might be for a particular user. The architecture proposed for this Multi-Faceted Trust Based Recommender (MFTBR) allows for extensibility - new facets of trust can be added without much effort - and dynamicity - trust facets are not weighed arbitrarily. Instead, the weights are optimised for the best result via a neural network. The trust facets considered here are local, global and category-wise trust. MFTBR performs significantly better than basic collaborative filtering - U2UCF (C. Desrosiers, 2011), as well as some established models in the domain of social and trust based recommendation systems - MTR (Mauro et al., 2019) and SocialFD (Yu et al., 2017). Thus, our model provides a better approximation of real-life recommendations, taking into account not only the impact of trust on recommendation, but the context in which trust is established.