pwc-archive/papers-with-abstracts huggingface dataset:
BAAAN: Backdoor Attacks Against Auto-encoder and GAN-Based Machine Learning Models
Authors: Ahmed Salem, Yannick Sautter, Michael Backes, Mathias Humbert, Yang Zhang
Date: 2021-01-01 00:00:00
Tasks: Backdoor Attack, BIG-bench Machine Learning, Fraud Detection

The tremendous progress of autoencoders and generative adversarial networks (GANs) has led to their application to multiple critical tasks, such as fraud detection and sanitized data generation. This increasing adoption has fostered the study of security and privacy risks stemming from these models. However, previous works have mainly focused on membership inference attacks. In this work, we explore one of the most severe attacks against machine learning models, namely the backdoor attack, against both autoencoders and GANs. The backdoor attack is a training time attack where the adversary implements a hidden backdoor in the target model that can only be activated by a secret trigger. State-of-the-art backdoor attacks focus on classification-based tasks. We extend the applicability of backdoor attacks to autoencoders and GAN-based models. More concretely, we propose the first backdoor attack against autoencoders and GANs where the adversary can control what the decoded or generated images are when the backdoor is activated. Our results show that the adversary can build a backdoored autoencoder that returns a target output for all backdoored inputs, while behaving perfectly normal on clean inputs. Similarly, for the GANs, our experiments show that the adversary can generate data from a different distribution when the backdoor is activated, while maintaining the same utility when the backdoor is not.
Effective Training of Sparse Neural Networks under Global Sparsity Constraint
Authors: Xiao Zhou, Weizhong Zhang, Tong Zhang
Date: 2021-01-01 00:00:00

Weight pruning is an effective technique to reduce the model size and inference time for deep neural networks in real world deployments. Although in earlier approaches, pruning was performed after the training of dense networks, in recent years, it was shown that the direct training of sparse neural networks is more effective. However, the performances of these two kinds of methods are limited due to a common challenge. That is, they always need to tune the pruning rate individually for each layer in a network either manually or by using handcrafted heuristic rules, since the magnitudes of the weight importance under their proposed criteria are quite different across layers. This paper proposes an effective method called ${\it probabilistic~masking}$ (ProbMask) to train sparse neural networks from scratch, where we develop a natural formulation under global sparsity constraint. The key idea is that we notice probability can be used as a global criterion for all layers to measure the weight importance. Therefore, we can encode the weight sparsity into an explicit global constraint on probabilities, leading to a constrained continuous minimization problem over the probability space. An appealing feature of ProbMask is that the amounts of weight redundancy can be learned automatically via our constraint and thus we avoid the problem of tuning pruning rates individually for different layers in a network. Extensive experimental results demonstrate that our method is much more effective than state-of-the-art methods and can outperform them by a significant margin under high pruning rates.
Early Stopping by Gradient Disparity
Authors: Mahsa Forouzesh, Patrick Thiran
Date: 2021-01-01 00:00:00

Validation-based early-stopping methods are one of the most popular techniques used to avoid over-training deep neural networks. They require to set aside a reliable unbiased validation set, which can be expensive in applications offering limited amounts of data. In this paper, we propose to use \emph{gradient disparity}, which we define as the $\ell_2$ norm distance between the gradient vectors of two batches drawn from the training set. It comes from a probabilistic upper bound on the difference between the classification errors over a given batch, when the network is trained on this batch and when the network is trained on another batch of points sampled from the same dataset. We empirically show that gradient disparity is a very promising early-stopping criterion when data is limited, because it uses all the training samples during training. Furthermore, we show in a wide range of experimental settings that gradient disparity is not only strongly related to the usual generalization error between the training and test sets, but that it is also much more informative about the level of label noise.
Sparse Binary Neural Networks
Authors: Riccardo Schiavone, Maria A Zuluaga
Date: 2021-01-01 00:00:00
Tasks: Sparse Learning and binarization

Quantized neural networks are gaining popularity thanks to their ability to solve complex tasks with comparable accuracy as full-precision Deep Neural Networks (DNNs), while also reducing computational power and storage requirements and increasing the processing speed. These properties make them an attractive alternative for the development and deployment of DNN-based applications in Internet-Of-Things (IoT) devices. Among quantized networks, Binary Neural Networks (BNNs) have reported the largest speed-up. However, they suffer from a fixed and limited compression factor that may result insufficient for certain devices with very limited resources. In this work, we propose Sparse Binary Neural Networks, a novel model and training scheme that allows to introduce sparsity in BNNs by using 0/1 binary weights, instead of the -1/+1 weights used by state-of-the-art binary networks. As a result, our method is able to achieve a high compression factor and reduces the number of operations and parameters at inference time. We study the properties of our method through experiments on linear and convolutional networks over MNIST and CIFAR-10 datasets. Experiments confirm that SBNNs can achieve high compression rates and good generalization, while further reducing the operations of BNNs, making it a viable option for deploying DNNs in cheap and low-cost IoT devices and sensors.
Accurate and fast detection of copy number variations from short-read whole-genome sequencing with deep convolutional neural network
Authors: Jiajin Li, Stephen Hwang, Luke Zhang, Jae Hoon Sul
Date: 2021-01-01 00:00:00
Tasks: Benchmarking

A copy number variant (CNV) is a type of genetic mutation where a stretch of DNA is lost or duplicated once or multiple times. CNVs play important roles in the development of diseases and complex traits. CNV detection with short-read DNA sequencing technology is challenging because CNVs significantly vary in size and are similar to DNA sequencing artifacts. Many methods have been developed but still yield unsatisfactory results with high computational costs. Here, we propose CNV-Net, a novel approach for CNV detection using a six-layer convolutional neural network. We encode DNA sequencing information into RGB images and train the convolutional neural network with these images. The fitted convolutional neural network can then be used to predict CNVs from DNA sequencing data. We benchmark CNV-Net with two high-quality whole-genome sequencing datasets available from the Genome in a Bottle Consortium, considered as gold standard benchmarking datasets for CNV detection. We demonstrate that CNV-Net is more accurate and efficient in CNV detection than current tools.
Meta Auxiliary Labels with Constituent-based Transformer for Aspect-based Sentiment Analysis
Authors: Ling Min Serena Khoo, Hai Leong Chieu
Date: 2021-01-01 00:00:00
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Auxiliary Learning, Sentiment Analysis

Aspect based sentiment analysis (ABSA) is a challenging natural language processing task that could benefit from syntactic information. Previous work exploit dependency parses to improve performance on the task, but this requires the existence of good dependency parsers.  In this paper, we build a constituent-based transformer for ABSA that can induce constituents without constituent parsers. We also apply meta auxiliary learning to generate labels on edges between tokens, supervised by the objective of the ABSA task.  Without input from dependency parsers, our models outperform previous work on three Twitter data sets and match previous work closely on two review data sets.
$Graph Embedding via Topology and Functional Analysis$
Authors: Phani raj Chinnalingu
Date: 2021-01-01 00:00:00
Tasks: BIG-bench Machine Learning, Graph Embedding

Graphs have been ubiquitous in Machine Learning due to their versatile nature in modelling real world situations .Graph embedding is an important precursor to using graphs in Machine Learning , and much of performance of algorithms developed later depends heavily on this. However very little theoretical work exists in this area ,  resulting in the proliferation of several benchmarks without any mathematical validation , which is detrimental .In this paper we present an analysis of deterministic graph embedding in general , using tools from Functional Analysis and Topology . We prove several important results pertaining to graph embedding  which may have practical importance .One limitation of our work in it's present form is it's applicable to deterministic embedding approaches only, although we strongly hope to extend it to random graph embedding methods as well in future.We sincerely hope that this work will be beneficial  to researchers working in field of  graph embedding.
Adversarial Deep Metric Learning
Authors: Thomas Kobber Panum, Zi Wang, Pengyu Kan, Earlence Fernandes, Somesh Jha
Date: 2021-01-01 00:00:00
Tasks: Metric Learning

Learning a distance metric between pairs of examples is widely important for various tasks. Deep Metric Learning (DML) utilizes deep neural network architectures to learn semantic feature embeddings where the distance between similar examples is close and dissimilar examples are far. While the underlying neural networks produce good accuracy on naturally occurring samples, they are vulnerable to adversarially-perturbed samples that can reduce their accuracy. To create robust versions of DML models, we introduce a robust training approach. A key challenge is that metric losses are not independent --- they depend on all samples in a mini-batch.  This sensitivity to samples, if not accounted for, can lead to incorrect robust training.  To the best of our knowledge, we are the first to systematically analyze this dependence effect and propose a principled approach for robust training of deep metric learning networks that accounts for the nuances of metric losses. Using experiments on three popular datasets in metric learning, we demonstrate the DML models trained using our techniques display robustness against strong iterative attacks while their performance on unperturbed (natural) samples remains largely unaffected.
DeepLTRS: A Deep Latent Recommender System based on User Ratings and Reviews
Authors: Dingge LIANG, Marco Corneli, Pierre Latouche, Charles Bouveyron
Date: 2021-01-01 00:00:00
Tasks: Recommendation Systems

We introduce a deep latent recommender system named deepLTRS in order to provide users with high quality recommendations based on observed user ratings and texts of product reviews. The underlying motivation is that, when a user scores only a few products, the texts used in the reviews represent a significant source of information. The addition of review information can alleviate data sparsity, thereby enhancing the predictive ability of the model. Our approach adopts a variational auto-encoder architecture as a generative deep latent variable model for both an ordinal matrix encoding users scores about products, and a document-term matrix encoding the reviews. Moreover, different from unique user-based or item-based models, deepLTRS assumes latent representations for both users and products. An alternated user/product mini-batching optimization structure is proposed to jointly capture user and product preferences. Numerical experiments on simulated and real-world data sets demonstrate that deepLTRS outperforms the state-of-the-art, in particular in contexts of extreme data sparsity.
DISE: Dynamic Integrator Selection to Minimize Forward Pass Time in Neural ODEs
Authors: Soyoung Kang, Ganghyeon Park, Kwang-Sung Jun, Noseong Park
Date: 2021-01-01 00:00:00

Neural ordinary differential equations (Neural ODEs) are appreciated for their ability to significantly reduce the number of parameters when constructing a neural network. On the other hand, they are blamed for their procrastinated forward-pass inference, which is incurred by solving integral problems. To improve the model accuracy, they rely on advanced solvers, such as the Dormand--Prince (DOPRI) method. To solve an integral problem, however, it requires at least tens (or sometimes hundreds) of steps in many Neural ODE experiments. In this work, we propose to i) directly regularize the step size of DOPRI to make the forward-pass faster and ii) dynamically choose a simpler integrator than DOPRI for a carefully selected subset of input. Because it is not the case that every input requires the advanced integrator, we design an auxiliary neural network to choose an appropriate integrator given input to decrease the overall inference time without significantly sacrificing accuracy. We consider the Euler method, the fourth-order Runge--Kutta (RK4) method, and DOPRI as selection candidates. We found that a non-trivial percentage of cases can be solved with simple integrators. Therefore, the overall number of functional evaluations (NFEs) decreases up to 78% with improved accuracy.
An Empirical Study of the Expressiveness of Graph Kernels and Graph Neural Networks
Authors: Giannis Nikolentzos, George Panagopoulos, Michalis Vazirgiannis
Date: 2021-01-01 00:00:00
Tasks: Graph Similarity

Graph neural networks and graph kernels have achieved great success in solving machine learning problems on graphs. Recently, there has been considerable interest in determining the expressive power mainly of graph neural networks and of graph kernels, to a lesser extent.  Most studies have focused on the ability of these approaches to distinguish non-isomorphic graphs or to identify specific graph properties. However, there is often a need for algorithms whose produced graph representations can accurately capture similarity/distance of graphs. This paper studies the expressive power of graph neural networks and graph kernels from an empirical perspective. Specifically, we compare the graph representations and similarities produced by these algorithms against those generated by a well-accepted, but intractable graph similarity function. We also investigate the impact of node attributes on the performance of the different models and kernels. Our results reveal interesting findings. For instance, we find that theoretically more powerful models do not necessarily yield higher-quality representations, while graph kernels are shown to be very competitive with graph neural networks.
Distributional Reinforcement Learning for Risk-Sensitive Policies
Authors: Shiau Hong Lim, Ilyas Malik
Date: 2021-01-01 00:00:00
Tasks: Distributional Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We address the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, we show that applying the distributional Bellman optimality operator with respect to a risk-based action-selection strategy overestimates the dynamic, Markovian CVaR. The resulting policies can however still be overly conservative and one often prefers to learn an optimal policy based on the static, non-Markovian CVaR. To this end, we propose a modification to the existing algorithm and show that it can indeed learn a proper CVaR-optimized policy. Our proposed approach is a simple extension of standard distributional RL algorithms and can therefore take advantage of many of the recent advances in deep RL. On both synthetic and real data, we empirically show that our proposed algorithm is able to produce a family of risk-averse policies that achieves a better tradeoff between risk and the expected return.
Generalized Universal Approximation for Certified Networks
Authors: Zi Wang, Aws Albarghouthi, Somesh Jha
Date: 2021-01-01 00:00:00

To certify safety and robustness of neural networks, researchers have successfully applied abstract interpretation, primarily using interval bound propagation. To understand the power of interval bounds, we present the abstract universal approximation (AUA) theorem, a generalization of the recent result by Baader et al. (2020) for ReLU networks to a large class of neural networks. The AUA theorem states that for any continuous function $f$, there exists a neural network that (1) approximates $f$ (universal approximation) and (2) whose interval bounds are an arbitrarily close approximation of the set semantics of $f$. The network may be constructed using any activation function from a rich class of functions---sigmoid, tanh, ReLU, ELU, etc.---making our result quite general. The key implication of the AUA theorem is that there always exists certifiably robust neural networks, which can be constructed using a wide range of activation functions.
Token-Level Contrast for Video and Language Alignment
Authors: Jianwei Yang, Yonatan Bisk, Jianfeng Gao
Date: 2021-01-01 00:00:00

Building video and language understanding models requires grounding linguistic concepts and video contents into a shared space. Most of previous works learn a holistic alignment between them while neglecting the token-level grounding. Masked token prediction can be used to learn token-level multi-modal representation, but it does not necessarily force lexical grounding on perception and also introduce a domain-shift between pretraining and fine-tuning. This paper introduces a simple token-level contrastive loss (ToCo) informed by syntactic classes (e.g., nouns and verbs) to force the model to prioritize grounding concrete semantic bearing words. ToCo does not mask inputs but poses both local (contextual token) and global (lexical type) pressures for multi-modal alignment in a contrastive manner. Our approach enables a simple vanilla BERT-based multimodal transformer to compete with or outperform existing heavily engineered multi-loss or large models on three benchmarks (YouCook2, MSR-VTT and CrossTask). Further, it is plug-n-play such that gains are made in both pretraining and downstream tasks solely, regardless of the underlying visual or textual feature representations.
Fighting Filterbubbles with Adversarial BERT-Training for News-Recommendation
Authors: Lukas Pfahler, Katharina Morik
Date: 2021-01-01 00:00:00
Tasks: Articles, News Recommendation

Recommender engines play a role in the emergence and reinforcement of filter bubbles. When these systems learn that a user prefers content from a particular site, the user will be less likely to be exposed to different sources or opinions and, ultimately, is more likely to develop extremist tendencies.
We trace the roots of this phenomenon to the way the recommender engine represents news articles. The vectorial features modern systems extract from the plain text of news articles are already highly predictive of the associated news outlet. We propose a new training scheme based on adversarial machine learning to tackle this issue . Our  experiments show that the features we can extract this way are significantly less predictive of the news outlet and thus offer the possibility to reduce the risk of manifestation of new filter bubbles. We validate our intuitions in a news recommendation task using a recent attention-based recommendation system.
Genetic Soft Updates for Policy Evolution in Deep Reinforcement Learning
Authors: Enrico Marchesini, Davide Corsi, Alessandro Farinelli
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: continuous-control, Continuous Control, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The combination of Evolutionary Strategies (ES) and Deep Reinforcement Learning (DRL) has been recently proposed to merge the benefits of both solutions. Existing mixed approaches, however, have been successfully applied only to actor-critic methods and present significant overhead. We address these issues by introducing a novel mixed framework that exploits a periodical genetic evaluation to soft update the weights of a DRL agent. The resulting approach is applicable with any DRL method and, in a worst-case scenario, it does not exhibit detrimental behaviours. Experiments in robotic applications and continuous control benchmarks demonstrate the versatility of our approach that significantly outperforms prior DRL, ES, and mixed approaches. Finally, we employ formal verification to confirm the policy improvement, mitigating the inefficient exploration and hyper-parameter sensitivity of DRL.
AutoCleansing: Unbiased Estimation of Deep Learning with Mislabeled Data
Authors: Koichi Kuriyama
Date: 2021-01-01 00:00:00
Tasks: Deep Learning

Mislabeled samples cause prediction errors. This study proposes a solution to the problem of incorrect labels, called AutoCleansing, to automatically capture the effect of incorrect labels and mitigate it without removing the mislabeled samples. AutoCleansing consists of a base network model and sample-category specific constants. Both parameters of the base model and sample-category constants are estimated simultaneously using the training data. Thereafter, predictions for test data are made using a base model without the constants capturing the mislabeled effects. A theoretical model for AutoCleansing is developed and theoretical analysis shows that the proposed method can estimate true parameters with mislabeled data if the model is correctly constructed. Experimental results show that AutoCleansing has better performance in test accuracy than previous studies for CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.
Spatially Structured Recurrent Modules
Authors: Nasim Rahaman, Anirudh Goyal, Muhammad Waleed Gondal, Manuel Wuthrich, Stefan Bauer, Yash Sharma, Yoshua Bengio, Bernhard Schölkopf
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Starcraft II, Video Prediction

Capturing the structure of a data-generating process by means of appropriate inductive biases can help in learning models that generalise well and are robust to changes in the input distribution. While methods that harness spatial and temporal structures find broad application, recent work has demonstrated the potential of models that leverage sparse and modular structure using an ensemble of sparingly interacting modules. In this work, we take a step towards dynamic models that are capable of simultaneously exploiting both modular and spatiotemporal structures. To this end, we model the dynamical system as a collection of autonomous but sparsely interacting sub-systems that interact according to a learned topology which is informed by the spatial structure of the underlying system. This gives rise to a class of models that are well suited for capturing the dynamics of systems that only offer local views into their state, along with corresponding spatial locations of those views. On the tasks of video prediction from cropped frames and multi-agent world modelling from partial observations in the challenging Starcraft2 domain, we find our models to be more robust to the number of available views and better capable of generalisation to novel tasks without additional training than strong baselines that perform equally well or better on the training distribution.