pwc-archive/papers-with-abstracts huggingface dataset:
A Dot Product Attention Free Transformer
Authors: Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, Joshua M. Susskind
Date: 2021-09-29 00:00:00
Tasks: Image Classification, Language Modelling

We introduce Dot Product Attention Free Transformer (DAFT), an efficient variant of Transformers \citep{transformer} that eliminates the query-key dot product in self attention. The core idea is to construct a decomposable attention map for each dimension of the query, key and value. This compositionality enables an implementation where the attention tensor does not to be computed or stored explicitly. A DAFT layer has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible with both large input and model sizes. We also introduce DAFT-conv, a model variant that takes advantage of locality and spatial weight sharing while maintaining global connectivity. We conduct experiments on ImageNet-1K classification, as well as CIFAR10 and Enwik8, two autoregressive modeling tasks. We show that DAFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.
Distribution-Driven Disjoint Prediction Intervals for Deep Learning
Authors: Jaehak Cho, Jae Myung Kim, Sungyeob Han, Jungwoo Lee
Date: 2021-09-29 00:00:00
Tasks: Deep Learning, Prediction, Prediction Intervals, regression, Uncertainty Quantification

This paper redefines prediction intervals (PIs) as the form of a union of disjoint intervals. PIs represent predictive uncertainty in the regression problem. Since previous PI methods assumed a single continuous PI (one lower and upper bound), it suffers from performance degradation in the uncertainty estimation when the conditional density function has multiple modes. This paper demonstrates that multimodality should be considered in regression uncertainty estimation. To address the issue, we propose a novel method that generates a union of disjoint PIs. Throughout UCI benchmark experiments, our method improves over current state-of-the-art uncertainty quantification methods, reducing an average PI width by over 27$\%$. Through qualitative experiments, we visualized that the multi-mode often exists in real-world datasets and why our method produces high-quality PIs compared to the previous PI.
Adaptive Filters for Low-Latency and Memory-Efficient Graph Neural Networks
Authors: Shyam A. Tailor, Felix Opolka, Pietro Lio, Nicholas Donald Lane
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

Scaling and deploying graph neural networks (GNNs) remains difficult due to their high memory consumption and inference latency. In this work we present a new type of GNN architecture that achieves state-of-the-art performance with lower memory consumption and latency, along with characteristics suited to accelerator implementation. Our proposal uses memory proportional to the number of vertices in the graph ($\mathcal{O}(V)$), in contrast to competing methods which require memory proportional to the number of edges ($\mathcal{O}(E)$); surprisingly, we find our efficient approach actually achieves higher accuracy than competing approaches across 6 large and varied datasets against strong baselines. We achieve our results by using a novel \textit{adaptive filtering} approach, which can be interpreted as enabling each vertex to have its own weight matrix, and is not directly related to attention. Following our focus on efficient hardware usage, we demonstrate that our method achieves lower latency and memory consumption for the same accuracy when compared to competing approaches.
Multilevel physics informed neural networks (MPINNs)
Authors: Elisa Riccietti, Valentin Mercier, Serge Gratton, Pierre Boudier
Date: 2021-09-29 00:00:00

In this paper we introduce multilevel physics informed neural networks (MPINNs). Inspired by classical multigrid methods for the solution of linear systems arising from the discretization of PDEs, our MPINNs are based on the classical correction scheme, which represents the solution as the sum of a fine and a coarse term that are optimized in an alternate way. We show that the proposed approach allows to reproduce in the neural network training the classical acceleration effect observed for classical multigrid methods, thus providing a PINN that shows improved performance compared to the state-of-the-art. Thanks to the support of the coarse model, MPINNs provide indeed a faster and improved decrease of the approximation error in the case both of elliptic and nonlinear equations.
Seq2Tok: Deep Sequence Tokenizer for Retrieval
Authors: Adhiraj Banerjee, Vipul Arora
Date: 2021-09-29 00:00:00
Tasks: Retrieval

Search over sequences is a fundamental problem. Very efficient solutions exist for text sequences, which are made up of discrete tokens chosen from a finite alphabet. Sequences, such as audio, video or sensor readings, are made up of continuous-valued samples with a large sampling rate, making similarity search inefficient. This paper proposes Seq2Tok, a deep sequence tokenizer that converts continuous-valued sequences to discrete tokens that are easier to retrieve via sequence queries. The only information available for training Seq2Tok is pairs of similar sequences, i.e., depending on how we form the pairs, the similarity semantics are learnt. Seq2Tok compresses the query and target sequences into short sequences of tokens that are faster to match. Experiments show consistent performance of Seq2Tok across various audio retrieval tasks, namely, music search (query by humming) and speech keyword search via audio query.
Network robustness as a mathematical property: training, evaluation and attack
Authors: Marco Casadio, Matthew L Daggitt, Ekaterina Komendantskaya, Wen Kokke, Robert Stewart
Date: 2021-09-29 00:00:00

Neural networks are widely used in AI for their ability to detect general patterns in noisy data. Paradoxically, by default they are also known to not be particularly robust, i.e. moving a small distance in the input space can result in the network's output changing significantly.
Many methods for improving neural network robustness have been proposed recently. This growing body of research gave rise to numerous explicit or implicit notions of robustness.  Connections between these notions are often subtle, and a systematic comparison of these different definitions was lacking in the literature.
In this paper we attempt to address this gap by performing an in-depth comparison of the different definitions of robustness, by analysing their relationships, assumptions, interpretability and verifiability.
By abstracting robustness as a stand-alone mathematical property, we are able to show that, having a choice of several  definitions of robustness, one can combine them in a modular way when defining training modes, evaluation metrics, and attacks on neural networks.
We also perform experiments to compare the applicability and efficacy of different training methods for ensuring the network obeys these different definitions.
Boosting Semantic Segmentation via Feature Enhancement
Authors: Liu Zhi, Xiaojie Guo, Zhang Yi
Date: 2021-09-29 00:00:00
Tasks: Segmentation, Semantic Segmentation

Semantic segmentation aims to map each pixel of an image into its correspond-ing semantic label. Most existing methods either mainly concentrate on high-levelfeatures or simple combination of low-level and high-level features from backboneconvolutional networks, which may weaken or even ignore the compensation be-tween different levels. To effectively take advantages from both shallow (textural)and deep (semantic) features, this paper proposes a novel plug-and-play module,namelyfeature enhancement module(FEM). The proposed FEM first aligns fea-tures from different stages through a learnable filter to extract desired information,and then enhances target features by taking in the extracted message. Two types ofFEM,i.e.detail FEM and semantic FEM, are customized. Concretely, the formertype strengthens textural information to protect key but tiny/low-contrast detailsfrom suppression/removal, while the other one highlights structural informationto boost segmentation performance. By equipping a given backbone network withFEMs,  there might contain two information flows,i.e.detail flow and seman-tic flow.  Extensive experiments on Cityscapes, PASCAL Context, and ADE20Kdatasets are conducted to validate the effectiveness of our design, and reveal itssuperiority over other state-of-the-art alternatives.
When Complexity Is Good: Do We Need Recurrent Deep Learning For Time Series Outlier Detection?
Authors: Alexander Capstick, Samaneh Kouchaki, Mazdak Ghajari, David J. Sharp, Payam M. Barnaghi
Date: 2021-09-29 00:00:00
Tasks: Home Activity Monitoring, Outlier Detection, Time Series, Time Series Analysis

Outlier detection is a critical part of understanding a dataset and extracting results. Outlier detection is used in different domains for various reasons; including detecting stolen credit cards, spikes of energy usage, web attacks, or in-home activity monitoring. Within this paper, we look at when it is appropriate to apply recurrent deep learning methods for time series outlier detection versus non-recurrent methods. Recurrent deep learning methods have a larger capacity for learning complex representations in time series data. We apply these methods to various synthetic and real-world datasets, including a dataset containing information about the in-home movement of people living with dementia in a clinical study cross-referenced with their recorded unplanned hospital admissions and infection episodes. We also introduce two new outlier detection methods, that can be useful in detecting contextual outliers in time series data where complex temporal relationships and local variations in the time series are important.
GLASS: GNN with Labeling Tricks for Subgraph Representation Learning
Authors: Xiyuan Wang, Muhan Zhang
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Graph Representation Learning, Representation Learning

Despite the remarkable achievements of Graph Neural Networks (GNNs) on graph representation learning, few works have tried to use them to predict properties of subgraphs in the whole graph. Existing state-of-the-art method SubGNN introduces an overly complicated subgraph-level GNN model which synthesizes 6 artificial properties, yet only has a marginal edge over a plain GNN. We find that the reason for the failure of directly applying a plain GNN to the whole graph and then pooling the representations of nodes within the subgraph is that it cannot tell whether nodes are in the subgraph or not  when exchanging messages between them. With this insight, we introduce an expressive and scalable labeling trick, namely max-zero-one, and propose GLASS (GNN with LAbel for SubgraphS). Compared with SubGNN, GLASS is more expressive , more scalable and easier to implement. Experiments show that GLASS outperforms the strongest baseline by 13% on average. And ablation analysis shows that our max-zero-one labeling trick can boost the performance of a plain GNN up to 23%. And training a GLASS model only takes 28% time needed for a SubGNN on average.
A Practical PAC-Bayes Generalisation Bound for Deep Learning
Authors: Diego Granziol, Mingtian Zhang, Nicholas Baskerville
Date: 2021-09-29 00:00:00
Tasks: Deep Learning

Under a PAC-Bayesian framework, we derive an implementation efficient parameterisation invariant metric to measure the difference between our true and empirical risk. We show that for solutions of low training loss, this metric can be approximated at the same cost as a single step of SGD. We investigate the usefulness of this metric on pathological examples, where traditional Hessian based sharpness metrics increase but generalisation also increases and find good experimental agreement. As a consequence of our PAC-Bayesian framework and theoretical arguments on the effect of sub-sampling the Hessian, we include a trace of Hessian term into our structural risk. We find that this term promotes generalisation on a variety of experiments using Wide-Residual Networks on the CIFAR datasets.
Revisiting Virtual Nodes in Graph Neural Networks for Link Prediction
Authors: EunJeong Hwang, Veronika Thost, Shib Sankar Dasgupta, Tengfei Ma
Date: 2021-09-29 00:00:00
Tasks: Clustering, Graph Classification, Link Prediction

It is well known that the graph classification performance of graph neural networks often improves by adding an artificial virtual node to the graphs, which is connected to all nodes in the graph. Intuitively, the virtual node provides a shortcut for message passing between nodes along the graph edges. Surprisingly, the impact of virtual nodes with other problems is still an open research question. 

In this paper, we adapt the concept of virtual nodes to the link prediction scenario, where we usually have much larger, often dense, and more heterogeneous graphs. In particular, we use multiple virtual nodes per graph and graph-based clustering to determine the connections to the graph nodes. We also investigate alternative clustering approaches (e.g., random or more advanced) and compare to the original model with a single virtual node. We conducted extensive experiments over different datasets of the Open Graph Benchmark (OGB) and analyze the results in detail. We show that our virtual node extensions yield rather stable performance increases and allow standard graph neural networks to compete with complex state-of-the-art models, as well as with the models leading the OGB leaderboards.
Are Vision Transformers Robust to Patch-wise Perturbations?
Authors: Jindong Gu, Volker Tresp, Yao Qin
Date: 2021-09-29 00:00:00
Tasks: image-classification, Image Classification

The recent advances in Vision Transformer (ViT) have demonstrated its impressive performance in image classification, which makes it a promising alternative to Convolutional Neural Network (CNN). Unlike CNNs, ViT represents an input image as a sequence of image patches. The patch-wise input image representation makes the following question interesting: How does ViT perform when individual input image patches are perturbed with natural corruptions or adversarial perturbations, compared to CNNs? In this work, we conduct a comprehensive study on the robustness of vision transformers to patch-wise perturbations. Surprisingly, we find that vision transformers are more robust to naturally corrupted patches than CNNs, whereas they are more vulnerable to adversarial patches. Based on extensive qualitative and quantitative experiments, we discover that ViT's stronger robustness to natural corrupted patches and higher vulnerability against adversarial patches are both caused by the attention mechanism. Specifically, the attention model can help improve the robustness of vision transformers by effectively ignoring natural corrupted patches. However, when vision transformers are attacked by an adversary, the attention mechanism can be easily fooled to focus more on the adversarially perturbed patches and cause a mistake.
Boundary-aware Pre-training for Video Scene Segmentation
Authors: Jonghwan Mun, Minchul Shin, Gunsoo Han, Sangho Lee, Seongsu Ha, Joonseok Lee, Eun-Sol Kim
Date: 2021-09-29 00:00:00
Tasks: Scene Segmentation, Self-Supervised Learning

Self-supervised learning has drawn attention through its effectiveness in learning in-domain representations with no ground-truth annotations; in particular, it is shown that properly designed pretext tasks (e.g., contrastive prediction task) bring significant performance gains for a downstream task (e.g., classification task). Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks. In our framework, we discover a pseudo-boundary from a sequence of shots by splitting it into two continuous, non-overlapping sub-sequences and leverage the pseudo-boundary to facilitate the pre-training. Based on this, we introduce three novel boundary-aware pretext tasks: 1) Shot-Scene Matching (SSM), 2) Contextual Group Matching (CGM) and 3) Pseudo-boundary Prediction (PP); SSM and CGM guide the model to maximize intra-scene similarity and inter-scene discrimination while PP encourages the model to identify transitional moments. Through comprehensive analysis, we empirically show that pre-training and transferring contextual representation are both critical to improving the video scene segmentation performance. Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark. The code will be released.
Efficient Semi-Supervised Adversarial Training without Guessing Labels
Authors: Huimin Wu, Heng Huang, Bin Gu
Date: 2021-09-29 00:00:00

Adversarial training has been proved to be  the most effective defensive strategy to protect models from adversarial attacks.  In the practical application scenario of adversarial training, we face not only labeled data, but also an enormous amount of unlabeled data. However, existing adversarial training methods are naturally targeting  supervised learning problems.  To adapt to  semi-supervised learning problems, they need to estimate labels for unlabeled data in advance,  which inevitably degenerates the performance of the learned model due to the bias on the estimation of labels for unlabeled data. 
To mitigate this degeneration, in this paper, we propose a new semi-supervised  adversarial training framework via maximizing  AUCs which is also a minimax problem but  treats the unlabeled samples as both positive and negative ones, so that we do not need to guess the labels for unlabeled data.  Unsurprisingly, the minimax problem can be solved via the traditional adversarial training algorithm by extending singly stochastic gradients to triply stochastic gradients, to adapt to the three (i.e. positive, negative, and unlabeled) data sources. To further accelerate the training procedure, we transform the minimax adversarial training problem into an equivalent minimization one based on the kernel perspective. For the minimization problem, we discuss scalable and efficient algorithms not only for deep neural networks  but also for kernel support vector machines.  Extensive experimental results show that our algorithms not only achieve better generalization performance against various adversarial attacks,  but also enjoy efficiency and scalability when considered from the kernel perspective.
Propagating Distributions through Neural Networks
Authors: Felix Petersen, Christian Borgelt, Mikhail Yurochkin, Hilde Kuehne, Oliver Deussen
Date: 2021-09-29 00:00:00
Tasks: Prediction, regression

We propose a new approach to propagating probability distributions through neural networks. To handle non-linearities, we use local linearization and show this to be an optimal approximation in terms of total variation for ReLUs. We demonstrate the advantages of our method over the moment matching approach popularized in prior works. In addition, we formulate new loss functions for training neural networks based on distributions. To demonstrate the utility of propagating distributions, we apply it to quantifying prediction uncertainties. In regression tasks we obtain calibrated confidence intervals, and in a classification setting we improve selective prediction on out-of-distribution data. We also show empirically that training with our uncertainty aware losses improve robustness to random and adversarial noise.
The Needle in the haystack: Out-distribution aware Self-training in an Open-World Setting
Authors: Maximilian Augustin, Matthias Hein
Date: 2021-09-29 00:00:00
Tasks: Out-of-Distribution Detection, Self-Learning

Traditional semi-supervised learning (SSL) has focused on the closed world assumption where all unlabeled samples are task-related. In practice, this assumption is often violated when leveraging data from very large image databases that contain mostly non-task-relevant samples. While standard self-training and other established methods fail in this open-world setting, we demonstrate that our out-distribution-aware self-learning (ODST) with a careful sample selection strategy can leverage unlabeled datasets with millions of samples, more than 1600 times larger than the labeled datasets, and which contain only about $2\%$ task-relevant inputs. Standard and open world SSL techniques degrade in performance when the ratio of task-relevant sample decreases and show a significant distribution shift which is problematic regarding AI safety while ODST outperforms them with respect to test performance, corruption robustness and out-of-distribution detection.
Self-Joint Supervised Learning
Authors: Navid Kardan, Mubarak Shah, Mitch Hill
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Out-of-Distribution Detection

Supervised learning is a fundamental framework used to train machine learning systems. A supervised learning problem is often formulated using an i.i.d. assumption that restricts model attention to a single relevant signal at a time when predicting. This contrasts with the human ability to actively use related samples as reference when making decisions. We hypothesize that the restriction to a single signal for each prediction in the standard i.i.d. framework contributes to well-known drawbacks of supervised learning: making overconfident predictions and vulnerability to overfitting, adversarial attacks, and out-of-distribution data. To address these limitations, we propose a new supervised learning paradigm called self-joint learning that generalizes the standard approach by modeling the joint conditional distribution of two observed samples, where each sample is an image and its label. Rather than assuming samples are independent, our models explicitly learn the sample-to-sample relation of conditional independence. Our framework can naturally incorporate auxiliary unlabeled data to further improve the performance. Experiments on benchmark image datasets show our method offers significant improvement over standard supervised learning in terms of accuracy, robustness against adversarial attacks, out-of-distribution detection, and overconfidence mitigation.
Model Compression via Symmetries of the Parameter Space
Authors: Iordan Ganev, Robin Walters
Date: 2021-09-29 00:00:00
Tasks: model, Model Compression

We provide a theoretical framework for neural networks in terms of the representation theory of quivers, thus revealing symmetries of the parameter space of neural networks. An exploitation of these symmetries leads to a model compression algorithm for radial neural networks based on an analogue of the QR decomposition. The algorithm is lossless; the compressed model has the same feedforward function as the original model. If applied before training, optimization of the compressed model by gradient descent is equivalent to a projected version of gradient descent on the original model.
Data-oriented Scene Recognition
Authors: Zhinan Qiao, Xiaohui Yuan, Chaoning Zhang, Jianfang Shi, Jian Xia
Date: 2021-09-29 00:00:00
Tasks: Scene Recognition

Most deep learning backbones are evaluated on ImageNet. Using scenery images as an example, we conducted extensive experiments to demonstrate the widely accepted principles in network design may result in dramatic performance differences when the data is altered. Exploratory experiments are engaged to explain the underlining cause of the differences. Based on our observation, this paper presents a novel network design methodology: data-oriented network design. In other words, instead of designing universal backbones, the scheming of the networks should treat the characteristics of data as a crucial component. We further proposed a Deep-Narrow Network and Lossless Pooling module, which improved the scene recognition performance using less than half of the computational resources compared to the benchmark network architecture ResNets.