pwc-archive/papers-with-abstracts huggingface dataset:
Azimuthal Rotational Equivariance in Spherical CNNs
Authors: Carl Toft, Georg Bökman, Fredrik Kahl
Date: 2021-01-01 00:00:00

In this work, we analyze linear operators from $L^2(S^2) \rightarrow L^2(S^2)$ which are equivariant to azimuthal rotations, that is, rotations around the z-axis. Several high-performing neural networks defined on the sphere are indeed equivariant to azimuthal rotations, but not $SO(3)$. Our main result is to show that a linear operator acting on band-limited functions on the sphere is equivariant to azimuthal rotations if and only if it can be realized as a block-diagonal matrix acting on the spherical harmonic expansion coefficients of its input. Further, we show that such an operation can be interpreted as a convolution or equivalently a correlation. Our theoretical findings are backed up with experimental results demonstrating that a state-of-the-art pipeline can be improved by making it fully equivariant to azimuthal rotations.
Stochastic Optimization with Non-stationary Noise: The Power of Moment Estimation
Authors: Jingzhao Zhang, Hongzhou Lin, Subhro Das, Suvrit Sra, Ali Jadbabaie
Date: 2021-01-01 00:00:00
Tasks: Stochastic Optimization

We investigate stochastic optimization under weaker assumptions on the distribution of noise  than those used in usual analysis. Our assumptions are motivated by empirical observations in training neural networks. In particular, standard results on optimal convergence rates for stochastic optimization assume either there exists a uniform bound on the moments of the gradient noise, or that the noise decays as the algorithm progresses. These assumptions do not match the empirical behavior of optimization algorithms used in neural network training where the noise level in stochastic gradients could even increase with time. We address this nonstationary behavior of noise by analyzing convergence rates of stochastic gradient methods subject to changing second moment (or variance) of the stochastic oracle. When the noise variation is known, we show that it is always beneficial to adapt the step-size and exploit the noise variability. When the noise statistics are unknown, we obtain similar improvements by developing an online estimator of the noise level, thereby recovering close variants of RMSProp~\citep{tieleman2012lecture}. Consequently, our results reveal why adaptive step size methods can outperform SGD, while still enjoying theoretical guarantees.
Learning Monotonic Alignments with Source-Aware GMM Attention
Authors: Tae Gyoon Kang, Ho-Gyeong Kim, Min-Joong Lee, Jihyun Lee, Seongmin Ok, Hoshik Lee, Young Sang Choi
Date: 2021-01-01 00:00:00
Tasks: speech-recognition, Speech Recognition

Transformers with soft attention have been widely adopted to various sequence-to-sequence tasks. While the soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which are crucial for monotonic sequence-to-sequence tasks. Learning monotonic alignments between input and output sequences could be beneficial for long-form and online inference applications that are still challenging problem for the conventional soft attention algorithm. In this paper, we focus on monotonic sequence-to-sequence task and propose Source-Aware Gaussian Mixture Model (SAGMM) attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence. We show that the proposed attention mechanism solves the online and long-form speech recognition problems without performance degradation in offline in-distribution speech recognition through experiments.
Learning to Generate Questions by Recovering Answer-containing Sentences
Authors: Seohyun Back, Akhil Kedia, Sai Chetan Chinthakindi, Haejun Lee, Jaegul Choo
Date: 2021-01-01 00:00:00
Tasks: Machine Reading Comprehension, Question Answering, Question Generation, Question-Generation, Reading Comprehension, Sentence

To train a question answering model based on machine reading comprehension (MRC), significant effort is required to prepare annotated training data composed of questions and their answers from contexts. To mitigate this issue, recent research has focused on synthetically generating a question from a given context and an annotated (or generated) answer by training an additional generative model, which can be utilized to augment the training data. In light of this research direction, we propose a novel pre-training approach that learns to generate contextually rich questions, by recovering answer-containing sentences. Our approach is composed of two novel components, (1) dynamically determining K answers from a given document and (2) pre-training the question generator on the task of generating the answer-containing sentence. We evaluate our method against existing ones in terms of the quality of generated questions as well as the fine-tuned MRC model accuracy after training on the data synthetically generated by our method. Experimental results demonstrate that our approach consistently improves the question generation capability of existing models such as UniLM, and shows state-of-the-art results on MS MARCO and NewsQA, and comparable results to the state-of-the-art on SQuAD. Additionally, we demonstrate that the data synthetically generated by our approach is beneficial for boosting up the downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modification to the existing MRC models. Furthermore, our experiments highlight that our method shines especially when a limited amount of training data is given, in terms of both pre-training and downstream MRC data.
A Distributional Perspective on Actor-Critic Framework
Authors: Daniel Wontae Nam, Younghoon Kim, Chan Youn Park
Date: 2021-01-01 00:00:00
Tasks: Distributional Reinforcement Learning

Recent distributional reinforcement learning methods, despite their successes, still contain fundamental problems that can lead to inaccurate representations of value distributions, such as distributional instability, action type restriction, and biased approximation. In this paper, we present a novel distributional actor-critic frame-work, GMAC, to address such problems.  Adopting a stochastic policy removes the first two problems, and the bias in approximation is alleviated by minimizing the Cramer distance between the value distribution and its Bellman target distribution. In addition, GMAC improves data efficiency by generating the Bellman target distribution through Sample-Replacement algorithm, denoted by SR(λ), which provides a distributional generalization of multi-step policy evaluation algorithms.We empirically show that our method captures the multimodality of value distributions and improves the performance of conventional actor-critic methods with low computational cost in both discrete and continuous action spaces, using ArcadeLearning Environment (ALE) and PyBullet environment.
Fully Convolutional Approach for Simulating Wave Dynamics
Authors: Mario Lino Valencia, Chris D Cantwell, Eduardo Pignatelli, Stathi Fotiadis, Anil Anthony Bharath
Date: 2021-01-01 00:00:00

In this work, we investigate the performance of fully convolutional networks to predict the motion and interaction of surface waves in open and closed complex geometries. We focus on a U-Net architecture and assess its ability to capture and extrapolate wave propagation in time as well as the reflection, interference and diffraction of waves.  We investigate how well the network generalises both to long-time predictions and to geometric configurations not seen during training. We demonstrate that our U-Net architecture is capable of accurately predicting the height distribution of waves on a liquid surface within curved and multi-faceted open and closed geometries, when only simple box and right-angled corner geometries were seen during training. We found the RMSE of the predictions remained of order $1\times10^{-4}$ times the characteristic length of the domain for at least 20 time-steps.
Robust Learning via Golden Symmetric Loss of (un)Trusted Labels
Authors: Amirmasoud Ghiassi, Robert Birke, Lydia Y. Chen
Date: 2021-01-01 00:00:00

Learning robust deep models against noisy labels becomes ever critical when today's data is commonly collected from open platforms and subject to adversarial corruption. The information on the label corruption process, i.e., corruption matrix, can greatly enhance the robustness of deep models but still fall behind in combating hard classes. In this paper, we propose to construct a golden symmetric loss (GSL) based on the estimated confusion matrix as to avoid overfitting to noisy labels and learn effectively from hard classes. GSL is the weighted sum of the corrected regular cross entropy and reverse cross entropy. By leveraging a small fraction of trusted clean data, we estimate the corruption matrix and use it to correct the loss as well as to determine the weights of GSL. We theoretically prove the robustness of the proposed loss function in the presence of dirty labels. We provide a heuristics to adaptively tune the loss weights of GSL according to the noise rate and diversity measured from the dataset. We evaluate our proposed golden symmetric loss on both vision and natural language deep models subject to different types of label noise patterns. Empirical results show that GSL can significantly outperform the existing robust training methods on different noise patterns, showing accuracy improvement up to 18% on CIFAR-100 and 1% on real world noisy dataset of Clothing1M.
Domain-Free Adversarial Splitting for Domain Generalization
Authors: Xiang Gu, Jiasun Feng, Jian Sun, Zongben Xu
Date: 2021-01-01 00:00:00
Tasks: Domain Generalization, Meta-Learning

Domain generalization is an approach that utilizes several source domains to train the learner to be generalizable to unseen target domain to tackle domain shift issue. It has drawn much attention in machine learning community. This paper aims to learn to generalize well to unseen target domain without relying on the knowledge of the number of source domains and domain labels. To achieve that goal, we unify adversarial training and meta-learning in a novel proposed Domain-Free Adversarial Splitting (DFAS) framework. In this framework, we model the domain generalization as a learning problem that enforces the learner to be able to generalize well for any train/val subsets splitting of the training dataset. This model can be further transformed to be a min-max optimization problem which can be solved by an iterative adversarial training process. In each iteration, it adversarially splits the training dataset into train/val subsets to maximize domain shift between them using current learner, and then updates the learner on this splitting to be able to generalize well from train-subset to val-subset using meta-learning approach. Extensive experiments on three benchmark datasets under three different settings on the source and target domains show that our method achieves state-of-the-art results and confirm the effectiveness of our method by ablation study. We also derive a generalization error bound for theoretical understanding of our method.
Neural Network Surgery: Combining Training with Topology Optimization
Authors: Elisabeth Schiessler, Roland Aydin, Kevin Linka, Christian Cyron
Date: 2021-01-01 00:00:00
Tasks: Decision Making, Neural Architecture Search

With ever increasing computational capacities, neural networks become more and more proficient at solving complex tasks. However, picking a sufficiently good network topology usually relies on expert human knowledge. Neural architecture search aims to reduce the extent of expertise that is needed. Modern architecture search techniques often rely on immense computational power, or apply trained meta controllers for decision making. We develop a framework for a genetic algorithm that is both computationally cheap and makes decisions based on mathematical criteria rather than trained parameters. It is a hybrid approach that fuses training and topology optimization together into one process. Structural modifications that are performed include adding or removing layers of neurons, with some re-training applied to make up for incurred change in input-output behaviour. Our ansatz is tested on both the SVHN and (augmented) CIFAR-10 datasets with limited computational overhead compared to training only the baseline. This algorithm can achieve a significant increase in accuracy (as compared to a fully trained baseline), rescue insufficient topologies that in their current state are only able to learn to a limited extent, and dynamically reduce network size without loss in achieved accuracy.
ColdExpand: Semi-Supervised Graph Learning in Cold Start
Authors: Il-Jae Kwon, Kyoung-Woon On, Dong-Geon Lee, Byoung-Tak Zhang
Date: 2021-01-01 00:00:00
Tasks: Graph Learning, Link Prediction, Node Classification, Prediction

Most real-world graphs are dynamic and eventually face the cold start problem. A fundamental question is how the new cold nodes acquire initial information in order to be adapted in to the existing graph. Here we propose a method that utilizes the deep autoencoding principle to extract structural information from the existing graph. In particular, we apply graph convolutional networks to solve the ”Expanded Semi-Supervised Graph Learning” problem in cold start situations. The proposed ColdExpand model classifies the cold nodes based on link prediction with multiple goals to tackle. We experimentally prove that by adding additional goal to existing link prediction method,  our method outperforms the baseline in both expanded semi-supervised link prediction(at most 24%) and node classification tasks(at most 15%).To the best of our knowledge this is the first study to address expansion of the semi-supervised learning to unseen nodes.
Information distance for neural network functions
Authors: Xiao Zhang, Dejing Dou, Ji Wu
Date: 2021-01-01 00:00:00

We provide a practical distance measure in the space of functions parameterized by neural networks. It is based on classic information distance and we replace the uncomputable Kolmogorov complexity in the original definition with information measured by codelength of prequential coding. Empirically, we show that information distance is invariant with respect to different parameterization of the functions with neural networks. We also verify that information distance can faithfully reflect similarities of neural network functions. Finally, we applied information distance to investigate the relationship between neural network models, and demonstrate the connection between information distance and multiple characteristics and behaviors of neural network models.
Perturbation Type Categorization for Multiple $\ell_p$ Bounded Adversarial Robustness
Authors: Pratyush Maini, Xinyun Chen, Bo Li, Dawn Song
Date: 2021-01-01 00:00:00
Tasks: Adversarial Robustness, Vocal Bursts Type Prediction

Despite the recent advances in $\textit{adversarial training}$ based defenses, deep neural networks are still vulnerable to adversarial attacks outside the perturbation type they are trained to be robust against. Recent works have proposed defenses to improve the robustness of a single model against the union of multiple perturbation types. However, when evaluating the model against each individual attack, these methods still suffer significant trade-offs compared to the ones specifically trained to be robust against that perturbation type. In this work, we introduce the problem of categorizing adversarial examples based on their $\ell_p$ perturbation types. Based on our analysis, we propose $\textit{PROTECTOR}$, a two-stage pipeline to improve the robustness against multiple perturbation types. Instead of training a single predictor, $\textit{PROTECTOR}$ first categorizes the perturbation type of the input, and then utilizes a predictor specifically trained against the predicted perturbation type to make the final prediction. We first theoretically show that adversarial examples created by different perturbation types constitute different distributions, which makes it possible to distinguish them. Further, we show that at test time the adversary faces a natural trade-off between fooling the perturbation type classifier and the succeeding predictor optimized with perturbation specific adversarial training. This makes it challenging for an adversary to plant strong attacks against the whole pipeline. In addition, we demonstrate the realization of this trade-off in deep networks by adding random noise to the model input at test time, enabling enhanced robustness  against strong adaptive attacks. Extensive experiments on MNIST and CIFAR-10 show that $\textit{PROTECTOR}$ outperforms prior adversarial training based defenses by over $5\%$, when tested against the union of $\ell_1, \ell_2, \ell_\infty$ attacks.
Model information as an analysis tool in deep learning
Authors: Xiao Zhang, Di Hu, Xingjian Li, Dejing Dou, Ji Wu
Date: 2021-01-01 00:00:00
Tasks: Deep Learning

Information-theoretic perspectives can provide an alternative dimension of analyzing the learning process and complements usual performance metrics. Recently several works proposed methods for quantifying information content in a model (which we refer to as "model information"). We demonstrate using model information as a general analysis tool to gain insight into problems that arise in deep learning. By utilizing model information in different scenarios with different control variables, we are able to adapt model information to analyze fundamental elements of learning, i.e., task, data, model, and algorithm. We provide an example in each domain that model information is used as a tool to provide new solutions to problems or to gain insight into the nature of the particular learning setting. These examples help to illustrate the versatility and potential utility of model information as an analysis tool in deep learning.
Cross-Probe BERT for Efficient and Effective Cross-Modal Search
Authors: Tan Yu, Hongliang Fei, Ping Li
Date: 2021-01-01 00:00:00
Tasks: Image Retrieval, Retrieval

Inspired by the great success of BERT in NLP tasks, many text-vision BERT models emerged recently. Benefited from cross-modal attention,  text-vision BERT models have achieved excellent performance in many language-vision tasks including text-image retrieval.   Nevertheless,  cross-modal attentions used in text-vision BERT models require too expensive computation cost when solving text-vision retrieval, which is impractical for large-scale search. In this work, we develop a novel architecture, Cross-Probe BERT.  It relies on devised text and vision probes, and cross-modal attentions are conducted on text and vision probes.  It takes lightweight computation cost, and meanwhile effectively exploits cross-modal attention.  Systematic experiments conducted on two public benchmarks demonstrate state-of-the-art effectiveness and efficiency of the proposed method.
Learning Safe Policies with Cost-sensitive Advantage Estimation
Authors: Bingyi Kang, Shie Mannor, Jiashi Feng
Date: 2021-01-01 00:00:00
Tasks: Reinforcement Learning (RL)

Reinforcement Learning (RL) with safety guarantee is critical for agents performing tasks in risky environments. Recent safe RL algorithms, developed based on Constrained Markov Decision Process (CMDP), mostly take the safety requirement as additional constraints when learning to maximize the return. However, they usually make unnecessary compromises in return for safety and only learn sub-optimal policies, due to the inability of differentiating safe and unsafe state-actions with high rewards. To address this, we propose Cost-sensitive Advantage Estimation (CSAE), which is simple to deploy for policy optimization and effective for guiding the agents to avoid unsafe state-actions by penalizing their advantage value properly. Moreover, for stronger safety guarantees, we develop a Worst-case Constrained Markov Decision Process (WCMDP) method to augment CMDP by constraining the worst-case safety cost instead of the average one. With CSAE and WCMDP, we develop new safe RL algorithms with theoretical justifications on their benefits for safety and performance of the obtained policies. Extensive experiments clearly demonstrate the superiority of our algorithms in learning safer and better agents under multiple settings.
Dense Global Context Aware RCNN for Object Detection
Authors: Wenchao Zhang, Haoyu Xie, Mai Zhu, Chong Fu
Date: 2021-01-01 00:00:00
Tasks: Object, object-detection, Object Detection

RoIPool/RoIAlign is an indispensable process for the typical two-stage object detection algorithm, it is used to rescale the object proposal cropped from the feature pyramid to generate a fixed size feature map. However, these cropped feature maps of local receptive fields will heavily lose global context information. To tackle this problem, in this paper, we propose a novel end-to-end trainable framework, called Dense Global Context Aware (DGCA) RCNN, aiming at assisting the neural network in strengthening the spatial correlation between the background and the foreground by fusing global context information. The core component of our DGCA framework is a context aware mechanism, in which both global feature pyramid and attention strategies are used for feature extraction and feature refinement, respectively. Specifically, we leverage the dense connection to fuse the global context information of different stages in the top-down process of FPN, and further leverage the attention mechanism to perform global context aware. Thus, the implicit relationship between object proposal and global features can be captured by neural networks to improve detection performance. Experimental results on COCO benchmark dataset demonstrate the significant advantages of our approach.
Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference
Authors: Byung-Kwan Lee, Youngjoon Yu, Yong Man Ro
Date: 2021-01-01 00:00:00
Tasks: Adversarial Defense, Adversarial Robustness, Bayesian Inference, Probabilistic Deep Learning, Variational Inference

Recent works have applied Bayesian Neural Network (BNN) to adversarial training, and shown the improvement of adversarial robustness via the BNN's strength of stochastic gradient defense. However, we have found that in general, the BNN loses its stochasticity after its training with the posterior. As a result, the lack of the stochasticity leads to weak regularization effect to the BNN, which increases KL divergence in ELBO from variational inference. In this paper, we propose an enhanced Bayesian regularizer through hierarchical variational inference in order to boost adversarial robustness against gradient-based attack. Furthermore, we also prove that the proposed method allows the BNN's stochasticity to be elevated with the reduced KL divergence. Exhaustive experiment results demonstrate the effectiveness of the proposed method by showing the improvement of adversarial robustness for the BNN, compared with adversarial training (Madry et al., 2018) and adversarial-BNN (Liu et al., 2019) under PGD attack and EOT-PGD attack to the $L_{\infty}$ perturbation on CIFAR-10/100, STL-10, and Tiny-ImageNet.