pwc-archive/papers-with-abstracts huggingface dataset:
PGD-2 can be better than FGSM + GradAlign
Authors: Tianhang Zheng, Baochun Li
Date: 2021-09-29 00:00:00

One major issue of adversarial training (AT) with the fast gradient sign method (FGSM AT) is the phenomenon of catastrophic overfitting, meaning that the trained model suddenly loses its robustness over a single epoch. In addition to FGSM AT, Andriushchenko & Flammarion (2020) observed that two-step projected gradient descent adversarial training (PGD-2 AT) also suffers from catastrophic overfitting for large $\ell_\infty$ perturbations. To prevent catastrophic overfitting, Andriushchenko & Flammarion (2020) proposed a gradient alignment regularization method (GradAlign) and claimed that GradAlign can prevent catastrophic overfitting in FGSM AT and PGD-2 AT. In this paper, we show that PGD-2 AT with random initialization (PGD-2-RS AT) and attack step size $\alpha=1.25\epsilon/2$ only needs approximately a half computational cost of FGSM + GradAlign AT and actually can avoid catastrophic overfitting for large $\ell_\infty$ perturbations. We hypothesize that, if FGSM-RS AT with $\alpha=1.25\epsilon/2$ can avoid catastrophic overfitting for $\ell_\infty$ perturbation size $\epsilon/2$, then PGD-2-RS AT with $\alpha=1.25\epsilon/2$ may be able to avoid catastrophic overfitting for $\ell_\infty$ perturbation size $\epsilon$. Our intuitions to justify this empirical hypothesis induce a more unexpected finding: If we apply random noise from the uniform distribution $\mathcal{U}(-\epsilon/2, \epsilon/2)$ to the perturbations before each step of PGD-2 with $\alpha=1.25\epsilon/2$, instead of initializing the perturbations with random noise from $\mathcal{U}(-\epsilon, \epsilon)$ at the beginning ({\em i.e.,} the conventional random initialization scheme), the corresponding AT method can also avoid catastrophic overfitting and even achieve better robust accuracy in most cases. We refer to this AT method as Qusai-PGD-2-RS AT. Extensive evaluations demonstrate that PGD-2-RS AT and Qusai-PGD-2-RS AT with $\alpha=1.25\epsilon/2$ achieve better performance and efficiency than FGSM + GradAlign AT. Notably, Qusai-PGD-2-RS AT achieves comparable robust accuracy against PGD-50-10 as PGD-3-RS AT on CIFAR10 and SVHN, and it also achieves approximately $18\%$ top-1 and $38\%$ top-5 robust accuracy against PGD-50-10 at $\epsilon=8/255$ on ImageNet.
RNAS: Robust Network Architecture Search beyond DARTS
Authors: Yaguan Qian, Shenghui Huang, Yuqi Wang, Simin Li
Date: 2021-09-29 00:00:00

The vulnerability of Deep Neural Networks (DNNs) (i.e., susceptibility to adversarial attacks) severely limits the application of DNNs. Most of the existing methods improve the robustness of the model from weights optimization, such as adversarial training and regularization. However, the architecture of DNNs is also a key factor to robustness, which is often neglected or underestimated. We propose a Robust Network Architecture Search (RNAS) to address this problem. In our method, we define a network vulnerability metric based on the features’ deviation between clean examples and adversarial examples. Through constraining this vulnerability, we search the robust architecture and solve it by iterative optimization. The extensive experiments conducted on CIFAR-10/100 and SVHN show that our model achieves the best performance under various adversarial attacks compared with extensive baselines and state-of-the-art methods.
LMSA: Low-relation Mutil-head Self-Attention Mechanism in Visual Transformer
Authors: JingJie Wang, Xiang Wei, Xiaoyu Liu
Date: 2021-09-29 00:00:00
Tasks: image-classification, Image Classification, Relation

The Transformer backbone network with the self-attention mechanism as the core has achieved great success in the field of natural language processing and computer vision. However, through the self-attention mechanism brings high performance, it also brings higher computational complexity compared to the classic visual feature extraction methods. To further reduce the complexity of self-attention mechanism and explore its lighter version in computer vision, in this paper, we design a novel lightweighted self-attention mechanism: Low-relation Mutil-head Self-Attention (LMSA), which is superior than the recent self-attention. Specifically, the proposed self-attention mechanism breaks the barrier of the dimensional consistency of the traditional self-attention mechanism, resulting in lower computational complexity and occupies less storage space. In addition, employing the new mechanism can release part of the computing consumption of the Transformer network and  make the best use of it. Experimental results show that the dimensional consistency inside the traditional self-attention mechanism is unnecessary. In particular, using Swin as the backbone model for training, the accuracy in CIFAR-10 image classification task is improved by 0.43$\%$, in the meanwhile, the consumption of a single self-attention resource is reduced by 64.58$\%$, and the number of model parameters and model size are reduced by more than 15$\%$. By appropriately compressing the dimensions of the self-attention relationship variables, the Transformer network can be more efficient and even perform better. The results prompt us to rethink the reason why the self-attention mechanism works.
Policy improvement by planning with Gumbel
Authors: Ivo Danihelka, Arthur Guez, Julian Schrittwieser, David Silver
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: reinforcement-learning, Reinforcement Learning (RL)

AlphaZero is a powerful reinforcement learning algorithm based on approximate policy iteration and tree search. However, AlphaZero can fail to improve its policy network, if not visiting all actions at the root of a search tree. To address this issue, we propose a policy improvement algorithm based on sampling actions without replacement. Furthermore, we use the idea of policy improvement to replace the more heuristic mechanisms by which AlphaZero selects and uses actions, both at root nodes and at non-root nodes. Our new algorithms, Gumbel AlphaZero and Gumbel MuZero, respectively without and with model-learning, match the state of the art on Go, chess, and Atari, and significantly improve prior performance when planning with few simulations.
Time-aware Relational Graph Attention Network for Temporal Knowledge Graph Embeddings
Authors: Chengjin Xu, Fenglong Su, Jens Lehmann
Date: 2021-09-29 00:00:00
Tasks: Entity Alignment, Graph Attention, Knowledge Graph Embeddings, Knowledge Graphs, Representation Learning

Embedding-based representation learning approaches for knowledge graphs (KGs) have been mostly designed for static data. However, many KGs involve temporal data, which creates the need for new representation learning approaches that can characterize and reason over time. In this work, we propose a Time-aware Relational Graph ATtention Network (TR-GAT) for temporal knowledge graph (TKG) embeddings, in which the initial feature of each entity is represented by fusing its embedding and the embeddings of its connected relations and timestamps as well as its neighboring entities. Different from the existing temporal GNN models which discretize temporal graphs into multiple snapshots, we treat timestamps as properties of links between entities. To further incorporate relation and time information into the graph structures, we utilize a self-attention mechanism which specifies different weights to different nodes according to the corresponding link features, i.e., embeddings of the relevant relations and timestamps within one neighborhood. Experimental results show that our approach achieves state-of-the-art performances regarding TKG completion and entity alignment tasks on several well-established TKG datasets due to the effective and efficient integration of time information.
C+1 Loss: Learn to Classify C Classes of Interest and the Background Class Differentially
Authors: Changhuai Chen, Xile Shen, Mengyu Ye, Yi Lu, Jun Che, ShiLiang Pu
Date: 2021-09-29 00:00:00
Tasks: Classification, Human Parsing, object-detection, Object Detection, Semantic Segmentation

There is one kind of problem all around the classification area, where we want to classify C+1 classes of samples, including C semantically deterministic classes which we call classes of interest and the (C+1)th semantically undeterministic class which we call background class. In spite of most classification algorithm use softmax-based cross-entropy loss to supervise the classifier training process without differentiating the background class from the classes of interest, it is unreasonable as each of the classes of interest has its own inherent characteristics, but the background class dosen’t. We figure out that the background class should be treated differently from the classes of interest during training. Motivated by this, firstly we define the C+1 classification problem. Then, we propose three properties that a good C+1 classifier should have: basic discriminability, compactness and background margin. Based on them we define a uniform general C+1 loss, composed of three parts, driving the C+1 classifier to satisfy those properties. Finally, we instantialize a C+1 loss and experiment it in semantic segmentation, human parsing and object detection tasks. The proposed approach shows its superiority over the traditional cross-entropy loss.
Revisiting Linear Decision Boundaries for Few-Shot Learning with Transformer Hypernetworks
Authors: Samrudhdhi Bharatkumar Rangrej, Kevin J Liang, Xi Yin, Guan Pang, Theofanis Karaletsos, Lior Wolf, Tal Hassner
Date: 2021-09-29 00:00:00
Tasks: Few-Shot Learning, image-classification, Image Classification

Few-shot learning (FSL) methods aim to generalize a model to new unseen classes using only a small number of support examples. In image classification settings, many FSL approaches utilize a similar architecture to standard supervised learning, learning a model composed of a feature extractor followed by a linear classifier head. A common choice for the classifier is ProtoNet-style nearest neighbor, but this may be suboptimal as it is context-independent. As an alternative, some methods train a parametric classifier (e.g. logistic regression, support vector machine) using embeddings from novel classes. However, task-specific training requires time and resources, and poses optimization challenges such as overfitting on only a few samples. Instead, we propose to generate linear classifiers for new classes using a transformer-based hypernetwork, performing context aggregation in permutation invariant manner. A transformer hypernetwork allows us to instantiate a new task-specific classifier without any additional training on novel tasks. Experiments conducted on 1-shot 5-way and 5-shot 5-way MiniImageNet, TieredImageNet, and CIFAR-FS demonstrate that transformer hypernetworks are capable of generating classifiers that achieve up to 1.4% higher accuracy than other commonly used linear classifiers. Among the group of methods that offer optimization-free meta-inference, we achieve new state-of-the-art in most cases.
Heterologous Normalization
Authors: Chunjie Luo, Jianfeng Zhan, Lei Wang, Wanling Gao
Date: 2021-09-29 00:00:00

Batch Normalization has become a standard technique for training modern deep networks. However, its effectiveness diminishes when the batch size becomes smaller since the batch statistics estimation becomes inaccurate. This paper proposes Heterologous Normalization, which computes normalization's mean and standard deviation from different pixel sets to take advantage of different normalization methods. Specifically, it calculates the mean like Batch Normalization to maintain the advantage of Batch Normalization. Meanwhile, it enlarges the number of pixels from which the standard deviation is calculated, thus alleviating the problem caused by the small batch size. Experiments show that Heterologous Normalization surpasses or achieves comparable performance to existing homologous methods, with large or small batch sizes on various datasets.
Coherence-Based Document Clustering
Authors: Anton Frederik Thielmann, Christoph Weisser, Thomas Kneib, Benjamin Saefken
Date: 2021-09-29 00:00:00
Tasks: Clustering, Hyperparameter Optimization, Keyword Extraction

Latent Dirichlet Allocation or Non-negative Matrix Factorization are just two widely used algorithms for extracting latent topics from large text corpora. While these algorithms differ in their modeling approach, they have in common that hyperparameter optimization is difficult and is mainly achieved by maximizing the extracted topic coherence scores via a grid search. Models using word-document embeddings can automatically detect the number of latent topics, but tend to have problems with smaller datasets and often require pre-trained embedding layers for successful topic extraction. We leverage widely used coherence scores by integrating them into a novel document-level clustering approach using keyword extraction methods. The metric by which most topic extraction methods optimize their hyperparameters is thus optimized during clustering, resulting in ultra-coherent clusters. Moreover, unlike traditional methods, the number of extracted topics or clusters does not need to be determined in advance, saving us an additional optimization step and a time- and computationally-intensive grid search. Additionally, the number of topics is detected much more accurately than by models leveraging word-document embeddings.
TaCE: Time-aware Convolutional Embedding Learning for Temporal Knowledge Graph Completion
Authors: Jin Luo, Hong Shen, YanFeng Hu, Chen Peng
Date: 2021-09-29 00:00:00
Tasks: Knowledge Graph Completion, Knowledge Graphs, Temporal Knowledge Graph Completion

Temporal knowledge graph completion (TKGC) is a challenging task to infer the missing component for quadruples. The key challenge lies at how to integrate time information into the embeddings of entities and relations. Recent TKGC methods tend to capture temporal patterns via linear or multilinear models, which are fast but not expressive enough. In this study, we propose a novel time-aware convolutional embedding model (TaCE) to represent the time-dependent facts in the task of TKGC. It highlights its novelty to feasibly convert timestamps as temporal convolutional filters to fully interact with entities and relations and learn temporal patterns in knowledge graphs (KGs). An extensive comparison proves that our model outperforms the state-of-the-art models on three public benchmark datasets of ICEWS14, ICEWS05-15 and GDELT. Results also demonstrate good temporal expressiveness and computation efficiency performed by our TaCE.
Generalized Maximum Entropy Reinforcement Learning via Reward Shaping
Authors: Feng Tao, Yongcan Cao
Date: 2021-09-29 00:00:00
Tasks: MuJoCo, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Entropy regularization is a commonly used technique in reinforcement learning to improve exploration and cultivate a better pre-trained policy for later adaptation. Recent studies further show that the use of entropy regularization can smooth the optimization landscape and simplify the policy optimization process, which indicates the value of integrating entropy into reinforcement learning. However, existing studies only consider the policy’s entropy at the current state as an extra regularization term in the policy gradient or in the objective function, while the topic of integrating the entropy into the reward function has not been investigated. In this paper, we propose a shaped reward that includes the agent’s policy entropy into the reward function. In particular, the agent’s entropy at the next state is added to the immediate reward associated with the current state. The addition of the agent’s policy entropy at the next state, instead of the policy entropy at the current state as used in the existing maximum entropy reinforcement learning framework, considers both state and action uncertainties. This distinguishes our work from the existing maximum entropy reinforcement learning framework via providing better action exploration and better control policies. We also show the addition of the agent’s policy entropy at the next state yields new soft Q function and state value function that are concise and modular. Hence, the new reinforcement learning framework can be easily applied to the existing standard reinforcement learning algorithms while inheriting the benefits of employing entropy regularization. We further present a soft stochastic policy gradient theorem based on the shaped reward and propose a new practical reinforcement learning algorithm. Finally, a few experimental studies are conducted in the MuJoCo environment to demonstrate that our method can outperform the existing state-of-the-art reinforcement learning approaches.
Equivariant Transformers for Neural Network based Molecular Potentials
Authors: Philipp Thölke, Gianni de Fabritiis
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Computational Efficiency

The prediction of quantum mechanical properties is historically plagued by a trade-off between accuracy and speed. Machine learning potentials have previously shown great success in this domain, reaching increasingly better accuracy while maintaining computational efficiency comparable with classical force fields. In this work we propose a novel equivariant Transformer architecture, outperforming state-of-the-art on MD17, ANI-1, and many QM9 targets in both accuracy and computational efficiency. Through an extensive attention weight analysis, we gain valuable insights into the black box predictor and show differences in the learned representation of conformers versus conformations sampled from molecular dynamics or normal modes. Furthermore, we highlight the importance of datasets including off-equilibrium conformations for the evaluation of molecular potentials.
ARMCMC: Online Bayesian Density Estimation of Model Parameters
Authors: Pedram Agand, Mo Chen, Hamid Taghirad
Date: 2021-09-29 00:00:00
Tasks: Density Estimation

Although the Bayesian paradigm provides a rigorous framework to estimate the full probability distribution over unknown parameters,  its online  implementation can be challenging due to heavy computational costs. This paper proposes Adaptive Recursive Markov Chain Monte Carlo (ARMCMC) which estimates full probability density of model parameters while alleviating  shortcomings of conventional online approaches. These shortcomings include: being solely able to account for Gaussian noise, being applicable to systems with linear in the parameters (LIP) constraint, or having requirements on persistence excitation (PE). In ARMCMC, we propose a variable jump distribution, which depends on a temporal forgetting factor.  This allows one to adjust the trade-off between exploitation and exploration, depending on whether there is an abrupt change to the parameter being estimated. We prove that ARMCMC requires fewer samples to achieve the same precision and reliability compared to conventional MCMC approaches.  We demonstrate our approach on two challenging benchmarks:  the estimation of parameters in a soft bending actuator and the Hunt-Crossley dynamic model. Our method shows at-least 70\% improvement in parameter point estimation accuracy and approximately 55\% reduction in tracking error of the value of interest compared to recursive least squares and conventional MCMC.
FedMorph: Communication Efficient Federated Learning via Morphing Neural Network
Authors: Guoqing Ma, Chuanting Zhang, Basem Shihada
Date: 2021-09-29 00:00:00
Tasks: Federated Learning, MORPH

The two fundamental bottlenecks in Federated Learning (FL) are communication and computation on heterogeneous edge networks, restricting both model capacity and user participation. 
To address these issues, we present FedMorph, an approach to automatically morph the global neural network to a sub-network to reduce both the communication and local computation overloads. 
FedMorph distills a fresh sub-network from the original one at the beginning of each communication round while keeps its `knowledge' as similar as the aggregated model from local clients in a federated average (FedAvg) like way.
The network morphing process considers the constraints, e.g., model size or computation flops, as an extra regularizer to the objective function.
To make the objective function solvable, we relax the model with the concept of soft-mask.
We empirically show that FedMorph, without any other tricks, reduces communication and computation overloads and increases the generalization accuracy.
E.g., it provides an $85\times$ reduction in server-to-client communication and $18\times$ reduction in local device computation on the MNIST dataset with ResNet8 as the training network.
With benchmark compression approaches, e.g., TopK sparsification, FedMorph collectively provides an $847\times$ reduction in upload communication.
Adversarial Training: A simple and efficient technique to Improving NLP Robustness
Authors: marwan omar
Date: 2021-09-29 00:00:00
Tasks: Adversarial Text, Attribute, Sentiment Analysis, Sentiment Classification, Text Generation

NLP models are shown to be prone to adversarial attacks which undermines their robustness, i.e. a small perturbation to the input text can fool an NLP model to incorrectly classify text. In this study, we present a new Adversarial Text Generation technique that, given an input text, generates adversarial texts through quickly and efficiently. For example, in order to attack a model for sentiment classification, we can use the product categories as the attribute which should not change the sentiment of the reviews. We conducted experiments on real-world NLP datasets to demonstrate that our technique can generate more meaningful and diverse adversarial texts, compared to many existing adversarial text generation approaches. We further use our generated adversarial examples to improve models through adversarial training, and we demonstrate that our generated attacks are more robust against model re training and different model architectures.
TransTCN: An Attention-based TCN Framework for Sequential Modeling
Authors: Yuan Chai, Liang He, Yang Zhao, Xueyan Li, Zhenxin Wang
Date: 2021-09-29 00:00:00
Tasks: Language Modelling, Time Series Analysis

Among the sequential modeling issues, the ability to model the long-term dependency remains a significant issue yet to be overcome. Although, recurrent networks extract this information via a recurrent connection, the training step also considers the temporal connection, which reduces the efficiency. However, a temporal connection network (TCN) exploits the benefit of parallelization of convolution and consequently models the sequential information via causal-dilated connection of layers. Moreover, Transformer has exhibited great ability to capture long-term dependency. Thus, in this study, based on the TCN model, the attention blocks in Transformer were introduced to form a model called TransTCN. TransTCN models the sequential information considering attention modules. The model was evaluated across a wide range of the tasks in time series, which are commonly used to the benchmark of TCN and recurrent networks. To the best of our knowledge, TransTCN is the first framework to combine the attention in transformer with TCN to achieve a SOTA performance. The experimental results showed that the perplexity of the word-level prediction on PennTreebank reached only $1.33$ while TCN achieved $87.90$, which is $66$ times of the original TCN. In addition, nearly all loss and perplexity/bpc was improved on other datasets that are commonly used in TCN, except for several datasets wherein our approach maintained performance similar to the original TCN. Furthermore, the training process of TransTCN converges faster than that of TCN.