pwc-archive/papers-with-abstracts huggingface dataset:
A self-explanatory method for the black problem on discrimination part of CNN
Authors: Jinwei Zhao, Qizhou Wang, Wanli Qiu, Guo Xie, Wei Wang, Xinhong Hei, Deyu Meng
Date: 2021-01-01 00:00:00

Convolution neural networks (CNNs) have surpassed human's abilities in some specific tasks. However, they are considered difficult to understand and explain. Recently, the black box problem of CNN, especially concerning the discrimination part, has been studied by different scientific communities. Many methods were proposed for extracting an interpretable model from the discrimination part, which can explain the prediction of the part. However, it is hard for the interpretable models to approximate the discrimination part because of the tradeoff problem between interpretability performance and generalization performance of the discrimination part. We suppose the tradeoff problem is mainly attributed to the fact that the sufficient and necessary condition for the consistent convergence of the both performances is hard to be guaranteed by tradition learning algorithm. So the tradeoff problem could be solved by shrinking the distance between the interpretable model and the discrimination part. This paper firstly introduces a Markov random field model (MRF), namely Deep Cognitive Learning Model (DCLM), which explains the causal relationship between the features(the weight matrixes in the first layer of the discrimination part of CNN can capture) and the output result of the discrimination part of CNN. A greedy algorithm is proposed for initiatively extracting the DCLM from the discrimination part by solving a MAX-SAT problem. And then, a game process between two MAP inferences is implemented for shrinking an interpretation distance which can evaluate how close the discrimination part is to the DCLM. Finally, the proposed self-explanatory approach is evaluated by some contrastive experiments with certain baseline methods on some standard image processing benchmarks. These experiments indicate that the proposed method can improve the interpretability performance of the discrimination part without largely reducing its generalization performance, the generalization performance of the DCLM also can be improved and the discrimination part can be explained by the DCLM in real time during the training process after the interpretation distance converges.
Counterfactual Thinking for Long-tailed Information Extraction
Authors: Guoshun Nan, Jiaqi Zeng, Rui Qiao, Wei Lu
Date: 2021-01-01 00:00:00
Tasks: Causal Inference, counterfactual, Event Detection, Graph Generation, image-classification, Image Classification, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Relation Extraction, Scene Graph Generation, Transfer Learning

Information Extraction (IE) aims to extract structured information from unstructured texts. However, in practice, the long-tailed and imbalanced data may lead to severe bias issues for deep learning models, due to very few training instances available for the tail classes. Existing works are mainly from computer vision society,  leveraging re-balancing, decoupling, transfer learning and causal inference to address this problem on image classification and scene graph generation. However, these approaches may not achieve good performance on textual data, which involves complex language structures that have been proven crucial for the IE tasks. To this end, we propose a novel framework (named Counterfactual-IE) based on language structure and causal reasoning with 
three key ingredients. First, by fusing the syntax information to various structured causal models for mainstream IE tasks including relation extraction (RE), named entity recognition (NER), and event detection (ED), our approach is able to learn the direct effect for classification from an imbalanced dataset. Second, counterfactuals are generated based on an explicit language structure to better calculate the direct effect during the inference stage. Third, we propose a flexible debiasing approach for more robust prediction during the inference stage. Experimental results on three IE tasks across five public datasets show that our model significantly outperforms the state-of-the-art models by a large margin in terms of Mean Recall and Macro F1, achieving a relative 30% improvement in Mean Recall for 14 tail classes on the  ACE2005 dataset. We also discuss some interesting findings based on our observations.
Adaptive and Generative Zero-Shot Learning
Authors: Yu-Ying Chou, Hsuan-Tien Lin, Tyng-Luh Liu
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Generalized Zero-Shot Learning, Zero-Shot Learning

We address the problem of generalized zero-shot learning (GZSL) where the task is to predict the class label of a target image whether its label belongs to the seen or unseen category. Similar to ZSL, the learning setting assumes that all class-level semantic features are given, while only the images of seen classes are available for training. By exploring the correlation between image features and the corresponding semantic features, the main idea of the proposed approach is to enrich the semantic-to-visual (S2V) embeddings via an elegant fusion of adaptive and generative learning. To this end, we extend the semantic features of each class by supplementing image-adaptive attention so that the learned S2V embedding can account for not only inter-class but also intra-class variations. In addition, to break the limit of training with images only from seen classes, we design a generative scheme to simultaneously generate virtual class labels and their visual features by sampling and interpolating over seen counterparts. In inference, a testing image will give rise to two different S2V embeddings, seen and virtual. The former is used to decide whether the underlying label is of the unseen category or otherwise a specific seen class; the latter is to predict an unseen class label. To demonstrate the effectiveness of our method, we report state-of-the-art results on four standard GZSL datasets, including an insightful ablation study of the proposed modules.
Prior Knowledge Representation for Self-Attention Networks
Authors: Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita
Date: 2021-01-01 00:00:00
Tasks: Translation

Self-attention networks (SANs) have shown promising empirical results in various natural language processing tasks. Typically, it gradually learning language knowledge on the whole training dataset in parallel and stacked ways, thereby modeling language representation. In this paper, we propose a simple and general representation method to consider prior knowledge related to language representation from the beginning of training. Also, the proposed method allows SANs to leverage prior knowledge in a universal way compatible with neural networks. Furthermore, we apply it to one prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data, respectively, thereby enhancing the language representation. Experimental results on WMT14 English-to-German and WMT17 Chinese-to-English translation tasks demonstrate the effectiveness and universality of the proposed method over a strong Transformer-based baseline.
Impact-driven Exploration with Contrastive Unsupervised Representations
Authors: Min Jae Song, Dan Kushnir
Date: 2021-01-01 00:00:00
Tasks: Contrastive Learning

Procedurally-generated sparse reward environments pose significant challenges for many RL algorithms. The recently proposed impact-driven exploration method (RIDE) by Raileanu & Rocktäschel (2020), which rewards actions that lead to large changes (measured by $\ell_2$-distance) in the observation embedding, achieves state-of-the-art performance on such procedurally-generated MiniGrid tasks. Yet, the definition of "impact" in RIDE is not conceptually clear because its learned embedding space is not inherently equipped with any similarity measure, let alone $\ell_2$-distance. We resolve this issue in RIDE via contrastive learning. That is, we train the embedding with respect to cosine similarity, where we define two observations to be similar if the agent can reach one observation from the other within a few steps, and define impact in terms of this similarity measure. Experimental results show that our method performs similarly to RIDE on the MiniGrid benchmarks while learning a conceptually clear embedding space equipped with the cosine similarity measure. Our modification of RIDE also provides a new perspective which connects RIDE and episodic curiosity (Savinov et al., 2019), a different exploration method which rewards the agent for visiting states that are unfamiliar to the agent's episodic memory. By incorporating episodic memory into our method, we outperform RIDE on the MiniGrid benchmarks.
Contrastive Self-Supervised Learning of Global-Local Audio-Visual Representations
Authors: Shuang Ma, Zhaoyang Zeng, Daniel McDuff, Yale Song
Date: 2021-01-01 00:00:00
Tasks: Classification, DeepFake Detection, Face Swapping, General Classification, Lip Reading, Self-Supervised Learning, Sound Classification, Sound Source Localization

Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks. However, existing approaches optimize for learning either global representations useful for high-level understanding tasks such as classification, or local representations useful for tasks such as audio-visual source localization and separation. While they produce satisfactory results in their intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose a versatile self-supervised approach to learn audio-visual representations that can generalize to both the tasks which require global semantic information (e.g. classification) and the tasks that require fine-grained spatio-temporal information (e.g. localization). We achieve this by optimizing two cross-modal contrastive objectives that together encourage our model to learn discriminative global-local visual information given the corresponding audio information. To show that our approach learns generalizable video representations, we evaluate it on various downstream scenarios including action/sound classification, lip reading, deepfake detection, and sound source localization.
Online Adversarial Purification based on Self-supervised Learning
Authors: Changhao Shi, Chester Holtz, Gal Mishne
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Adversarial Purification, Representation Learning, Self-Supervised Learning

Deep neural networks are known to be vulnerable to adversarial examples, where a perturbation in the input space leads to an amplified shift in the latent network representation. In this paper, we combine canonical supervised learning with self-supervised representation learning, and present Self-supervised Online Adversarial Purification (SOAP), a novel defense strategy that uses a self-supervised loss to purify adversarial examples at test-time. Our approach leverages the label-independent nature of self-supervised signals and counters the adversarial perturbation with respect to the self-supervised tasks. SOAP yields competitive robust accuracy against state-of-the-art adversarial training and purification methods, with considerably less training complexity. In addition, our approach is robust even when adversaries are given the knowledge of the purification defense strategy. To the best of our knowledge, our paper is the first that generalizes the idea of using self-supervised signals to perform online test-time purification.
Demystifying Learning of Unsupervised Neural Machine Translation
Authors: Guanlin Li, Lemao Liu, Taro Watanabe, Conghui Zhu, Tiejun Zhao
Date: 2021-01-01 00:00:00
Tasks: Machine Translation, Translation

Unsupervised Neural Machine Translation or UNMT has received great attention
in recent years. Though tremendous empirical improvements have been achieved,
there still lacks theory-oriented investigation and thus some fundamental
questions like \textit{why} certain training protocol can work or not under
\textit{what} circumstances have not yet been well understood. This paper
attempts to provide theoretical insights for the above questions. Specifically,
following the methodology of comparative study, we leverage two perspectives,
i) \textit{marginal likelihood maximization} and ii) \textit{mutual information}
from information theory, to understand the different learning effects from the
standard training protocol and its variants. Our detailed analyses reveal
several critical conditions for the successful training of UNMT.
Searching for Convolutions and a More Ambitious NAS
Authors: Nicholas Carl Roberts, Mikhail Khodak, Tri Dao, Liam Li, Nina Balcan, Christopher Re, Ameet Talwalkar
Date: 2021-01-01 00:00:00
Tasks: Neural Architecture Search

An important goal of neural architecture search (NAS) is to automate-away the design of neural networks on new tasks in under-explored domains, thus helping to democratize machine learning. However, current NAS research largely focuses on search spaces consisting of existing operations---such as different types of convolution---that are already known to work well on well-studied problems---often in computer vision. Our work is motivated by the following question: can we enable users to build their own search spaces and discover the right neural operations given data from their specific domain? We make progress towards this broader vision for NAS by introducing a space of operations generalizing the convolution that enables search over a large family of parameterizable linear-time matrix-vector functions. Our flexible construction allows users to design their own search spaces adapted to the nature and shape of their data, to warm-start search methods using convolutions when they are known to perform well, or to discover new operations from scratch when they do not. We evaluate our approach on several novel search spaces over vision and text data, on all of which simple NAS search algorithms can find operations that perform better than baseline layers.
not-so-big-GAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
Authors: Seungwook Han, Akash Srivastava, Cole Lincoln Hurwitz, Prasanna Sattigeri, David Daniel Cox
Date: 2021-01-01 00:00:00
Tasks: Decoder, Image Generation, Image Super-Resolution, Super-Resolution

State-of-the-art models for high-resolution image generation, such as BigGAN and VQVAE-2, require an incredible amount of compute resources and/or time (512 TPU-v3 cores) to train, putting them out of reach for the larger research community. On the other hand, GAN-based image super-resolution models, such as ESRGAN, can not only upscale images to high dimensions, but also are efficient to train. In this paper, we present not-so-big-GAN (nsb-GAN), a simple yet cost-effective two-step training framework for deep generative models (DGMs) of high-dimensional natural images. First, we generate images in low-frequency bands by training a sampler in the wavelet domain. Then, we super-resolve these images from the wavelet domain back to the pixel-space with our novel wavelet super-resolution decoder network. Wavelet-based down-sampling method preserves more structural information than pixel-based methods, leading to significantly better generative quality of the low-resolution sampler (e.g., 64×64). Since the sampler and decoder can be trained in parallel and operate on much lower dimensional spaces than end-to-end models, the training cost is substantially reduced. On ImageNet 512×512, our model achieves a Fréchet Inception Distance (FID) of 10.59 – beating the baseline BigGAN model – at half the compute (256 TPU-v3 cores).
Outlier Robust Optimal Transport
Authors: Debarghya Mukherjee, Aritra Guha, Justin Solomon, Yuekai Sun, Mikhail Yurochkin
Date: 2021-01-01 00:00:00
Tasks: Outlier Detection

Optimal transport (OT) provides a way of measuring distances between distributions that depends on the geometry of the sample space. In light of recent advances in solving the OT problem, OT distances are widely used as loss functions in minimum distance estimation. Despite its prevalence and advantages, however, OT is extremely sensitive to outliers. A single adversarially-picked outlier can increase OT distance arbitrarily. To address this issue, in this work we propose an outlier-robust OT formulation. Our formulation is convex but challenging to scale at a first glance. We proceed by deriving an equivalent formulation based on cost truncation that is easy to incorporate into modern stochastic algorithms for regularized OT. We demonstrate our model applied to mean estimation under the Huber contamination model in simulation as well as outlier detection on real data.
Private Split Inference of Deep Networks
Authors: Mohammad Samragh, Hossein Hosseini, Kambiz Azarian, Joseph Soriaga
Date: 2021-01-01 00:00:00

Splitting network computations between the edge device and the cloud server is a promising approach for enabling low edge-compute and private inference of neural networks. Current methods for providing the privacy train the model to minimize information leakage for a given set of private attributes. In practice, however, the test queries might contain private attributes that are not foreseen during training. 
We propose an alternative solution, in which, instead of obfuscating the information corresponding to a set of attributes, the edge device discards the information irrelevant to the main task. To this end, the edge device runs the model up to a split layer determined based on its computational capacity and then removes the activation content that is in the null space of the next layer of the model before sending it to the server. It can further remove the low-energy components of the remaining signal to improve the privacy at the cost of reducing the accuracy. The experimental results show that our methods provide privacy while maintaining the accuracy and introducing only a small computational overhead.
Uniform-Precision Neural Network Quantization via Neural Channel Expansion
Authors: Seongmin Park, Beomseok Kwon, Kyuyoung Sim, Jieun Lim, Tae-Ho Kim, Jungwook Choi
Date: 2021-01-01 00:00:00
Tasks: Neural Architecture Search, Quantization

Uniform-precision neural network quantization has gained popularity thanks to its simple arithmetic unit densely packed for high computing capability. However, it ignores heterogeneous sensitivity to the impact of quantization across the layers, resulting in sub-optimal inference accuracy. This work proposes a novel approach to adjust the network structure to alleviate the impact of uniform-precision quantization. The proposed neural architecture search selectively expands channels for the quantization sensitive layers while satisfying hardware constraints (e.g., FLOPs). We provide substantial insights and empirical evidence that the proposed search method called neural channel expansion can adapt several popular networks' channels to achieve superior 2-bit quantization accuracy on CIFAR10 and ImageNet. In particular, we demonstrate the best-to-date Top-1/Top-5 accuracy for 2-bit ResNet50 with smaller FLOPs and the parameter size.
Efficient Graph Neural Architecture Search
Authors: Huan Zhao, Lanning Wei, Quanming Yao, Zhiqiang He
Date: 2021-01-01 00:00:00
Tasks: Neural Architecture Search, Transfer Learning

Recently, graph neural networks (GNN) have been demonstrated effective in various graph-based tasks.  To obtain state-of-the-art (SOAT) data-specific GNN architectures, researchers turn to the neural architecture search (NAS) methods. However, it remains to be a challenging problem to conduct efficient architecture search for GNN. In this work, we present a novel framework for Efficient GrAph Neural architecture search (EGAN). By designing a novel and expressive search space, an efficient one-shot NAS method based on stochastic relaxation and natural gradient is proposed. Further, to enable architecture search in large graphs, a transfer learning paradigm is designed. Extensive experiments, including node-level and graph-level tasks, are conducted. The results show that the proposed EGAN can obtain SOTA data-specific architectures, and reduce the search cost by two orders of magnitude compared to existing NAS baselines.
Speeding up Deep Learning Training by Sharing Weights and Then Unsharing
Authors: Shuo Yang, Le Hou, Xiaodan Song, Qiang Liu, Denny Zhou
Date: 2021-01-01 00:00:00

It has been widely observed that increasing  deep learning model sizes often leads to significant performance improvements  on a variety of natural language processing and computer vision tasks. In the meantime, however, computational costs and training time would dramatically increase when models get larger. In this paper, we propose a simple approach to speed up  training for a particular kind of deep networks which contain repeated structures,  such as the transformer module. In our method, we first train such a deep network with the weights shared across all the repeated layers.  Once an unsharing condition is triggered, we stop weight sharing  and continue training until convergence. Empirical results show that our method is able to reduce the training time of BERT  by 50%. We also conduct a preliminary theoretic analysis which motivates our approach.
Robust Multi-Agent Reinforcement Learning Driven by Correlated Equilibrium
Authors: Yizheng Hu, Kun Shao, Dong Li, Jianye Hao, Wulong Liu, Yaodong Yang, Jun Wang, Zhanxing Zhu
Date: 2021-01-01 00:00:00
Tasks: Adversarial Robustness, Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), SMAC, SMAC+

In this paper we deal with robust cooperative multi-agent reinforcement learning (CMARL). While CMARL has many potential applications, only a trained policy that is robust enough can be confidently deployed in real world. Existing works on robust MARL mainly apply vanilla adversarial training in centralized training and decentralized execution paradigm. We, however, find that if a CMARL environment contains an adversarial agent, the performance of decentralized equilibrium might perform significantly poor for achieving such adversarial robustness. To tackle this issue, we suggest that when execution the non-adversarial agents must jointly make the decision to improve the robustness, therefore solving correlated equilibrium instead. We theoretically demonstrate the superiority of correlated equilibrium over the decentralized one in adversarial MARL settings. Therefore, to achieve robust CMARL, we introduce novel strategies to encourage agents to learn correlated equilibrium while maximally preserving the convenience of the decentralized execution. The global variables with mutual information are proposed to help agents learn robust policies with MARL algorithms. The experimental results show that our method can dramatically boost performance on the SMAC environments.