pwc-archive/papers-with-abstracts huggingface dataset:
The Unreasonable Effectiveness of the Class-reversed Sampling in Tail Sample Memorization
Authors: Benyi Hu, Chi Zhang, Yuehu Liu, Le Wang, Li Liu
Date: 2021-01-01 00:00:00
Tasks: Memorization

Long-tailed visual class recognition poses significant challenges to traditional machine learning and emerging deep networks due to its inherent class imbalance. A common belief is that tail classes with few samples cannot exhibit enough regularity for pattern extraction. What makes things worse, the limited cardinality may lead to low exposure of tail classes in the training stage. Resampling methods, especially those who naively enlarge the exposure frequency, eventually fail with head classes under-represented and tail classes overfitted.
Arguing that long-tailed learning involves a trade-off between head class pattern extraction and tail class memorizing, we propose a simple yet effective combinational sampling method motivated by the recent success of a series works on the memorization-generalization mechanism. We empirically demonstrate that a naive switching of instance-balanced sampler to class-reversed sampler for the last several epochs of training can help neural networks better memorize tail classes of low-regularity. In our experiments, the proposed method can reach the state-of-the-art performance more efficiently than current methods, on several datasets. Further experiments also validate the superior performance of the proposed sampling strategy, implying that the long-tailed learning trade-off could be effectively tackled only in the memorization stage with a small learning rate and over-exposure of tail samples.
Weakly Supervised Scene Graph Grounding
Authors: Yizhou Zhang, Zhaoheng Zheng, Yan Liu
Date: 2021-01-01 00:00:00

 Recent researches have achieved substantial advances in learning structured representations from images. However, current methods rely heavily on the annotated mapping between the nodes of scene graphs and object bounding boxes inside images. Here, we explore the problem of learning the mapping between scene graph nodes and visual objects under weak supervision. Our proposed method learns a metric among visual objects and scene graph nodes by incorporating information from both object features and relational features. Extensive experiments on Visual Genome (VG) and Visual Relation Detection (VRD) datasets verify that our model post an improvement on scene graph grounding task over current state-of-the-art approaches. Further experiments on scene graph parsing task verify the grounding found by our model can reinforce the performance of the existing method.
Deep Learning Solution of the Eigenvalue Problem for Differential Operators
Authors: Ido Ben-Shaul, Leah Bar, Nir Sochen
Date: 2021-01-01 00:00:00
Tasks: Deep Learning

Solving the eigenvalue problem for differential operators is a common problem in many scientific fields. Classical numerical methods rely on intricate domain discretization, and yield non-analytic or non-smooth approximations. We introduce a novel Neural Network (NN)-based solver for the eigenvalue problem of differential self-adjoint operators where the eigenpairs are learned in an unsupervised end-to-end fashion. We propose three different training procedures, for solving increasingly challenging tasks towards the general eigenvalue problem. The proposed solver is able to find the M smallest eigenpairs for a general differential operator.

We demonstrate the method on the Laplacian eigenproblem which is of particular interest in image processing, computer vision, shape analysis among many other applications. 
Unlike other numerical methods such as finite differences, the partial derivatives of the network approximation of the eigenfunction can be analytically calculated to any order. Therefore, the proposed framework enables the solution of higher order operators and on free shape domain or even on a manifold. Non-linear operators can be investigated by this approach as well.
Contextual Image Parsing via Panoptic Segment Sorting
Authors: Jyh-Jing Hwang, Tsung-Wei Ke, Stella Yu
Date: 2021-01-01 00:00:00
Tasks: Metric Learning, Panoptic Segmentation, Segmentation

Visual context is versatile and hard to describe or label precisely.  We aim to leverage the densely labeled task, image parsing, a.k.a panoptic segmentation, to learn a model that encodes and discovers object-centric context.  Most existing approaches based on deep learning tackle image parsing via fusion of pixel-wise classification and instance masks from two sub-networks.  Such approaches isolate things from stuff and fuse the semantic and instance masks in the later stage.   To encode object-centric context inherently, we propose a metric learning framework, Panoptic Segment Sorting, that is directly trained with stuff and things jointly.  Our key insight is to make the panoptic embeddings separate every instance so that the model automatically learns to leverage visual context as many instances across different images appear similar.  We show that the context of our model's retrieved instances is more consistent relatively by 13.7%, further demonstrating its ability to discover novel context unsupervisedly.  Our overall framework also achieves competitive performance across standard panoptic segmentation metrics amongst the state-of-the-art methods on two large datasets, Cityscapes and PASCAL VOC.  These promising results suggest that pixel-wise embeddings can not only inject new understanding into panoptic segmentation but potentially serve for other tasks such as modeling instance relationships.
Optimizing Information Bottleneck in Reinforcement Learning: A Stein Variational Approach
Authors: Pei Yingjun, Hou Xinwen, Li Jian, Lei Wang
Date: 2021-01-01 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The information bottleneck (IB) principle is an elegant and useful learning framework for extracting relevant information that an input feature contains about the target. The principle has been widely used in supervised and unsupervised learning. In this paper, we investigate the effectiveness of the IB framework in reinforcement learning (RL). We first derive the objective based on IB in reinforcement learning, then we analytically derive the optimal conditional distribution of the optimization problem. Following the variational information bottleneck (VIB), we provide a variational lower bound using a prior distribution. Unlike VIB, we propose to utilize the amortized Stein variational gradient method to optimize the lower bound. We incorporate this framework in two popular RL algorithms: the advantageous actor critic algorithm (A2C) and the proximal policy optimization algorithm (PPO). Our experimental results show that our framework can improve the sample efficiency of vanilla A2C and PPO. We also show that our method achieves better performance than VIB and mutual information neural estimation (MINE), two other popular approaches to optimize the information bottleneck framework in supervised learning.
Counterfactual Self-Training
Authors: Ruijiang Gao, Max Biggs, Wei Sun, Ligong Han
Date: 2021-01-01 00:00:00
Tasks: counterfactual, Domain Adaptation, Marketing

Unlike traditional supervised learning, in many settings only partial feedback is available. We may only observe outcomes for the chosen actions, but not the counterfactual outcomes associated with other  alternatives. Such settings encompass a wide variety of applications including pricing, online marketing and precision medicine. A key challenge is that observational data are influenced by historical policies deployed in the system, yielding a biased data distribution. We approach this task as a domain adaptation problem and propose a self-training algorithm which imputes outcomes for the unseen actions in the observational data to simulate a randomized trial. We offer a theoretical motivation for this approach by providing an upper bound on the generalization error defined on a randomized trial under the self-training objective. We empirically demonstrate the effectiveness of the proposed algorithms on both synthetic and real datasets.
EXPLORING VULNERABILITIES OF BERT-BASED APIS
Authors: Xuanli He, Lingjuan Lyu, Lichao Sun, Xiaojun Chang, Jun Zhao
Date: 2021-01-01 00:00:00
Tasks: Attribute, Inference Attack, Model extraction, text-classification, Text Classification, Text Generation

Natural language processing (NLP) tasks, ranging from text classification to text generation, have been revolutionised by pretrained BERT models. This allows corporations to easily build powerful APIs by encapsulating fine-tuned BERT models. These BERT-based APIs are often designed to not only provide reliable service but also protect intellectual properties or privacy-sensitive information of the training data. However, a series of privacy and robustness issues may still exist when a fine-tuned BERT model is deployed as a service.  In this work, we first present an effective model extraction attack, where the adversary can steal a BERT-based API (the victim model), without knowing the victim model’s architecture, parameters or the training data distribution. We then demonstrate how the extracted model can be exploited to develop effective attribute inference attack to expose sensitive information of the training data. We also show that the extracted model can lead to highly transferable adversarial attacks against the original model. Extensive experiments on multiple benchmark datasets under various settings validate the potential privacy issues and adversarial vulnerabilities of BERT-based APIs.
Learning Robust Models by Countering Spurious Correlations
Authors: Haohan Wang, Zeyi Huang, Eric Xing
Date: 2021-01-01 00:00:00
Tasks: Domain Adaptation

Machine learning has demonstrated remarkable prediction accuracy over i.i.d data, but the accuracy often drops when tested with data from another distribution. One reason behind this accuracy drop is the reliance of models on the features that are only associated with the label in the training distribution, but not the test distribution. This problem is usually known as spurious correlation, confounding factors, or dataset bias. In this paper, we formally study the generalization error bound for this setup with the knowledge of how the spurious features are associated with the label. We also compare our analysis to the widely-accepted domain adaptation error bound and show that our bound can be tighter, with more assumptions that we consider realistic. Further, our analysis naturally offers a set of solutions for this problem, linked to established solutions in various topics about robustness in general, and these solutions all require some understandings of how the spurious features are associated with the label. Finally, we also briefly discuss a method that does not require such an understanding.
On the Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations
Authors: Haohan Wang, Zeyi Huang, Xindi Wu, Eric Xing
Date: 2021-01-01 00:00:00
Tasks: Data Augmentation

Data augmentation is one of the most popular techniques for improving the robustness of neural networks. In addition to directly training the model with original samples and augmented samples, a torrent of methods regularizing the distance between embeddings/representations of the original samples and their augmented counterparts have been introduced. In this paper, we explore these various regularization choices, seeking to provide a general understanding of how we should regularize the embeddings. Our analysis suggests how the ideal choices of regularization correspond to various assumptions. With an invariance test, we show that regularization is important if the model is to be used in a broader context than the in-lab setting because non-regularized approaches are limited in learning the concept of invariance, despite equally high accuracy. Finally, we also show that the generic approach we identified (squared $\ell_2$ norm regularized augmentation) performs better than several recent methods, which are each specially designed for one task and significantly more complicated than ours, over three different tasks.
Identifying Coarse-grained Independent Causal Mechanisms with Self-supervision
Authors: Xiaoyang Wang, Klara Nahrstedt, Oluwasanmi O Koyejo
Date: 2021-01-01 00:00:00
Proceeding: 1st Conference on Causal Learning and Reasoning 2022 2

Current approaches for learning disentangled representations assume that independent latent variables generate the data through a single data generation process. In contrast, this manuscript considers independent causal mechanisms (ICM), which, unlike disentangled representations, directly model multiple data generation processes in a coarse granularity. In this work, we aim to learn a model that isolates each mechanism and approximates the ground-truth ICM from observational data. We outline sufficient conditions under which the ICM can be learned and isolated using a single self-supervised generative model with a mixture prior, simplifying previous methods. Moreover, we implement a generative model with an identifiable structural latent space by combining the ICM with a shared latent space. We compare this ICM approach to disentangled representations on various downstream tasks, showing that the ICM is more robust to intervention, co-variant shift, and noise due to the isolation between the data generation processes.
BDS-GCN: Efficient Full-Graph Training of Graph Convolutional Nets with Partition-Parallelism and Boundary Sampling
Authors: Cheng Wan, Youjie Li, Nam Sung Kim, Yingyan Lin
Date: 2021-01-01 00:00:00

Graph Convolutional Networks (GCNs) have emerged as the state-of-the-art model for graph-based learning tasks. However, it is still challenging to train GCNs at scale, limiting their applications to real-world large graphs and hindering the exploration of deeper and more sophisticated GCN architectures. While it can be natural to leverage graph partition and distributed training for tackling this challenge, this direction has only been slightly touched on previously due to the unique challenge posed by the GCN structures, especially the excessive amount of boundary nodes in each partitioned subgraph, which can easily explode the required memory and communications for distributed training of GCNs. To this end, we propose BDS-GCN, a method that adopts unbiased boundary sampling strategy to enable efficient and scalable distributed GCN training while maintaining the full-graph accuracy. Empirical evaluations and ablation studies validate the effectiveness of the proposed BDS-GCN, e.g., boosting the throughput by up-to 500% and reducing the memory usage by up-to 58% for distributed GCN training, while achieving the same accuracy, as compared with the state-of-the-art methods. We believe our BDS-GCN would open up a new paradigm for enabling GCN training at scale. All code will be released publicly upon acceptance.
An Unsupervised Deep Learning Approach for Real-World Image Denoising
Authors: Dihan Zheng, Sia Huat Tan, Xiaowen Zhang, Zuoqiang Shi, Kaisheng Ma, Chenglong Bao
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Decoder, Deep Learning, Denoising, Image Denoising

Designing an unsupervised image denoising approach in practical applications is a challenging task due to the complicated data acquisition process. In the real-world case, the noise distribution is so complex that the simplified additive white Gaussian (AWGN) assumption rarely holds, which significantly deteriorates the Gaussian denoisers' performance. To address this problem, we apply a deep neural network that maps the noisy image into a latent space in which the AWGN assumption holds, and thus any existing Gaussian denoiser is applicable. More specifically, the proposed neural network consists of the encoder-decoder structure and approximates the likelihood term in the Bayesian framework. Together with a Gaussian denoiser, the neural network can be trained with the input image itself and does not require any pre-training in other datasets. Extensive experiments on real-world noisy image datasets have shown that the combination of neural networks and Gaussian denoisers improves the performance of the original Gaussian denoisers by a large margin. In particular, the neural network+BM3D method significantly outperforms other unsupervised denoising approaches and is competitive with supervised networks such as DnCNN, FFDNet, and CBDNet.
PhraseTransformer: Self-Attention using Local Context for Semantic Parsing
Authors: Phuong Minh Nguyen, Vu Tran, Minh Le Nguyen
Date: 2021-01-01 00:00:00
Tasks: Machine Translation, Semantic Parsing, Sentence, Translation

Semantic parsing is a challenging task whose purpose is to convert a natural language utterance to machine-understandable information representation. Recently, solutions using Neural Machine Translation have achieved many promising results, especially Transformer because of the ability to learn long-range word dependencies. However, the one drawback of adapting the original Transformer to the semantic parsing is the lack of detail in expressing the information of sentences. Therefore, this work proposes a PhraseTransformer architecture that is capable of a more detailed meaning representation by learning the phrase dependencies in the sentence. The main idea is to incorporate Long Short-Term Memory (LSTM) into the Self-Attention mechanism of the original Transformer to capture more local context of phrases. Experimental results show that the proposed model captures the detailed meaning better than Transformer, raises local context awareness and achieves strong competitive performance on Geo, MSParS datasets, and leads to new state-of-the-art (SOTA) performance on Atis dataset.
A Geometric Analysis of Deep Generative Image Models and Its Applications
Authors: Binxu Wang, Carlos R Ponce
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Image-Variation

Generative adversarial networks have emerged as a powerful unsupervised method to model the statistical patterns of real-world data sets, such as natural images. These networks are trained to map random inputs in their latent space to new samples representative of the learned data. However, the structure of the latent space is hard to intuit due to its high dimensionality and the non-linearity of the generator, which limits the usefulness of the models. Understanding the latent space requires a way to identify input codes for existing real-world images (inversion), and a way to identify directions with known image transformations (interpretability). Here, we use a geometric framework to address both issues simultaneously. We develop an architecture-agnostic method to compute the Riemannian metric of the image manifold created by GANs. The eigen-decomposition of the metric isolates axes that account for different levels of image variability. An empirical analysis of several pretrained GANs shows that image variation around each position is concentrated along surprisingly few major axes (the space is highly anisotropic) and the directions that create this large variation are similar at different positions in the space (the space is homogeneous). We show that many of the top eigenvectors correspond to interpretable transforms in the image space, with a substantial part of eigenspace corresponding to minor transforms which could be compressed out. This geometric understanding unifies previous results of GAN inversion and interpretation. We show that the use of this metric allows for more efficient optimization in the latent space (e.g. GAN inversion) and facilitates unsupervised discovery of interpretable axes. Our results show that defining the geometry of the GAN image manifold can serve as a general framework for understanding GANs.
Natural World Distribution via Adaptive Confusion Energy Regularization
Authors: Yen-Chi Hsu, Cheng-Yao Hong, Wan-Cyuan Fan, Ding-Jie Chen, Ming-Sui Lee, Davi Geiger, Tyng-Luh Liu
Date: 2021-01-01 00:00:00
Tasks: Fine-Grained Image Classification

We introduce a novel and adaptive batch-wise regularization based on the proposed Batch Confusion Norm (BCN) to flexibly address the natural world distribution which usually involves fine-grained and long-tailed properties at the same time. The Fine-Grained Visual Classification (FGVC) problem is notably characterized by two intriguing properties, significant inter-class similarity and intra-class variations, which cause learning an effective FGVC classifier a challenging task. Existing techniques attempt to capture the discriminative parts by their modified attention mechanism. The long-tailed distribution of visual classification poses a great challenge for handling the class imbalance problem. Most of existing solutions usually focus on the class-balancing strategies, classifier normalization, or alleviating the negative gradient of tailed categories. Depart from the conventional approaches, we propose to tackle both problems simultaneously with the adaptive confusion concept. When inter-class similarity prevails in a batch, the BCN term can alleviate possible overfitting due to exploring image features of fine details. On the other hand, when inter-class similarity is not an issue, the class predictions from different samples would unavoidably yield a substantial BCN loss, and prompt the network learning to further reduce the cross-entropy loss. More importantly, extending the existing confusion energy-based framework to account for long-tailed scenario, BCN can learn to exert proper distribution of confusion strength over tailed and head categories to improve classification performance. While the resulting FGVC model by the BCN technique is effective, the performance can be consistently boosted by incorporating extra attention mechanism. In our experiments, we have obtained state-of-the-art results on several benchmark FGVC datasets, and also demonstrated that our approach is competitive on the popular natural world distribution dataset, iNaturalist2018.
Oblivious Sketching-based Central Path Method for Solving Linear Programming Problems
Authors: Zhao Song, Zheng Yu
Date: 2021-01-01 00:00:00

In this work, we propose a sketching-based central path method for solving linear programmings, whose running time matches the state of art results [Cohen, Lee, Song STOC 19; Lee, Song, Zhang COLT 19]. Our method opens up the iterations of the central path method and deploys an "iterate and sketch" approach towards the problem by introducing a new coordinate-wise embedding technique, which may be of independent interest. Compare to previous methods, the work [Cohen, Lee, Song STOC 19] enjoys feasibility while being non-oblivious, and [Lee, Song, Zhang COLT 19] is oblivious but infeasible, and relies on $\mathit{dense}$ sketching matrices such as subsampled randomized Hadamard/Fourier transform matrices. Our method enjoys the benefits of being both oblivious and feasible, and can use $\mathit{sparse}$ sketching matrix [Nelson, Nguyen FOCS 13] to speed up the online matrix-vector multiplication. Our framework for solving LP naturally generalizes to a broader class of convex optimization problems including empirical risk minimization.