pwc-archive/papers-with-abstracts huggingface dataset:
Representational correlates of hierarchical phrase structure in deep language models
Authors: Matteo Alleman, Jonathan Mamou, Miguel A Del Rio, Hanlin Tang, Yoon Kim, SueYeon Chung
Date: 2021-01-01 00:00:00
Tasks: Sensitivity, Sentence

While contextual representations from Transformer-based architectures have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings. In particular, it is not entirely clear what aspects of sentence-level syntax are captured by these representations, nor how (if at all) they are built along the stacked layers of the network. In this paper, we aim to address such questions with a general class of input perturbation-based analyses of representations from Transformer networks pretrained on self-supervised objectives. Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of Transformer representations to several kinds of structure in sentences. Each probe involves swapping words in a sentence and comparing the representations from perturbed sentences against the original. We experiment with three different perturbations: (1) random permutations of n-grams of varying width, to test the scale at which a representation is sensitive to word position; (2) swapping of two spans which do or do not form a syntactic phrase, to test sensitivity to global phrase structure; and (3) swapping of two adjacent words which do or do not break apart a syntactic phrase, to test sensitivity to local phrase structure. We also connect our probe results to the Transformer architecture by relating the attention mechanism to syntactic distance between two words. Results from the three probes collectively suggest that Transformers build sensitivity to larger parts of the sentence along their layers, and that hierarchical phrase structure plays a role in this process. In particular, sensitivity to local phrase structure increases along deeper layers. Based on our analysis of attention, we show that this is at least partly explained by generally larger attention weights between syntactically distant words.
ProGAE: A Geometric Autoencoder-based Generative Model for Disentangling Protein Dynamics
Authors: Norman Joseph Tatro, Payel Das, Pin-Yu Chen, Vijil Chenthamarakshan, Rongjie Lai
Date: 2021-01-01 00:00:00
Tasks: Property Prediction

Understanding molecular dynamics is critical, especially for large flexible molecules like proteins. Protein function, as well as modulations thereof due to ligand binding or changes in environment, are intimately connected with these dynamics. This work focuses on learning a generative neural network on molecular simulation trajectories to characterize the distinct dynamics of a protein bound to various drug molecules. Specifically, we use a  geometric autoencoder framework to learn separate latent space encodings of  the intrinsic and extrinsic geometries of the system. For this purpose, the proposed Protein Geometric AutoEncoder (ProGAE) model is trained on  the length of the alpha-carbon pseudobonds and the  orientation of the backbone bonds of the protein. Using ProGAE latent embeddings, we reconstruct protein simulation trajectories at or near the experimental resolution. Empowered by the disentangled latent space learning, the extrinsic latent embedding is successfully used for classification or property prediction of  different drugs bound to a specific protein. Additionally, ProGAE is able to be transferred to a trajectory of a different state of the same protein or to a completely different protein, where only the dense layer decoding from the latent representation needs to be retrained. Results show that our geometric learning-based method enjoys both accuracy and efficiency for generating complex molecular dynamics, charting the path toward scalable and improved approaches for analyzing and enhancing molecular simulations.
Feature Integration and Group Transformers for Action Proposal Generation
Authors: He-Yen Hsieh, Ding-Jie Chen, Tung-Ying Lee, Tyng-Luh Liu
Date: 2021-01-01 00:00:00
Tasks: Temporal Action Proposal Generation

The task of temporal action proposal generation (TAPG) aims to provide high-quality video segments, i.e., proposals that potentially contain action events. The performance of tackling the TAPG task heavily depends on two key issues, feature representation and scoring mechanism. To simultaneously take account of both aspects, we introduce an attention-based model, termed as FITS, to address the issues for retrieving high-quality proposals. We first propose a novel Feature-Integration (FI) module to seamlessly fuse two-stream features concerning their interaction to yield a robust video segment representation. We then design a group of Transformer-driven Scorers (TS) to gain the temporal contextual supports over the representations for estimating the starting or ending boundary of an action event. Unlike most previous work to estimate action boundaries without considering the long-range temporal neighborhood, the proposed action-boundary co-estimation mechanism in TS leverages the bi-directional contextual supports for such boundary estimation, which shows the advantage of removing several false-positive boundary predictions. We conduct experiments on two challenging datasets, ActivityNet-1.3 and THUMOS-14. The experimental results demonstrate that the proposed FITS model consistently outperforms state-of-the-art TAPG methods.
Diversity Augmented Conditional Generative Adversarial Network for Enhanced Multimodal Image-to-Image Translation
Authors: Yunlong MENG, Lin Xu
Date: 2021-01-01 00:00:00
Tasks: Diversity, Generative Adversarial Network, Image-to-Image Translation, Translation

Conditional generative adversarial networks (cGANs) play an important role in multimodal image-to-image translation. We propose Diversity Augmented conditional Generative Adversarial Network (DivAugGAN), a highly effective solution to further resolve the mode collapse problem and enhance the diversity for the generated images. DivAugGAN functions as a regularizer to maximize the distinction of the generating samples when different noise vectors are injected. We also exert extra constraint on the generator to ensure the relative variation consistency in the translation process. This guarantees that the changing scale of the generated images in the image space is coherent to the difference of the injected noise vectors in the latent space. It also reduces the chances to bring about unexpected mode override and mode fusion issues. Experimental results on both two-domain and multi-domain multimodal image-to-image translation tasks demonstrate its effectiveness. DivAugGAN leads to consistent diversity augmentations and visual quality improvements for the developed models. We also achieves state-of-the-art performances on multiple datasets in terms of widely used quantitative evaluation metrics. DivAugGAN can be easily integrated into any objectives in conditional generative models as a regularizer for diversity augmentations and quality enhancements without any additional computation overheads compromise The source code and pre-trained models of our method is available at https://github.com/anomymous-gan/DivAugGAN.
Illuminating Dark Knowledge via Random Matrix Ensembles
Authors: Anthony Ndirango
Date: 2021-01-01 00:00:00
Tasks: Learning Theory

It is all but certain that machine learning models based on deep neural networks will soon feature ubiquitously in a wide variety of critical products and services that people rely on. This should be a major cause of concern given that we still lack a rigorous understanding of the failure modes of these systems, and can hardly make guarantees about the conditions under which the models are expected to work. In particular, we would like to understand how these models manage to generalize so well, even when seemingly overparametrized, effectively evading many of the intuitions expected from statistical learning theory. We argue that Distillation (Caruana et al., 2006, Hinton et al., 2014)  provides us with a rich playground for understanding what enables generalization in a concrete setting.  We carry out a precise high-dimensional analysis of generalization under distillation in a real world setting, eschewing ad hoc assumptions, and instead consider models actually encountered in the wild.
DiP Benchmark Tests: Evaluation Benchmarks for Discourse Phenomena in MT
Authors: Prathyusha Jwalapuram, Barbara Rychalska, Shafiq Joty, Dominika Basaj
Date: 2021-01-01 00:00:00
Tasks: Machine Translation, Translation

Despite increasing instances of machine translation (MT) systems including extrasentential context information, the evidence for translation quality improvement is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark testsets that aim to track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. We also introduce evaluation methods for these tasks, and evaluate several competitive baseline MT systems on the curated datasets. Surprisingly, we find that existing context-aware models do not improve discourse-related translations consistently across languages and phenomena. Our evaluation benchmark is available as a leaderboard at <dipbenchmark1.github.io>.
Cut out the annotator, keep the cutout: better segmentation with weak supervision
Authors: Sarah Hooper, Michael Wornow, Ying Hang Seah, Peter Kellman, Hui Xue, Frederic Sala, Curtis Langlotz, Christopher Re
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Data Augmentation, Few-Shot Learning, Image Segmentation, Segmentation, Semantic Segmentation, Weakly-supervised Learning

 Constructing large labeled datasets for training segmentation models is an expensive and labor-intensive process. This is a common challenge in machine learning, addressed by methods that require few or no labeled data points such as few-shot learning (FSL) and weakly-supervised learning (WS). Such techniques, however, have limitations when applied to image segmentation---it is difficult to inject multiple forms of knowledge into FSL and other limited label learning techniques, while WS models struggle to fully exploit rich image information. We propose a framework that fuses limited label learning and weak supervision for segmentation tasks, enabling users to train high-performing segmentation CNNs with very few hand-labeled training points. We use CNNs trained with limited labels as weak sources, requiring a very small set of reference labeled images, and introduce a new WS model that focuses on key areas---areas with contention among noisy labels---of the image to fuse these weak sources. Empirically, we evaluate our proposed approach over seven well-motivated segmentation tasks. We show that our methods can achieve within 2 Dice points compared to fully supervised networks while only requiring five hand-labeled training points. Compared to existing limited label supervision methods, including solo few-shot and data augmentation approaches, our approach improves performance by a mean 5 Dice points of the next best method. Finally, we explore the tradeoffs of these various supervision methods, including how well each approach generalizes to new test sets and how well each approach leverages additional unlabeled training data.
Meta-Learning in Reproducing Kernel Hilbert Space
Authors: Yufan Zhou, Zhenyi Wang, Jiayi Xian, Changyou Chen, Jinhui Xu
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Meta-Learning

  Model Agnostic Meta Learning (MAML) has emerged as a standard framework for meta learning, where a meta-model is learned with the ability of fast adapting to new tasks. However, as a double-looped optimization problem, MAML needs to differentiate through the whole inner-loop optimization path for every outer-loop training step, which may lead to both computational inefficiency and sub-optimal solutions. In this paper, we generalize MAML to allow meta learning to be defined in function spaces, and propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta model's Neural Tangent Kernel (NTK). Within this paradigm, we introduce two meta learning algorithms in RKHS, which no longer need an explicit inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory. Extensive experimental studies demonstrate the superiority of our paradigm in both efficiency and quality of solutions compared to related meta-learning algorithms. Another interesting feature of our proposed methods is that they are much more robust to adversarial attacks and out-of-distribution adaptation than existing approaches, as demonstrated by our experiments.
One Vertex Attack on Graph Neural Networks-based Spatiotemporal Forecasting
Authors: Fuqiang Liu, Luis Miranda Moreno, Lijun Sun
Date: 2021-01-01 00:00:00
Tasks: Graph Classification

Spatiotemporal forecasting plays an essential role in intelligent transportation systems (ITS) and numerous applications, such as route planning, navigation, and automatic driving. Deep Spatiotemporal Graph Neural Networks, which capture both spatial and temporal patterns, have achieved great success in traffic forecasting applications. Though Deep Neural Networks (DNNs) have been proven to be vulnerable to carefully designed perturbations in multiple domains like objection classification and graph classification, these adversarial works cannot be directly applied to spatiotemporal GNNs because of their causality and spatiotemporal mechanism. There is still a lack of studies on the vulnerability and robustness of spatiotemporal GNNs. Particularly, if spatiotemporal GNNs are vulnerable in real-world traffic applications, a hacker can easily cause serious traffic congestion and even a city-scale breakdown. To fill this gap, we design One Vertex Attack to break deep spatiotemporal GNNs by attacking a single one vertex. To achieve this, we apply the genetic algorithm with a universal attack method as the evaluation function to locate the weakest vertex; then perturbations are generated by solving an optimization problem with the inverse estimation. Empirical studies prove that perturbations in one vertex can be diffused into most of the graph when spatiotemporal GNNs are under One Vertex Attack.
Maximum Categorical Cross Entropy (MCCE): A noise-robust alternative loss function to mitigate racial bias in Convolutional Neural Networks (CNNs) by reducing overfitting
Authors: Nidhi Gowdra, Roopak Sinha, Stephen MacDonell, WeiQi Yan
Date: 2021-01-01 00:00:00
Tasks: Benchmarking, General Classification, image-classification, Image Classification, Multi-class Classification

Categorical Cross Entropy (CCE) is the most commonly used loss function in deep neural networks such as Convolutional Neural Networks (CNNs) for multi-class classification problems. In spite of the fact that CCE is highly susceptible to noise; CNN models trained without accounting for the unique noise characteristics of the input data, or noise introduced during model training, invariably suffer from overfitting affecting model generalizability. The lack of generalizability becomes especially apparent in the context of ethnicity/racial image classification problems encountered in the domain of computer vision. One such problem is the unintended discriminatory racial bias that CNN models trained using CCE fail to adequately address. In other words, CNN models trained using CCE offer a skewed representation of classification performance favoring lighter skin tones.

In this paper, we propose and empirically validate a novel noise-robust extension to the existing CCE loss function called Maximum Categorical Cross-Entropy (MCCE), which utilizes CCE loss and a novel reconstruction loss, calculated using the Maximum Entropy (ME) measures of the convolutional kernel weights and input training dataset. We compare the use of MCCE with CCE-trained models on two benchmarking datasets, colorFERET and UTKFace, using a Residual Network (ResNet) CNN architecture. MCCE-trained models reduce overfitting by 5.85% and 4.3% on colorFERET and UTKFace datasets respectively. In cross-validation testing, MCCE-trained models outperform CCE-trained models by 8.8% and 25.16% on the colorFERET and UTKFace datasets respectively. MCCE addresses and mitigates the persistent problem of inadvertent racial bias for facial recognition problems in the domain of computer vision.
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms
Authors: Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, Yi Wu
Date: 2021-01-01 00:00:00
Tasks: Benchmarking, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Starcraft

We benchmark commonly used multi-agent deep reinforcement learning (MARL) algorithms on a variety of cooperative multi-agent games. While there has been significant innovation in MARL algorithms, algorithms tend to be tested and tuned on a single domain and their average performance across multiple domains is less characterized. Furthermore, since the hyperparameters of the algorithms are carefully tuned to the task of interest, it is unclear whether hyperparameters can easily be found that allow the algorithm to be repurposed for other cooperative tasks with different reward structure and environment dynamics. To investigate the consistency of the performance of MARL algorithms, we build an open-source library of multi-agent algorithms including DDPG/TD3/SAC with centralized Q functions, PPO with centralized value functions, and QMix and test them across a range of tasks that vary in coordination difficulty and agent number. The domains include the particle-world environments, starcraft micromanagement challenges, the Hanabi challenge, and the hide-and-seek environments. Finally, we investigate the ease of hyper-parameter tuning for each of the algorithms by tuning hyper-parameters in one environment per domain and re-using them in the other environments within the domain.
Systematic generalisation with group invariant predictions
Authors: Faruk Ahmed, Yoshua Bengio, Harm van Seijen, Aaron Courville
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

We consider situations where the presence of dominant simpler correlations with the target variable in a training set can cause an SGD-trained neural network to be less reliant on more persistently-correlating complex features. When the non-persistent, simpler correlations correspond to non-semantic background factors, a neural network trained on this data can exhibit dramatic failure upon encountering systematic distributional shift, where the correlating background features are recombined with different objects. We perform an empirical study showing that group invariance methods across inferred partitionings of the training set can lead to significant improvements at such test-time situations. We suggest a new invariance penalty, showing with experiments on three synthetic datasets that it can perform better than alternatives. We find that even without assuming access to any systematic-shift validation sets, one can still find improvements over an ERM-trained reference model.
Counterfactual Fairness through Data Preprocessing
Authors: Haoyu Chen, Wenbin Lu, Rui Song, Pulak Ghosh
Date: 2021-01-01 00:00:00
Tasks: BIG-bench Machine Learning, counterfactual, Decision Making, Fairness

Machine learning has become more important in real-life decision-making but people are concerned about the ethical problems it may bring when used improperly. Recent work brings the discussion of machine learning fairness into the causal framework and elaborates on the concept of Counterfactual Fairness. In this paper, we develop the Fair Learning through dAta Preprocessing (FLAP) algorithm to learn counterfactually fair decisions from biased training data and formalize the conditions where different data preprocessing procedures should be used to guarantee counterfactual fairness. We also show that Counterfactual Fairness is equivalent to the conditional independence of the decisions and the sensitive attributes given the processed non-sensitive attributes, which enables us to detect discrimination in the original decision using the processed data. The performance of our algorithm is illustrated using simulated data and a real-world application in loan assessment.
Achieving Explainability in a Visual Hard Attention Model through Content Prediction
Authors: Samrudhdhi Bharatkumar Rangrej, James J. Clark
Date: 2021-01-01 00:00:00
Tasks: Hard Attention, image-classification, Image Classification

A visual hard attention model actively selects and observes a sequence of subregions in an image to make a prediction. Unlike in the deep convolution network, in hard attention it is explainable which regions of the image contributed to the prediction. However, the attention policy used by the model to select these regions is not explainable. Moreover, a suboptimal parameterization of the policy leads to a suboptimal model. In this paper, we attempt at designing an efficient hard attention model for the image classification task. The attention policy used by our model is non-parametric and explainable. The model estimates expected information gain (EIG) obtained from attending various regions by predicting their content ahead of time. It compares EIGs using Bayesian Optimal Experiment Design and attends the region with maximum EIG. The model aggregates features of the observed regions in a recurrent state, which is used by a classifier and a variational autoencoder with the normalizing flows to predict the class label and the image content, respectively. We train our model with a differentiable objective, optimized using gradient descent, and test it on several datasets. The performance of our model is comparable to or better than the baseline models.
Dynamic Graph Representation Learning with Fourier Temporal State Embedding
Authors: Yihan He, Wei Cao, Shun Zheng, Zhifeng Gao, Jiang Bian
Date: 2021-01-01 00:00:00
Tasks: Graph Embedding, Graph Representation Learning, Representation Learning

Static graph representation learning has been applied in many tasks over the years thanks to the invention of unsupervised graph embedding methods and more recently, graph neural networks (GNNs). However, in many cases, we are to handle dynamic graphs where the structures of graphs and labels of the nodes are evolving steadily with time. This has posed a great challenge to existing methods in time and memory efficiency. In this work, we present a new method named Fourier Temporal State Embedding (FTSE) to address the temporal information in dynamic graph representation learning. FTSE offered time and memory-efficient solution through applying signal processing techniques to the temporal graph signals. We paired the Fourier Transform with an efficient edge network and provided a new prototype of modeling dynamic graph evolution with high precision. FTSE can also prevent the 'history explosion' that exists in sequential models. The empirical study shows that our proposed approach achieves significantly better performance than previous approaches on public datasets across multiple tasks.