pwc-archive/papers-with-abstracts huggingface dataset:
Generalizing Graph Convolutional Networks
Authors: Jialin Zhao, Yuxiao Dong, Jie Tang, Ming Ding, Kuansan Wang
Date: 2021-01-01 00:00:00

Graph convolutional networks (GCNs) have emerged as a powerful framework for mining and learning with graphs. A recent study shows that GCNs can be simplified as a linear model by removing nonlinearities and weight matrices across all consecutive layers, resulting the simple graph convolution (SGC) model. In this paper, we aim to understand GCNs and generalize SGC as a linear model via heat kernel (HKGCN), which acts as a low-pass filter on graphs and enables the aggregation of information from extremely large receptive fields. We theoretically show that HKGCN is in nature a continuous propagation model and GCNs without nonlinearities (i.e., SGC) are the discrete versions of it. Its low-pass filter and continuity properties facilitate the fast and smooth convergence of feature propagation. Experiments on million-scale networks show that the linear HKGCN model not only achieves consistently better results than SGC but also can match or even beat advanced GCN models, while maintaining SGC’s superiority in efficiency.
iPTR: Learning a representation for interactive program translation retrieval
Authors: Binger Chen, Ziawasch Abedjan
Date: 2021-01-01 00:00:00
Tasks: Code Translation, Retrieval, Translation

Program translation contributes to many real world scenarios, such as porting codebases written in an obsolete or deprecated language to a modern one or re-implementing existing projects in one's preferred programming language. Existing data-driven approaches either require large amounts of training data or neglect significant characteristics of programs. In this paper, we present iPTR for interactive code translation retrieval from Big Code. iPTR uses a novel code representation technique that encodes structural characteristics of a program and a predictive transformation technique to transform the representation into the target programming language. The transformed representation is used for code retrieval from Big Code. With our succinct representation, the user can easily update and correct the returned results to improve the retrieval process. Our experiments show that iPTR outperforms supervised baselines in terms of program accuracy.
Mitigating Mode Collapse by Sidestepping Catastrophic Forgetting
Authors: Karttikeya Mangalam, Rohin Garg, Jathushan Rajasegaran, Taesung Park
Date: 2021-01-01 00:00:00
Tasks: Continual Learning

Generative Adversarial Networks (GANs) are a class of generative models used for various applications, but they have been known to suffer from the mode collapse problem, in which some modes of the target distribution are ignored by the generator.  Investigative study using a new data generation procedure indicates that the mode collapse of the generator is driven by the discriminator’s inability to maintain classification accuracy on previously seen samples, a phenomenon called Catastrophic Forgetting in continual learning. Motivated by this observation, we introduce a novel training procedure that dynamically spawns additional dis-criminators to remember previous modes of generation. On several datasets, we show that our training scheme can be plugged-in to existing GAN frameworks to mitigate mode collapse and improve standard metrics for GAN evaluation.
Ego-Centric Spatial Memory Networks
Authors: Daniel James Lenton, Stephen James, Ronald Clark, Andrew Davison
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: CPU, GPU, Inductive Bias, Semantic Segmentation, Spatial Reasoning

Egocentric spatial memory is cornerstone to our understanding of biological perception. Spatial memory is also essential for embodied AI systems acting from image sensors in partially observable (PO) environments.  For image-to-action learning, the most common approach for dealing with PO is to incorporate general memory architectures, which do not explicitly leverage any spatial inductive bias. Examples are Long Short-Term Memory, self-attention, or differentiable external memory. While other works have demonstrated the benefits of a learnt spatial memory, these are typically targeted at 2D navigation,  and encode the memory into aerial 2D maps. We propose Egocentric Spatial Memory Networks (ESMN), which encode the memory in an ego-sphere around the agent, enabling more expressive 3D representations. ESMN improves both training efficiency and final performance against other memory baselines on both drone and manipulator visuomotor control tasks. We see a benefit for both body-fixed and scene-fixed cameras, and show that ESMN can be trained via either imitation or reinforcement learning.  The explicit ego-centric geometry also enables us to seamlessly combine the learnt controller with other non-learnt modalities, such as local obstacle avoidance, which is not possible using less structured memory. We further show applications to semantic segmentation on the ScanNet dataset, where ESMN naturally combines image-level and map-level inference modalities. Finally,  when combined with a visual tracker, we show that the module can operate as a real-time real-world mapping system, reaching 30Hz on either GPU or CPU, and producing high-quality and robust ego-centric maps.  With our broad demonstrations, we show that ESMN represents a useful and general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures.
A Universal Learnable Audio Frontend
Authors: Neil Zeghidour, Olivier Teboul, Félix de Chaumont Quitry, Marco Tagliasacchi
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Audio Classification

Mel-filterbanks are fixed, engineered audio features which emulate human perception and have lived through the history of audio understanding up to today. However, their undeniable qualities are counterbalanced by the fundamental limitations of handmade representations. In this work we show that we can train a single, universal learnable frontend that outperforms mel-filterbanks over a wide range of audio domains, including speech, music, audio events, and animal sounds, providing an unprecedented general purpose learned frontend for audio. To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-filterbanks. Our system learns all operations of audio features extraction, from filtering to pooling, compression and normalization, and can be integrated into any neural network at a negligible parameter cost. We perform multi-task training on 8 diverse audio classification tasks, and show consistent improvements of our model over mel-filterbanks and previous learnable alternatives. Moreover, our system is competitive with the current state-of-the-art learnable frontend on Audioset, with orders of magnitude fewer parameters.
The Impact of the Mini-batch Size on the Dynamics of SGD: Variance and Beyond
Authors: Xin Qian, Diego Klabjan
Date: 2021-01-01 00:00:00
Tasks: regression

We study mini-batch stochastic gradient descent (SGD) dynamics under linear regression and deep linear networks by focusing on the variance of the gradients only given the initial weights and mini-batch size, which is the first study of this nature. In the linear regression case, we show that in each iteration the norm of the gradient is a decreasing function of the mini-batch size $b$ and thus the variance of the stochastic gradient estimator is a decreasing function of $b$. For deep neural networks with $L_2$ loss we show that the variance of the gradient is a polynomial in $1/b$. The results theoretically back the important intuition that smaller batch sizes yield larger variance of the stochastic gradients and lower loss function values which is a common believe among the researchers. The proof techniques exhibit a relationship between stochastic gradient estimators and initial weights, which is useful for further research on the dynamics of SGD. We empirically provide insights to our results on various datasets and commonly used deep network structures. We further discuss possible extensions of the approaches we build in studying the generalization ability of the deep learning models.
Simple Augmentation Goes a Long Way: ADRL for DNN Quantization
Authors: Lin Ning, Guoyang Chen, Weifeng Zhang, Xipeng Shen
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Deep Reinforcement Learning, Quantization, Reinforcement Learning (RL)

Mixed precision quantization improves DNN performance by assigning different layers with different bit-width values. Searching for the optimal bit-width for each layer, however, remains a challenge. Deep Reinforcement Learning (DRL) shows some recent promise. It however suffers instability due to function approximation errors, causing large variances in the early training stages, slow convergence, and suboptimal policies in the mixed-precision quantization problem. This paper proposes augmented DRL (ADRL) as a way to alleviate these issues. This new strategy augments the neural networks in DRL with a complementary scheme to boost the performance of learning. The paper examines the effectiveness of ADRL both analytically and empirically, showing that it can produce more accurate quantized models than the state of the art DRL-based quantization while improving the learning speed by 4.5-64 times.
Multi-View Disentangled Representation
Authors: Zongbo Han, Changqing Zhang, Huazhu Fu, QinGhua Hu, Joey Tianyi Zhou
Date: 2021-01-01 00:00:00
Tasks: Disentanglement

Learning effective representations for data with multiple views is crucial in machine learning and pattern recognition. Recently great efforts have focused on learning unified or latent representations to integrate information from different views for specific tasks. These approaches generally assume simple or implicit relationships between different views and as a result are not able to flexibly and explicitly depict the correlations among these views. To address this, we firstly propose the definition and conditions for multi-view disentanglement providing general instructions for disentangling representations between different views. Furthermore, a novel objective function is derived to explicitly disentangle the multi-view data into a shared part across different views and a (private) exclusive part within each view. Experiments on a variety of multi-modal datasets demonstrate that our objective can effectively disentangle information from different views while satisfying the disentangling conditions.
Polynomial Graph Convolutional Networks
Authors: Luca Pasa, Nicolò Navarin, Alessandro Sperduti
Date: 2021-01-01 00:00:00
Tasks: Graph Classification, Graph Neural Network

Graph Convolutional Neural Networks (GCNs) exploit convolution operators, based on some neighborhood aggregating scheme, to compute representations of graphs. The most common convolution operators only exploit local topological information. To consider wider topological receptive fields, the mainstream approach is to non-linearly stack multiple Graph Convolutional (GC) layers. In this way, however, interactions among GC parameters at different levels pose a bias on the flow of topological information. In this paper,  we propose a different strategy, considering a single graph convolution layer that independently exploits neighbouring nodes at different topological distances, generating decoupled representations for each of them. These representations are then  processed by subsequent readout layers. We implement this strategy introducing the Polynomial Graph Convolution (PGC) layer, that we prove being more expressive than the most common convolution operators and their linear stacking. Our contribution is not limited to the definition of a convolution operator with a larger receptive field, but we prove both theoretically and experimentally that the common way  multiple non-linear graph convolutions are stacked limits the neural network expressiveness. Specifically, we show that a Graph Neural Network architecture with a single PGC layer achieves state of the art performance on many commonly adopted graph classification benchmarks.
The inductive bias of ReLU networks on orthogonally separable data
Authors: Mary Phuong, Christoph H Lampert
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Inductive Bias

We study the inductive bias of two-layer ReLU networks trained by gradient flow. We identify a class of easy-to-learn (`orthogonally separable') datasets, and characterise the solution that ReLU networks trained on such datasets converge to. Irrespective of network width, the solution turns out to be a combination of two max-margin classifiers: one corresponding to the positive data subset and one corresponding to the negative data subset.
The proof is based on the recently introduced concept of extremal sectors, for which we prove a number of properties in the context of orthogonal separability. In particular, we prove stationarity of activation patterns from some time $T$ onwards, which enables a reduction of the ReLU network to an ensemble of linear subnetworks.
Recycling sub-optimial Hyperparameter Optimization models to generate efficient Ensemble Deep Learning
Authors: Pierrick Pochelu, Bruno Conche, Serge G. Petiton
Date: 2021-01-01 00:00:00
Tasks: Hyperparameter Optimization

Ensemble Deep Learning improves accuracy over a single model by combining predictions from multiple models.  It has established itself to be the core strategy for tackling the most difficult problems, like winning Kaggle challenges. Due to the lack of consensus to design a successful deep learning ensemble, we introduce Hyperband-Dijkstra, a new workflow that automatically explores neural network designs with Hyperband and efficiently combines them with Dijkstra's algorithm.  This workflow has the same training cost than standard Hyperband running except sub-optimal  solutions are stored  and  are  candidates  to  be  selected in  the  ensemble  selection  step (recycling). Next, to predict on new data, the user gives to Dijkstra the maximum number of models wanted in  the  ensemble to control the tradeoff between accuracy and inference time.  
Hyperband is a very efficient algorithm allocating exponentially more resources to the most promising configurations. It is also capable to propose diverse models due to its pure-exploration nature, which allows Dijkstra algorithm with a smart combination of diverse models to achieve a strong variance and bias reduction. The exploding number of possible combinations generated by Hyperband increases the probability that Dijkstra finds an accurate combination which fits the dataset and generalizes on new data.
The two experimentation on CIFAR100 and on our unbalanced microfossils dataset show that our new workflow generates an ensemble far more accurate than any other ensemble of any ResNet models from ResNet18 to ResNet152.
Detecting Adversarial Examples by Additional Evidence from Noise Domain
Authors: Song Gao, Shui Yu, Shaowen Yao
Date: 2021-01-01 00:00:00
Tasks: Steganalysis

Deep neural networks are widely adopted powerful tools for perceptual tasks. However, recent research indicated that they are easily fooled by adversarial examples, which are produced by adding imperceptible adversarial perturbations to clean examples. In this paper, we utilize the steganalysis rich model (SRM) to generate noise feature maps, and combine them with RGB images to discover the difference between adversarial examples and clean examples. In particular, we propose a two-stream pseudo-siamese network and train it end-to-end to detect adversarial examples. Our approach fuses the subtle difference in RGB images with the noise inconsistency in noise features. The proposed method has strong detection capability and transferability, and can be combined with any classifier without modifying its architecture or training procedure. Our extensive empirical experiments show that, compared with the state-of-the-art detection methods, the proposed method achieves excellent performance in distinguishing adversarial samples generated by popular attack methods on different real datasets. Moreover, our method has good generalization, it trained by a specific adversary can generalize to other adversaries.
Monte-Carlo Planning and Learning with Language Action Value Estimates
Authors: Youngsoo Jang, Seokin Seo, Jongmin Lee, Kee-Eung Kim
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Natural Language Understanding, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Interactive Fiction (IF) games provide a useful testbed for language-based reinforcement learning agents, posing significant challenges of natural language understanding, commonsense reasoning, and non-myopic planning in the combinatorial search space. Agents based on standard planning algorithms struggle to play IF games due to the massive search space of language actions. Thus, language-grounded planning is a key ability of such agents, since inferring the consequence of language action based on semantic understanding can drastically improve search. In this paper, we introduce Monte-Carlo planning with Language Action Value Estimates (MC-LAVE) that combines a Monte-Carlo tree search with language-driven exploration. MC-LAVE invests more search effort into semantically promising language actions using locally optimistic language value estimates, yielding a significant reduction in the effective search space of language actions. We then present a reinforcement learning approach via MC-LAVE, which alternates between MC-LAVE planning and supervised learning of the self-generated language actions. In the experiments, we demonstrate that our method significantly outperforms the state-of-the-art on the extensive set of IF games.
A Simple and General Strategy for Referential Problem in Low-Resource Neural Machine Translation
Authors: Yatu Ji, Nier Wu, Hongxu Hou
Date: 2021-01-01 00:00:00
Tasks: Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Low Resource NMT, Machine Translation, NMT, Translation

This paper aims to solve a series of referential problems in sequence decoding caused by data sparsity and corpus scarce in low-resource Neural Machine Translation (NMT), including pronoun missing, reference error, bias and so on. It is difficult to find the essential reason of these problems because they are only shown in the prediction results and involve all aspects of the model. Different from the usual solutions based on complex mathematical rule setting and adding artificial features, we expect to turn the problems in the predictions into noise as much as possible, and use adversarial training to make the model find the balance between the noise and the golden samples, instead of exploring the reason of the problem during the complex training. In this paper, only a simple noise-based preprocessing operation and a slight modification of the adversarial training can make the model generalize to a series of referential problems in low-resource NMT task. On Korean-Chinese, Mongolian-Chinese and Arabic-Chinese tasks, the evaluation of BLEU score and the accuracy of pronouns in sequence have been significantly improved.
Adaptive Tree Wasserstein Minimization for Hierarchical Generative Modeling
Authors: ZiHao Wang, Xu Zhao, Tam Le, Hao Wu, Yong Zhang, Makoto Yamada
Date: 2021-01-01 00:00:00
Tasks: Domain Adaptation, Unsupervised Domain Adaptation

Optimal Transport(OT) is a machine learning problem with applications including distribution comparison, generative adversarial networks, unsupervised domain adaptation, and to name a few. For deep learning literatures, the Wasserstein distance, which can be estimated by using linear programming and/or the Sinkhorn algorithm, is widely used to compare two distributions. However, those optimization algorithms require complexity over $O(n^2)$, where $n$ is the number of samples for data sets. Thus, it is in general hard to use the traditional algorithms for large-scale problems. Recently, researchers try to speed up the Wasserstein computation by using a closed-form solution, which has lower complexity than $O(n^2)$. For example, sliced Wasserstein solves OT in $O(n\log n)$ time in one-dimensional space by using sorting samples but it suffers from the information loss when projected into one-dimensional space. In this work, we consider OT over tree metrics, which is more general than the sliced Wasserstein and includes the sliced Wasserstein as a special case, and we propose a fast minimization algorithm in $O(n)$ for the optimal Wasserstein-1 transport plan between two distributions in the tree structure. To this end, we propose an online algorithm for adaptive tree construction during the minimization of Wasserstein loss. Thanks to the nice property of the proposed algorithm, we can obtain the alignment in $O(n\log n)$ time. We then apply the proposed Wasserstein minimization algorithm for generative model estimation and propose a tree Wasserstein Auto Encoder. Through extensive experiments on 2D toy examples, high-dimension Mixture Gaussian, and real-world datasets, we show that our algorithm is compatible with the one-dimensional slice Wasserstein in computational time but more effective in higher-dimensional problems. Moreover, by using real-datasets, we demonstrate that the efficacy of the proposed algorithms.
Grid cell modeling with mapping representation of self-motion for path integration
Authors: Anonymous
Date: 2021-01-01 00:00:00

The representation of grid cells in the medial entorhinal cortex (MEC) region is crucial for path integration. In this paper, we proposed a grid cell modeling mechanism by mapping the agent’s self-motion in Euclidean space to the neuronal activity of grid cells. Our representational model can achieve hexagonal patterns of grid cells from recurrent neural network (RNN) and enables multi-scale path integration for 1D, 2D and 3D spaces. Different from the existing works which need to learn weights of RNN to get the vector representation of grid cells, our method can obtain weights by direct matrix operations. Moreover, compared with the classical models based on continuous attractor network (CAN), our model avoids the connection matrix’s symmetry limitation and spatial representation redundancy problems. In this paper, we also discuss the connection pattern between grid cells and place cells to demonstrate grid cells’  functioning as a metric for coding space.