pwc-archive/papers-with-abstracts huggingface dataset:
A Spectral Perspective of Neural Networks Robustness to Label Noise
Authors: Oshrat Bar, Amnon Drory, Raja Giryes
Date: 2021-01-01 00:00:00

Deep networks usually require a massive amount of labeled data for their training. Yet, such data may include some mistakes in the labels. Interestingly, networks have been shown to be robust to such errors. This work uses recent developments in the analysis of neural networks function space to provide an explanation for their robustness. In particular, we relate the smoothness regularization that usually exists in conventional training to attenuation of high frequencies, which mainly characterize noise. By using a connection between the smoothness and the spectral norm of the network weights, we suggest that one may further improve robustness via spectral normalization. Empirical experiments validate our claims and show the advantage of this normalization for classification with label noise.
On Flat Minima, Large Margins and Generalizability
Authors: Daniel Lengyel, Nicholas Jennings, Panos Parpas, Nicholas Kantas
Date: 2021-01-01 00:00:00

The intuitive connection to robustness and convincing empirical evidence have made the flatness of the loss surface an attractive measure of generalizability for neural networks. 
     Yet it suffers from various problems such as computational difficulties, reparametrization issues, and a growing concern that it may only be an epiphenomenon of optimization methods. 
     We provide empirical evidence that under the cross-entropy loss once a neural network reaches a non-trivial training error, the flatness correlates (via Pearson Correlation Coefficient) well to the classification margins, which allows us to better reason about the concerns surrounding flatness. 
     Our results lead to the practical recommendation that when assessing generalizability one should consider a margin-based measure instead, as it is computationally more efficient, provides further insight, and is highly correlated to flatness. 
     We also use our insight to replace the misleading folklore that small-batch methods generalize better because they are able to escape sharp minima. Instead, we argue that large-batch methods did not have enough time to maximize margins and hence generalize worse.
Improving the Unsupervised Disentangled Representation Learning with VAE Ensemble
Authors: Nanxiang Li, Shabnam Ghaffarzadegan, Liu Ren
Date: 2021-01-01 00:00:00
Tasks: Disentanglement, Representation Learning

Variational Autoencoder (VAE) based frameworks have achieved the state-of-the-art performance on the unsupervised disentangled representation learning. A recent theoretical analysis shows that such success is mainly due to the VAE implementation choices that encourage a PCA-like behavior locally on data samples. Despite this implied model identifiability, the VAE based disentanglement frameworks still face the trade-off between the local orthogonality and data reconstruction. As a result, models with the same architecture and hyperparameter setting can sometime learn entangled representations. To address this challenge, we propose a simple yet effective VAE ensemble framework consisting of multiple VAEs. It is based on the assumption that entangled representations are unique in their own ways, and the disentangled representations are "alike" (similar up to a signed permutation transformation). In the proposed VAE ensemble, each model not only maintains its original objective, but also encodes to and decodes from other models through pair-wise linear transformations between the latent representations. We show both theoretically and experimentally, the VAE ensemble objective encourages the linear transformations connecting the VAEs to be trivial transformations, aligning the latent representations of different models to be "alike". We compare our approach with the state-of-the-art unsupervised disentangled representation learning approaches and show the improved performance.
Neural Networks Preserve Invertibility Across Iterations: A Possible Source of Implicit Data Augmentation
Authors: Arushi Gupta
Date: 2021-01-01 00:00:00
Tasks: Data Augmentation

Determining what kind of representations neural networks learn, and how this may relate to generalization, remains a challenging problem. Previous work has utilized a rich set of methods to invert layer representations of neural networks, i.e. given some reference activation $\Phi_0$ and a layer function $r_{\ell}$, find $x$ which minimizes $||\Phi_0 - r_{\ell}(x)||^2$ . We show that neural networks can preserve invertibility across several iterations. That is, it is possible to interpret activations produced in some later iteration in the context of the layer function of the current iteration. For convolutional and fully connected networks, the lower layers maintain such a consistent representation for several iterations, while in the higher layers invertibility holds for fewer iterations. Adding skip connections such as those found in Resnet allows even higher layers to preserve invertibility across several iterations. We believe the fact that higher layers may interpret weight changes made by lower layers as changes to the data may produce implicit data augmentation. This implicit data augmentation may eventually yield some insight into why neural networks can generalize even with so many parameters.
Meta-Learned Confidence for Transductive Few-shot Learning
Authors: Seong Min Kye, Hae Beom Lee, Hoirin Kim, Sung Ju Hwang
Date: 2021-01-01 00:00:00
Tasks: Few-Shot Learning, Meta-Learning

Transductive inference is an effective means of tackling the data deficiency problem in few-shot learning settings. A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples, or confidence-weighted average of all the query samples. However, a caveat here is that the model confidence may be unreliable, which may lead to incorrect predictions. To tackle this issue, we propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries such that they improve the model’s transductive inference performance on unseen tasks. We achieve this by meta-learning an input-adaptive distance metric over a task distribution under various model and data perturbations, which will enforce consistency on the model predictions under diverse uncertainties for unseen tasks. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets, on which it largely outperforms strong recent baselines and obtains new state-of-the-art results. Further application on semi-supervised few shot learning tasks also yields significant performance improvements over the baselines.
Semantic Segmentation Based Unsupervised Domain Adaptation via Pseudo-Label Fusion
Authors: Chen-Hao Chao, Bo-Wun Cheng, Chien Feng, Chun-Yi Lee
Date: 2021-01-01 00:00:00
Tasks: Pseudo Label, Segmentation

In this paper, we propose a pseudo label fusion framework (PLF), a learning framework developed to deal with the domain gap between a source domain and a target domain for performing semantic segmentation based UDA in the unseen target domain. PLF fuses the pseudo labels generated by an ensemble of teacher models. The fused pseudo labels are then used by a student model to distill out the information embedded in these fused pseudo labels to perform semantic segmentation in the target domain. To examine the effectiveness of PLF, we perform a number of experiments on both  GTA5 to Cityscapes and SYNTHIA to Cityscapes benchmarks to quantitatively and qualitatively inspect the improvements achieved by employing PLF in performing semantic segmentation in the target domain. Moreover, we provide a number of parameter analyses to validate that the choices made in the design of PLF is both practical and beneficial. Our experimental results on both benchmarks shows that PLF indeed offers adequate performance benefits in performing semantic segmentation in the unseen domain, and is able to achieve competitive performance when compared to the contemporary UDA techniques.
Self-Supervised Learning of Compressed Video Representations
Authors: Youngjae Yu, Sangho Lee, Gunhee Kim, Yale Song
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Computational Efficiency, Self-Supervised Learning

Self-supervised learning of video representations has recently received great attention. Existing methods typically require frames to be decoded before being processed, which increases compute and storage requirements and ultimately hinders large-scale training. In this work, we propose a self-supervised approach to learn compressed video representations, eliminating the expensive decoding step. We use a three-stream video architecture that directly encodes I-frames and P-frames of a compressed video. Unlike existing approaches that encode I-frames and P-frames individually, we propose to jointly encode them by establishing bidirectional dynamic connections across streams. We show that our approach outperforms existing compressed video approaches in the supervised regime while maintaining computational efficiency. To enable self-supervised learning, we propose two pretext tasks that leverage the multimodal nature (RGB, motion vector, residuals) and the internal GOP structure of compressed videos. The first task asks our network to predict zeroth-order motion statistics in a spatio-temporal pyramid; the second task asks correspondence types between I-frames and P-frames after applying temporal transformation. We show that our approach achieves competitive performance on self-supervised learning of video representations with a considerable improvement in speed compared to the traditional methods.
FLAGNet : Feature Label based Automatic Generation Network for symbolic music
Authors: SeongHyeon Go
Date: 2021-01-01 00:00:00
Tasks: Generative Adversarial Network, Music Generation

The technology for automatic music generation has been very actively studied in recent years. However, almost in these studies, handling domain knowledge of music was omitted or considered a difficult task. In particular, research that analyzes and utilizes the characteristics of each bar of music is very rare, even though it is essential in the human composition. We propose a model that generate music with musical characteristics of bars by conditional generative adversarial network, and analyze the good combination of the sequence of which characterized bars  for symbolic-domain music generation by Recurrent Neural Network with Long short term memory layer. Also, by analyzing symbolic music data as image-like based on relational pitch approach, it increases the utilization of the data set with arbitrary chord scales and enables the use of generational results extensively. The resulting model FLAGNet generates music with the understanding of musical domain knowledge while handling inputs like minimum unit of note, length of music, chart scales, and chord condition.
Self-Supervised Continuous Control without Policy Gradient
Authors: Hao Sun, Ziping Xu, Meng Fang, Yuhang Song, Jiechao Xiong, Bo Dai, Zhengyou Zhang, Bolei Zhou
Date: 2021-01-01 00:00:00
Tasks: continuous-control, Continuous Control, MuJoCo, Policy Gradient Methods, reinforcement-learning, Reinforcement Learning (RL), Self-Supervised Learning

Despite the remarkable progress made by the policy gradient algorithms in reinforcement learning (RL), sub-optimal policies usually result from the local exploration property of the policy gradient update. In this work, we propose a method called Zeroth-Order Supervised Policy Improvement (ZOSPI) that exploits the estimated value function $Q$ globally while preserving the local exploitation of the policy gradient methods. Experiments show that ZOSPI achieves competitive results on the MuJoCo benchmarks with a remarkable sample efficiency. Moreover, different from the conventional policy gradient methods, the policy learning of ZOSPI is conducted in a self-supervised manner. We show such a self-supervised learning paradigm has the flexibility of including optimistic exploration as well as adopting a non-parametric policy.
Discrete Predictive Representation for Long-horizon Planning
Authors: Thanard Kurutach, Julia Peng, Yang Gao, Stuart Russell, Pieter Abbeel
Date: 2021-01-01 00:00:00
Tasks: Deep Reinforcement Learning, Object, Reinforcement Learning (RL)

Discrete representations have been key in enabling robots to plan at more abstract levels and solve temporally-extended tasks more efficiently for decades. However, they typically require expert specifications. On the other hand, deep reinforcement learning aims to learn to solve tasks end-to-end, but struggles with long-horizon tasks. In this work, we propose Discrete Object-factorized Representation Planning (DORP), which learns temporally-abstracted discrete representations from exploratory video data in an unsupervised fashion via a mutual information maximization objective. DORP plans a sequence of abstract states for a low-level model-predictive controller to follow. In our experiments, we show that DORP robustly solves unseen long-horizon tasks. Interestingly, it discovers independent representations per object and binary properties such as a key-and-door.
ME-MOMENTUM: EXTRACTING HARD CONFIDENT EXAMPLES FROM NOISILY LABELED DATA
Authors: Yingbin Bai, Tongliang Liu
Date: 2021-01-01 00:00:00
Proceeding: ICCV 2021 10
Tasks: Learning with noisy labels, Memorization

Examples that are close to the decision boundary—that we term hard examples, are essential to shaping accurate classifiers.   Extracting confident examples has been widely studied in the community of learning with noisy labels.  However, it remains elusive how to extract hard confident examples from the noisy training data.  In this paper, we propose a deep learning paradigm to solve this problem, which is built on the memorization effect of deep neural networks that they would first learn simple patterns,  i.e.,  which are defined by these shared by multiple training examples. To extract hard confident examples that contain non-simple patterns and are entangled with the inaccurately labeled examples, we borrow the idea of momentum from physics. Specifically, we alternately update the confident examples and refine the classifier.  Note that the extracted confident examples in the previous round can be exploited to learn a better classifier and that the better classifier will help identify better (and hard) confident examples.  We call the approach the “Momentum of Memorization” (Me-Momentum). Empirical results on benchmark-simulated and real-world label-noise data illustrate the effectiveness of Me-Momentum for extracting hard confident examples, leading to better classification performance.
Robust Temporal Ensembling
Authors: Abel Brown, Benedikt Schifferer, Robert DiPietro
Date: 2021-01-01 00:00:00
Tasks: RTE

Successful training of deep neural networks with noisy labels is an essential capability as most real-world datasets contain some amount of mislabeled data.  Left unmitigated, label noise can sharply degrade typical supervised learning approaches.  In this paper, we present robust temporal ensembling (RTE), a simple supervised learning approach which combines robust task loss, temporal pseudo-labeling, and a new ensemble consistency regularization term to achieve noise-robust learning.  We demonstrate that RTE achieves state-of-the-art performance across the CIFAR-10, CIFAR-100, and ImageNet datasets, without any label filtering or label fixing. In particular, RTE achieves 93.64% accuracy on CIFAR-10 and 66.43% accuracy on CIFAR-100 under 80% label corruption, and achieves 76.72% accuracy on ImageNet under 40% corruption. These are substantial gains over previous state-of-the-art accuracies of 86.6%, 60.2%, and 71.31%, respectively, achieved using three distinct methods. Finally, we show that RTE retains competitive corruption robustness to unforeseen input noise using CIFAR-10-C, obtaining a mean corruption error (mCE) of 13.50% even in the presence of an 80% noise ratio, versus 26.9% mCE with standard methods on clean data.
EMTL: A Generative Domain Adaptation Approach
Authors: Jianfeng Zhang, Illyyne Saffar, Aladin Virmaux, Balázs Kégl
Date: 2021-01-01 00:00:00
Tasks: Density Estimation, Domain Adaptation, Unsupervised Domain Adaptation

We propose an unsupervised domain adaptation approach based on generative models. We show that when the source probability density function can be learned, one-step Expectation–Maximization iteration plus an additional marginal density function constraint will produce a proper mediator probability density function to bridge the gap between the source and target domains. The breakthrough is based on modern generative models (autoregressive mixture density nets) that are competitive to discriminative models on moderate-dimensional classification problems. By decoupling the source density estimation from the adaption steps, we can design a domain adaptation approach where the source data is locked away after being processed only once, opening the door to transfer when data security or privacy concerns impede the use of traditional domain adaptation. We demonstrate that our approach can achieve state-of-the-art performance on synthetic and real data sets, without accessing the source data at the adaptation phase.
Quantum and Translation Embedding for Knowledge Graph Completion
Authors: Panfeng Chen, Yisong Wang, Renyan Feng, Xiaomin Yu, Quan Yu
Date: 2021-01-01 00:00:00
Tasks: Knowledge Graph Completion, Translation

Knowledge Graph Completion (KGC) mainly devotes to link predicting for an entity pair in Knowledge Graph (KG) according to known facts. In this work, we present a novel model for this end. In this model, Quantum and Translation Embedding are used as components for logical and structural feature capturing in the same vector subspace, respectively. The two components have synergy with each other and achieve impressive performance at low cost which is close to the efficient model TransE. Surprisingly, the performance on challenging datasets such as fb15k237 and WN18RR is up to 94.89% and 92.79% in metric Hits@1 while the dimension of embedding is only 4 in the process of training. The insight of this work enlightens the notion of dense feature model design for KGC which is a new alternative to Deep Neural networks (DNN) in this task or even a better choice.
MULTI-SPAN QUESTION ANSWERING USING SPAN-IMAGE NETWORK
Authors: Tarik Arici, Hayreddin Ceker, Ismail Baha Tutar
Date: 2021-01-01 00:00:00
Tasks: Question Answering, valid

Question-answering (QA) models aim to find an answer given a question and con- text. Language models like BERT are used to associate question and context to find an answer span. Prior art on QA focuses on finding the best answer. There is a need for multi-span QA models to output the top-K likely answers to questions such as "Which companies Elon Musk started?" or "What factors cause global warming?" In this work, we introduce Span-Image architecture that can learn to identify multiple answers in a context for a given question. This architecture can incorporate prior information about the span length distribution or valid span patterns (e.g., end index has to be larger than start index), thus eliminating the need for post-processing. Span-Image architecture outperforms the state-of-the-art in top-K answer accuracy on SQuAD dataset and in multi-span answer accuracy on an Amazon internal dataset.
Learning Discrete Adaptive Receptive Fields for Graph Convolutional Networks
Authors: Xiaojun Ma, Ziyao Li, Lingjun Xu, Guojie Song, Yi Li, Chuan Shi
Date: 2021-01-01 00:00:00

Different nodes in a graph neighborhood generally yield different importance. In previous work of Graph Convolutional Networks (GCNs), such differences are typically modeled with attention mechanisms. However, as we prove in our paper, soft attention weights suffer from over-smoothness in large neighborhoods. To address this weakness, we introduce a novel framework of conducting graph convolutions, where nodes are discretely selected among multi-hop neighborhoods to construct adaptive receptive fields (ARFs). ARFs enable GCNs to get rid of the over-smoothness of soft attention weights, as well as to efficiently explore long-distance dependencies in graphs. We further propose GRARF (GCN with Reinforced Adaptive Receptive Fields) as an instance, where an optimal policy of constructing ARFs is learned with reinforcement learning. GRARF achieves or matches state-of-the-art performances on public datasets from different domains. Our further analysis corroborates that GRARF is more robust than attention models against neighborhood noises.
Sparse Coding-inspired GAN for Weakly Supervised Hyperspectral Anomaly Detection
Authors: Tao Jiang, Weiying Xie, Jie Lei, Yunsong Li, Zan Li
Date: 2021-01-01 00:00:00
Tasks: Anomaly Detection, Decoder, Generative Adversarial Network, Weakly-supervised Learning

Anomaly detection (AD) on hyperspectral images (HSIs) is of great importance in both space exploration and earth observations. However, the challenges caused by insufficient datasets, no labels, and noise corruption substantially downgrade the quality of detection. For solving these problems, this paper proposes a sparse coding-inspired generative adversarial network (GAN) for weakly supervised HAD, named sparseHAD. It can learn a discriminative latent reconstruction with small errors for background samples and large errors for anomaly samples.  First, we design a novel background-category searching step to eliminate the difficulty of data annotation and prepare for weakly supervised learning. Then, a sparse coding-inspired regularized network is integrated into an end-to-end GAN to form a weakly supervised spectral mapping model consisting of two encoders, a decoder, and a discriminator. This model not only makes the network more robust and interpretable both experimentally and theoretically but also develops a new sparse coding-inspired path for HAD. Subsequently, the proposed sparseHAD detect anomalies in latent space rather than original space, which also contributes to the robustness of the network against noise. Quantitative assessments and experiments over real HSIs demonstrate the unique promise of such an approach.
Dissecting graph measures performance for node clustering in LFR parameter space
Authors: Vladimir Ivashkin, Pavel Chebotarev
Date: 2021-01-01 00:00:00
Tasks: Clustering, Node Clustering, Open-Ended Question Answering

Graph measures can be used for graph node clustering using metric clustering algorithms. There are multiple measures applicable to this task, and which one performs better is an open question. We study the performance of 25 graph measures on generated graphs with different parameters. While usually measure comparisons are limited to general measure ranking on a particular dataset, we aim to explore the performance of various measures depending on graph features. Using an LFR generator of graphs with preferential attachment, we create a dataset of ~7500 graphs covering the whole LFR parameter space. For each graph, we assess the quality of clustering with k-means algorithm for every considered measure. We determine the best measure for every area of the parameter space. We find that the parameter space consists of distinct zones where one particular measure is the best. We analyze the geometry of the resulting zones and describe it with simple criteria. Given particular graph parameters, this allows us to choose the best measure to use for clustering.