pwc-archive/papers-with-abstracts huggingface dataset:
Learning the Connections in Direct Feedback Alignment
Authors: Matthew Bailey Webster, Jonghyun Choi, changwook Ahn
Date: 2021-01-01 00:00:00
Tasks: Image Classification

Feedback alignment was proposed to address the biological implausibility of the backpropagation algorithm which requires the transportation of the weight transpose during the backwards pass. The idea was later built upon with the proposal of direct feedback alignment (DFA), which propagates the error directly from the output layer to each hidden layer in the backward path using a fixed random weight matrix. This contribution was significant because it allowed for the parallelization of the backwards pass by the use of these feedback connections. However, just as feedback alignment, DFA does not perform well in deep convolutional networks. We propose to learn the backward weight matrices in DFA, adopting the methodology of Kolen-Pollack learning, to improve training and inference accuracy in deep convolutional neural networks by updating the direct feedback connections such that they come to estimate the forward path. The proposed method improves the accuracy of learning by direct feedback connections and reduces the gap between parallel training to serial training by means of backpropagation.
Gradient Descent Resists Compositionality
Authors: Yuanpeng Li, Liang Zhao, Joel Hestness, Kenneth Church, Mohamed Elhoseiny
Date: 2021-01-01 00:00:00

In this paper, we argue that gradient descent is one of the reasons that make compositionality learning hard during neural network optimization. We find that the optimization process imposes a bias toward non-compositional solutions. This is caused by gradient descent, trying to use all available and redundant information from input, violating the conditional independence property of compositionality. Based on this finding, we suggest that compositionality learning approaches considering only model architecture design are unlikely to achieve complete compositionality. This is the first work to investigate the relation between compositional learning and gradient descent. We hope this study provides novel insights into compositional generalization, and forms a basis for new research directions to equip machine learning models with such skills for human-level intelligence.
Learning a unified label space
Authors: Xingyi Zhou, Vladlen Koltun, Philipp Kraehenbuehl
Date: 2021-01-01 00:00:00
Tasks: Instance Segmentation, Object, object-detection, Object Detection, Semantic Segmentation

How do we build a general and broad object detection system? We use all labels of all concepts ever annotated. These labels span many diverse datasets with potentially inconsistent semantic labels. In this paper, we show how to integrate these datasets and their semantic taxonomies in a completely automated fashion. Once integrated, we train an off-the-shelf object detector on the union of the datasets. This unified recognition system performs as well as dataset-specific models on each training domain, but generalizes much better to new unseen domains. Entries based on the presented methodology ranked first in the object detection and instance segmentation tracks of the ECCV 2020 Robust Vision Challenge.
AggMask: Exploring locally aggregated learning of mask representations for instance segmentation
Authors: Tao Wang, Jun Hao Liew, Yu Li, Yunpeng Chen, Jiashi Feng
Date: 2021-01-01 00:00:00
Tasks: Instance Segmentation, Segmentation, Semantic Segmentation

 Recently proposed one-stage instance segmentation models (\emph{e.g.}, SOLO) learn  to directly predict location-specific object mask   with fully-convolutional networks. They perform comparably well as the  traditional two-stage Mask R-CNN model, yet enjoying  much simpler architecture and   higher efficiency.  However, an intrinsic limitation of these models  is that they tend to generate similar mask predictions for a single object at nearby  locations, while most of them are directly discarded by non-maximum suppression, leading to a waste of  some useful predictions that can supplement the final result. In this work, we aim to explore how the model can benefit from better leveraging the neighboring predictions while maintaining the architectural simplicity and efficiency.   To this end, we develop a novel learning-based aggregation framework that learns to aggregate the neighboring predictions. Meanwhile, unlike original location-based masks, the segmentation model is implicitly supervised to learn location-aware \textit{mask representations} that  encode the geometric structure of nearby objects and complements adjacent representations with context.  Based on the aggregation framework, we further introduce a mask interpolation mechanism that enables sharing mask representations for nearby spatial locations, thus allowing the model to generate much fewer representations for computation and memory saving.  We experimentally show that by simply augmenting the baseline model with our proposed aggregation framework, the instance segmentation performance is significantly improved. For instance, it improves a SOLO model with ResNet-101 backbone by 2.0 AP on the COCO benchmark, with only about 2\% increase of computation. {Code and models} are available at   anonymous repository: {\url{https://github.com/advdfacd/AggMask}}.
Faster and Smarter AutoAugment: Augmentation Policy Search Based on Dynamic Data-Clustering
Authors: Jonghyun Bae, Ji-Hoon Kim
Date: 2021-01-01 00:00:00
Tasks: Autonomous Driving, Clustering, Data Augmentation, Diversity, GPU

Data augmentation tuned to datasets and tasks has had great success in various AI applications, such as computer vision, natural language processing, autonomous driving, and bioinformatics. However, most of the specific parameter-based augmentation strategies are inefficient in finding suitable augmentation parameters to improve the model performance whenever the dataset changes. We introduce a dynamic data augmentation strategy called Faster and Smarter AutoAugment (FSAA) that separates the data augmentation method through the initial policy search result. Based on our policy branching principle, the augmentation policy dynamically clusters data points within the dataset according to the degree of performance change at the data split stage. With extensive experimentation on various datasets for image recognition task, we show that FSAA dramatically reduces the GPU computation cost, a problem in the existing automatic data augmentation strategies, by up to over 90%. It also ensures diversity and generalization of the dataset augmentation by searching for more diverse policies than existing autonomous methods in less time, resulting in consistent performance gains and robustness across multiple models.
Connection- and Node-Sparse Deep Learning: Statistical Guarantees
Authors: Johannes Lederer
Date: 2021-01-01 00:00:00
Tasks: Deep Learning, Prediction

Neural networks are becoming increasingly popular in applications, but a comprehensive mathematical understanding of their potentials and limitations is still missing. In this paper, we study the prediction accuracies of neural networks from a statistical point of view. In particular, we establish statistical prediction guarantees for deep learning with different types of sparsity-inducing regularization. Our bounds feature a mild dependence on network widths and depths, and, therefore, support the current trend toward wide and deep networks. The tools that we use in our derivations are uncommon in deep learning and, hence, might be of additional interest.
Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors
Authors: Linfeng Zhang, Kaisheng Ma
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: image-classification, Image Classification, Knowledge Distillation, Model Compression, object-detection, Object Detection

Knowledge distillation, in which a student model is trained to mimic a teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, designed for image classification, have failed on more challenging tasks, such as object detection. In this paper, we suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and background and (2) lack of distillation on the relation between different pixels. Observing the above reasons, we propose attention-guided distillation and non-local distillation to address the two problems, respectively.  Attention-guided distillation is proposed to find the crucial pixels of foreground objects with attention mechanism and then make the students take more effort to learn their features. Non-local distillation is proposed to enable students to learn not only the feature of an individual pixel but also the relation between different pixels captured by non-local modules. Experiments show that our methods achieve excellent AP improvements on both one-stage and two-stage, both anchor-based and anchor-free detectors. For example, Faster RCNN101 with our distillation achieves 43.9 AP on COCO2017, which is 4.1 higher than the baseline. Codes are will be released on Github.
GraphSAD: Learning Graph Representations with Structure-Attribute Disentanglement
Authors: Minghao Xu, Hang Wang, Bingbing Ni, Wenjun Zhang, Jian Tang
Date: 2021-01-01 00:00:00
Tasks: Attribute, Disentanglement, Graph Classification

Graph Neural Networks (GNNs) learn effective node/graph representations by aggregating the attributes of neighboring nodes, which commonly derives a single representation mixing the information of graph structure and node attributes. However, these two kinds of information might be semantically inconsistent and could be useful for different tasks. In this paper, we aim at learning node/graph representations with Structure-Attribute Disentanglement (GraphSAD). We propose to disentangle graph structure and node attributes into two distinct sets of representations, and such disentanglement can be done in either the input or the embedding space. We further design a metric to quantify the extent of such a disentanglement. Extensive experiments on multiple datasets show that our approach can indeed disentangle the semantics of graph structure and node attributes, and it achieves superior performance on both node and graph classification tasks.
DynaTune: Dynamic Tensor Program Optimization in Deep Neural Network Compilation
Authors: Minjia Zhang, Menghao Li, Chi Wang, Mingqin Li
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Decision Making, Uncertainty Quantification

Recently, the DL compiler, together with Learning to Compile has proven to be a powerful technique for optimizing deep learning models. However, existing methods focus on accelerating the convergence speed of the individual tensor operator rather than the convergence speed of the entire model, which results in long optimization time to obtain a desired latency.

In this paper, we present a new method called DynaTune, which provides significantly faster convergence speed to optimize a DNN model. In particular, we consider a Multi-Armed Bandit (MAB) model for the tensor program optimization problem. We use UCB to handle the decision-making of time-slot-based optimization, and we devise a Bayesian belief model that allows predicting the potential performance gain of each operator with uncertainty quantification, which guides the optimization process. We evaluate and compare DynaTune with the state-of-the-art DL compiler. The experiment results show that DynaTune is 1.2--2.4 times faster to achieve the same optimization quality for a range of models across different hardware architectures.
Learning Latent Topology for Graph Matching
Authors: Tianshu Yu, Runzhong Wang, Junchi Yan, Baoxin Li
Date: 2021-01-01 00:00:00
Tasks: Graph Generation, Graph Matching, Graph Representation Learning, Representation Learning

Graph matching (GM) has been traditionally modeled as a deterministic optimization problem characterized by an affinity matrix under pre-defined graph topology. Though there have been several attempts on learning more effective node-level affinity/representation for matching, they still heavily rely on the initial graph structure/topology which is typically obtained through heuristic ways (e.g. Delauney or $k$-nearest) and will not be adjusted during the learning process to adapt to problem-specific patterns. We argue that a standalone graph representation learning is insufficient for GM task, whereby a GM solver may favor some latent topology other than pre-defined one. Motivated by this hypothesis, we propose to learn latent graph topology in replacement of the fixed topology as input. To this end, we devise two types of latent graph generation procedures in deterministic and generative fashion, respectively. Particularly, the generative procedure emphasizes the across-graph consistency and thus can be viewed as a \textbf{co-generative} model. Our methods show superior performance over previous state-of-the-arts on several benchmarks, thus strongly supporting our hypothesis.
Weak NAS Predictor Is All You Need
Authors: Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen, Lu Yuan
Date: 2021-01-01 00:00:00
Tasks: All, Neural Architecture Search

Neural Architecture Search (NAS) finds the best network architecture by exploring the architecture-to-performance manifold. It often trains and evaluates a large amount of architectures, causing tremendous computation cost. Recent predictor-based NAS approaches attempt to solve this problem with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor. Existing predictors attempt to model the performance distribution over the whole architecture space, which could be too challenging given limited samples. Instead, we envision that this ambitious goal may not be necessary if the final aim is to find the best architecture. We present a novel framework to estimate weak predictors progressively. Rather than expecting a single strong predictor to model the whole space, we seek a progressive line of weak predictors that can connect a path to the best architecture, thus greatly simplifying the learning task of each predictor. It is based on the key property of the predictors that their probabilities of sampling better architectures will keep increasing. We thus only sample a few well-performed architectures guided by the predictive model, to estimate another better weak predictor. By this coarse-to-fine iteration, the ranking of sampling space is refined gradually, which helps find the optimal architectures eventually. Experiments demonstrate that our method costs fewer samples to find the top-performance architectures on NAS-Bench-101 and NAS-Bench-201, and it achieves the state-of-the-art ImageNet performance on the NASNet search space.
Incremental Learning on Growing Graphs
Authors: Yutong Feng, Jianwen Jiang, Yue Gao
Date: 2021-01-01 00:00:00
Tasks: Graph Learning, Incremental Learning, Link Prediction, Node Classification

Graphs have attracted numerous attention in varied areas and are dynamic in many scenarios. Among dynamic graphs, growing graphs with frequently expanding vertex and edge sets are typical and widely existed, e.g. the rapidly growing social networks. Confronting such growing data, existing methods on either static or dynamic graphs take the entire graph as a whole and may suffer from high computation cost and memory usage due to the continual growth of graphs. To tackle this problem, we introduce incremental graph learning (IGL), a general framework to formulate the learning on growing graphs in an incremental manner, where traditional graph learning method could be deployed as a basic model. We first analyze the problems of directly finetuning on the incremental part of graph, and theoretically discuss the unbiased and edge-preserved conditions of IGL. In our method, when the graph grows with new-coming data, we select or generate vertices and edges within restricted sizes from the previous graph to update current model together with the new data. Here, two strategies, i.e. sample-based and cluster-based, are proposed for learning with restricted time and space complexity. We conduct experiments on the node classification and link prediction tasks of multiple datasets. Experimental results and comparisons show that our method achieves satisfying performance with high efficiency on growing graphs.
Unity of Opposites: SelfNorm and CrossNorm for Model Robustness
Authors: Zhiqiang Tang, Yunhe Gao, Yi Zhu, Zhi Zhang, Mu Li, Dimitris N. Metaxas
Date: 2021-01-01 00:00:00
Tasks: Object Recognition, Unity

Studies have demonstrated the fundamental behavioral difference between human vision and Convolutional Neural Networks (CNNs). Human vision can make robust object recognition mainly using object shape (content). In contrast, CNNs are not only texture-biased but also sensitive to texture (style) noises, giving rise to security concerns in practice. To reduce CNNs' texture sensitivity and bias, this paper proposes SelfNorm and CrossNorm, respectively. SelfNorm uses attention to recalibrate feature styles by emphasizing primary styles and suppressing trivial ones. CrossNorm exchanges styles between feature channels to perform style augmentation, diversifying the content and style mixtures. SelfNorm and CrossNorm explore the opposite directions in utilizing style. Nevertheless, they complement each other and seek the same goal: model robustness. Surprisingly, they can boost the performance of corrupted and clean test data simultaneously. Extensive experiments on different tasks (classification and segmentation) and settings (supervised and semi-supervised)  show their effectiveness.
On the Marginal Regret Bound Minimization of Adaptive Methods
Authors: Wenjie Li, Guang Cheng
Date: 2021-01-01 00:00:00
Tasks: Deep Learning, Open-Ended Question Answering

Numerous adaptive algorithms such as AMSGrad and Radam have been proposed and applied to deep learning recently. However, these modifications do not improve the convergence rate of adaptive algorithms and whether a better algorithm exists still remains an open question. In this work, we propose a new motivation for designing the proximal function of adaptive algorithms, named as marginal regret bound minimization. Based on such an idea, we propose a new class of adaptive algorithms that not only achieves marginal optimality but can also potentially converge much faster than any existing adaptive algorithms in the long term. We show the superiority of the new class of adaptive algorithms both theoretically and empirically using experiments in deep learning.
Self-Pretraining for Small Datasets by Exploiting Patch Information
Authors: Zhang Chunyang
Date: 2021-01-01 00:00:00

  Deep learning tasks with small datasets are often tackled by pretraining models with large datasets on relevent tasks. Although pretraining methods mitigate the problem of overfitting, it can be difficult to find appropriate pretrained models sometimes. In this paper, we proposed a self-pretraininng method by exploiting patch information in the dataset itself without pretraining on other datasets. Our experiments show that the self-pretraining method leads to better performance than training from scratch both in the condition of not using other data.
Dual Graph Complementary Network
Authors: Chenhua Liu, Kun Zhan
Date: 2021-01-01 00:00:00
Tasks: Representation Learning

As a powerful representation learning method on graph data, graph neural networks (GNNs) have shown great popularity in tackling graph analytic problems. Although many attempts have been made in literatures to find strategies about extracting better embedding of the target nodes, few of them consider this issue from a comprehensive perspective. Most of current GNNs usually employ some single method which can commendably extract a certain kind of feature but some equally important features are often ignored. In this paper, we develop a novel dual graph complementary network (DGCN) to learn representation complementarily. We use two different branches, and inputs of the two branches are the same, which are composed of structure and feature information. At the same time, there is also a complementary relationship between the two branches. Beyond that, our extensive experiments show that DGCN outperforms state-of-the-art methods on five public benchmark datasets.
Graph View-Consistent Learning Network
Authors: Zhuolin Liao, Kun Zhan
Date: 2021-01-01 00:00:00
Tasks: Node Classification, Pseudo Label

Recent years, methods based on neural networks have made great achievements in solving large and complex graph problems. However, high efficiency of these methods depends on large training and validation sets, while the acquisition of ground-truth labels is expensive and time-consuming. In this paper, a graph view-consistent learning network (GVCLN) is specially designed for the semi-supervised learning when the number of the labeled samples is very small. We fully exploit the neighborhood aggregation capability of GVCLN and use dual views to obtain different representations. Although the two views have different viewing angles, their observation objects are the same, so their observation representations need to be consistent. For view-consistent representations between two views, two loss functions are designed besides a supervised loss. The supervised loss uses the known labeled set, while a view-consistent loss is applied to the two views to obtain the consistent representation and a pseudo-label loss is designed by using the common high-confidence predictions. GVCLN with these loss functions can obtain the view-consistent representations of the original feature. We also find that preprocessing the node features with specific filter before training is good for subsequent classification tasks. Related experiments have been done on the three citation network datasets of Cora, Citeseer, and PubMed. On several node classification tasks, GVCLN achieves state-of-the-art performance.
Robust Loss Functions for Complementary Labels Learning
Authors: Defu Liu, Guowu Yang
Date: 2021-01-01 00:00:00

In ordinary-label learning, the correct label is given to each training sample. Similarly, a complementary label is also provided for each training sample in complementary-label learning. A complementary label indicates a class that the example does not belong to. Robust learning of classifiers has been investigated from many viewpoints under label noise, but little attention has been paid to complementary-label learning. In this paper, we present a new algorithm of complementary-label learning with the robustness of loss function. We also provide two sufficient conditions on a loss function so that the minimizer of the risk for complementary labels is theoretically guaranteed to be consistent with the minimizer of the risk for ordinary labels. Finally, the empirical results validate our method’s superiority to current state-of-the-art techniques. Especially in cifar10, our algorithm achieves a much higher test accuracy than the gradient ascent algorithm, and the parameters of our model are less than half of the ResNet-34 they used.