pwc-archive/papers-with-abstracts huggingface dataset:
NAS-Bench-Zero: A Large Scale Dataset for Understanding Zero-Shot Neural Architecture Search
Authors: Hanlin Chen, Ming Lin, Xiuyu Sun, Hao Li
Date: 2021-09-29 00:00:00
Tasks: Benchmarking, Neural Architecture Search

Zero-shot Neural Architecture Search (ZS-NAS) is a recently developed low-cost NAS framework which identifies top-performer neural architectures from a large candidate pool without training their parameters. Despite its popularity in recent NAS literatures, the effectiveness of ZS-NAS has not been comprehensively understood. Previous works analyze ZS-NAS methods on NAS benchmark datasets such as NAS-Bench-101/201/301 which are initially designed for learning network topology with irregular connections. However, most modern state-of-the-art networks as well as popular classical ones are designed in more conventional, well-established search spaces such as ResNet (RS) and MobileNet (MB) search space. This imposes a significant gap between the benchmark dataset and real-world practice, hindering a deeper understanding of ZS-NAS. In this work, we aim to bridge the gap systematically. First, we collect a novel large-scale dataset termed NAS-Bench-Zero for benchmarking and understanding popular ZS-NAS methods in the conventional RS/MB search spaces. Then the characteristics of these ZS-NAS methods are extensively examined from various aspects. Notably, we find that: 1) the performance of ZS-NAS on NAS-Bench-101/201/301 cannot transfer to RS/MB search spaces; 2) A proxy with higher ranking correlation score may actually perform worse in constrained NAS; 3) existing zero-shot proxies cannot outperform naive proxies such as FLOPs/params in RS/MB search spaces; 4) Top best zero-shot proxies as well as FLOPs/params compensate each other. Based on these new discoveries, we propose i) a novel hybrid zero-shot proxy which outperforms existing ones by a large margin and is transferable among popular search spaces; ii) a new index for better measuring the true performance of ZS-NAS proxies in constrained NAS. Source code and the NAS-Bench-Zero dataset will be released after publication.
Hot-Refresh Model Upgrades with Regression-Free Compatible Training in Image Retrieval
Authors: Binjie Zhang, Yixiao Ge, Yantao Shen, Yu Li, Chun Yuan, Xuyuan Xu, Yexin Wang, Ying Shan
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Image Retrieval, regression, Retrieval

The task of hot-refresh model upgrades of image retrieval systems plays an essential role in the industry but has never been investigated in academia before. Conventional cold-refresh model upgrades can only deploy new models after the gallery is overall backﬁlled, taking weeks or even months for massive data. In contrast, hot-refresh model upgrades deploy the new model immediately and then gradually improve the retrieval accuracy by backﬁlling the gallery on-the-ﬂy. Compatible training has made it possible, however, the problem of model regression with negative ﬂips poses a great challenge to the stable improvement of user experience. We argue that it is mainly due to the fact that new-to-old positive query-gallery pairs may show less similarity than new-to-new negative pairs. To solve the problem, we introduce a Regression-Free Compatible Training (RFCT) method to properly constrain the feature compatibility while reducing negative ﬂips. The core is to encourage the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs. An efﬁcient uncertainty-based backﬁlling strategy is further introduced to fasten accuracy improvements. Extensive experiments on large-scale retrieval benchmarks (e.g., Google Landmark) demonstrate that our RFCT effectively alleviates the model regression for one more step towards seamless model upgrades.
Learning Representations of Partial Subgraphs by Subgraph InfoMax
Authors: Dongkwan Kim, Jiho Jin, Jaimeen Ahn, Alice Oh
Date: 2021-09-29 00:00:00

Subgraphs are important substructures of graphs, but learning their representations has not been studied well. Particularly, when we have partial subgraphs, existing node- or subgraph-level message-passing is likely to produce suboptimal representations. In this paper, we propose Intra- and Inter-Subgraph InfoMax, a model that learns subgraph representations under incomplete observation. Our model employs subgraph summaries at two different levels while maximizing the mutual information between the subgraph summaries and the node representations. By doing so, we reconstruct the representation of the underlying subgraph and improve its expressiveness from different angles of the local-global structure. We conduct experiments on three real-world datasets under training and evaluation protocols designed for this problem. Experimental results show that our model outperforms baselines in all settings.
Relative Instance Credibility Inference for Learning with Noisy Labels
Authors: Yikai Wang, Xinwei Sun, Yanwei Fu
Date: 2021-09-29 00:00:00
Tasks: Learning with noisy labels

The existence of noisy labels usually leads to the degradation of generalization and robustness of neural networks in supervised learning. In this paper, we propose to use a simple theoretically guaranteed sample selection framework as a plug-in module to handle noisy labels. Specifically, we re-purpose a sparse linear model with incidental parameters as a unified Relative Instance Credibility Inference (RICI) framework, which will detect and remove outliers in the forward pass of each mini-batch and use the remaining instances to train the network. The credibility of instances is measured by the sparsity of incidental parameters, which can be ranked among other instances within each mini-batch to get a relatively consistent training mini-batch. The proposed RICI framework yields two variants that enjoy superior performance on the symmetric and asymmetric noise settings, respectively. We prove that our RICI can theoretically recover the clean data. Experimental results on several benchmark datasets and a real-world noisy dataset show the effectiveness of our framework.
An Improved Composite Functional Gradient Learning by Wasserstein Regularization for Generative adversarial networks
Authors: Chang Wan, Yanwei Fu, Ke Fan, Jinshan Zeng, Ming Zhong, Riheng Jia, MingLu Li, ZhongLong Zheng
Date: 2021-09-29 00:00:00
Tasks: Image Generation, regression

Generative adversarial networks (GANs) are usually trained by a minimax game which is notoriously and empirically known to be unstable. Recently, a totally new methodology called 
Composite Functional Gradient Learning (CFG) provides an alternative
theoretical foundation for training GANs more stablely by employing
a strong discriminator with logistic regression and functional gradient learning for the generator. 
However, the discriminator using logistic regression from the CFG framework is gradually 
hard to discriminate between real and fake images while the training steps go on.
To address this problem, our key idea and contribution are to introduce
the Wasserstein distance regularization into the CFG framework for the discriminator. This gives us a novel
improved CFG formulation with more competitive generate image quality. In particular, we provide an intuitive explanation using logistic regression with Wasserstein regularization. The method helps to enhance the model gradients 
in training GANs to archives better image quality. Empirically, we compare our improved CFG with the original
version. We show that the standard CFG is easy to stick into mode collapse problem, while our improved CFG works much better
thanks to the newly added Wasserstein  distance regularization. We conduct extensive
experiments for image generation on different benchmarks, and it shows
the efficacy of our improved CFG method.
PhaseFool: Phase-oriented Audio Adversarial Examples via Energy Dissipation
Authors: Ziyue Jiang, Yi Ren, Zhou Zhao
Date: 2021-09-29 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Sentence, speech-recognition, Speech Recognition

Audio adversarial attacks design perturbations onto inputs that lead an automatic speech recognition (ASR) model to predict incorrect outputs. Current audio adversarial attacks optimize perturbations with different constraints (e.g. lp-norm for waveform or the principle of auditory masking for magnitude spectrogram) to achieve their imperceptibility. Since phase is not relevant for speech recognition, the existing audio adversarial attacks neglect the influence of phase spectrogram. In this work, we propose a novel phase-oriented algorithm named PhaseFool that can efficiently construct imperceptible audio adversarial examples with energy dissipation. Specifically, we leverage the spectrogram consistency of short-time Fourier transform (STFT) to adversarially transfer phase perturbations to the adjacent frames of magnitude spectrogram and dissipate the energy that is crucial for ASR systems. Moreover, we propose a weighted loss function to improve the imperceptibility of PhaseFool. Experimental results demonstrate that PhaseFool can inherently generate full-sentence imperceptible audio adversarial examples with the 100% targeted success rate within 500 steps on average (9.24x speed-up over current state-of-the-art imperceptible counterparts), which is verified through a human study. Most importantly, our PhaseFool is the first to exploit the phase-oriented energy dissipation in the audio adversarial examples rather than add perturbations on the audio waveform like most previous works.
Continual Learning Using Task Conditional Neural Networks
Authors: Honglin Li, Frieder Ganz, David J. Sharp, Payam M. Barnaghi
Date: 2021-09-29 00:00:00
Tasks: Continual Learning, Mixture-of-Experts

Conventional deep learning models have limited capacity in learning multiple tasks sequentially. The issue of forgetting the previously learned tasks in continual learning is known as catastrophic forgetting or interference. When the input data or the goal of learning changes, a continual model will learn and adapt to the new status. However, the model will not remember or recognise any revisits to the previous states. This causes performance reduction and re-training curves in dealing with periodic or irregularly reoccurring changes in the data or goals. Dynamic approaches, which assign new neuron resources to the upcoming tasks, are introduced to address this issue. However, most of the dynamic methods need task information about the upcoming tasks during the inference phase to activate the corresponding neurons. To address this issue, we introduce Task Conditional Neural Network which allows the model to identify the task information automatically. The proposed model can continually learn and embed new tasks into the model without losing the information about previously learned tasks. We evaluate the proposed model combined with the mixture of experts approach on the MNIST and CIFAR100 datasets and show how it significantly improves the continual learning process without requiring task information in advance.
Mutual Information Continuity-constrained Estimator
Authors: Tsun-An Hsieh, Cheng Yu, Ying Hung, Chung-Ching Lin, Yu Tsao
Date: 2021-09-29 00:00:00
Tasks: Density Estimation

The estimation of mutual information (MI) is vital to a variety of applications in machine learning. Recent developments in neural approaches have shown encouraging potential in estimating the MI between high-dimensional variables based on their latent representations. However, these estimators are prone to high variances owing to the inevitable outlier events. Recent approaches mitigate the outlier issue by smoothing the partition function using clipping or averaging strategies; however, these estimators either break the lower bound condition or sacrifice the level of accuracy. Accordingly, we propose Mutual Information Continuity-constrained Estimator (MICE). MICE alternatively smooths the partition function by constraining the Lipschitz constant of the log-density ratio estimator, thus alleviating the induced variances without clipping or averaging. Our proposed estimator outperforms most of the existing estimators in terms of bias and variance in the standard benchmark. In addition, we propose an experiment extension based on the standard benchmark, where variables are drawn from a multivariate normal distribution with correlations between each sample in a batch. The experimental results imply that when the i.i.d. assumption is unfulfilled, our proposed estimator can be more accurate than the existing approaches in which the MI tends to be underestimated. Finally, we demonstrate that MICE mitigates mode collapse in the kernel density estimation task.
Learning Universal User Representations via Self-Supervised Lifelong Behaviors Modeling
Authors: Bei Yang, Ke Liu, Xiaoxiao Xu, Renjun Xu, Hong Liu, Huan Xu
Date: 2021-09-29 00:00:00
Tasks: Contrastive Learning, Dimensionality Reduction, Representation Learning, Self-Supervised Learning

Universal user representation is an important research topic in industry, and is widely used in diverse downstream user analysis tasks, such as user profiling and user preference prediction. With the rapid development of Internet service platforms, extremely long user behavior sequences have been accumulated. However, existing researches have little ability to model universal user representation based on lifelong behavior sequences since user registration. In this study, we propose a novel framework called Lifelong User Representation Model (LURM) to tackle this challenge. Specifically, LURM consists of two cascaded sub-models: (i) Bag of Interests (BoI) encodes user behaviors in any time period into a sparse vector with super-high dimension (eg. 10^5); (ii) Self-supervised Multi-anchor Encoder Network (SMEN) maps sequences of BoI features to multiple low-dimensional user representations by contrastive learning. SMEN achieves almost lossless dimensionality reduction with the main help of a novel multi-anchor module which can learn different aspects of user preferences. Experiments on several benchmark datasets show that our approach can outperform state-of-the-art unsupervised representation methods in downstream tasks.
Metrics Matter: A Closer Look on Self-Paced Reinforcement Learning
Authors: Pascal Klink, Haoyi Yang, Jan Peters, Joni Pajarinen
Date: 2021-09-29 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Curriculum reinforcement learning (CRL) allows to solve complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. However, the generation of such task sequences is largely governed by application assumptions, often preventing a theoretical investigation of existing approaches. Recently, Klink et al. (2021) showed how self-paced learning induces a principled interpolation between task distributions in the context of RL, resulting in high learning performance. So far, this interpolation is unfortunately limited to Gaussian distributions. Here, we show that on one side, this parametric restriction is insufficient in many learning cases but that on the other, the interpolation of self-paced RL (SPRL) can be degenerate when not restricted to this parametric form. We show that the introduction of concepts from optimal transport into SPRL prevents aforementioned issues. Experiments demonstrate that the resulting introduction of metric structure into the curriculum allows for a well-behaving non-parametric version of SPRL that leads to stable learning performance across tasks.
Post-Training Quantization Is All You Need to Perform Cross-Platform Learned Image Compression
Authors: Dailan He, Ziming Yang, Yan Wang, Yuan Chen, Qi Zhang, Hongwei Qin
Date: 2021-09-29 00:00:00
Tasks: All, Image Compression, Quantization

It has been witnessed that learned image compression has outperformed conventional image coding techniques and tends to be practical in industrial applications. One of the most critical issues preventing it from being practical is the non-deterministic calculation, which makes the probability prediction cross-platform inconsistent and frustrates successful decoding. We propose to solve this problem by introducing well-developed post-training quantization and making the model inference integer-arithmetic-only, which is much simpler than presently existing training and fine-tuning based approaches yet still keeps the superior rate-distortion performance of learned image compression. Based on that, we further improve the discretization of the entropy parameters and extend the deterministic inference to fit Gaussian mixture models. With our proposed methods, the current state-of-the-art image compression models can infer in a cross-platform consistent manner, which makes the further development and practice of learned image compression more promising.
On The Quality Assurance Of Concept-Based Representations
Authors: Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Mateja Jamnik
Date: 2021-09-29 00:00:00
Tasks: Disentanglement

Recent work on Explainable AI has focused on concept-based explanations, where deep learning models are explained in terms of high-level units of information, referred to as concepts. In parallel, the field of disentanglement learning has explored the related notion of finding underlying factors of variation in the data that have interpretability properties. Despite their overlapping purpose, the metrics to evaluate the quality of concepts and factors of variation in the two fields are not aligned, hindering a systematic comparison. In this paper we consider factors of variation as concepts and thus unify the notations in concept and disentanglement learning. Next, we propose metrics for evaluating the quality of concept representations in both approaches, in the presence and in the absence of ground truth concept labels. Via our proposed metrics, we benchmark state-of-the-art methods from both families, and propose a set of guidelines to determine the impact that supervision may have on the quality of learnt concept representations.
Causal Triple Attention Time Series Forecasting
Authors: Zhixuan Chu, Tan Yan, Yue Wu, Yi Xu, Cheng Zhang, Yulin kang
Date: 2021-09-29 00:00:00
Tasks: Causal Inference, Time Series, Time Series Analysis, Time Series Forecasting

Time series forecasting has historically been a key area of academic research and industrial applications. In multi-horizon and multi-series forecasting tasks, accurately capturing the local information in a sequence and effectively sharing global information across different sequences are very challenging, due to the complex dependencies over time in a long sequence and the heterogeneous nature across multiple time series. In this paper, from the perspective of causal inference, we give a theoretical analysis of these difficulties and establish a causal graph to identify the confounding relationship that generates harmful bias and misleads the time series model to capture the spurious correlations. We propose a causal triple attention time series forecasting model with three interpretable attention modules, which leverages the front-door adjustment to remove the confounding effect and help the model effectively utilize the local and global temporal information.  We evaluate the performance of our model on four benchmark datasets and the results demonstrate the superiority over the state-of-the-art methods.
Mutual Information Estimation as a Difference of Entropies for Unsupervised Representation Learning
Authors: Jaeill Kim, Wonjong Rhee
Date: 2021-09-29 00:00:00
Tasks: Mutual Information Estimation, Representation Learning

Contrastive loss has been successfully exploited in the latest visual unsupervised representation learning methods. Contrastive loss is based on a lower-bound estimation of mutual information where its known limitations include batch size dependency expressed as $O(log (n))$. It is also commonly known as negative sampling size problem. To cope with the limitation, non-contrastive methods have been proposed and they have been shown to achieve outstanding performance. The non-contrastive methods, however, are limited in that they are not based on principled designs and their learning dynamics can be unstable. In this work, we derive a principled non-contrastive method where mutual information is estimated as a difference of entropies and thus no need for negative sampling. With our best knowledge, this is the first successful implementation of difference of entropies for visual unsupervised representation learning. Our method performs on par with or better than the state-of-the-art contrastive and non-contrastive methods. The main idea of our approach is to extend Shannon entropy $H(\rmZ)$ to von Neumann entropy $S(\rmZ)$. The von Neumann entropy can be shown to be a lower bound of Shannon entropy and it can be stably estimated with a small sample size. Additionally, we prove that the conditional entropy term $H(\rmZ_1|\rmZ_2)$ is upper bounded by the negative cosine similarity for the case of weak Gaussian noise augmentation. Even though the derivation is limited to a special case of augmentation, it provides a justification of cosine similarity as the measure between positive samples.
Modality Laziness: Everybody's Business is Nobody's Business
Authors: Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Yue Wang, Yang Yuan, Hang Zhao
Date: 2021-09-29 00:00:00

Models fusing multiple modalities receive more information and can outperform their uni-modal counterparts. However, existing multi-modal training approaches often suffer from learning insufficient representations of each modality.  We theoretically analyze this phenomenon and prove that with more modalities, the models quickly saturate and ignore the features that are hard-to-learn but important. We name this problem of multi-modal training, \emph{Modality Laziness}. The solution to this problem depends on a notion called paired feature. If there exist no paired features in the data, one may simply run independent training on each modality. Otherwise, we propose Uni-Modal Teacher (UMT), which distills the pre-trained uni-modal features to the corresponding parts in multi-modal models, as a pushing force to tackle the laziness problem. We empirically verify that we can achieve competitive performance on various multi-modal datasets in light of this dichotomy.
Disentangled Mask Attention in Transformer
Authors: Jen-Tzung Chien, Yu-Han Huang
Date: 2021-09-29 00:00:00
Tasks: Disentanglement, Machine Translation, Translation

Transformer conducts self attention which has achieved state-of-the-art performance in many applications. Multi-head attention in transformer basically gathers the features from individual tokens in input sequence to form the mapping to output sequence. There are twofold weaknesses in learning representation using transformer. First, due to the natural property that attention mechanism would mix up the features of different tokens in input and output sequences, it is likely that the representation of input tokens contains redundant information. Second, the patterns of attention weights between different heads tend to be similar, the representation capacity of the model might be bounded. To strengthen the sequential learning representation, this paper presents a new disentangled mask attention in transformer where the redundant features are reduced and the semantic information is enriched. Latent disentanglement in multi-head attention is learned. The attention weights are filtered by a mask which is optimized by semantic clustering. The proposed attention mechanism is implemented for sequential learning according to the clustered disentanglement objective. The experiments on machine translation show the merit of this disentangled transformer in sequence-to-sequence learning tasks.