pwc-archive/papers-with-abstracts huggingface dataset:
Efficient Regularization for Adversarially Robustness Deep ReLU Networks
Authors: Charles Jin, Martin Rinard
Date: 2021-09-29 00:00:00

We present a regularization functional for deep neural networks with ReLU activations, and propose regularizers that encourage networks which are smooth not only in their predictions but also their decision boundaries. We evaluate the stability of our networks against the standard set of $\ell_2$ and $\ell_\infty$ norm-bounded adversaries, as well as several recently proposed perception-based adversaries, including spatial, recoloring, JPEG, and a learned neural threat model. Crucially, our models are simultaneously robust against multiple state-of-the-art adversaries, suggesting that the robustness generalizes well to \textit{unseen} adversaries. Furthermore, our techniques do not rely on adversarial training and are thus very efficient, incurring overhead on par with two additional parallel passes through the network. On CIFAR-10, we obtain our results after training for only 4 hours, while the next-best performing baseline requires nearly 25 hours of training. To the best of our knowledge, this work presents the first technique to achieve robustness against adversarial perturbations \textit{without} adversarial training.
TS-BERT: A fusion model for Pre-trainning Time Series-Text Representations
Authors: Jiahao Qin, Lu Zong
Date: 2021-09-29 00:00:00
Tasks: Prediction, Time Series, Time Series Analysis

There are many tasks to use news text information and stock data to predict the crisis. In the existing research, the two usually play one master and one follower in the prediction task.
Use one of the news text and the stock data as the primary information source for the prediction task and the other as the auxiliary information source.
This paper proposes a fusion model for pre-training time series-Text representations, in which news text and stock data have the same status and are treated as two different modes to describe crises. Our model has achieved the best results in the task of predicting financial crises.
Manifold Distance Judge, an Adversarial Samples Defense Strategy Based on Service Orchestration
Authors: Mengxin Zhang, Xiaofeng QIU
Date: 2021-09-29 00:00:00

Deep neural networks (DNNs) are playing an increasingly significant role in the modern world. However, they are weak to adversarial examples that are generated by adding specially crafted perturbations. Most defenses against adversarial examples focused on refining the DNN models, which often sacrifice the performance and computational cost of models on benign samples. In this paper, we propose a manifold distance detection method to distinguish between legitimate samples and adversarial samples by measuring the different distances on the manifold. The manifold distance detection method neither modifies the protected models nor requires knowledge of the process for generating adversarial samples. Inspired by the effectiveness of the manifold distance detection, we demonstrated a well-designed orchestrated defense strategy, named Manifold Distance Judge (MDJ), which selects the best image processing method that will effectively expand the manifold distance between legitimate and adversarial samples, and thus, enhances the performance of the following manifold distance detection method. Tests on the ImageNet dataset, the MDJ is effective against the most adversarial samples under whitebox, graybox, and blackbox attack scenarios. We show empirically that the orchestration strategy MDJ is significantly better than Feature Squeezing on the recall rate. Meanwhile, MDJ achieves high detection rates against CW attack and DI-FGSM attack.
Discovering the neural correlate informed nosological relation among multiple neuropsychiatric disorders through dual utilisation of diagnostic information
Authors: Wenjun Bai, Tomoki Tokuda, Okito Yamashita, Junichiro Yoshimoto
Date: 2021-09-29 00:00:00
Tasks: Density Estimation, Diagnostic, Functional Connectivity, Relation

The unravelled nosological relation among diverse types of neuropsychiatric disorders serves as an important precursor in advocating the dimensional approach to psychiatric classification. Leveraging high-dimensional abnormal resting-state functional connectivity, the crux of mining corresponded nosological relations is to derive a low-dimensional embedding space that preserves the diagnostic attributes of represented disorders. To accomplish this goal, we seek to exploit the available diagnostic information in learning the optimal embedding space by proposing a novel type of conditional variational auto-encoder that incorporates dual utilisation of diagnostic information. Encouraged by the achieved promising results in challenging the conventional approaches in low dimensional density estimation of synthetic functional connectivity features, we further implement our approach on two empirical neuropsychiatric neuroimaging datasets and discover a reliable nosological relation among autism spectrum disorder, major depressive disorder, and schizophrenia.
Unsupervised Domain Adaptation By Optimal Transportation Of Clusters Between Domains
Authors: Yang Liu, Zhipeng Zhou, Lei Shang, Baigui Sun, Hao Li, Rong Jin
Date: 2021-09-29 00:00:00
Tasks: Attribute, Clustering, Domain Adaptation, Transfer Learning, Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) aims to transfer the knowledge from a labeled source domain to an unlabeled target domain. Typically, to guarantee desirable knowledge transfer, aligning the distribution  between source and target domain from a global perspective is widely adopted in UDA. Recent researchers further point out the importance of local-level alignment and borrow the experience from Optimal Transport (OT) theory to construct instance-pair alignment.
 However, existing OT-based algorithms are limited to resolve class imbalance challenge and require a huge computation cost when considering a  large-scale training situation. 
In this paper, we address these two issues by proposing a Clustering-based Optimal Transport (COT) algorithm, which formulates the alignment procedure as an Optimal Transport problem by capturing the fine-grained attribute alignment. Concretely, COT innovatively designs the loss derived from discrete Kantorovich dual form to construct a mapping between clustering centers in source and target domain, which simultaneously eliminates the negative effect brought by class imbalance and reduces the computation cost on the basis of theoretical analysis. Finally, our COT together with some previous UDA methods achieve superior performance on several benchmarks.
OUT-OF-DISTRIBUTION CLASSIFICATION WITH ADAPTIVE LEARNING OF LOW-LEVEL CONTEXTUAL FEATURES
Authors: Neeraj Tiwari, Tushar Bangoria, Rajan Vaja
Date: 2021-09-29 00:00:00

Humans can inherently identify what is unknown to them, but the existing Neural Network (NN) is still lacking in this aspect. Out-of-Distribution (OOD) classification is an incredibly challenging problem for any NN model. In General, any model tries to predict the OOD samples from the labels used for training only, but that is not acceptable for AGI (Artificial General Intelligence) [Fjelland(2020)]. There are several kinds of research already done to avoid this issue and build a model to predict the OOD samples, existing baseline work like 1) Thresholding SoftMax, 2) train model by adding extra OOD class as a label, or 3) Mahalanobis distance-based approach. All existing approach uses the CNN to get the spatial feature information and channel-wise information within the local receptive field at each layer. Here in this paper, we have proposed a method to learn the features of In-class and OOD sample’s features with global receptive field among channels to learn the spatial relationship with modified SEnet block. Broadly, our model learns the interdependencies between channels with adaptive recalibration of the weights of stacked channels at each layer. To give more weightage to the In-class samples, we uniformly normalized the OOD samples with the total number of known class samples and trained our model to suppress the OOD class probability with a simple and effective loss function. We did our experiments for our model with MNIST and F-MNIST as In-class samples and EMNIST, KMNIST, not-MNIST, Omniglot, Uniform Noise, Gaussian Noise as OOD samples.
NETWORK INSENSITIVITY TO PARAMETER NOISE VIA PARAMETER ATTACK DURING TRAINING
Authors: Julian Büchel, Fynn Firouz Faber, Dylan Richard Muir
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

Neuromorphic neural network processors, in the form of compute-in-memory crossbar arrays of memristors, or in the form of subthreshold analog and mixed-signal ASICs, promise enormous advantages in compute density and energy efficiency for NN-based ML tasks. However, these technologies are prone to computational non-idealities, due to process variation and intrinsic device physics. This degrades the task performance of networks deployed to the processor, by introducing parameter noise into the deployed model. While it is possible to calibrate each device, or train networks individually for each processor, these approaches are expensive and impractical for commercial deployment. Alternative methods are therefore needed to train networks that are inherently robust against parameter variation, as a consequence of network architecture and parameters. We present a new network training algorithm that attacks network parameters during training, and promotes robust performance during inference in the face of random parameter variation. Our approach introduces a loss regularization term that penalizes the susceptibility of a network to weight perturbation. We compare against previous approaches for producing parameter insensitivity such as dropout, weight smoothing and introducing parameter noise during training. We show that our approach produces models that are more robust to random mismatch-induced parameter variation as well as to targeted parameter variation. Our approach finds minima in flatter locations in the weight-loss landscape compared with other approaches, highlighting that the networks found by our technique are less sensitive to parameter perturbation. Our work provides an approach to deploy neural network architectures to inference devices that suffer from computational non-idealities, with minimal loss of performance. This method will enable deployment at scale to novel energy-efficient computational substrates, promoting cheaper and more prevalent edge inference.
NASPY: Automated Extraction of Automated Machine Learning Models
Authors: Xiaoxuan Lou, Shangwei Guo, Jiwei Li, Yaoxin Wu, Tianwei Zhang
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: BIG-bench Machine Learning, Model extraction, Neural Architecture Search

We present NASPY, an end-to-end adversarial framework to extract the networkarchitecture of deep learning models from Neural Architecture Search (NAS). Existing works about model extraction attacks mainly focus on conventional DNN models with very simple operations, or require heavy manual analysis with lots of domain knowledge.  In contrast, NASPY introduces seq2seq models to automatically identify novel and complicated operations (e.g., separable convolution,dilated convolution) from hardware side-channel sequences. We design two models (RNN-CTC and transformer), which can achieve only 3.2% and 11.3% error rates for operation prediction.  We further present methods to recover the model hyper-parameters and topology from the operation sequence .  With these techniques, NASPY is able to extract the complete NAS model architecture with high fidelity and automation, which are rarely analyzed before.
Aug-ILA: More Transferable Intermediate Level Attacks with Augmented References
Authors: Chiu Wai Yan, Dit-yan Yeung
Date: 2021-09-29 00:00:00
Tasks: Adversarial Attack, Image Augmentation

An intriguing property of deep neural networks is that adversarial attacks can transfer across different models. Existing methods such as the Intermediate Level Attack (ILA) further improve black-box transferability by fine-tuning a reference adversarial attack, so as to maximize the perturbation on a pre-specified layer of the source model. In this paper, we revisit ILA and evaluate the effect of applying augmentation to the images before passing them to ILA. We start by looking into the effect of common image augmentation techniques and exploring novel augmentation with the aid of adversarial perturbations. Based on the observations, we propose Aug-ILA, an improved method that enhances the transferability of an existing attack under the ILA framework. Specifically, Aug-ILA has three main characteristics: typical image augmentation such as random cropping and resizing applied to all ILA inputs, reverse adversarial update on the clean image, and interpolation between two attacks on the reference image. Our experimental results show that Aug-ILA outperforms ILA and its subsequent variants, as well as state-of-the-art transfer-based attacks, by achieving $96.99\%$ and $87.84\%$ average attack success rates with perturbation budgets $0.05$ and $0.03$, respectively, on nine undefended models.
AdaAug: Learning Class- and Instance-adaptive Data Augmentation Policies
Authors: Tsz-Him Cheung, Dit-yan Yeung
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Data Augmentation

Data augmentation is an effective way to improve the generalization capability of modern deep learning models. However, the underlying augmentation methods mostly rely on handcrafted operations. Moreover, an augmentation policy useful to one dataset may not transfer well to other datasets. Therefore, Automated Data Augmentation (AutoDA) methods, like \textit{AutoAugment} and \textit{Population-based Augmentation}, have been proposed recently to automate the process of searching for optimal augmentation policies. However, the augmentation policies found are not adaptive to the dataset used, hindering the effectiveness of these AutoDA methods. In this paper, we propose a novel AutoDA method called \texttt{AdaAug} to efficiently learn adaptive augmentation policies in a class-dependent and potentially instance-dependent manner. Our experiments show that the adaptive augmentation policies learned by our method transfer well to unseen datasets such as the Oxford Flowers, Oxford-IIT Pets, FGVC Aircraft, and Stanford Cars datasets when compared with other AutoDA baselines. In addition, our method also achieves state-of-the-art performance on the CIFAR-10, CIFAR-100, and SVHN datasets.
Privacy-preserving Task-Agnostic Vision Transformer for Image Processing
Authors: Boah Kim, Jeongsol Kim, Jong Chul Ye
Date: 2021-09-29 00:00:00
Tasks: Multi-Task Learning, Privacy Preserving

Distributed collaborative learning approaches such as federated and split learning have attracted significant attention lately due to their ability to train neural networks using data from multiple sources without sharing data. However, they are not usually suitable in applications where each client carries out different tasks with its own data. Inspired by the recent success of Vision Transformer (ViT), here we present a new distributed learning framework for image processing applications, allowing clients to learn multiple tasks with their private data. The key idea arises from a novel task-agnostic Vision Transformer that is introduced to learn the global attention independent of specific tasks. Specifically, by connecting task-specific heads and tails at client sides to a task-agnostic Transformer body at a server side, each client learns a translation from its own task to a common representation, while the Transformer body learns global attention between the features embedded in the common representation. To enable decomposition between the task-specific and common representation, we propose an alternating training strategy in which task-specific learning for the heads and tails is run on the clients by fixing the Transformer, which alternates with task-agnostic learning for the Transformer on the server by freezing the heads and tails. Experimental results on multi-task learning for various image processing show that our method synergistically improves the performance of the task-specific network of each client while maintaining privacy.
Unifying Top-down and Bottom-up for Recurrent Visual Attention
Authors: Gang Chen
Date: 2021-09-29 00:00:00
Tasks: Q-Learning

The idea of using the recurrent neural network for visual attention has gained popularity in computer vision community. Although the recurrent visual attention model (RAM) leverages the glimpses with more large patch size to increasing its scope, it may result in high variance and instability. For example, we need the Gaussian policy with high variance to explore object of interests in a large image, which may cause randomized search and unstable learning. In this paper, we propose to unify the top-down and bottom-up attention together for recurrent visual attention. Our model exploits the image pyramids and Q-learning to select regions of interests in the top-down attention mechanism, which in turn to guide the policy search in the bottom-up approach. In addition, we add another two constraints over the bottom-up recurrent neural networks for better exploration. We train our model in an end-to-end reinforcement learning framework, and evaluate our method on visual classification tasks. The experimental results outperform convolutional neural networks (CNNs) baseline and the bottom-up recurrent models with visual attention.
Learning Neural Processes on the Fly
Authors: Younghwa Jung, Zhenyuan Yuan, Seung-Woo Seo, Minghui Zhu, Seong-Woo Kim
Date: 2021-09-29 00:00:00
Tasks: Meta-Learning

Deep neural networks (DNNs) have performed impressively on a wide range of tasks, but they usually require a significant number of training samples to achieve good performance. Thus, DNNs do not work well in low-data regimes because they tend to overfit a small dataset and make poor predictions. In contrast, shallow neural networks (SNNs) generally are robust against overfitting in low-data regimes and converge more quickly than DNNs, but they struggle to represent very complex systems. Hence, DNNs and SNNs have a complementary relationship, and combining their benefits can provide fast-learning capability with high asymptotic performance, as meta-learning does. However, aggregating heterogeneous methods with opposite properties is not trivial, as it can make the combined method inferior to each base method. In this paper, we propose a new algorithm called anytime neural processes that combines DNNs and SNNs and can work in both low-data and high-data regimes. To combine heterogeneous models effectively, we propose a novel aggregation method based on a generalized product-of-exports and a winner-take-all gate network. Moreover, we discuss the theoretical basis of the proposed method. Experiments on a public dataset show that the proposed method achieves comparable performance with other state-of-the-art methods.
SCformer: Segment Correlation Transformer for Long Sequence Time Series Forecasting
Authors: Dazhao Du, Bing Su, Zhewei Wei
Date: 2021-09-29 00:00:00
Tasks: Management, Time Series, Time Series Analysis, Time Series Forecasting

Long-term time series forecasting is widely used in real-world applications such as financial investment, electricity management and production planning. Recently, transformer-based models with strong sequence modeling ability have shown the potential in this task. However, most of these methods adopt point-wise dependencies discovery, whose complexity increases quadratically with the length of time series, which easily becomes intractable for long-term prediction. This paper proposes a new Transformer-based model called SCformer, which replaces the canonical self-attention with efficient segment correlation attention (SCAttention) mechanism. SCAttention divides time series into segments by the implicit series periodicity and utilizes correlations between segments to capture long short-term dependencies. Besides, we design a dual task that restores past series with the predicted future series to make SCformer more stable. Extensive experiments on several datasets in various fields demonstrate that our SCformer outperforms other Transformer-based methods and training with the additional dual task can enhance the generalization ability of the prediction model.
A Two-Stage Data-Free Adversarial Patch Generation Framework
Authors: Jiawei Liu, Hang Gao, Yunfeng Hu, Xun Gong
Date: 2021-09-29 00:00:00
Tasks: Vocal Bursts Valence Prediction

General adversarial patch generation (APG) methods rely on training datasets of target models and are not applicable to data-free scenarios. This article presents a two-stage APG framework that exploits a determined proxy dataset in place of an unknown training dataset. The proxy dataset selection stage calculates the proposed average patch saliency (APS) of each available dataset to select a high-APS proxy dataset that can guarantee patches' fooling abilities. Then, the patch generation stage applies the proposed data-free Expectation over Transformation (DF-EoT) as the APG method in case only low-APS datasets are available. Evaluation results show that the determined high-APS proxy datasets enable EoT (benchmark APG method) to generate patches of comparable fooling abilities to patches utilising training datasets, and DF-EoT can further improve the fooling abilities for both low-APS and high-APS proxy datasets. Specifically, DF-EoT enhances average targeted fooling rates (ATFR) of patches utilising a low-APS dataset from 42.71% of EoT to 78.34% on target model VGG-19 and increases ATFR from 62.57% to 84.33% with a high-APS dataset on Inception-v1.
S$^3$ADNet: Sequential Anomaly Detection with Pessimistic Contrastive Learning
Authors: Quexuan Zhang, Yukio Ohsawa
Date: 2021-09-29 00:00:00
Tasks: Anomaly Detection, Contrastive Learning

Anomalies are commonly found in sequential data generated by real-world applications, such as cyberattacks in network traffic, human activity changes in wearable sensors. Thanks to the development of computing technology, many impressive results have been obtained from deep learning-based anomaly detection approaches in recent years. This paper proposes a simple neural network framework for detecting anomalies on sequential data, called $S$elf-$S$upervised $S$equential $A$nomaly $D$etection $N$etwork (S$^3$ADNet). S$^3$ADNet first extracts the representations from each data point by performing feature augmentation for contrastive learning; then captures the contextual information from the sequential data points for estimating anomaly probabilities by optimizing the context-adaptive objective. Here, we design a novel loss function based on a pessimistic policy, considering that only anomalies can affect the contextual relationships in sequences. Our proposed method outperformed other state-of-the-art approaches on the benchmark datasets by F1-score with a more straightforward architecture.