pwc-archive/papers-with-abstracts huggingface dataset:
Iterative Bilinear Temporal-Spectral Fusion for Unsupervised Representation Learning in Time Series
Authors: Ling Yang, Shenda Hong, Luxia Zhang
Date: 2021-09-29 00:00:00
Tasks: Anomaly Detection, Contrastive Learning, Data Augmentation, Representation Learning, Time Series, Time Series Analysis

Unsupervised representation learning for multivariate time series has practical significances, but it is also a challenging problem because of its complex dynamics and sparse annotations. Existing works mainly adopt the framework of contrastive learning and involve the data augmentation techniques to sample positives and negatives for contrastive training. However, their designs of representation learning framework have two drawbacks. First, we revisit the augmentation methods for time series of existing works and note that they mostly use segment-level augmentation derived from time slicing, which may bring about sampling bias and incorrect optimization with false negatives due to the loss of global context. Second, they all pay no attention to incorporate the spectral information and temporal-spectral relations in feature representation. To address these problems, we propose a novel framework, namely Bilinear Temporal-Spectral Fusion (BTSF).
In contrast to segment-level augmentation, we utilize the instance-level augmentation by simply applying dropout on the entire time series for better preserving global context and capturing long-term dependencies. Also, an iterative bilinear temporal-spectral fusion module is devised to explicitly encode the affinities of abundant time-frequency pairs and iteratively refine representations of time series through cross-domain interactions with Spectrum-to-Time (S2T) and Time-to-Spectrum (T2S) Aggregation modules. Finally, we make sufficient assessments including alignment and uniformity to prove the effectiveness of our bilinear feature representations produced by BTSF. Extensive experiments are conducted on three major practical tasks for time series such as classification, forecasting and anomaly detection, which is the first to evaluate on all three tasks. Results shows that our BTSF achieves the superiority over the state-of-the-art methods and surpasses them by a large margin across downstream tasks. Code will be released.
Enhancing the Transferability of Adversarial Attacks via Scale Ensemble
Authors: Xianfeng Gao, Zhikai Chen, Bo Zhang
Date: 2021-09-29 00:00:00
Tasks: Diversity

There is a line of works on adversarial example generation in computer vision, which makes deep learning suffers a lot. Driven by the transferability decrease among models with different input sizes, we present a novel attack method by using a scale input ensemble framework to enhance the transferability of adversarial images, which is named Scale Ensemble Method(SEM). Our method can preserve the characteristic textures of the original image via zooming the surrogate model's input in and out in a specific sequence during generating adversarial examples. The superior texture feature highlights the important attacking region and increases the diversity of adversarial perturbations for assisting a more aggressive attack. The experiments on ImageNet show that our method successfully mitigates the gap of transferability between models with different input sizes and achieves about 8% higher success rate comparing with the state-of-the-art input transformation methods. And we also demonstrate that our method can integrate with existing methods and bypass a variety of defense methods with over 90% success rate.
Generalizable Person Re-identification Without Demographics
Authors: Yifan Zhang, Feng Li, Zhang Zhang, Liang Wang, DaCheng Tao, Tieniu Tan
Date: 2021-09-29 00:00:00
Tasks: Generalizable Person Re-identification, Person Re-Identification

Generalizable Person Re-Identification (DG ReID) aims to learn ready-to-use cross-domain representations for direct cross-data evaluation. It typically fully exploit demographics information, e.g. the domain information and camera IDs to learn features that are domain-invariant. However, the protected demographic features are not often accessible due to privacy and regulation issues. Under this more realistic setting, distributionally robust optimization (DRO) provides a promising way for learning robust models that are able to perform well on a collection of possible data distributions (the ``uncertainty set”) without demographics. However, the convex condition of KL DRO may not hold for overparameterized neural networks, such that applying KL DRO often fails to generalize under distribution shifts in real scenarios. Instead, by applying the change-of-measure technique and the analytical solution of KL DRO, we propose a simple yet efficient approach, Unit DRO. Unit DRO minimizes the loss over a reweighted dataset where important samples (i.e. samples on which models perform poorly) will be upweighted and others will be downweighted. Empirical results show that Unit DRO achieves superior performance on large-scale DG ReID and cross-domain ReID benchmarks compared to standard baselines.
Network calibration by weight scaling
Authors: Lior Frenkel, Jacob Goldberger
Date: 2021-09-29 00:00:00
Tasks: Decision Making, Prediction

Calibrating neural networks is crucial in applications where the decision making depends on the predicted probabilities. Modern neural networks are not well calibrated and they tend to overestimate probabilities when compared to the expected accuracy. This results in a misleading reliability that corrupts our decision policy. We define a weight scaling calibration method that computes a convex combination of the network output class distribution and the uniform distribution. The weights controls the confidence of the calibrated prediction. Since the goal of calibration is making the confidence prediction more accurate, the most suitable
weight is found as a function of the given confidence. We derive an optimization method that is based on a closed form solution for the optimal weight scaling in each bin of a discretized value of the prediction confidence. We report extensive experiments on a variety of image datasets and network architectures. This approach achieves state-of-the-art calibration with a guarantee that the classification accuracy is not altered.
Improved Generalization Risk Bounds for Meta-Learning with PAC-Bayes-kl Analysis
Authors: Jiechao Guan, Zhiwu Lu, Yong liu
Date: 2021-09-29 00:00:00
Tasks: Generalization Bounds, Learning Theory, Meta-Learning

 By incorporating knowledge from observed tasks, PAC-Bayes meta-learning algorithms aim to construct a hyperposterior from which an informative prior is sampled for fast adaptation to novel tasks. The goal of PAC-Bayes meta-learning theory is thus to propose an upper bound on the generalization risk over a novel task of the learned hyperposterior. In this work, we first generalize the tight PAC-Bayes-kl bound from independently and identically distributed (i.i.d.) setting to non-i.i.d. meta-learning setting. Based on the extended PAC-Bayes-kl bound, we further provide three improved PAC-Bayes generalization bounds for meta-learning, leading to better asymptotic behaviour than existing results. By minimizing objective functions derived from the improved bounds, we develop three PAC-Bayes meta-learning algorithms for classification. Moreover, we employ localized PAC-Bayes analysis for meta-learning to yield insights into the role of hyperposterior for learning a novel task. In particular, we identify that when the number of training task is large, utilizing a prior generated from an informative hyperposterior can achieve the same order of PAC-Bayes-kl bound as that obtained through setting a localized distribution-dependent prior for a novel task. Experiments with deep neural networks show that minimizing our bounds can achieve competitive performance on novel tasks w.r.t. previous PAC-Bayes meta-learning methods as well as PAC-Bayes single-task learning methods with localized prior.
L2BGAN: An image enhancement model for image quality improvement and image analysis tasks without paired supervision
Authors: Jhilik Bhattacharya, Gianni Ramponi, Leonardo Gregorat, Shatrughan Modi
Date: 2021-09-29 00:00:00
Tasks: Image Enhancement

The paper presents an image enhancement model,
L2BGAN, to translate low light images to bright images
without a paired supervision. We introduce the use of geo-
metric and lighting consistency along with a contextual loss
criterion. These when combined with multiscale color, tex-
ture and edge discriminators prove to provide competitive
results. We perform extensive experiments on benchmark
datasets to compare our results visually as well as objec-
tively. We observe the performance of L2BGAN on real time
driving datasets which are subject to motion blur, noise and
other artifacts. We further demonstrate the application of
image understanding tasks on our enhanced images using
DarkFace and ExDark datasets.
Learning rate optimization through step sampling
Authors: Brendon Eby, Suhel Jaber
Date: 2021-09-29 00:00:00

Modern machine learning models require selecting hyper-parameters prior to training; important variables that define the way in which the model can learn, but which cannot be learned by the model itself and instead need to be assigned in advance. Of the hyper-parameters that must be selected when configuring a model, arguably the most important is the “learning rate” of the model, the step size that the model uses when learning its parameters with gradient descent. Here we propose a method to deliberately select a learning rate by training a model for a small number of steps over a variety of learning rates and resetting both the model parameters and dataset between each trial. A curve of the log of those rates vs the losses achieved for each is used to select a viable range for an optimal learning rate, and we compare several methods of selecting an optimal point within that range. The performance of the selections from these methods are then evaluated using a full grid search, and in our experiments, they reliably select learning rates that achieve a good accuracy for any given model.
Faster Neural Net Inference via Forests of Sparse Oblique Decision Trees
Authors: Yerlan Idelbayev, Arman Zharmagambetov, Magzhan Gabidolla, Miguel A. Carreira-Perpinan
Date: 2021-09-29 00:00:00
Tasks: Quantization

 It is widely established that large neural nets can be considerably compressed by techniques such as pruning, quantization or low-rank factorization. We show that neural nets can be further compressed by replacing layers of it with a special type of decision forest. This consists of sparse oblique trees, trained with the Tree Alternating Optimization (TAO) algorithm, using a teacher-student approach. We find we can replace the fully-connected and some convolutional layers of standard architectures with a decision forest containing very few, shallow trees so that the prediction accuracy is preserved or improved, but the number of parameters and especially the inference time is greatly reduced. For example, replacing last 7 layers of VGG16 with a single tree reduces the inference FLOPs by 7440$\times$ with a marginal increase in the test error, and a boosted ensemble of nine trees can match the network's performance while still reducing the FLOPs 6289$\times$. The idea is orthogonal to other compression approaches, which can also be used on other parts of the net not being replaced by a forest.
New Perspective on the Global Convergence of Finite-Sum Optimization
Authors: Lam M. Nguyen, Trang H. Tran, Marten van Dijk
Date: 2021-09-29 00:00:00

Deep neural networks (DNNs) have shown great success in many machine learning tasks. Their training is challenging since the loss surface of the network architecture is generally non-convex, or even non-smooth. How and under what assumptions is guaranteed convergence to a \textit{global} minimum possible? We propose a reformulation of the minimization problem allowing for a new recursive algorithmic framework. By using bounded style assumptions, we prove convergence to an $\varepsilon$-(global) minimum using $\mathcal{\tilde{O}}(1/\varepsilon^2)$ gradient computations. Our theoretical foundation motivates  further study, implementation, and optimization of the new  algorithmic framework and further investigation of its non-standard bounded style assumptions. This new direction broadens our understanding of why and under what circumstances  training of a DNN converges to a global minimum.
Differentiable Scaffolding Tree for Molecule Optimization
Authors: Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W. Coley, Jimeng Sun
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Combinatorial Optimization, Drug Discovery, Graph Neural Network

The structural design of functional molecules, also called molecular optimization, is an essential chemical science and engineering task with important applications, such as drug discovery. Deep generative models and combinatorial optimization methods achieve initial success but still struggle with directly modeling discrete chemical structures and often heavily rely on brute-force enumeration. The challenge comes from the discrete and non-differentiable nature of molecule structures. To address this, we propose differentiable scaffolding tree (DST) that utilizes a learned knowledge network to convert discrete chemical structures to locally differentiable ones. DST enables a gradient-based optimization on a chemical graph structure by back-propagating the derivatives from the target properties through a graph neural network (GNN). Our empirical studies show the gradient-based molecular optimizations are both effective and sample efficient. Furthermore, the
learned graph parameters can also provide an explanation that helps domain experts understand the model output.
Towards Robust Point Cloud Models with Context-Consistency Network and Adaptive Augmentation
Authors: Guanlin Li, Guowen Xu, Han Qiu, Ruan He, Jiwei Li, Tianwei Zhang
Date: 2021-09-29 00:00:00
Tasks: Data Augmentation

3D point cloud models based on deep neural networks were proven to be vulnerable to adversarial examples, with a quantity of novel attack techniques proposed by researchers recently. It is of paramount importance to preserve the robustness of 3D models under adversarial environments, considering their broad application in safety- and security-critical tasks. Unfortunately, defenses for 3D models are much less studied compared to 2D image models. In this paper, we reason about the vulnerability of 3D models based on the mutual information theory.  Furthermore, we design an effective defense methodology, consisting of two innovations. (1) We introduce CCDGN, a novel 3D DNN architecture which includes robust and light-weight modules to alleviate adversarial examples. (2) We propose AA-AMS a novel data augmentation strategy to adaptively balance the model usability and robustness. Extensive evaluations indicate the integration of the two techniques provides much more robustness than existing defense solutions for 3D models.
Tabular Data Imputation: Choose KNN over Deep Learning
Authors: Florian Lalande, Kenji Doya
Date: 2021-09-29 00:00:00
Tasks: Common Sense Reasoning, Deep Learning, Imputation, Missing Values

As databases are ubiquitous nowadays, missing values constitute a pervasive problem for data analysis. Over the last 70 years, various imputation algorithms for tabular data have been developed and shown useful at estimating missing values. Besides, recent infatuations for Artificial Neural Networks have led to the development of complex and powerful algorithms for data imputation. 
This study is the first to compare state-of-the-art deep-learning models with the well-established KNN algorithm (1951). By using real-world and generated datasets in various missing data scenarios, we claim that the good old KNN algorithm is still competitive (nay better) than powerful deep-learning algorithms for tabular data imputation.
This work advocates for an appropriate and reasonable use of machine learning, in a world where overconsumption, performances and rapidity unfortunately often prevails over sustainability and common sense.
D$^2$ETR: Decoder-Only DETR with Computationally Efficient Cross-Scale Attention
Authors: Junyu Lin, Xiaofeng Mao, Yuefeng Chen, Lei Xu, Yuan He, Hui Xue'
Date: 2021-09-29 00:00:00
Tasks: Decoder

DETR is the first fully end-to-end detector that predicts a final set of predictions without post-processing. However, it suffers from problems such as low performance and slow convergence. A series of works aim to tackle these issues in different ways, but the computational cost is yet expensive due to the sophisticated encoder-decoder architecture. To alleviate this issue, we propose a decoder-only detector called D$^2$ETR. In the absence of encoder, the decoder directly attends to the fine-fused feature maps generated by the Transformer backbone with a novel computationally efficient cross-scale attention module. D$^2$ETR demonstrates low computational complexity and high detection accuracy in evaluations on the COCO benchmark, outperforming DETR and its variants.
TexRel: a Green Family of Datasets for Emergent Communication with Relations
Authors: Hugh Perkins
Date: 2021-09-29 00:00:00

We propose a new dataset TexRel as a playground for the study of emergent communications, in particular for relations.
TexRel  provides rapid training and experimentation, whilst being sufficiently large to avoid overfitting.
We use TexRel to compare Sender models, compare with a related dataset, examine the effect of changing meaning space size, and perform a case-study for using \sys/ in place of symbolic input.
Icy: A benchmark for measuring compositional inductive bias of emergent communication models
Authors: Hugh Perkins
Date: 2021-09-29 00:00:00
Tasks: Inductive Bias

We present a benchmark \textsc{Icy} for measuring the compositional inductive bias of models in the context of emergent communications. We devise corrupted compositional grammars that probe for limitations in the compositional inductive bias of frequently used models. We use these corrupted compositional grammars to compare and contrast a wide range of models. We propose a hierarchical model, HU-RNN, which might show an inductive bias towards relocatable atomic groups of tokens, thus potentially encouraging the emergence of words. We experiment with probing for the compositional inductive bias of sender networks in isolation, and also placed end-to-end, with a receiver, as an auto-encoder. We propose a metric of compositionality, Compositional Entropy, that is fast to calculate, and broadly applicable.
Zero-shot detection of daily objects in YCB video dataset
Authors: Wanqing Xia
Date: 2021-09-29 00:00:00
Tasks: Object, object-detection, Object Detection, Zero-Shot Object Detection

To let robots be able to manipulate objects, they have to sense the location of objects. With the development of visual data collecting and processing technology, robots are gradually evolving to localize objects in a greater field of view rather than being limited to a small space where the object could appear. To train such a robot vision system, pictures of all the objects need to be taken under various orientations and illumination. In the traditional manufacturing environment, this is applicable since objects involved in the production process does not change frequently. However, in the vision of smart manufacturing and high-mix-low-volume production, parts and products for robots to handle may change frequently. Thus, it is unrealistic to re-training the vision system for new products and tasks. Under this situation, we discovered the necessity to introduce a hot concept which is zero-shot object detection. Zero-shot object detection is a subset of unsupervised learning, and it aims to detect novel objects in the image with the knowledge learned from and only from seen objects. With zero-shot object detection algorithm, time can be greatly saved from collecting training data and training the vision system. Previous works focus on detecting objects in outdoor scenes, such as bikes, car, people, and dogs. The detection of daily objects is actually more challenging since the knowledge can be learned from each object is very limited. In this work, we explore the zero-shot detection of daily objects in indoor scenes since the objects’ size and environment are closely related to the manufacturing setup. The YCB Video Dataset is used in this work, which contains 21 objects in various categories. To the best of our knowledge, no previous work has explored zero-shot detection in this object size level and on this dataset.
Towards Better Understanding and Better Generalization of Low-shot Classification in Histology Images with Contrastive Learning
Authors: Jiawei Yang, Hanbo Chen, Jiangpeng Yan, Xiaoyu Chen, Jianhua Yao
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Contrastive Learning, Representation Learning, Self-Supervised Learning

Low-shot learning is an established topic in natural images for years, but few work is attended to histology images, which is of high clinical value since well-labeled datasets and rare abnormal samples are expensive to collect. Here, we pioneer the study of low-shot learning for histology images by setting up three cross-domain tasks that reflect real clinics problems. To enable label-efficient learning and better generalizability, we propose to incorporate contrastive learning (CL) with latent augmentation (LA) to build a few-shot system. CL learns useful representations without manual labels, while LA transfers semantic variations of base dataset in an unsupervised way. These two components fully exploit unlabeled training data and can scale gracefully to other label-hungry problems. In experiments, we find: i) models learned by CL generalize fairly better than supervised learning for histology images, and ii) LA brings consistent gains over baselines. Prior studies of self-supervised representation learning mainly focus on ImageNet-like images, which only present a dominant object in their centers. Recent attention has been paid to images with multi-objects and multi-textures (Chen & Li, 2020). Histology images are a natural choice for such study. We show the superiority of CL over supervised learning in terms of generalization for such data, and provide our empirical understanding for this observation. The findings in this work could contribute to understanding how model generalizes in the context of both self-supervised learning and histology images.