pwc-archive/papers-with-abstracts huggingface dataset:
DJMix: Unsupervised Task-agnostic Augmentation for Improving Robustness
Authors: Ryuichiro Hataya, Hideki Nakayama
Date: 2021-01-01 00:00:00
Tasks: Data Augmentation, Semantic Segmentation

Convolutional Neural Networks (CNNs) are vulnerable to unseen noise on input images at the test time, and thus improving the robustness is crucial. In this paper, we propose DJMix, a data augmentation method to improve the robustness by mixing each training image and its discretized one. Discretization is done in an unsupervised manner by an autoencoder, and the mixed images are nearly impossible to distinguish from the original images. Therefore, DJMix can easily be adapted to various image recognition tasks. We verify the effectiveness of our method using classification, semantic segmentation, and detection using clean and noisy test images.
Semi-Supervised Learning via Clustering Representation Space
Authors: Yen-Chieh Huang, Yuh-Jye Lee, Chih-Chi Wu, Yi-Wei Chiu, Yong-Xiang Lin, CHENG-YING LI, Po-Hung Ko
Date: 2021-01-01 00:00:00
Tasks: Clustering

We proposed a novel loss function that combines supervised learning with clustering in deep neural networks. Taking advantage of the data distribution and the existence of some labeled data, we construct a meaningful latent space.  Our loss function consists of three parts, the quality of the clustering result, the margin between clusters, and the classification error of labeled instances. Our proposed model is trained to minimize our loss function by backpropagation, avoiding the need for pre-training or additional networks. This guides our network to classify labeled samples correctly while able to find good clusters simultaneously. We applied our proposed method on MNIST, USPS, ETH-80, and COIL-100; the comparison results confirm our model's outstanding performance over semi-supervised learning.
A General Computational Framework to Measure the Expressiveness of Complex Networks using a Tight Upper Bound of Linear Regions
Authors: Yutong Xie, Gaoxiang Chen, Quanzheng Li
Date: 2021-01-01 00:00:00

The expressiveness of deep neural network (DNN) is a perspective to understand the surprising performance of DNN. The number of linear regions, i.e. pieces that a piece-wise-linear function represented by a DNN, is generally used to measure the expressiveness. And the upper bound of regions number partitioned by a rectifier network, instead of the number itself, is a more practical measurement of expressiveness of a rectifier DNN. In this work, we propose a new and tighter upper bound of regions number. Inspired by the proof of this upper bound and the framework of matrix computation in \citet{hinz2019framework}, we propose a general computational approach to compute a tight upper bound of regions number for theoretically any network structures (e.g. DNN with all kind of skip connections and residual structures). Our experiments show our upper bound is tighter than existing ones, and explain why skip connections and residual structures can improve network performance.
Towards Principled Representation Learning for Entity Alignment
Authors: Lingbing Guo, Zequn Sun, Mingyang Chen, Wei Hu, Huajun Chen
Date: 2021-01-01 00:00:00
Tasks: Entity Alignment, Machine Translation, Representation Learning

Knowledge graph (KG) representation learning for entity alignment has recently received great attention. Compared with conventional methods, these embedding-based ones are considered to be robuster for highly-heterogeneous and cross-lingual entity alignment scenarios as they do not rely on the quality of  machine translation or feature extraction. Despite the significant improvement that has been made, there is little understanding of how the embedding-based entity alignment methods actually work. Most existing methods rest on the foundation that a small number of pre-aligned entities can serve as anchors to connect the embedding spaces of two KGs. But no one investigates the rationality of such foundation. In this paper, we define a typical paradigm abstracted from the existing methods, and analyze how the representation discrepancy between two potentially-aligned entities is implicitly bounded by a predefined margin in the scoring function for embedding learning. However, such a margin cannot guarantee to be tight enough for alignment learning. We mitigate this problem by proposing a new approach that explicitly learns KG-invariant and principled entity representations, meanwhile preserves the original infrastructure of existing methods. In this sense, the model not only pursues the closeness of aligned entities on geometric distance, but also aligns the neural ontologies of two KGs to eliminate the discrepancy in feature distribution and underlying ontology knowledge. Our experiments demonstrate consistent and significant improvement in performance against the existing embedding-based entity alignment methods, including several state-of-the-art ones.
Adaptive Stacked Graph Filter
Authors: Hoang NT, Takanori Maehara, Tsuyoshi Murata
Date: 2021-01-01 00:00:00
Tasks: Classification, General Classification

We study Graph Convolutional Networks (GCN) from the graph signal processing viewpoint by addressing a difference between learning graph filters with fully-connected weights versus trainable polynomial coefficients. We find that by stacking graph filters with learnable polynomial parameters, we can build a highly adaptive and robust vertex classification model. Our treatment here relaxes the low-frequency (or equivalently, high homophily) assumptions in existing vertex classification models, resulting a more ubiquitous solution in terms of spectral properties. Empirically, by using only one hyper-parameter setting, our model achieves strong results on most benchmark datasets across the frequency spectrum.
Learning to Learn with Smooth Regularization
Authors: Yuanhao Xiong, Cho-Jui Hsieh
Date: 2021-01-01 00:00:00
Tasks: Decision Making, Few-Shot Learning

Recent decades have witnessed great prosperity of deep learning in tackling various problems such as classification and decision making. The rapid development stimulates a novel framework, Learning-to-Learn (L2L), in which an automatic optimization algorithm (optimizer) modeled by neural networks is expected to learn rules for updating the target objective function (optimizee). Despite its advantages for specific problems, L2L still cannot replace classic methods due to its instability. Unlike hand-engineered algorithms, neural optimizers may suffer from the instability issue---when provided with similar states (a combination of some metrics to describe the optimizee), the same neural optimizer can produce quite different updates. Motivated by the stability property that should be satisfied by an ideal optimizer,  we propose a regularization term that can  enforce the smoothness and stability of the learned neural optimizers. Comprehensive experiments on the neural network training tasks demonstrate that the proposed regularization consistently improve the learned neural optimizers even when transferring to tasks with different architectures and data. Furthermore, we show that our regularizer can improve the performance of neural optimizers on few-shot learning tasks.
Adversarial Training using Contrastive Divergence
Authors: Hongjun Wang, Guanbin Li, Liang Lin
Date: 2021-01-01 00:00:00

To protect the security of machine learning models against adversarial examples, adversarial training becomes the most popular and powerful strategy against various adversarial attacks by injecting adversarial examples into training data. However, it is time-consuming and requires high computation complexity to generate suitable adversarial examples for ensuring the robustness of models, which impedes the spread and application of adversarial training. In this work, we reformulate adversarial training as a combination of stationary distribution exploring, sampling, and training. Each updating of parameters of DNN is based on several transitions from the data samples as the initial states in a Hamiltonian system. Inspired by our new paradigm, we design a new generative method for adversarial training by using Contrastive Divergence (ATCD), which approaches the equilibrium distribution of adversarial examples with only few iterations by building from small modifications of the standard Contrastive Divergence (CD). Our adversarial training algorithm achieves much higher robustness than any other state-of-the-art adversarial training acceleration method on the ImageNet, CIFAR-10, and MNIST datasets and reaches a balance between performance and efficiency.
Can Students Outperform Teachers in Knowledge Distillation based Model Compression?
Authors: Xiang Deng, Zhongfei Zhang
Date: 2021-01-01 00:00:00
Tasks: Knowledge Distillation, Model Compression, Transfer Learning

Knowledge distillation (KD) is an effective technique to compress a large model (teacher) to a compact one (student) by knowledge transfer. The ideal case is that the teacher is compressed to the small student without any performance dropping. However, even for the state-of-the-art (SOTA) distillation approaches, there is still an obvious performance gap between the student and the teacher. The existing literature usually attributes this to model capacity differences between them. However, model capacity differences are unavoidable in model compression. In this work, we systematically study this question. By designing exploratory experiments, we find that model capacity differences are not necessarily the root reason, and the distillation data matters when the student capacity is greater than a threshold. In light of this, we propose to go beyond in-distribution distillation and accordingly develop KD+. KD+ is superior to the original KD as it outperforms KD and the other SOTA approaches substantially and is more compatible with the existing approaches to further improve their performances significantly.
Is Robustness Robust? On the interaction between augmentations and corruptions
Authors: Eric Mintun, Alexander Kirillov, Saining Xie
Date: 2021-01-01 00:00:00

Invariance to a broad array of image corruptions, such as warping, noise, or color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed that significantly improve performance on ImageNet-C, a benchmark of such corruptions. However, there is still a lack of basic understanding on the relationship between data augmentations and test-time corruptions. To this end, we develop a new metric called Minimal Sample Distance to measure the perceptual similarity between image transforms, and then apply it to augmentations and corruptions to demonstrate there is a strong correlation between similarity and performance. We then investigate recent data augmentations and observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C under our metric. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may risk overfitting to the existing benchmark. We hope our results and tools will allow for more robust progress towards improving robustness to image corruptions.
Uncovering the impact of learning rate for global magnitude pruning
Authors: Janice Lan, Rudy Chin, Alexei Baevski, Ari S. Morcos
Date: 2021-01-01 00:00:00

A common paradigm in model pruning is to train a model, prune, and then fine-tune or reinitialize and retrain in the lottery ticket framework. However, prior work has implicitly assumed that the best training configuration for model performance was also the best configuration for mask discovery. In this paper, we propose a simple idea: what if a training configuration which yields worse performance actually yields a better mask? To test this, we decouple the learning rates for mask discovery (LR_find) and mask evaluation (LR_eval). Using a ResNet-50 on Tiny ImageNet, we discovered the counterintuitive ``decoupled LR phenomenon," in which smaller LR_find values lead to models which have lower performance, but generate masks with substantially higher eventual performance compared to using the same learning rate for both stages. We show that this phenomenon holds across a number of models, datasets, configurations, and also for one-shot structured pruning. Finally, we demonstrate that smaller LR_find values yield masks with materially different layerwise pruning ratios and that the decoupled LR phenomenon is causally mediated by these ratios. Our results demonstrate the practical utility of decoupling learning rates and provide clear insights into the mechanisms underlying this counterintuitive effect.
MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-Shot Learning
Authors: Nanyi Fei, Zhiwu Lu, Tao Xiang, Songfang Huang
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Few-Shot Learning, Meta-Learning

Most recent few-shot learning (FSL) approaches are based on episodic training whereby each episode samples few training instances (shots) per class to imitate the test condition. However, this strict adhering to test condition has a negative side effect, that is, the trained model is susceptible to the poor sampling of few shots. In this work, for the first time, this problem is addressed by exploiting inter-episode relationships. Specifically, a novel meta-learning via modeling episode-level relationships (MELR) framework is proposed. By sampling two episodes containing the same set of classes, MELR is designed to ensure that the meta-learned model is robust against the presence of poorly-sampled shots in one episode or both. This is achieved through two key components: (1) a Cross-Episode Attention Module (CEAM) to alleviate the effects of poorly-sampled shots, and (2) a Cross-Episode Consistency Regularization (CECR) to enforce that the two classifiers learned from the two episodes are consistent even when there are unrepresentative instances. Extensive experiments on two benchmarks with three different backbones show that our MELR achieves the new state-of-the-art.
Partitioned Learned Bloom Filters
Authors: Kapil Vaidya, Eric Knorr, Michael Mitzenmacher, Tim Kraska
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

Bloom filters are space-efficient probabilistic data structures that are used to test whether an element is a member of a set, and may return false positives.  Recently, variations referred to as learned Bloom filters were developed that can provide improved performance in terms of the rate of false positives, by using a learned model for the represented set.  However, previous methods for learned Bloom filters do not take full advantage of the learned model.  Here we show how to frame the problem of optimal model utilization as an optimization problem, and using our framework derive algorithms that can achieve near-optimal performance in many cases.
An Empirical Exploration of Open-Set Recognition via Lightweight Statistical Pipelines
Authors: Shu Kong, Deva Ramanan
Date: 2021-01-01 00:00:00
Tasks: open-set classification, Open Set Learning, Outlier Detection, Retrieval, Semantic Segmentation

Machine-learned safety-critical systems need to be self-aware and reliably know their unknowns in the open-world. This is often explored through the lens of anomaly/outlier detection or out-of-distribution modeling. One popular formulation is that of open-set classification, where an image classifier trained for 1-of-$K$ classes should also recognize images belonging to a $(K+1)^{th}$ "other" class, not present in the training set. Recent work has shown that, somewhat surprisingly, most if not all existing open-world methods do not work well on high-dimensional open-world images (Shafaei et al., 2019). In this paper, we carry out an empirical exploration of open-set classification, and find that combining classic statistical methods with carefully computed features can dramatically outperform prior work. We extract features from off-the-shelf (OTS) state-of-the-art (SOTA) networks for the underlying $K$-way closed-world task. We leverage insights from the retrieval community for computing feature descriptors that are low-dimensional (via pooling and PCA) and normalized (via L2-normalization), enabling the modeling of training data densities via classic statistical tools such as kmeans and Gaussian Mixture Models (GMMs). Finally, we (re)introduce the task of open-set semantic segmentation, which requires classifying individual pixels into one of $K$ known classes or an "other" class. In this setting, our feature-based statistical models noticeably outperform prior open-world methods.
SAFENet: A Secure, Accurate and Fast Neural Network Inference
Authors: Qian Lou, Yilin Shen, Hongxia Jin, Lei Jiang
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

The advances in neural networks have driven many companies to provide prediction services to users in a wide range of applications. However, current prediction systems raise privacy concerns regarding the user's private data. A cryptographic neural network inference service is an efficient way to allow two parties to execute neural network inference without revealing either party’s data or model. Nevertheless, existing cryptographic neural network inference services suffer from huge running latency; in particular, the latency of communication-expensive cryptographic activation function is 3 orders of magnitude higher than plaintext-domain activation function. And activations are the necessary components of the modern neural networks. Therefore, slow cryptographic activation has become the primary obstacle of efficient cryptographic inference. 

In this paper, we propose a new technique, called SAFENet, to enable a Secure, Accurate and Fast nEural Network inference service. To speedup secure inference and guarantee inference accuracy, SAFENet includes channel-wise activation approximation with multiple-degree options. This is implemented by keeping the most useful activation channels and replacing the remaining, less useful, channels with various-degree polynomials. SAFENet also supports mixed-precision activation approximation by automatically assigning different replacement ratios to various layer; further increasing the approximation ratio and reducing inference latency. Our experimental results show SAFENet obtains the state-of-the-art inference latency without a decrease in accuracy, reducing latency by $38\% \sim 61\%$ over prior techniques on various encrypted datasets.
Dataset Curation Beyond Accuracy
Authors: Johan Bjorck, Carla P Gomes
Date: 2021-01-01 00:00:00
Tasks: Self-Driving Cars

Neural networks are known to be data-hungry, and collecting large labeled datasets is often a crucial step in deep learning deployment. Researchers have studied dataset aspects such as distributional shift and labeling cost, primarily using downstream prediction accuracy for evaluation. In sensitive real-world applications such as medicine and self-driving cars, not only is the accuracy important, but also the calibration -- the extent that model uncertainty reflects the actual correctness likelihood. It has recently been shown that modern neural networks are ill-calibrated. In this work, we take a complementary approach -- studying how dataset properties, rather than architecture, affects calibration. For the common issue of dataset imbalance, we show that calibration varies significantly among classes, even when common strategies to mitigate class imbalance are employed. We also study the effects of label quality, showing how label noise dramatically increases calibration error. Furthermore, poor calibration can come from small dataset sizes, which we motive via results on network expressivity. Our experiments demonstrate that dataset properties can significantly affect calibration and suggest that calibration should be measured during dataset curation.
Improving Local Effectiveness for Global Robustness Training
Authors: Jingyue Lu, M. Pawan Kumar
Date: 2021-01-01 00:00:00

Despite its increasing popularity,  deep neural networks are easily fooled.   Toalleviate this deficiency, researchers are actively developing new training strategies,which encourage models that are robust to small input perturbations.   Severalsuccessful robust training methods have been proposed. However, many of themrely on strong adversaries, which can be prohibitively expensive to generate whenthe input dimension is high and the model structure is complicated. We adopt a newperspective on robustness and propose a novel training algorithm that allows a moreeffective use of adversaries. Our method improves the model robustness at eachlocal patch and then, by combining these patches through a global term, achievesoverall robustness.  We demonstrate that, by maximizing the use of adversaries,we achieve high robust accuracy with weak adversaries. Specifically, our methodreaches a similar robust accuracy level to the state of the art approaches trained onstrong adversaries on MNIST, CIFAR-10 and CIFAR-100. As a result, the overalltraining time is reduced. Furthermore, when trained with strong adversaries, ourmethod matches with the current state of the art on MNIST and outperforms themon CIFAR-10 and CIFAR-100
NAHAS: Neural Architecture and Hardware Accelerator Search
Authors: Yanqi Zhou, Xuanyi Dong, Daiyi Peng, Ethan Zhu, Amir Yazdanbakhsh, Berkin Akin, Mingxing Tan, James Laudon
Date: 2021-01-01 00:00:00
Tasks: Neural Architecture Search

Neural architectures and hardware accelerators have been two driving forces for the rapid progress in deep learning.
Although previous works have optimized either neural architectures given fixed hardware, or hardware given fixed neural architectures, none has considered optimizing them jointly. In this paper, we study the importance of co-designing neural architectures and hardware accelerators. To this end, we propose NAHAS, an automated hardware design paradigm that jointly searches for the best configuration for both neural architecture and accelerator. In NAHAS, accelerator hardware design is conditioned on the dynamically explored neural networks for the targeted application, instead of fixed architectures, thus providing better performance opportunities. Our experiments with an industry-standard edge accelerator show that NAHAS consistently outperforms previous platform-aware neural architecture search and state-of-the-art EfficientNet on all latency targets by 0.5% - 1% ImageNet top-1 accuracy, while reducing latency by about 20%. Joint optimization reduces the search samples by 2x and reduces the latency constraint violations from 3 violations to 1 violation per 4 searches, compared to independently optimizing the two sub spaces.
SyncTwin: Transparent Treatment Effect Estimation under Temporal Confounding
Authors: Zhaozhi Qian, Yao Zhang, Ioana Bica, Angela Wood, Mihaela van der Schaar
Date: 2021-01-01 00:00:00

Estimating causal treatment effects using observational data is a problem with few solutions when the confounder has a temporal structure, e.g. the history of disease progression might impact both treatment decisions and clinical outcomes. For such a challenging problem, it is desirable for the method to be transparent --- the ability to pinpoint a small subset of data points that contributes most to the estimate and to clearly indicate whether the estimate is reliable or not. This paper develops a new method, SyncTwin, to overcome temporal confounding in a transparent way. SyncTwin estimates the treatment effect of a target individual by comparing the outcome with its synthetic twin, which is constructed to closely match the target in the representation of the temporal confounders. SyncTwin achieves transparency by enforcing the synthetic twin to only depend on the weighted combination of few other individuals in the dataset. Moreover, the quality of the synthetic twin can be assessed by a performance metric, which also indicates the reliability of the estimated treatment effect. Experiments demonstrate that SyncTwin outperforms the benchmarks in clinical observational studies while still being transparent.