pwc-archive/papers-with-abstracts huggingface dataset:
Model-Based Robust Adaptive Semantic Segmentation
Authors: Jun Wang, Yiannis Kantaros
Date: 2021-09-29 00:00:00
Tasks: Autonomous Vehicles, Image Segmentation, model, Segmentation, Semantic Segmentation

Semantic image segmentation enjoys a wide range of applications such as autonomous vehicles and medical imaging while it is typically accomplished by deep neural networks (DNNs). Nevertheless, DNNs are known to be fragile to input perturbations that are adversarially crafted or occur due to natural variations, such as changes in weather or lighting conditions. This issue of lack of robustness prevents the application of learning-based semantic segmentation methods on safety-critical applications. To mitigate this challenge, in this paper, we propose model-based robust adaptive training algorithm (MRTAdapt), a new training algorithm to enhance the robustness of DNN-based semantic segmentation methods against natural variations that leverages model-based robust training algorithms and generative adversarial networks. Natural variation effects are minimized from both image and label sides. We provide extensive experimental results on both real-world and synthetic datasets demonstrating that model-based robust adaptive training algorithm outperforms multiple state-of-the-art models under various natural variations.
Ambiguity Adaptive Inference and Single-shot based Channel Pruning for Satellite Processing Environments
Authors: Minsu Jeon, Kyungno Joo, Changha Lee, Taewoo Kim, SeongHwan Kim, Chan-Hyun Youn
Date: 2021-09-29 00:00:00
Tasks: GPU

In a restricted computing environment like satellite on-board systems, running DL models has limitation on high-speed processing due to the problems such as restriction of available power to consume compared to the relatively high computational complexity. In particular, the latest GPU resources shows high computing performance but also shows relatively high power consumption, whereas in restricted environments such as satellite systems, reconfigurable resources like FPGA or low power embedded GPU are generally adopted due to their relatively low power consumption compared to computing capability. In such a constrained computing environment, in order to overcome the problem of too huge model size to fit in reconfigurable resources or limitation on high-speed processing, we propose a reconfigurable DL accelerating system where the computing complexity and size of DL model are compressed by pruning and can be adapted to the FPGA or low power GPU resources. Therefore, in this paper, we mainly address an ambiguity adaptive inference model that can enhance overall accuracy in inference step directly for mission critical task, a new method for single-shot based channel pruning that can accelerate inference of DL model through compressing the model as much as possible with maintaining accuracy performance under constrained accelerator resources. From the experimental evaluation, for the satellite image analysis model as an example application, our method can achieve up to x8.53 compression while keeping the accuracy, and verified that our method can deploy and accelerate the DL model with high computational complexity on FPGA/GPU resources.
Inductive Lottery Ticket Learning for Graph Neural Networks
Authors: Yongduo Sui, Xiang Wang, Tianlong Chen, Xiangnan He, Tat-Seng Chua
Date: 2021-09-29 00:00:00
Tasks: Graph Classification, Node Classification, Transductive Learning

Deep graph neural networks (GNNs) have gained increasing popularity, while usually suffer from unaffordable computations for real-world large-scale applications. Hence, pruning GNNs is of great need but largely unexplored. A recent work, UGS, studies lottery ticket learning for GNNs, aiming to find a subset of model parameters and graph structure that can best maintain the GNN performance. However, it is tailed for the transductive setting, failing to generalize to unseen graphs, which are common in inductive tasks like graph classification. In this work, we propose a simple and effective learning paradigm, Inductive Co-Pruning of GNNs (ICPG), to endow graph lottery tickets with inductive pruning capacity. To prune the input graphs, we design a generative probabilistic model to generate importance scores for each edge based on the input; to prune the model parameters, it views the weight's magnitude as their importance scores. Then we design an iterative co-pruning strategy to trim the graph edges and GNN weights based on their importance scores. Although it might be strikingly simple, ICPG surpasses the existing pruning method and can be universally applicable in both inductive and transductive learning settings. On ten graph-classification and two node-classification benchmarks, ICPG achieves the same performance level with $14.26\%\sim43.12\%$ sparsity for graphs and $48.80\%\sim91.41\%$ sparsity for the model.
Predictive Maintenance for Optical Networks in Robust Collaborative Learning
Authors: Khouloud Abdelli, JOO YEON CHO
Date: 2021-09-29 00:00:00
Tasks: Federated Learning

Machine learning (ML) has recently emerged as a powerful tool to enhance the proactive optical network maintenance and thereby, improve network reliability and operational efficiency, and reduce unplanned downtime and maintenance costs. However, it is challenging to develop an accurate and reliable ML based prognostic models due mainly to the unavailability of sufficient amount of training data since the device failure does not occur often in optical networks. Federated learning (FL) is a promising candidate to tackle the aforementioned challenge by enabling the development of a global ML model using datasets owned by many vendors without revealing their business-confidential  data. While FL greatly enhances the data privacy, a global model can be strongly affected by a malicious local model. We propose a robust collaborative learning framework for predictive maintenance on cross-vendor in a dishonest setting. Our experiments confirm that a global ML model can be accurately built with sensitive datasets in federated learning even when a subset of vendors behave dishonestly.
Einops: Clear and Reliable Tensor Manipulations with Einstein-like Notation
Authors: Alex Rogozhnikov
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Management

Tensor computations underlie modern scientific computing and deep learning.
A number of tensor frameworks emerged varying in execution model, hardware support, memory management, model definition, etc.
However, available tensor operations follow the same paradigm in all frameworks.
Recent neural network architectures demonstrate demand for higher expressiveness of tensor operations.
The current paradigm is not suited to write readable, reliable, or easy-to-modify code for multidimensional tensor manipulations. 
Moreover, some commonly used operations do not provide sufficient checks and can break a tensor structure.
These mistakes are elusive as no tools or tests can detect them.
Independently, API discrepancies complicate code transfer between frameworks.
We propose einops notation: a uniform and generic way to manipulate tensor structure, that significantly improves code readability and flexibility by focusing on structure of input and output tensors.
We implement einops notation in a Python package that efficiently supports multiple widely used frameworks and provides framework-independent minimalist API for tensor manipulations.
A Topological View of Rule Learning in Knowledge Graphs
Authors: Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, Chao Chen
Date: 2021-09-29 00:00:00
Tasks: Inductive Relation Prediction, Knowledge Graph Completion, Knowledge Graphs, Relation, Relation Prediction

Inductive relation prediction is an important learning task for knowledge graph completion. One can use the existence of rules, namely a sequence of relations, to predict the relation between two entities. Previous works view rules as paths and primarily focus on the searching of paths between entities. The space of paths is huge, and one has to sacrifice either efficiency or accuracy. In this paper, we consider rules in knowledge graphs as cycles and show that the space of cycles has a unique structure based on the theory of algebraic topology. By exploring the linear structure of the cycle space, we can improve the searching efficiency of rules. We propose to collect cycle bases that span the space of cycles. We build a novel GNN framework on the collected cycles to learn the representations of cycles, and to predict the existence/non-existence of a relation. Our method achieves state-of-the-art performance on benchmarks.
FED-$\chi^2$: Secure Federated Correlation Test
Authors: Lun Wang, Qi Pang, Shuai Wang, Dawn Song
Date: 2021-09-29 00:00:00

In this paper, we propose the first secure federated $\chi^2$-test protocol, FED-$\chi^2$. We recast $\chi^2$-test as a problem of the second moment estimation and use stable projection to encode the local information in a short vector. Due to the fact that such encodings can be aggregated with summation, secure aggregation can smoothly be applied to conceal the individual updates. We formally establish the security guarantee of FED-$\chi^2$ by demonstrating that the joint distribution is hidden in a subspace containing exponentially possible distributions. Our evaluation results show that FED-$\chi^2$ achieves good accuracy with small client-side computation overhead. FED-$\chi^2$ performs comparably to the centralized $\chi^2$-test in several real-world case studies. The code for evaluation is in the supplementary material.
Inductive Relation Prediction Using Analogy Subgraph Embeddings
Authors: Jiarui Jin, Yangkun Wang, Kounianhua Du, Weinan Zhang, Zheng Zhang, David Wipf, Yong Yu, Quan Gan
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Inductive Bias, Inductive Relation Prediction, Knowledge Graph Completion, Prediction, Relation, Relation Prediction

Prevailing methods for relation prediction in heterogeneous graphs aim at learning latent representations (i.e., embeddings) of observed nodes and relations, and thus are limited to the transductive setting where the relation types must be known during training.  Here,  we propose ANalogy  SubGraphEmbeddingLearning (GraphANGEL), a novel relation prediction framework that predicts relations5between each node pair based on the subgraphs containing the pair, as well as other  (analogy)  subgraphs with the same graph patterns.   Each graph pattern explicitly represents a specific logical rule, which contributes to an inductive bias that facilitates generalization to unseen relations and leads to more explainable predictive models. Moreover, our method also removes the limited neighborhood constraint of graph neural networks. Our model consistently outperforms existing models on heterogeneous graph based recommendation as well as knowledge graph completion.  We also empirically demonstrate our model’s capability in generalizing to new relations while producing explainable heat maps of attention scores across the discovered logic.
Generating Transferable Adversarial Patch by Simultaneously Optimizing its Position and Perturbations
Authors: Xingxing Wei, Ying Guo, Jie Yu, Huanqian Yan, Bo Zhang
Date: 2021-09-29 00:00:00
Tasks: Face Recognition, Position

Adversarial patch is one kind of important form to perform adversarial attacks in the real world and brings serious risks to the robustness of deep neural networks. Previous methods generate adversarial patches by either optimizing their perturbation values while fixing the position on the image or manipulating the position while fixing the content of the patch.
In this paper, we propose a method to simultaneously optimize the position and perturbation to generate transferable adversarial patches, and thus obtain high attack success rates in the black-box setting. We adjust the transferability by taking the position, weights of surrogate models in the ensemble attack and the attack step size as parameters, and utilize the reinforcement learning framework to simultaneously solve these parameters based on the reward information obtained from the target model with a small number of queries.
Extensive experiments are conducted on the Face Recognition (FR) task, and the results on four representative FR models demonstrate that our method can significantly improve the attack success rate and the query efficiency. Besides, experiments on the commercial FR service and physical environments confirm the practical application value of our method.
Conceptron: a probabilistic deep one-class classification method
Authors: Erika Gardini, Andrea Cavalli, Sergio Decherchi
Date: 2021-09-29 00:00:00
Tasks: Classification, One-Class Classification

One-class learning through deep architectures is a particularly challenging task; in this scenario the crasis of kernel methods and deep networks can represent a viable strategy to empower already effective methods. In this contribution we present Conceptron, a probabilistic and deep one-class classification method. The proposed algorithm is a hybridization of the Nystrom version of the Import Vector Domain Description (IVDD) to deep learning layers rendering the approach highly scalable (via batch stochastic gradient optimization) and automatically learning the underlying feature space. Further we modify the cost function to allow to get a Laplace distribution of the samples probabilities. Experiments on MNIST, CIFAR-10 and other benchmark datasets show that Conceptron (and/or variations) performs comparably or better with competing state-of-the-art methods with the additional capability of providing probabilities (through a logistic model) and avoiding any degeneracy in the training process.
Interpretable Multi-hop Reasoning for Forecasting Future Links on Temporal Knowledge Graphs
Authors: Liang Zongwei, Junan Yang, Keju Huang, Hui Liu, Lin Cui, Lingzhi Qu, Xiang Li
Date: 2021-09-29 00:00:00
Tasks: Knowledge Graphs, Question Answering

Temporal knowledge graphs (KGs) have recently attracted growing attention. The temporal KG forecasting task, which plays a crucial role in applications such as event prediction, is predicting future links based on historical facts. The interpretability of the current temporal KG forecasting models is manifested in providing the reasoning paths. However, the comparison of reasoning paths is operated under the black box. Inspired by the observation that reasoning based on multi-hop paths is equivalent to answering questions step by step, this paper designs an Interpretable Multi-hop Reasoning (IMR) model for temporal KG forecasting. IMR transforms reasoning based on path searching into step-by-step question answering. Moreover, IMR designs three indicators according to the characteristics of temporal KGs and reasoning paths: question matching degree, answer completing level and path confidence. Unlike other models that can only utilize paths with a specified hop, IMR can effectively integrate paths of different hops; IMR can provide the reasoning paths like other interpretable models and further explain the basis for path comparison. While being more explainable, IMR has achieved state-of-the-art on four baseline datasets.
Sharp Attention for Sequence to Sequence Learning
Authors: Pei Zhang, Hua Liu
Date: 2021-09-29 00:00:00
Tasks: Hard Attention, Scene Text Recognition

Attention mechanism has been widely applied to tasks that output some sequence from an input image. Its success comes from the ability to align relevant parts of the encoded image with the target output. However, most of the existing methods fail to build clear alignment because the aligned parts are unable to well represent the target. In this paper we seek clear alignment in attention mechanism through a \emph{sharpener} module. Since it deliberately locates the target in an image region and refines representation to be target-specific, the alignment and interpretability of attention can be significantly improved. Experiments on synthetic handwritten digit as well as real-world scene text recognition datasets show that our approach outperforms the mainstream ones such as soft and hard attention.
$k$-Mixup Regularization for Deep Learning via Optimal Transport
Authors: Kristjan Greenewald, Anming Gu, Mikhail Yurochkin, Justin Solomon, Edward Chien
Date: 2021-09-29 00:00:00
Tasks: Adversarial Robustness, Deep Learning

Mixup is a popular regularization technique for training deep neural networks that can improve generalization  and increase adversarial robustness. It perturbs input training data in the direction of other randomly-chosen instances in the training set. To better leverage the structure of the data, we extend mixup to $k$-mixup by perturbing $k$-batches of training points in the direction of other $k$-batches using displacement interpolation, i.e. interpolation under the Wasserstein metric. We demonstrate theoretically and in simulations that $k$-mixup preserves cluster and manifold structures, and we extend theory studying the efficacy of standard mixup to the $k$-mixup case. Our empirical results show that training with $k$-mixup further improves generalization and robustness across several network architectures and benchmark datasets of differing modalities.
Protecting Proprietary Data: Poisoning for Secure Dataset Release
Authors: Liam H Fowl, Ping-Yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Amit Bansal, Wojciech Czaja, Tom Goldstein
Date: 2021-09-29 00:00:00
Tasks: Data Poisoning

Large organizations such as social media companies continually release data, for example user images.  At the same time, these organizations leverage their massive corpora of released data to train proprietary models that give them an edge over their competitors.  These two behaviors can be in conflict as an organization wants to prevent competitors from using their own data to replicate the performance of their proprietary models.  We solve this problem by developing a data poisoning method by which publicly released data can be minimally modified to prevent others from training models on it.  Moreover, our method can be used in an online fashion so that companies can protect their data in real time as they release it.  We demonstrate the success of our approach on ImageNet classification and on facial recognition.
Towards simple time-to-event modeling: optimizing neural networks via rank regression
Authors: Hyunjun Lee, Junhyun Lee, Taehwa Choi, Jaewoo Kang, Sangbum Choi
Date: 2021-09-29 00:00:00
Tasks: regression, Representation Learning, Survival Analysis, Time-to-Event Prediction

Time-to-event analysis, also known as survival analysis, aims to predict the first occurred event time, conditional on a set of features.
However, the presence of censorship brings much complexity in learning algorithms due to data incompleteness.
Hazard-based models (e.g. Cox's proportional hazards) and accelerated failure time (AFT) models are two popular tools in time-to-event modeling, requiring the proportional hazards and linearity assumptions, respectively. 
In addition, AFT models require pre-specified parametric distributional assumptions in most cases. 
To alleviate such strict assumptions and improve predictive performance, there have been many deep learning approaches for hazard-based models in recent years. 
However, compared to hazard-based methods, AFT-based representation learning has received limited attention in neural network literature, despite its model simplicity and interpretability. 
In this work, we introduce a Deep AFT Rank-regression for Time-to-event prediction model (DART), which is a deep learning-based semiparametric AFT model, and propose a $l_1$-type rank loss function that is more suitable for optimizing neural networks. 
Unlike existing neural network-based AFT models, the proposed model is semiparametric in that any distributional assumption is not imposed for the survival time distribution without requiring further hyperparameters or complicated model architectures. 
We verify the usefulness of DART via quantitative analysis upon various benchmark datasets. 
The results show that our method has considerable potential to model high-throughput censored time-to-event data.
A partial theory of Wide Neural Networks using WC functions and its practical implications
Authors: Dario Balboni, Davide Bacciu
Date: 2021-09-29 00:00:00

We present a framework based on the theory of Polyak-Łojasiewicz functions to explain the properties of convergence and generalization of overparameterized feed-forward neural networks. We introduce the class of Well-Conditioned (WC) reparameterizations, which are closed under composition and preserve the class of Polyak-Łojasiewicz functions, thus enabling compositionality of the framework results which can be studied separately for each layer and in an architecture-neutral way. We show that overparameterized neural layers are WC and can therefore be composed to build easily optimizable functions. We expose a pointwise stability bound implying that overparameterization in WC models leads to a tighter convergence around a global minimizer. Our framework allows to derive quantitative estimates for the terms that govern the optimization process of neural networks.  We leverage this aspect to empirically evaluate the predictions set forth by some relevant published theories concerning conditioning, training speed, and generalization of the neural networks training process.  Our contribution aims to encourage the development of mixed theoretical-practical approaches, where the properties postulated by the theory can also find empirical confirmation.
A Unified Knowledge Distillation Framework for Deep Directed Graphical Models
Authors: Yizhuo Chen, Kaizhao Liang, Zhe Zeng, Yifei Yang, Shuochao Yao, Huajie Shao
Date: 2021-09-29 00:00:00
Proceeding: CVPR 2023 1
Tasks: Continual Learning, Federated Learning, Knowledge Distillation, Model Compression

 Knowledge distillation (KD) is a technique that transfers the knowledge from a large teacher network to a small student network. It has been widely applied to many different tasks, such as model compression and federated learning. However, the existing KD methods fail to generalize to general \textit{deep directed graphical models (DGMs)} with arbitrary layers of random variables. We refer by \textit{deep} DGMs to DGMs whose conditional distributions are parameterized by deep neural networks. In this work, we propose a novel unified knowledge distillation framework for deep DGMs on various applications. Specifically, we leverage the reparameterization trick to hide the intermediate latent variables, resulting in a compact DGM. Then we develop a surrogate distillation loss to reduce error accumulation through multiple layers of random variables. Moreover, we present the connections between our method and some existing knowledge distillation approaches. The proposed framework is evaluated on three applications: deep generative models compression, discriminative deep DGMs compression, and VAE continual learning. The results show that our distillation method outperforms the baselines in data-free compression of deep generative models, including variational autoencoder (VAE), variational recurrent neural networks (VRNN), and Helmholtz Machine (HM). Moreover, our method achieves good performance for discriminative deep DGMs compression. Finally, we also demonstrate that it significantly improves the continual learning performance of VAE.