pwc-archive/papers-with-abstracts huggingface dataset:
Efficient Winning Tickets Drawing over Fine-Grained Structured Sparsity
Authors: Sai Qian Zhang, Bradley McDanel
Date: 2021-09-29 00:00:00

The fine-grained structured sparsity has been proposed as a middle-ground between unstructured sparsity, where weights are pruned independently, and coarse-grained structured sparsity, where entire blocks of weights are pruned. Specifically, N:M fine-grained structured sparsity allows for at most N nonzero weights across a group of M consecutive weights. A recent implementation of 2:4 sparsity (N=2 and M=4) in Sparse Tensor Cores of Nvidia A100 GPUs shows significant improvement in throughput compared to unstructured sparsity while maintaining similar performance (e.g., accuracy). However, despite its potential for superior computational performance, how to efficiently train DNNs with N:M fine-grained structured sparsity remains a challenging problem. In this work, we leverage the recent advance of~\textit{Lottery Ticket Hypothesis} (LTH) and propose an iterative pruning algorithm for N:M fine-grained structured sparsity. By leveraging the N:M sparsity constraint, we can identify the unimportant weights across each group of M weights at earlier stages of iterative pruning, which significantly lowers the cost of iterative training compared to conventional unstructured pruning.
Federated Contrastive Representation Learning with Feature Fusion and Neighborhood Matching
Authors: Yawen Wu, Zhepeng Wang, Dewen Zeng, Meng Li, Yiyu Shi, Jingtong Hu
Date: 2021-09-29 00:00:00
Tasks: Contrastive Learning, Federated Learning, Representation Learning, Self-Supervised Learning

Federated learning (FL) enables distributed clients to learn a shared model for prediction while keeping the training data local on each client. However, existing FL requires fully-labeled data for training, which is inconvenient or sometimes infeasible to obtain due to the high labeling cost and the requirement of expertise. The lack of labels makes FL impractical in many realistic settings. Self-supervised learning can address this challenge by learning from unlabeled data such that FL can be widely used. Contrastive learning (CL), a self-supervised learning approach, can effectively learn data representations from unlabeled data. However, the distributed data collected on clients are usually not independent and identically distributed (non-IID) among clients, and each client may only have few classes of data, which degrades the performance of CL and learned representations. To tackle this problem, we propose a federated contrastive learning framework consisting of two approaches: feature fusion and neighborhood matching, by which a unified feature space among clients is learned for better data representations. Feature fusion provides remote features as accurate contrastive information to each client for better local learning. Neighborhood matching further aligns each client’s local features to the remote features such that well-clustered features among clients can be learned. Extensive experiments show the effectiveness of the proposed framework. It outperforms other methods by 11% on IID data and matches the performance of centralized learning.
Neuro-Symbolic Ontology-Mediated Query Answering
Authors: Medina Andresel, Daria Stepanova, Trung-Kien Tran, Csaba Domokos, Pasquale Minervini
Date: 2021-09-29 00:00:00
Tasks: Data Augmentation, Knowledge Graphs

Recently, low-dimensional vector space representations of Knowledge Graphs (KGs) have been applied to find answers to logical queries over incomplete KGs. However, the current methods only focus on inductive reasoning, i.e. answering such queries by predicting facts based on patterns learned from the data, and lack the ability of deductive reasoning, the task of computing logical entailments using expert domain knowledge. To address this shortcoming, we investigate how existing embedding models for query answering over incomplete KGs can be adapted to incorporate domain knowledge in the form of  ontologies. We propose two novel datasets, based on LUBM and NELL KGs, as well as various training strategies to integrate domain knowledge into prominent representatives of embedding models for query answering. Our strategies involve (1) different ontology-driven data augmentation techniques and (2) adaptation of the loss function using query-rewriting methods. The achieved improvements in the settings that require both inductive and deductive reasoning, are from 20% to 50% in HITS@3.
Hybrid Cloud-Edge Networks for Efficient Inference
Authors: Anil Kag, Igor Fedorov, Aditya Gangrade, Paul Whatmough, Venkatesh Saligrama
Date: 2021-09-29 00:00:00

Although deep neural networks (DNNs) achieve state-of-the-art accuracy on large-scale and fine-grained prediction tasks, they are high capacity models and often cannot be deployed on edge devices. As such, two distinct paradigms have emerged in parallel: 1) edge device inference for low-level tasks, 2) cloud-based inference for large-scale tasks. We propose a novel hybrid option, which marries these extremes and seeks to bring the latency and computational cost benefits of edge device inference to tasks currently deployed in the cloud. Our proposed method is an end-to-end approach, and involves architecting and training two networks in tandem. The first network is a low-capacity network that can be deployed on an edge device, whereas the second is a high-capacity network deployed in the cloud. When the edge device encounters challenging inputs, these inputs are transmitted and processed on the cloud. Empirically, on the ImageNet classification dataset, our proposed method leads to substantial decrease in the number of floating point operations (FLOPs) used compared to a well-designed high-capacity network, while suffering no excess classification loss. A novel aspect of our method is that, by allowing abstentions on a small fraction of examples ($<20\%$), we can increase accuracy without increasing the edge device memory and FLOPs substantially (up to $7$\% higher accuracy and $3$X fewer FLOPs on ImageNet with $80$\% coverage), relative to MobileNetV3 architectures.
Learning-to-Count by Learning-to-Rank: Weakly Supervised Object Counting & Localization Using Only Pairwise Image Rankings
Authors: Adriano C. D'Alessandro, Ali Mahdavi Amiri, Ghassan Hamarneh
Date: 2021-09-29 00:00:00
Tasks: Learning-To-Rank, Object, Object Counting

Object counting and localization in dense scenes is a challenging class of image analysis problems that typically requires labour intensive annotations to learn to solve. We propose a form of weak supervision that only requires object-based pairwise image rankings. These annotations can be collected rapidly with a single click per image pair and supply a weak signal for object quantity. However, the problem of actually extracting object counts and locations from rankings is challenging. Thus, we introduce adversarial density map generation, a strategy for regularizing the features of a ranking network such that the features correspond to an object proposal map where each proposal must be a Gaussian blob that integrates to 1. This places a soft integer and soft localization constraint on the representation, which encourages the network to satisfy the provided ranking constraints by detecting objects. We then demonstrate the effectiveness of our method for exploiting pairwise image rankings as a weakly supervised signal for object counting and localization on several datasets, and show results with a performance that approaches that of fully supervised methods on many counting benchmark datasets while relying on data that can be collected with a fraction of the annotation burden.
Understanding Graph Learning with Local Intrinsic Dimensionality
Authors: Xiaojun Guo, Xingjun Ma, Yisen Wang
Date: 2021-09-29 00:00:00
Tasks: Graph Learning

Many real-world problems can be formulated as graphs and solved by graph learning techniques. Whilst the rise of Graph Neural Networks (GNNs) has greatly advanced graph learning, there is still a lack of understanding of the intrinsic properties of graph data and their impact on graph learning. In this paper, we narrow the gap by studying the intrinsic dimension of graphs with \emph{Local Intrinsic Dimensionality (LID)}. The LID of a graph measures the expansion rate of the graph as the local neighborhood size of the nodes grows.
With LID, we estimate and analyze the intrinsic dimensions of node features, graph structure and representations learned by GNNs. We first show that feature LID (FLID) and structure LID (SLID) are well correlated with the complexity of synthetic graphs. Following this, we conduct a comprehensive analysis of 12 popular graph datasets of diverse categories and show that 1) graphs of lower FLIDs and SLIDs are generally easier to learn; 2) GNNs learn by mapping graphs (feature and structure together) to low-dimensional manifolds that are of much lower representation LIDs (RLIDs), i.e., RLID $\ll$ FLID/SLID; and 3) when the layers go deep in message-passing based GNNs, the underlying graph will converge to a complete graph of $\operatorname{SLID}=0.5$, losing structural information and causing the over-smoothing problem.
AutoNF: Automated Architecture Optimization of Normalizing Flows Using a Mixture Distribution Formulation
Authors: Yu Wang, Jan Drgona, Jiaxin Zhang, Karthik Somayaji NS, Frank Y Liu, Malachi Schram, Peng Li
Date: 2021-09-29 00:00:00

Although various flow models based on different transformations have been proposed, there still lacks a quantitative analysis of performance-cost trade-offs between different flows as well as a systematic way of constructing the best flow architecture. To tackle this challenge, we present an automated normalizing flow (NF) architecture search method. Our method aims to find the optimal sequence of transformation layers from a given set of unique transformations with three folds. First, a mixed distribution is formulated to enable efficient architecture optimization originally on the discrete space without violating the invertibility of the resulting NF architecture. Second, the mixture NF is optimized with an approximate upper bound which has a more preferable global minimum. Third, a block-wise alternating optimization algorithm is proposed to ensure efficient architecture optimization of deep flow models.
Automated hypothesis generation via Evolutionary Abduction
Authors: Roberto Pietrantuono
Date: 2021-09-29 00:00:00
Tasks: Causal Inference, Intent Recognition, Medical Diagnosis

Abduction is a powerful form of causal inference employed in many artificial intelligence tasks, such as medical diagnosis, criminology, root cause analysis, intent recognition. Given an effect, the abductive reasoning allows advancing a plausible set of explanatory hypotheses for its causes. This paper presents a new evolutionary strategy - called Evolutionary Abduction (EVA) - for automated abductive inference, aiming at effectively generating sets of hypotheses for explaining an occurred effect and/or predicting an effect that could occur in the future. EVA defines a set of abductive operators to repeatedly construct hypothetical cause-effect instances, and then automatically assesses their plausibility as well as their novelty with respect to already known instances - a mechanism mimicking the human reasoning employed whenever we need to select the best candidates from a set of hypotheses. Experiments with four datasets confirm that, given a background knowledge, EVA can construct new and realistic multiple-cause hypotheses for a given effect. EVA outperforms alternative strategies based on causal structure discovery, generating closer-to-real instances in most settings and datasets.
Towards Robust Domain Generalization in 2D Neural Audio Processing
Authors: Byeonggeun Kim, Seunghan Yang, Jangho Kim, Hyunsin Park, Jun-Tae Lee, Simyung Chang
Date: 2021-09-29 00:00:00
Tasks: Acoustic Scene Classification, Domain Generalization, Keyword Spotting, Scene Classification, Speaker Verification

While using two-dimensional convolutional neural networks (2D-CNNs) in image processing, it is possible to manipulate domain information using channel statistics, and instance normalization has been a promising way to get domain-invariant features. Although 2D image features represent spatial information, 2D audio features like log-Mel spectrogram represent two different temporal and spectral information. Unlike image processing, we analyze that domain-relevant information in the audio feature is dominant in frequency statistics rather than channel statistics. Motivated by our analysis, we introduce RFN, a plug-and-play, explicit normalization module along the frequency axis, eliminating instance-specific domain discrepancy in the audio feature while relaxing undesirable loss of useful discriminative information. Empirically, simply adding RFN to networks shows clear margins compared to previous domain generalization approaches on acoustic scene classification, keyword spotting, and speaker verification tasks and yields improved robustness to audio-device, speaker-ID, or genre.
On the approximation properties of recurrent encoder-decoder architectures
Authors: Zhong Li, Haotian Jiang, Qianxiao Li
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Decoder

Encoder-decoder architectures have recently gained popularity in sequence to sequence modelling, featuring in state-of-the-art models such as transformers. However, a mathematical understanding of their working principles remains limited. In this paper, we study the approximation properties of recurrent encoder-decoder architectures. Prior work established theoretical results for classical RNNs in the linear setting, where approximation capabilities can be related to smoothness and memory. Here, we find that the encoder and decoder together form a particular “temporal product structure” which determines the approximation efficiency. Moreover, the encoder-decoder architecture generalises RNNs with the capability to learn time-inhomogeneous relationships. Our results provide the theoretical understanding of approximation properties of the recurrent encoder-decoder architecture, which characterises, in the considered setting, the types of temporal relationships that can be efficiently learned.
Learning to Estimate Epistemic Uncertainty in Neural Networks
Authors: Katherine Elizabeth Brown, Doug Talbert
Date: 2021-09-29 00:00:00
Tasks: regression, Uncertainty Quantification

Epistemic uncertainty quantification provides useful insight into a deep neural network's understanding of the relationship between its training distribution and unseen instances. A Bayesian-based approaches have been shown to quantify this relationship better than softmax probabilities. Unfortunately, however, those approaches to uncertainty quantification require multiple Monte-Carlo samples of a neural network, augmenting the neural network to learn distributions for its weights, or utilizing an ensemble of neural networks. Such extra calculations are problematic in time-critical, resource-limited scenarios such as trauma triage. In this work, we propose a technique that allows epistemic uncertainty to be estimated using learned regression algorithms. We find that this technique, once trained, allows epistemic uncertainty to be effectively and efficiently predicted.
Few-Shot Classification with Task-Adaptive Semantic Feature Learning
Authors: Meihong Pan, Chunqiu Xia, Hongyi Xin, Yang Yang, Xiaoyong Pan, Hong-Bin Shen
Date: 2021-09-29 00:00:00
Tasks: Classification

Few-shot classification aims to learn a classifier that categorizes objects of unseen classes with limited samples. One general approach is to mine as much information as possible from limited samples. This can be achieved by incorporating data aspects from multiple modals. However, existing multi-modality methods only use additional modality in support samples while adhering to a single modal in query samples. Such approach could lead to information imbalance between support and query samples, which confounds model generalization from support to query samples. Towards this problem, we propose a task-adaptive semantic feature learning mechanism to incorporates semantic features for both support and query samples. The semantic feature learner is trained episodic-wisely by regressing from the feature vectors of support samples. Then the query samples can obtain the semantic features with this module. Such method maintains a consistent training scheme between support and query samples and enables direct model transfer from support to query datasets, which significantly improves model generalization. We develop two modality combination implementations: feature concatenation and feature fusion, based on the semantic feature learner. Extensive experiments conducted on four benchmarks demonstrate that our method outperforms state-of-the-arts, proving the effectiveness of our method.
Pairwise Adversarial Training for Unsupervised Class-imbalanced Domain Adaptation
Authors: Weili Shi, Ronghang Zhu, Sheng Li
Date: 2021-09-29 00:00:00
Tasks: Domain Adaptation, Transfer Learning, Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) has become an appealing approach for knowledge transfer from a labeled source domain to an unlabeled target domain. However, when the classes in source and target domains are imbalanced, most existing UDA methods experience significant performance drop, as the decision boundary usually favors the majority classes. Some recent class-imbalanced domain adaptation (CDA) methods aim to tackle the challenge of biased label distribution by exploiting pseudo-labeled target data during training process. However, these methods may be challenged with the problem of unreliable pseudo labels and error accumulation during training. In this paper, we propose a pairwise adversarial training approach to augment training data for unsupervised class-imbalanced domain adaptation. Unlike conventional adversarial training in which the adversarial samples are obtained from the $\ell_p$ ball of the original data, we obtain the semantic adversarial samples from the interpolated line of the aligned pair-wise samples from source domain and target domain. Experimental results and ablation study show that our method can achieve considerable improvements on the CDA benchmarks compared with the state-of-art methods focusing on the same problem.
Variable Length Variable Quality Audio Steganography
Authors: Seungmo Ku
Date: 2021-09-29 00:00:00
Tasks: Image Reconstruction

Steganography is the task of hiding and recovering secret data inside a non-secret container data while making imperceptible changes to the container. When using steganography to hide audio inside an image, current approaches neither allow the encoding of a signal with variable length nor allow making a trade-off between secret data reconstruction quality and imperceptibility in the changes made to the container image. To address this problem, we propose VLVQ (Variable Length Variable Quality Audio Steganography), a deep learning based steganographic framework capable of hiding variable-length audio inside an image by training the network to iteratively encode and decode the audio data from the container image. Complementary to the standard reconstruction loss, we propose an optional conditional loss term that allows the users to make quality trade-offs between audio and image reconstruction on inference time, without needing to train a separate model for each trade-off setups. Our experiments on ImageNet and AudioSet demonstrate VLVQ’s ability to retain reasonable image quality (28.99 $psnr$) and audio reconstruction quality (23.79 $snrseg$) while encoding 19 seconds of audio. We also show VLVQ’s capability to generalize to signals longer than what is seen during training.
Faking Interpolation Until You Make It
Authors: Alasdair Paren, Rudra Poudel, M. Pawan Kumar
Date: 2021-09-29 00:00:00

Deep over-parameterized neural networks exhibit the interpolation property on many data sets. That is, these models are able to achieve approximately zero loss on all training samples simultaneously. Recently, this property has been exploited to develop novel optimisation algorithms for this setting. These algorithms use the fact that the optimal loss value is known to employ a variation of a Polyak step-size calculated on a stochastic batch of data. We introduce a novel extension of this idea to tasks where the interpolation property does not hold. As we no longer have access to the optimal loss values a priori, we instead estimate these for each sample online. To realise this, we introduce a simple but highly effective heuristic for approximating the optimal value based on previous loss evaluations. This heuristic starts by setting the approximate optimal values to a known lower bound on the loss function, typically zero. It then updates them at fixed intervals through training in the direction of the best iterate visited so far. We provide rigorous experimentation on a wide range of problems including two natural language processing tasks, popular vision benchmarks and the challenging ImageNet classification data set. From our empirical analysis we demonstrate the effectiveness of our approach, which in the non-interpolating setting, outperforms state of the art baselines, namely adaptive gradient and line search methods.
Provably Robust Transfer
Authors: Anna-Kathrin Kopetzki, Jana Obernosterer, Aleksandar Bojchevski, Stephan Günnemann
Date: 2021-09-29 00:00:00
Tasks: Adversarial Robustness, Transfer Learning

Knowledge transfer is an effective tool for learning, especially when labeled data is scarce or when training from scratch is prohibitively costly. The overwhelming majority of transfer learning literature is focused on obtaining accurate models, neglecting the issue of adversarial robustness. Yet, robustness is essential, particularly when transferring to safety-critical domains. We analyze and improve the robustness of a popular transfer learning framework consisting of two parts: a feature extractor and a classifier which is re-trained on the target domain. Our experiments show how adversarial training on the source domain affects robustness on source and target domain, and we propose the first provably robust transfer learning models. We obtain strong robustness guarantees by bounding the worst-case change in the extracted features while controlling the Lipschitz constant of the classifier. Our models maintain high accuracy while significantly improving provable robustness.
Clustered Task-Aware Meta-Learning by Learning from Learning Paths
Authors: Danni Peng, Sinno Pan
Date: 2021-09-29 00:00:00
Tasks: Clustering, Few-Shot Image Classification, image-classification, Image Classification, Meta-Learning

To enable effective learning of new tasks with only few samples, meta-learning acquires common knowledge from the existing tasks with a globally shared meta-learner. To further address the problem of task heterogeneity, recent developments balance between customization and generalization by incorporating task clustering to generate the task-aware modulation to be applied on the global meta-learner. However, these methods learn task representation mostly from the features of input data, while the task-specific optimization process with respect to the base-learner model is often neglected. In this work, we propose a Clustered Task-Aware Meta-Learning (CTML) framework with task representation learned from its own learning path. We first conduct a rehearsed task learning from the common initialization, and collect a set of geometric quantities that adequately describes this learning path. By inputting this set of values into a meta path learner, we automatically abstract path representation optimized for the downstream clustering and modulation. To further save the computational cost incurred by the additional rehearsed learning, we devise a shortcut tunnel to directly map between the path and feature cluster assignments. Extensive experiments on two real-world application domains: few-shot image classification and cold-start recommendation demonstrate the superiority of CTML compared to state-of-the-art baselines.