pwc-archive/papers-with-abstracts huggingface dataset:
DAG-GPs: Learning Directed Acyclic Graph Structure For Multi-Output Gaussian Processes
Authors: Benjamin J. Ayton, Brian Williams
Date: 2021-01-01 00:00:00
Tasks: Gaussian Processes

Multi-output Gaussian processes (MOGPs) introduce correlations between outputs, but are subject to negative transfer, where learned correlations associate an output with another that is actually unrelated, leading to diminished predictive accuracy. Negative transfer may be countered by structuring the MOGP to follow conditional independence statements so that independent outputs do not correlate, but to date the imposed structures have been hand selected for specific applications. We introduce the DAG-GP model, which linearly combines latent Gaussian processes so that MOGP outputs follow a directed acyclic graph structure. Our method exposes a deep connection between MOGPs and Gaussian directed graphical models, which has not been explored explicitly in prior work. We propose to learn the graph from data prior to training the MOGP, so that correlations between outputs are only introduced when justified. Automated structure learning means no prior knowledge of the conditional independence between outputs is required, and training multiple MOGPs to identify the best structure is no longer necessary. Graph structure is learned by applying existing structure learning algorithms developed for graphical models to a downselected set of MOGP training data. Experiments on real world data sets show that with sufficiently expressive kernels, prediction error and likelihood are improved when using the DAG-GP model compared to state of the art exact MOGP methods.
CompOFA – Compound Once-For-All Networks for Faster Multi-Platform Deployment
Authors: Manas Sahni, Shreya Varshini, Alind Khare, Alexey Tumanov
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: All, GPU

The emergence of CNNs in mainstream deployment has necessitated methods to design and train efficient architectures tailored to maximize the accuracy under diverse hardware & latency constrains. To scale these resource-intensive tasks with an increasing number of deployment targets, Once-For-All (OFA) proposed an approach to jointly train several models at once with a constant training cost. However, this cost remains as high as $O(10^3)$ GPU hours and also suffers from a combinatorial explosion of potentially sub-optimal model configurations. We find that the cost of this one-shot training is dependent on the size of the model design space, and hence seek to speed up the training by constraining the design space to configurations with better accuracy-latency trade-offs. We incorporate the insights of compound relationships between model depth & width to build CompOFA, a design space smaller by several orders of magnitude. We demonstrate a 50% cut in the training time, dollar cost, and CO2 emissions over OFA due to the decreased interference in training this smaller design space. We also show that this smaller design space is dense enough to support equally accurate models for similar diversity of hardware and latency targets, as well as simplifying training and searching 
procedures.
Transferable Feature Learning on Graphs Across Visual Domains
Authors: Ronghang Zhu, Xiaodong Jiang, Jiasen Lu, Sheng Li
Date: 2021-01-01 00:00:00
Tasks: Domain Adaptation, Unsupervised Domain Adaptation

Unsupervised domain adaptation has attracted increasing attention in recent years, which adapts classifiers to an unlabeled target domain by exploiting a labeled source domain. To reduce discrepancy between source and target domains, adversarial learning methods are typically selected to seek domain-invariant representations by confusing the domain discriminator. However, classifiers may not be well adapted to such a domain-invariant representation space, as the sample-level and class-level data structures could be distorted during adversarial learning. In this paper, we propose a novel Transferable Feature Learning approach on Graphs (TFLG) for unsupervised adversarial domain adaptation, which jointly incorporates sample-level and class-level structure information across two domains. TFLG first constructs graphs for mini-batch samples, and identifies the class-wise correspondence across domains. A novel cross-domain graph convolutional operation is designed to jointly align the sample-level and class-level structures in two domains. Moreover, a memory bank is designed to further exploit the class-level information. Extensive experiments on benchmark datasets demonstrate the effectiveness of our approach, compared to the representative unsupervised domain adaptation methods.
Implicit Regularization of SGD via Thermophoresis
Authors: Mingwei Wei, David J. Schwab
Date: 2021-01-01 00:00:00

A central ingredient in the impressive predictive performance of deep neural networks is optimization via stochastic gradient descent (SGD). While some theoretical progress has been made, the effect of SGD in neural networks is still unclear, especially during the early phase of training. Here we generalize the theory of thermophoresis from statistical mechanics and show that there exists an effective entropic force from SGD that pushes to reduce the gradient variance. We study this effect in detail in a simple two-layer model, where the thermophoretic force functions to decreases the weight norm and activation rate of the units. The strength of this effect is proportional to squared learning rate and inverse batch size, and is more effective during the early phase of training when the model's predictions are poor. Lastly we test our quantitative predictions with experiments on various models and datasets.
FOC OSOD: Focus on Classification One-Shot Object Detection
Authors: Hanqing Yang, Huaijin Pi, SABA GHORBANI BARZEGAR, Yu Zhang
Date: 2021-01-01 00:00:00
Tasks: Classification, General Classification, Object, object-detection, Object Detection, One-Shot Object Detection

One-shot object detection (OSOD) aims at detecting all instances that are consistent with the category of the single reference image. OSOD achieves object detection by comparing the query image and the reference image. We observe that the essential problem behind the limited performance of OSOD is that OSOD generates a lot of false positives due to its poor classification ability. This paper analyzes the serious false positive problem in OSOD and proposes a Focus on Classification One-Shot Object Detection (FOC OSOD) framework, which is improved in two important aspects: (1) classification cascade head with the fixed IoU threshold can enhance the robustness of classification by comparing multiple close regions; (2) classification region deformation on the query feature and the reference feature to obtain a more effective comparison region. Without bells and whistles, a single FOC obtains 1.8% AP and 1.3% AP improvement on the seen classes and the unseen classes over a Siamese Faster R-CNN baseline on the MS-COCO dataset in the one-shot setting. The code will be available.
Max-sliced Bures Distance for Interpreting Discrepancies
Authors: Austin J. Brockmeier, Claudio Cesar Claros, Carlos H. Mendoza-Cardenas, Yüksel Karahan, Matthew S. Emigh, Luis Gonzalo Sanchez Giraldo
Date: 2021-01-01 00:00:00

We propose the max-sliced Bures distance, a lower bound on the max-sliced Wasserstein-2 distance, to identify the instances associated with the maximum discrepancy between two samples. By scoring instances by an energy-based witness function, the proposed divergence can identify the specific subsets that are not well matched between the distributions. Localizing these discrepancies can be used to detect and correct for covariate shift and to evaluate generative adversarial networks. Similar to the Fréchet distance, the max-sliced Bures distance can be computed using the first and second-order moments and is able to capture changes in higher-order statistics through a non-linear mapping. We explore two types of non-linear mappings: positive semidefinite kernels where the witness functions belong to a reproducing kernel Hilbert space, and task-relevant mappings learned by a neural network. In the context of samples of natural images, our approach provides an interpretation of the Fréchet inception distance by identifying the synthetic and natural instances that are either over-represented or under-represented with respect to the other sample. We apply the proposed measure to detect imbalances in class distributions in various data sets and to critique generative models.
Blind Pareto Fairness and Subgroup Robustness
Authors: Natalia Martinez, Martin Bertran, Afroditi Papadaki, Miguel R. D. Rodrigues, Guillermo Sapiro
Date: 2021-01-01 00:00:00
Tasks: Fairness

With the wide adoption of machine learning algorithms across various application domains, there is a growing interest in the fairness properties of such algorithms. The vast majority of the activity in the field of group fairness addresses disparities between predeﬁned groups based on protected features such as gender, age, and race, which need to be available at train, and often also at test, time. These approaches are static and retrospective, since algorithms designed to protect groups identified  a priori cannot anticipate and protect the needs of different at-risk groups in the future. In this work we analyze the space of solutions for worst-case fairness beyond demographics, and propose Blind Pareto Fairness (BPF), a method that leverages no-regret dynamics to recover a fair minimax classiﬁer that reduces worst-case risk of  any potential subgroup of sufﬁcient size, and guarantees that the remaining population receives the best possible level of service. BPF addresses  fairness beyond demographics, that is, it  does not rely on predeﬁned notions of at-risk groups, neither at train nor at test time. Our experimental results show that the proposed framework improves worst-case risk in multiple standard datasets, while simultaneously providing better levels of service for the remaining population, in comparison to competing methods.
Meta Gradient Boosting Neural Networks
Authors: Manqing Dong, Lina Yao, Xianzhi Wang, Xiwei Xu, Liming Zhu
Date: 2021-01-01 00:00:00
Tasks: Meta-Learning, regression

Meta-optimization is an effective approach that learns a shared set of parameters across tasks for parameter initialization in meta-learning.
A key challenge for meta-optimization based approaches is to determine whether an initialization condition can be generalized to tasks with diverse distributions to accelerate learning. 
To address this issue, we design a meta-gradient boosting framework that uses a base learner to learn shared information across tasks and a series of gradient-boosted modules to capture task-specific information to fit diverse distributions.
We evaluate the proposed model on both regression and classification tasks with multi-mode distributions. 
The results demonstrate both the effectiveness of our model in modulating task-specific meta-learned priors and its advantages on multi-mode distributions.
When Are Neural Pruning Approximation Bounds Useful?
Authors: Mitchell A Gordon
Date: 2021-01-01 00:00:00
Tasks: Network Pruning

   Approximation bounds for neural network pruning attempt to predict the trade-off between sparsity and fidelity while shrinking neural networks. In the first half of this paper, we empirically evaluate the predictive power of two recently proposed methods based on coreset algorithms. We identify several circumstances in which the bounds are loose or impractical to use and provide a brief analysis of the components of the bounds that contribute to those short-comings. In the second half, we examine the role of fine-tuning in prunability and observe that even tight approximation bounds would be poor predictors of accuracy after fine-tuning. This is because fine-tuning can recover large amounts of accuracy while simultaneously maintaining or increasing approximation error. We discuss the implications of these finding on the application of coreset-based pruning methods in practice and the role of approximation in the pruning community. Our code is available in the attached supplementary material.
MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training
Authors: Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, Christopher Re
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Efficient Neural Network, Language Modeling, Language Modelling, Recommendation Systems, Scheduling

Recent advances by practitioners in the deep learning community have breathed new life into Locality Sensitive Hashing (LSH), using it to reduce memory and time bottlenecks in neural network (NN) training. However, while LSH has sub-linear guarantees for approximate near-neighbor search in theory, it is known to have inefficient query time in practice due to its use of random hash functions. Moreover, when model parameters are changing, LSH suffers from update overhead. This work is motivated by an observation that model parameters evolve slowly, such that the changes do not always require an LSH update to maintain performance. This phenomenon points to the potential for a reduction in update time and allows for a modified learnable version of data-dependent LSH to improve query time at a low cost. We use the above insights to build MONGOOSE, an end-to-end LSH framework for efficient NN training. In particular, MONGOOSE is equipped with a scheduling algorithm to adaptively perform LSH updates with provable guarantees and learnable hash functions to improve query efficiency. Empirically, we validate MONGOOSE on large-scale deep learning models for recommendation systems and language modeling. We find that it achieves up to $8\%$ better accuracy compared to previous LSH approaches, with $3 \times$ speed-up.
GenQu: A Hybrid Framework for Learning Classical Data in Quantum States
Authors: Samuel A. Stein, Ray Marie Tischio, Betis Baheri, YiWen Chen, Ying Mao, Qiang Guan, Ang Li, Bo Fang
Date: 2021-01-01 00:00:00

Deep neural network-powered artificial intelligence has rapidly changed our daily life with various applications. However, as one of the essential steps of deep neural networks, training a heavily-weighted network requires a tremendous amount of computing resources. Especially in the post Moore's Law era, the limit of semiconductor fabrication technology has restricted the development of learning algorithms to cope with the increasing high-intensity training data. Meanwhile, quantum computing has exhibited its significant potential in terms of speeding up the traditionally compute-intensive workloads. For example, Google illustrates quantum supremacy by completing a sampling calculation task in 200 seconds, which is otherwise impracticable on the world's largest supercomputers. To this end, quantum-based learning becomes an area of interest, with the promising of a quantum speedup. In this paper, we propose GenQu, a hybrid and general-purpose quantum framework for learning classical data through quantum states. We evaluate GenQu with real datasets and conduct experiments on both simulations and real quantum computer IBM-Q. Our evaluation demonstrates that, comparing with classical solutions, the proposed models running on GenQu framework achieve similar accuracy with a much smaller number of qubits, while significantly reducing the parameter size by up to 95.8\% and converging speedup by 66.67% faster.
PolarNet: Learning to Optimize Polar Keypoints for Keypoint Based Object Detection
Authors: Wu Xiongwei, Doyen Sahoo, Steven Hoi
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: object-detection, Object Detection

A variety of anchor-free object detectors have been actively proposed as possible alternatives to the mainstream anchor-based detectors that often rely on complicated design of anchor boxes. Despite achieving promising performance at par with anchor-based detectors, the existing anchor-free detectors such as FCOS or CenterNet predict objects based on standard Cartesian coordinates, which often yield poor quality keypoints. Further, the bounding box regression is also scale-sensitive. In this paper, we propose a new anchor-free keypoint based detector ``"PolarNet", where keypoints are represented as a set of Polar coordinates instead of Cartesian coordinates. The ``PolarNet detector learns offsets pointing to the corners of objects in order to learn high quality keypoints. Additionally, PolarNet uses corner points to localize objects, making the localization scale-insensitive. Finally in our experiments, we show that PolarNet, an anchor-free detector, outperforms the existing anchor-free detectors, and is able to achieve highly competitive result on COCO test-dev benchmark ($48.0\%$ AP under the single-model single-scale testing) which is at par with the state-of-the-art two-stage anchor-based object detectors.
Robust Multi-view Representation Learning
Authors: Sibi Venkatesan, Kyle Miller, Artur Dubrawski
Date: 2021-01-01 00:00:00
Tasks: Representation Learning, Self-Driving Cars

Multi-view data has become ubiquitous, especially with multi-sensor systems like self-driving cars or medical patient-side monitors.

We look at modeling multi-view data through robust representation learning, with the goal of leveraging relationships between views and building resilience to missing information.
We propose a new flavor of multi-view AutoEncoders, the Robust Multi-view AutoEncoder, which explicitly encourages robustness to missing views.
The principle we use is straightforward: we apply the idea of drop-out to the level of views.
During training, we leave out views as input to our model while forcing it to reconstruct all of them.
We also consider a flow-based generative modeling extension of our approach in the case where all the views are available.

We conduct experiments for different scenarios: directly using the learned representations for reconstruction, as well as a two-step process where the learned representation is subsequently used as features for the data for a down-stream application.
Our synthetic and real-world experiments show promising results for the application of these models to robust representation learning.
Differentially Private Generative Models Through Optimal Transport
Authors: Tianshi Cao, Alex Bie, Karsten Kreis, Sanja Fidler
Date: 2021-01-01 00:00:00

Although machine learning models trained on massive data have led to breakthroughs in several areas, their deployment in privacy-sensitive domains remains limited due to restricted access to data. Generative models trained with privacy constraints on private data can sidestep this challenge and provide indirect access to the private data instead. We propose DP-Sinkhorn, a novel optimal transport-based generative method for learning data distributions from private data with differential privacy. DP-Sinkhorn relies on minimizing the Sinkhorn divergence---a computationally efficient approximation to the exact optimal transport distance---between the model and the data in a differentially private manner and also uses a novel technique for conditional generation in the Sinkhorn framework. Unlike existing approaches for training differentially private generative models, which are mostly based on generative adversarial networks, we do not rely on adversarial objectives, which are notoriously difficult to optimize, especially in the presence of noise imposed by the privacy constraints. Hence, DP-Sinkhorn is easy to train and deploy. Experimentally, despite our method's simplicity we improve upon the state-of-the-art on multiple image modeling benchmarks. We also show differentially private synthesis of informative RGB images, which has not been demonstrated before by differentially private generative models without the use of auxiliary public data.
R-LAtte: Attention Module for Visual Control via Reinforcement Learning
Authors: Mandi Zhao, Qiyang Li, Aravind Srinivas, Ignasi Clavera, Kimin Lee, Pieter Abbeel
Date: 2021-01-01 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Unsupervised Pre-training

Attention mechanisms are generic inductive biases that have played a critical role in improving the state-of-the-art in supervised learning, unsupervised pre-training and generative modeling for multiple domains including vision, language and speech. However, they remain relatively under-explored for neural network architectures typically used in reinforcement learning (RL) from high dimensional inputs such as pixels. In this paper, we propose and study the effectiveness of augmenting a simple attention module in the convolutional encoder of an RL agent. Through experiments on the widely benchmarked DeepMind Control Suite environments, we demonstrate that our proposed module can (i) extract interpretable task-relevant information such as agent locations and movements without the need for data augmentations or contrastive losses; (ii) significantly improve the sample-efficiency and final performance of the agents. We hope our simple and effective approach will serve as a strong baseline for future research incorporating attention mechanisms in reinforcement learning and control.
Re-examining Routing Networks for Multi-task Learning
Authors: Limeng Cui, Aaron Jaech
Date: 2021-01-01 00:00:00
Tasks: Multi-Task Learning, reinforcement-learning, Reinforcement Learning (RL), Transfer Learning

We re-examine Routing Networks, an approach to multi-task learning that uses reinforcement learning to decide parameter sharing with the goal of maximizing knowledge transfer between related tasks while avoiding task interference. These benefits come with the cost of solving a more difficult optimization problem. We argue that the success of this model depends on a few key assumptions and, when they are not satisfied, the difficulty of learning a good route can outweigh the benefits of the approach. In these cases, a simple unlearned routing strategy, which we propose, achieves the best results.
A Unified Spectral Sparsification Framework for Directed Graphs
Authors: Ying Zhang, Zhiqiang Zhao, Zhuo Feng
Date: 2021-01-01 00:00:00

Recent spectral graph sparsification research allows constructing nearly-linear-sized subgraphs that can well preserve the spectral (structural) properties of the original graph, such as the first few eigenvalues and eigenvectors of the graph Laplacian, leading to the development of a variety of nearly-linear time numerical and graph algorithms. However, there is not a unified approach that allows for truly scalable spectral sparsification of both directed and undirected graphs. For the first time, we prove the existence of linear-sized spectral sparsifiers for general directed
graphs and introduce a practically-efficient and unified spectral graph sparsification approach that allows sparsifying real-world, large-scale directed and undirected graphs with guaranteed preservation of the original graph spectra. By exploiting a highly-scalable (nearly-linear complexity) spectral matrix perturbation analysis framework for constructing nearly-linear sized (directed) subgraphs, it enables us to well preserve the key eigenvalues and eigenvectors of the original (directed) graph
Laplacians. The proposed method has been validated using various kinds of directed graphs obtained from public domain sparse matrix collections, showing promising results for solving directed graph Laplacians, spectral embedding, and partitioning of general directed graphs, as well as approximately computing (personalized) PageRank vectors.