pwc-archive/papers-with-abstracts huggingface dataset:
Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-Task Model Fusion
Authors: Nghia Hoang, Thanh Lam, Bryan Kian Hsiang Low, Patrick Jaillet 
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Knowledge Distillation

Model fusion is an emerging study in collective learning where heterogeneous experts with private data and learning architectures need to combine their black-box knowledge for better performance. Existing literature achieves this via a local knowledge distillation scheme that transfuses the predictive patterns of each pre-trained expert onto a white-box imitator model, which can be incorporated efficiently into a global model. This scheme however does not extend to multi-task scenarios where different experts were trained to solve different tasks and only part of their distilled knowledge is relevant to a new task. To address this multi-task challenge, we develop a new fusion paradigm that represents each expert as a distribution over a spectrum of predictive prototypes, which are isolated from task-specific information encoded within the prototype distribution. The task-agnostic prototypes can then be reintegrated to generate a new model that solves a new task encoded with a different prototype distribution. The fusion and adaptation performance of the proposed framework is demonstrated empirically on several real-world benchmark datasets.
On hyperparameter tuning in general clustering problemsm
Authors: Xinjie Fan, Yuguang Yue, Purnamrita Sarkar, Y. X. Rachel Wang
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Clustering, Community Detection, Model Selection

Tuning hyperparameters for unsupervised learning problems is difficult in general due to the lack of ground truth for validation. However, the success of most clustering methods depends heavily on the correct choice of the involved hyperparameters. Take for example the Lagrange multipliers of penalty terms in semidefinite programming (SDP) relaxations of community detection in networks, or the bandwidth parameter needed in the Gaussian kernel used to construct similarity matrices for spectral clustering. Despite the popularity of these clustering algorithms, there are not many provable methods for tuning these hyperparameters. In this paper, we provide a overarching framework with provable guarantees for tuning hyperparameters in the above class of problems under two different models. Our framework can be augmented with a cross validation procedure  to do model selection  as well. In a variety of simulation and real data experiments, we show that our framework outperforms other widely used tuning procedures in a broad range of parameter settings.
Optimal Differential Privacy Composition for Exponential Mechanisms
Authors: Jinshuo Dong, David Durfee, Ryan Rogers
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Composition is one of the most important properties of differential privacy (DP), as it allows algorithm designers to build complex private algorithms from DP primitives. We consider precise composition bounds of the overall privacy loss for exponential mechanisms, one of the fundamental classes of mechanisms in DP. Exponential mechanism has also become a fundamental building block in private machine learning, e.g. private PCA and hyper-parameter selection. We give explicit formulations of the optimal privacy loss for both the adaptive and non-adaptive composition of exponential mechanism. For the non-adaptive setting in which each mechanism has the same privacy parameter, we give an efficiently computable formulation of the optimal privacy loss. In the adaptive case, we derive a recursive formula and an efficiently computable upper bound. These precise understandings about the problem lead to a 40\% saving of the privacy budget in a practical application. Furthermore, the algorithm-specific analysis shows a difference in privacy parameters of adaptive and non-adaptive composition, which was widely believed to not exist based on the evidence from general analysis.
Recht-Re Noncommutative Arithmetic-Geometric Mean Conjecture is False
Authors: Zehua Lai, Lek-Heng Lim
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Stochastic Optimization

Stochastic optimization algorithms have become indispensable in machine learning. An unresolved foundational question in this area is the difference between with-replacement sampling and without-replacement sampling --- does the latter have superior convergence rate compared to the former? A groundbreaking result of Recht and Re reduces the problem to a noncommutative analogue of the arithmetic-geometric mean inequality where positive numbers are replaced by n positive definite matrices. If this inequality holds for all n, then without-replacement sampling indeed outperforms with-replacement sampling. The conjectured Recht--Re inequality has so far only been established for n = 2 and a special case of n = 3. We will show that the Recht--Re conjecture is false for general n. Our approach relies on the noncommutative positivstellensatz, which allows us to reduce the conjectured inequality to a semidefinite program and the validity of the conjecture to certain bounds for the optimum values, which we show are false as soon as n = 5.
Understanding and Stabilizing GANs' Training Dynamics Using Control Theory
Authors: Kun Xu, Chongxuan Li, Jun Zhu, Bo Zhang
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Generative adversarial networks (GANs) are effective in generating realistic images but the training is often unstable.
There are existing efforts that model the training dynamics of GANs in the parameter space but the analysis cannot directly motivate practically effective stabilizing methods.
To this end, we present a conceptually novel perspective from control theory to directly model the dynamics of GANs in the frequency domain and provide simple yet effective methods to stabilize GAN's training.
We first analyze the training dynamic of a prototypical Dirac GAN and adopt the widely-used closed-loop control (CLC) to improve its stability. We then extend CLC to stabilize the training dynamic of normal GANs, which can be implemented as an L2 regularizer on the output of the discriminator. Empirical results show that our method can effectively stabilize the training and obtain state-of-the-art performance on data generation tasks.
Uniform Convergence of Rank-weighted Learning
Authors: Liu Leqi, Justin Khim, Adarsh Prasad, Pradeep Ravikumar
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: BIG-bench Machine Learning, Decision Making, regression

The decision-theoretic foundations of classical machine learning models have largely focused on estimating model parameters that minimize the expectation of a given loss function. However, as machine learning models are deployed in varied contexts, such as in high-stakes decision-making and societal settings, it is clear that these models are not just evaluated by their average performances. In this work, we study a novel notion of L-Risk based on the classical idea of rank-weighted learning. These L-Risks, induced by rank-dependent weighting functions with bounded variation, is a unification of popular risk measures such as conditional value-at-risk and those defined by cumulative prospect theory. We give uniform convergence bounds of this broad class of risk measures and study their consequences on a logistic regression example.
Variance Reduction and Quasi-Newton for Particle-Based Variational Inference
Authors: Michael Zhu, Chang Liu, Jun Zhu
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Bayesian Inference, Riemannian optimization, Variational Inference

Particle-based Variational Inference methods (ParVIs), like Stein Variational Gradient Descent, are nonparametric variational inference methods that optimize a set of particles to best approximate a target distribution. ParVIs have been proposed as efficient approximate inference algorithms and as potential alternatives to MCMC methods. However, to our knowledge, the quality of the posterior approximation of particles from ParVIs has not been examined before for challenging, large-scale Bayesian inference problems. In this paper, we find that existing ParVI approaches converge insufficiently fast under sample quality metrics, and we propose a novel variance reduction and quasi-Newton preconditioning framework for all ParVIs, by leveraging the Riemannian structure of the Wasserstein space and advanced Riemannian optimization algorithms. Experimental results demonstrate the accelerated convergence of ParVIs for accurate posterior inference in large-scale and ill-conditioned problems.
Variational Inference for Sequential Data with Future Likelihood Estimates
Authors: Geon-Hyeong Kim, Youngsoo Jang, Hongseok Yang, Kee-Eung Kim
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Variational Inference

The recent development of flexible and scalable variational inference algorithms has popularized the use of deep probabilistic models in a wide range of applications. However, learning and reasoning about high-dimensional models with non-differentiable densities are still a challenge. For such a model, inference algorithms struggle to estimate the gradients of variational objectives accurately, due to high variance in their estimates. To tackle this challenge, we present a novel variational inference algorithm for sequential data, which performs well even when the density from the model is not differentiable, for instance, due to the use of discrete random variables. The key feature of our algorithm is that it estimates future likelihoods at all time steps. The estimated future likelihoods form the core of our new low-variance gradient estimator. We formally analyze our gradient estimator from the perspective of variational objective, and show the effectiveness of our algorithm with synthetic and real datasets.
When Demands Evolve Larger and Noisier: Learning and Earning in a Growing Environment
Authors: Feng Zhu, Zeyu Zheng
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We consider a single-product dynamic pricing problem under a specific non-stationary setting, where the demand grows over time in expectation and possibly gets noisier. The decision maker dynamically sets price and learns the unknown price elasticity, with the goal of maximizing expected cumulative revenue. We prove matching upper and lower bounds on regret and provide near-optimal pricing policies. We show how the change in demand uncertainty over time affects the optimal policy design and demonstrate how the order of optimal regret depends on the magnitude of demand uncertainty evolvement. Moreover, we distinguish between the \textit{any-time} situation and the \textit{fixed-time} situation by whether the seller knows the total number of time periods $T$ in advance or not, and show that they surprisingly render different optimal regret orders. We then extend the demand model to a more general case allowing for an additional intercept term and present a novel and near-optimal algorithm for the extended model. Finally, we consider an analogous non-stationary setting in the canonical multi-armed bandit problem, and points out that the \textit{any-time} situation and the \textit{fixed-time} situation render the same optimal regret order in a simple form, in contrast to the dynamic pricing problem.
Cost-effective Interactive Attention Learning with Neural Attention Process
Authors: Jay Heo, Junhyeon Park, Hyewon Jeong, Kwang Joon Kim, Juho Lee, Eunho Yang, Sung Ju Hwang
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Reranking, Time Series, Time Series Analysis

We propose a novel interactive learning framework which we refer to as Interactive Attention Learning (IAL), in which the human supervisors interactively manipulate the allocated attentions, to correct the model's behavior by updating the attention-generating network. However, such a model is prone to overfitting due to scarcity of human annotations, and requires costly retraining. Moreover, it is almost infeasible for the human annotators to examine attentions on tons of instances and features. We tackle these challenges by proposing a sample-efficient attention mechanism and a cost-effective reranking algorithm for instances and features. First, we propose Neural Attention Process (NAP), which is an attention generator that can update its behavior by incorporating new attention-level supervisions without any retraining. Secondly, we propose an algorithm which prioritizes the instances and the features by their negative impacts, such that the model can yield large improvements with minimal human feedback. We validate IAL on various time-series datasets from multiple domains (healthcare, real-estate, and computer vision) on which it significantly outperforms baselines with conventional attention mechanisms, or without cost-effective reranking, with substantially less retraining and human-model interaction cost.
Few-shot Relation Extraction via Bayesian Meta-learning on Task Graphs
Authors: Meng Qu, Tianyu Gao, Louis-Pascal Xhonneux, Jian Tang
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Graph Neural Network, Meta-Learning, Relation, Relation Extraction, Sentence

This paper studies few-shot relation extraction, which aims at predicting the relation for a pair of entities in a sentence by training with a few labeled examples in each relation. To more effectively generalize to new relations/tasks, in this paper we study the relationships between different tasks and propose to leverage a global task graph. We propose a novel Bayesian meta-learning approach to effectively learn the posterior distributions of the prototype vectors of tasks, where the initial prior of the prototype vectors is parameterized with a graph neural network on the global task graph. Moreover, to effectively optimize the posterior distributions of the prototype vectors, we propose to use the stochastic gradient Langevin dynamic, which can be related to the MAML algorithm but is able to handle the uncertainty of the prototype vectors. The whole framework can be effectively and efficiently optimized in an end-to-end fashion. Experiments on two benchmark datasets prove the effectiveness of our proposed approach against competitive baselines in both the few-shot and zero-shot settings.
Learning General-Purpose Controllers via Locally Communicating Sensorimotor Modules
Authors: Wenlong Huang, Igor Mordatch, Deepak Pathak
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Reinforcement learning is typically concerned with learning control policies tailored to a particular agent. We investigate whether there exists a single policy that generalizes to controlling a wide variety of agent morphologies - ones in which even dimensionality of state and action spaces changes. Such a policy would distill general and modular sensorimotor patterns that can be applied to control arbitrary agents. We propose a policy expressed as a collection of identical modular neural network components for each of the agent’s actuators. Every module is only responsible for controlling its own actuator and receives information from its local sensors. In addition, messages are passed between modules, propagating information between distant modules. A single modular policy can successfully generate locomotion behaviors for over 20 planar morphologies such as monopod hoppers, quadrupeds, bipeds and generalize to variants not seen during training - a process that would normally require training and manual hyper-parameter tuning for each morphology. We observe a wide variety of drastically diverse locomotion styles across morphologies as well as centralized coordination emerging via message passing between decentralized modules purely from the reinforcement learning objective.
One-shot Distributed Ridge Regression in High Dimensions
Authors: Yue Sheng, Edgar Dobriban
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Distributed Computing, regression, Unity, Vocal Bursts Intensity Prediction

To scale up data analysis, distributed and parallel computing approaches are increasingly needed. Here we study a fundamental problem in this area: How to do ridge regression in a distributed computing environment? We study one-shot methods constructing weighted combinations of ridge regression estimators computed on each machine. By analyzing the mean squared error in a high dimensional model where each predictor has a small effect, we discover several new phenomena including that the efficiency depends strongly on the signal strength, but does not degrade with many workers, the risk decouples over machines, and the unexpected consequence that the optimal weights do not sum to unity. We also propose a new optimally weighted one-shot ridge regression algorithm. Our results are supported by simulations and real data analysis.
Parameter-free, Dynamic, and Strongly-Adaptive Online Learning
Authors: Ashok Cutkosky
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We provide a new online learning algorithm that for the first time combines several disparate notions of adaptivity. First, our algorithm obtains a ``parameter-free'' regret bound that adapts to the norm of the comparator and the squared norm of the size of the gradients it observes. Second, it obtains a ``strongly-adaptive'' regret bound, so that for any given interval of length $N$, the regret over the interval is $\tilde O(\sqrt{N})$. Finally, our algorithm obtains an optimal ``dynamic'' regret bound: for any sequence of comparators with path-length $P$, our algorithm obtains regret $\tilde O(\sqrt{PN})$ over intervals of length $N$. Our primary technique for achieving these goals is a new method of combining constrained online learning regret bounds that does not rely on an expert meta-algorithm to aggregate learners.
Pseudo-Masked Language Models for Unified Language Model Pre-Training
Authors: Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, Hsiao-Wuen Hon
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Decoder, Language Modeling, Language Modelling, Natural Language Understanding, Position

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-relations between corrupted tokens and context via autoencoding, and pseudo masks to learn intra-relations between masked spans via partially autoregressive modeling. With well-designed position embeddings and self-attention masks, the context encodings are reused to avoid redundant computation. Moreover, conventional masks used for autoencoding provide global masking information, so that all the position embeddings are accessible in partially autoregressive language modeling. In addition, the two tasks pre-train a unified language model as a bidirectional encoder and a sequence-to-sequence decoder, respectively. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.
The Performance Analysis of Generalized Margin Maximizers on Separable Data
Authors: Fariborz Salehi, Ehsan Abbasi, Babak Hassibi
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Binary Classification

Logistic models are commonly used for binary classification tasks. The success of such models has often been attributed to their connection to maximum-likelihood estimators. It has been shown that SGD algorithms , when applied on the logistic loss, converge to the max-margin classifier (a.k.a. hard-margin SVM). The performance of hard-margin SVM has been recently analyzed in~\cite{montanari2019generalization, deng2019model}. Inspired by these results, in this paper, we present and study a more general setting, where the underlying parameters of the logistic model possess certain structures (sparse, block-sparse, low-rank, etc.) and introduce a more general framework (which is referred to as “Generalized Margin Maximizer”, GMM). While classical max-margin classifiers minimize the 2-norm of the parameter vector subject to linearly separating the data, GMM minimizes any arbitrary convex function of the parameter vector. We provide a precise performance analysis of the generalization error of such methods and show improvement over the max-margin method which does not take into account the structure of the model. In future work we show that mirror descent algorithms, with a properly tuned step size, can be exploited to achieve GMM classifiers. Our theoretical results are validated by extensive simulation results across a range of parameter values, problem instances, and model structures.
Active World Model Learning in Agent-rich Environments with Progress Curiosity
Authors: Kuno Kim, Megumi Sano, Julian De Freitas, Nick Haber, Daniel Yamins
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

World models are a family of predictive models that solve self-supervised problems on how the world evolves. Humans learn world models by curiously exploring their environment, in the process acquiring compact abstractions of high bandwidth sensory inputs, the ability to plan across long temporal horizons, and an understanding of the behavioral patterns of other agents. In this work, we study how to design such a curiosity-driven Active World Model Learning (AWML) system. To do so, we simulate a curious agent building world models while visually exploring a 3D physical environment rich with distillations of representative real-world stimuli. We propose an AWML system driven by $\gamma$-Progress: a scalable and effective learning progress-based curiosity signal. We show that $\gamma$-Progress is robust to "white noise" and naturally gives rise to an exploration policy that allocates attention in a balanced manner, with a preference towards agents displaying complex yet learnable behaviors. As a result, our $\gamma$-Progress driven controller achieves significantly higher AWML performance than baseline controllers equipped with state-of-the-art exploration strategies such as Random Network Distillation and Model Disagreement.
Bounding the fairness and accuracy of classifiers from population statistics
Authors: Sivan Sabato, Elad Yom-Tov
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Fairness

We consider the study of a classification model whose properties are impossible to estimate using a validation set, either due to the absence of such a set or because access to the classifier, even as a black-box, is impossible. Instead, only aggregate statistics on the rate of positive predictions in each of several sub-populations are available, as well as the true rates of positive labels in each of these sub-populations.  We show that these aggregate statistics can be used to lower-bound the discrepancy of a classifier, which is a measure that balances inaccuracy and unfairness. To this end, we define a new measure of unfairness, equal to the fraction of the population on which the classifier behaves differently, compared to its global, ideally fair behavior, as defined by the measure of equalized odds.  We propose an efficient and practical procedure for finding the best possible lower bound on the discrepancy of the classifier, given the aggregate statistics, and demonstrate in experiments the empirical tightness of this lower bound, as well as its possible uses on various types of problems, ranging from estimating the quality of voting polls to measuring the effectiveness of patient identification from internet search queries. The code and data are available at https://github.com/sivansabato/bfa.