pwc-archive/papers-with-abstracts huggingface dataset:
CURL: Contrastive Unsupervised Representation Learning for Reinforcement Learning
Authors: Michael Laskin, Pieter Abbeel, Aravind Srinivas
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Contrastive Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Representation Learning

Reinforcement Learning for control tasks where the agent learns from raw high dimensional pixels has proven to be difficult and sample-inefficient.  Operating on high-dimensional observational input poses a challenging credit assignment problem, which hinders the agent’s ability to learn optimal policies quickly. One promising approach to improve the sample efficiency of image-based RL algorithms is to learn low-dimensional representations from the raw input using unsupervised learning. To that end, we propose a new model: Contrastive Unsupervised Representation Learning for Reinforcement Learning (CURL). CURL extracts high level features from raw pixels using a contrastive learning objective and performs off-policy control on top of the extracted features. CURL achieves state-of-the-art performance and is the first image based algorithm across both model-free and model-based settings to nearly match the sample-efficiency and performance of state-based features on five out of the six DeepMind control benchmarks.
ECLIPSE: An Extreme-Scale Linear Program Solver for Web-Applications
Authors: Kinjal Basu, Amol Ghoting, Rahul Mazumder, Yao Pan
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Key problems arising in web applications (with millions of users and thousands of items) can be formulated as Linear Programs (LP) involving billions to trillions of decision variables and constraints. Despite the appeal of LP formulations, solving problems at these scales is well beyond the capabilities of existing LP solvers. Often ad-hoc decomposition rules are used to approximately solve these LPs, which have limited optimality guarantees and lead to sub-optimal performance in practice. In this work, we propose a distributed solver that solves a perturbation of the LP problems at scale. We propose a gradient-based algorithm on the smooth dual of the perturbed LP with computational guarantees. The main workhorses of our algorithm are distributed matrix-vector multiplications (with load balancing) and efficient projection operations on distributed machines. Experiments on real-world data show that our proposed LP solver, ECLIPSE, can solve problems with $10^{12}$ decision variables -- well beyond the capabilities of current solvers.
Evaluating Machine Accuracy on ImageNet
Authors: Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, Ludwig Schmidt
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We perform an in-depth evaluation of human accuracy on the ImageNet dataset. First, three expert labelers re-annotated 30,000 images from the original ImageNet validation set and the ImageNetV2 replication experiment with multi-label annotations to enable a semantically coherent accuracy measurement. Then we evaluated five trained humans on both datasets. The median of the five labelers outperforms the best publicly released ImageNet model by 1.5% on the original validation set and by 6.2% on ImageNetV2. Moreover, the human labelers see a substantially smaller drop in accuracy between the two datasets compared to the best available model (less than 1% vs 5.4%). Our results put claims of superhuman performance on ImageNet in context and show that robustly classifying ImageNet at human-level performance is still an open problem.
Generalized Neural Policies for Relational MDPs
Authors: Sankalp Garg, Aniket Bajpai, Mausam 
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

A Relational Markov Decision Process (RMDP) is a first-order representation to express all instances of a single probabilistic planning domain with possibly unbounded number of objects. 
Early work in RMDPs outputs generalized (instance-independent) first-order policies or value functions as a means to solve all instances of a domain at once. Unfortunately, this line of work met with limited success due to inherent limitations of the representation space used in such policies or value functions. Can neural models provide the missing link by easily representing more complex generalized policies, thus making them effective on all instances of a given domain?

We present the first neural approach for solving RMDPs, expressed in the probabilistic planning language of RDDL. Our solution first converts an RDDL instance into a ground DBN. We then extract a graph structure from the DBN. We train a relational neural model that computes an embedding for each node in the graph and also scores each ground action as a function over the first-order action variable and object embeddings on which the action is applied. In essence, this represents a neural generalized policy for the whole domain. Given a new test problem of the same domain, we can compute all node embeddings using trained parameters and score each ground action to choose the best action using a single forward pass without any retraining. Our experiments on nine RDDL domains from IPPC demonstrate that neural generalized policies are significantly better than random and sometimes even more effective than training a state-of-the-art deep reactive policy from scratch.
Global Decision-Making via Local Economic Transactions
Authors: Michael Chang, Sid Kaushik, S. Matthew Weinberg, Sergey Levine, Thomas Griffiths
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Decision Making

This paper seeks to establish a mechanism for directing a collection of simple, specialized, self-interested agents to solve what traditionally are posed as monolithic single-agent sequential decision problems with a central global objective. What makes it challenging to use a decentralized approach to collectively optimize a central objective is the difficulty in characterizing the equilibrium strategy profile of non-cooperative games. To overcome this challenge, we design a mechanism for defining the learning environment of each primitive agent for which we know that the optimal solution for the global objective coincides with a Nash equilibrium strategy profile of the agents optimizing their own local objectives. We then derive a learning algorithm for the system and empirically test to what extent the desired equilibrium is achieved. The system functions as an economy of agents that learn the credit assignment process itself by buying and selling to each other the right to operate on the environment state. We also show that redundancy not only enforces credit conservation but also improves robustness against suboptimal equilibria.
Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: Joint Gradient Estimation and Tracking
Authors: Haoran Sun, Songtao Lu, Mingyi Hong
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Stochastic Optimization

Many modern large-scale machine learning problems benefit from decentralized and stochastic optimization. Recent works have shown that utilizing both decentralized computing and local stochastic gradient estimates can outperform state-of-the-art centralized algorithms, in applications involving highly non-convex problems, such as training deep neural networks. 	
	
In this work, we propose a decentralized stochastic algorithm to deal with certain smooth non-convex problems where there are $m$ nodes in the system, and each node has a large number of samples (denoted as $n$). Differently from the majority of the existing decentralized learning algorithms for either stochastic or finite-sum problems, our focus is given to {\it both} reducing the total communication rounds among the nodes, while accessing the minimum number of local data samples. In particular, we propose an algorithm named D-GET (decentralized gradient estimation and tracking), which jointly performs decentralized gradient estimation (which estimates the local gradient using a subset of local samples) {\it and} gradient tracking (which tracks the global full gradient using local estimates). We show that, to achieve certain $\epsilon$  stationary solution of the deterministic finite sum problem, the proposed algorithm achieves an $\mathcal{O}(mn^{1/2}\epsilon^{-1})$ sample complexity and an $\mathcal{O}(\epsilon^{-1})$ communication complexity. These bounds significantly improve upon the best existing bounds of $\mathcal{O}(mn\epsilon^{-1})$ and $\mathcal{O}(\epsilon^{-1})$, respectively. Similarly, for online problems, the proposed method achieves an $\mathcal{O}(m \epsilon^{-3/2})$ sample complexity and  an $\mathcal{O}(\epsilon^{-1})$ communication complexity, while the best existing bounds are  $\mathcal{O}(m\epsilon^{-2})$ and $\mathcal{O}(\epsilon^{-2})$.
Inferring DQN structure for high-dimensional continuous control
Authors: Andrey Sakryukin, Chedy Raissi, Mohan Kankanhalli
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: continuous-control, Continuous Control, Deep Reinforcement Learning, MuJoCo, Vocal Bursts Intensity Prediction

Despite recent advancements in the field of Deep Reinforcement Learning, Deep Q-network (DQN) models still show lackluster performance on problems with high-dimensional action spaces.
The problem is even more pronounced for cases with high-dimensional continuous action spaces due to a combinatorial increase in the number of the outputs. 
Recent works approach the problem by dividing the network into multiple parallel or sequential (action) modules responsible for different discretized actions. 
However, there are drawbacks to both the parallel and the sequential approaches.
Parallel module architectures lack coordination between action modules, leading to extra complexity in the task, while a sequential structure can result in the vanishing gradients problem and exploding parameter space. 
In this work, we show that the compositional structure of the action modules has a significant impact on model performance.
We propose a novel approach to infer the network structure for DQN models operating with high-dimensional continuous actions.
Our method is based on the uncertainty estimation techniques introduced in the paper.
Our approach achieves state-of-the-art performance on MuJoCo environments with high-dimensional continuous action spaces.
Furthermore, we demonstrate the improvement of the introduced approach on a realistic AAA sailing simulator game.
Learning to Navigate in Synthetically Accessible Chemical Space Using Reinforcement Learning
Authors: Sai Krishna Gottipati, Boris Sattarov, Sufeng. Niu, Hao-Ran Wei, Yashaswi Pathak, Shengchao Liu, Simon Blackburn, Karam Thomas, Connor Coley, Jian Tang, Sarath Chandar, Yoshua Bengio
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Drug Discovery, Navigate, reinforcement-learning, Reinforcement Learning (RL), valid

Over the last decade, there has been significant progress in the field of machine learning-based de novo drug discovery, particularly in generative modeling of chemical structures. However, current generative approaches exhibit a significant challenge: they do not ensure the synthetic accessibility nor provide the synthetic routes of the proposed small molecules which limits their applicability. In this work, we propose a novel reinforcement learning (RL) setup for drug discovery that addresses this challenge by embedding the concept of synthetic accessibility directly into the de novo compound design system. In this setup, the agent learns to navigate through the immense synthetically accessible chemical space by subjecting initial commercially available molecules to valid chemical reactions at every time step of the iterative virtual synthesis process. The proposed environment for drug discovery provides a highly challenging test-bed for RL algorithms owing to the large state space and high-dimensional continuous action space with hierarchical actions. Our end-to-end approach achieves state-of-the-art performance when compared against other generative approaches for drug discovery. Moreover, we leverage our approach in a proof-of-concept that mimics the drug discovery process by generating novel HIV drug candidates. Finally, we describe how the end-to-end training conceptualized in this study represents an important paradigm in radically expanding the synthesizable chemical space and automating the drug discovery process.
Multiclass Neural Network Minimization via Tropical Newton Polytope Approximation
Authors: Georgios Smyrnis, Petros Maragos
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

The field of tropical algebra is closely linked with the domain of neural networks with piecewise linear activations, since their output can be described via tropical polynomials in the max-plus semiring.  In this work, we attempt to make use of methods stemming from a form of approximate division of such polynomials, which relies on the approximation of their Newton Polytopes, in order to minimize networks trained for multiclass classification problems. We make theoretical contributions in this domain, by proposing and analyzing methods which seek to reduce the size of such networks. In addition, we make experimental evaluations on the MNIST and Fashion-MNIST datasets, with our results demonstrating a significant reduction in network size, while retaining adequate performance.
Network Pruning by Greedy Subnetwork Selection
Authors: Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, Qiang Liu
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Network Pruning

Recent works on network pruning show that large deep neural networks are often highly redundant and one can find much smaller subnetworks with much lower computational cost without a significant drop of accuracy. Most existing methods of network pruning are based on eliminating unnecessary neurons from the large networks. In this work, we study a greedy forward selection approach following the opposite direction, which starts from an empty network, and gradually adds good neurons from the large network.  Theoretically, we show that the small networks pruned using our method achieve provably lower loss than small networks trained from scratch with the same size. It implies that the learned weight of large networks is important to the small pruned models. Practically, for architectures in mobile setting, we find that fine-tuning networks pruned using our method outperforms training them from scratch. Our method improves all the prior arts on learning compact networks, using architectures such as ResNet, MobilenetV2, MobileNetV3 and ProxylessNet on ImageNet. Our theory and empirical results highlight the benefits of fine-tuning networks from large models over training from scratch, which is different from the findings of Liu et al. (2019b).
A Flexible Latent Space Model for Multilayer Networks
Authors: Xuefei Zhang, Songkai Xue, Ji Zhu
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: model, parameter estimation

Entities often interact with each other through multiple types of relations, which are often represented as multilayer networks.  Multilayer networks among the same set of nodes usually share common structures, while each layer can possess its distinct node connecting behaviors. This paper proposes a flexible latent space model for multilayer networks for the purpose of capturing such characteristics. Specifically, the proposed model embeds each node with a latent vector shared among layers and a layer-specific effect for each layer; both elements together with a layer-specific connectivity matrix determine edge formations. To fit the model, we develop a projected gradient descent algorithm for efficient parameter estimation.  We also establish theoretical properties of the maximum likelihood estimators and show that the upper bound of the common latent structure's estimation error is inversely proportional to the number of layers under mild conditions. The superior performance of the proposed model is demonstrated through simulation studies and applications to two real-world data examples.
Bisection-Based Pricing for Repeated Contextual Auctions against Strategic Buyer
Authors: Anton Zhiyanov, Alexey Drutsa
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We are interested in learning algorithms that optimize revenue in repeated contextual posted-price auctions where a single seller faces a single strategic buyer. In our setting, the buyer  maximizes his expected cumulative discounted surplus, and his valuation of a good is assumed to be a fixed function of a $d$-dimensional context (feature) vector. We introduce a novel deterministic learning algorithm that is based on ideas of the Bisection method and has strategic regret upper bound of $O(\log^2 T)$. Unlike previous works, our algorithm does not require any assumption on the distribution of context information, and the regret guarantee holds for any realization of feature vectors (adversarial upper bound). To construct our algorithm we non-trivially adopted techniques of integral geometry to act against buyer strategicness and improved the penalization trick to work in contextual auctions.
Double-Loop Unadjusted Langevin Algorithm
Authors: Paul Rolland, Armin Eftekhari, Ali Kavis, Volkan Cevher
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

A well-known first-order method for sampling from  log-concave probability distributions is the Unadjusted Langevin Algorithm (ULA). This work 
proposes a new annealing step-size schedule for ULA, which allows to prove new convergence guarantees for sampling from a smooth log-concave distribution, which are not covered by existing state-of-the-art convergence guarantees. To establish this result, we derive a new theoretical bound that relates the Wasserstein distance to total variation distance between any two log-concave distributions that complements the reach of Talagrand $T_2$ inequality. Moreover, applying this new step size schedule to an existing constrained sampling algorithm, we show state-of-the-art convergence rates for sampling from a constrained log-concave distribution, as well as improved dimension dependence.
Extra-gradient with player sampling for faster convergence in n-player games
Authors: Samy Jelassi, Carles Domingo-Enrich, Damien Scieur, Arthur Mensch, Joan Bruna
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e.g. when training GANs. In this paper, we analyse a new extra-gradient method for Nash equilibrium finding, that performs gradient extrapolations and updates on a random subset of players at each iteration. This approach provably exhibits a better rate of convergence than full extra-gradient for non-smooth convex games with noisy gradient oracle. We propose an additional variance reduction mechanism to obtain speed-ups in smooth convex games. Our approach makes extrapolation amenable to massive multiplayer settings, and brings empirical speed-ups, in particular when using a heuristic cyclic sampling scheme. Most importantly, it allows to train faster and better GANs and mixtures of GANs.
Gradient-free Online Learning in Continuous Games with Delayed Rewards
Authors: Amélie Héliou, Panayotis Mertikopoulos, Zhengyuan Zhou
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Multi-Armed Bandits, Recommendation Systems

Motivated by applications to online advertising and recommender systems, we consider a game-theoretic model with delayed rewards and asynchronous, payoff-based feedback. In contrast to previous work on delayed multi-armed bandits, we focus on games with continuous action spaces, and we examine the long-run behavior of strategic agents that follow a no-regret learning policy (but are otherwise oblivious to the game being played, the objectives of their opponents, etc.). To account for the lack of a consistent stream of information (for instance, rewards can arrive out of order and with an a priori unbounded delay), we introduce a gradient-free learning policy where payoff information is placed in a priority queue as it arrives. Somewhat surprisingly, we find that under a standard diagonal concavity assumption, the induced sequence of play converges to Nash Equilibrium (NE) with probability 1, even if the delay between choosing an action and receiving the corresponding reward is unbounded.
Information Particle Filter Tree: An Online Algorithm for POMDPs with Belief-Based Rewards on Continuous Domains
Authors: Johannes Fischer, Ömer Sahin Tas
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Partially Observable Markov Decision Processes (POMDPs) inherently gather the information necessary to act optimally under uncertainties. The framework can be extended to model pure information gathering tasks by considering belief-based rewards. This allows us to use reward shaping to guide POMDP planning to informative beliefs by using a weighted combination of the original reward and the expected information gain as the objective. In this work we propose a novel online algorithm, Information Particle Filter Tree (IPFT), to solve problems with belief-dependent rewards on continuous domains. It simulates particle-based belief trajectories in a Monte Carlo Tree Search (MCTS) approach to construct a search tree in the belief space. The evaluation shows that the consideration of information gain greatly improves the performance in problems where information gathering is an essential part of the optimal policy.
Inter-domain Deep Gaussian Processes with RKHS Fourier Features
Authors: Tim G. J. Rudner, Dino Sejdinovic, Yarin Gal
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Gaussian Processes

Inter-domain Gaussian processes (GPs) allow for high flexibility and low computational cost when performing approximate inference in GP models. They are particularly suitable for modeling data exhibiting global function behavior but are limited to stationary covariance functions and thus fail to model non-stationary data effectively. We propose Inter-domain Deep Gaussian Processes with RKHS Fourier Features, an extension of shallow inter-domain GPs that combines the advantages of inter-domain and deep Gaussian processes (DGPs) and demonstrate how to leverage existing approximate inference approaches to perform simple and scalable approximate inference on Inter-domain Deep Gaussian Processes. We assess the performance of our method on a wide range of prediction problems and demonstrate that it outperforms inter-domain GPs and DGPs on challenging large-scale and high-dimensional real-world datasets exhibiting both global behavior as well as a high-degree of non-stationarity.
Low Bias Low Variance Gradient Estimates for Hierarchical Boolean Stochastic Networks
Authors: Adeel Pervez, Taco Cohen, Efstratios Gavves
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Stochastic neural networks with discrete random variables are an important class of models for their expressiveness and interpretability. Since direct differentiation and backpropagation is not possible, Monte Carlo gradient estimation techniques are a popular alternative. Efficient stochastic gradient estimators, such Straight-Through and Gumbel-Softmax, work well for shallow stochastic models. Their performance, however, suffers with hierarchical, more complex models. We focus on hierarchical stochastic networks with multiple layers of Boolean latent variables. To analyze such networks, we introduce the framework of harmonic analysis for Boolean functions to derive an analytic formulation for the bias and variance in the Straight-Through estimator. Exploiting these formulations, we propose \emph{FouST}, a low-bias and low-variance gradient estimation algorithm that is just as efficient. Extensive experiments show that FouST performs favorably compared to state-of-the-art biased estimators and is much faster than unbiased ones.