pwc-archive/papers-with-abstracts huggingface dataset:
Learning State Representations via Temporal Cycle-Consistency Constraint in Model-Based Reinforcement Learning
Authors: Changmin Yu, Dong Li, Hangyu Mao, Jianye Hao, Neil Burgess
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: continuous-control, Continuous Control, Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL), Representation Learning

Representation learning is a popular approach for reinforcement learning (RL) tasks with partially observable Markov decision processes. Existing works on learning representations utilise the dynamics models in model-based RL to perform training through model predictive reconstruction in a temporally forward fashion. However, temporally backward state predictions also yield useful supervision signals as they convey information about the future states given the action choices. We argue that combining them with forward passes will facilitate stronger representation learning and improve the sample efficiency of RL. Here we propose a general framework for learning state representations for RL tasks, utilising both forward and backward passes by imposing temporal cycle-consistency constraints, which can be integrated with any model-based RL algorithms leveraging a latent dynamics model. We show improved empirical performance in terms of sample-efficiency and convergence score over several baselines on continuous control benchmarks.
Learning to Explore a Class of Multiple Reward-Free Environments
Authors: Mirco Mutti, Mattia Mancassola, Marcello Restelli
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Several recent works have been dedicated to the pure exploration of a single reward-free environment. Along this line, we address the problem of learning to explore a class of multiple reward-free environments with a unique general strategy, which aims to provide a universal initialization to subsequent reinforcement learning problems specified over the same class. Notably, the problem is inherently multi-objective as we can trade off the exploration performance between environments in many ways. In this work, we foster an exploration strategy that is sensitive to the most adverse cases within the class. Hence, we cast the exploration problem as the maximization of the mean of a critical percentile of the state visitation entropy induced by the exploration strategy over the class of environments. Then, we present a policy gradient algorithm, MEMENTO, to optimize the introduced objective through mediated interactions with the class. Finally, we empirically demonstrate the ability of the algorithm in learning to explore challenging classes of continuous environments and we show that reinforcement learning greatly benefits from the pre-trained exploration strategy when compared to learning from scratch.
Minimum Description Length Skills for Accelerated Reinforcement Learning
Authors: Jesse Zhang, Karl Pertsch, Jiefan Yang, Joseph J Lim
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Humans can quickly learn new tasks by reusing a large number of previously acquired skills. How can we discover such reusable skills for artificial agents when given a large dataset of prior experience? Past works leverage extensive human supervision to define skills or use simple skill heuristics that limit their expressiveness. In contrast, we propose a principled, unsupervised objective for skill discovery from large, offline datasets based on the Minimum Description Length principle: we show that a "code book" of skills that can maximally compress the training data can be reused to efficiently learn new tasks. By minimizing description length we strike an optimal balance between the number of extracted skills and their complexity. We show that our approach outperforms alternative approaches that heuristically define skills on a complex, long-horizon maze navigation task.
Less Suboptimal Learning and Control in Variational POMDPs
Authors: Baris Kayalibay, Atanas Mirchev, Patrick van der Smagt, Justin Bayer
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Variational Inference

A recently uncovered pitfall in learning generative models with amortised variational inference, the conditioning gap, questions common practices in model-based reinforcement learning. Withholding a part of the quantities that the true posterior depends on from the inference network leads to a biased generative model and an approximate posterior that underestimates uncertainty. We examine the effect of the conditioning gap on model-based reinforcement learning with variational world models. We study the effect in three settings with known dynamics, which enables us to compare to a near-optimal policy. Our finding is that the impact of the conditioning gap becomes severe in systems where the state is hard to estimate.
Reinforcement Learning with Prototypical Representations
Authors: Anonymous
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: continuous-control, Continuous Control, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Representation Learning

Learning effective representations in image-based environments is crucial for sample efficient Reinforcement Learning (RL). Unfortunately, in RL, representation learning is confounded with the exploratory experience of the agent -- learning a useful representation requires diverse data, while effective exploration is only possible with coherent representations. Furthermore, we would like to learn representations that not only generalize across tasks but also accelerate downstream exploration for efficient task-specific training. To address these challenges we propose Proto-RL, a self-supervised framework that ties representation learning with exploration through prototypical representations. These prototypes simultaneously serve as a summarization of the exploratory experience of an agent as well as a basis for representing observations. We pre-train these task-agnostic representations and prototypes on environments without downstream task information. This enables state-of-the-art downstream RL on a set of difficult continuous control tasks.
Critic-Guided Learning to Segment Rewarding Objects in First-Person Views
Authors: Anonymous
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5

We train a U-Net model to generate masks over reward-related objects in images. Our approach allows to train the U-Net model without explicit label information, but only using feedback from a critic model which learned to estimate the expected-reward value of an image observation. The masking is learned in contrastive fashion with image pairs using an adversarial scheme for employing the critic score gradient with respect to the mask operation: The pair consists of two images, where the first has a high and the second a low critic value. Training with such pairs enables the U-Net model to produce masks that decrease the critic value in the first image and increase the critic value in the second image when transferring pixels in the masked segment from the first to the second image. The training of the U-Net model is based on an imitation database from the NeurIPS 2020 MineRL Competition Track, where our agent took the ?-place winning entry. Video demonstration: www.rebrand.ly/Rewarding-Objects-mp4
LOCO: Adaptive exploration in reinforcement learning via local estimation of contraction coefficients
Authors: Manfred Diaz, Liam Paull, Pablo Samuel Castro
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: reinforcement-learning, Reinforcement Learning (RL)

We offer a novel approach to balance exploration and exploitation in reinforcement learning (RL). To do so, we characterize an environment’s exploration difficulty via the Second Largest Eigenvalue Modulus (SLEM) of the Markov chain induced by uniform stochastic behaviour. Specifically, we investigate the connection of state-space coverage with the SLEM of this Markov chain and use the theory of contraction coefficients to derive estimates of this eigenvalue of interest. Furthermore, we introduce a method for estimating the contraction coefficients on a local level and leverage it to design a novel exploration algorithm. We evaluate our algorithm on a series of GridWorld tasks of varying sizes and complexity.
Automatic Goal Generation using Dynamical Distance Learning
Authors: Anonymous
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: Decision Making, Reinforcement Learning (RL), Sequential Decision Making

Reinforcement Learning (RL) agents can learn to solve complex sequential decision making tasks by interacting with the environment. However, sample efficiency remains a major challenge. In the field of multi-goal RL, where agents are required to reach multiple goals to solve complex tasks, improving sample efficiency can be especially challenging. On the other hand, humans or other biological agents learn such tasks in a much more strategic way, following a curriculum where tasks are sampled with increasing difficulty level in order to make gradual and efficient learning progress. In this work, we propose a method for automatic goal generation using a dynamical distance function (DDF) in a self-supervised fashion. DDF is a function which predicts the dynamical distance between any two states within a markov decision process (MDP). With this, we generate a curriculum of goals at the appropriate difficulty level to facilitate efficient learning throughout the training process. We evaluate this approach on several goal-conditioned robotic manipulation and navigation tasks, and show improvements in sample efficiency over a baseline method which only uses random goal sampling.
Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos
Authors: Anonymous
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: Model Predictive Control

We are motivated by the goal of generalist robotic agents that can complete a wide range of tasks across many environments. Critical to this is the robot’s ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language, collecting high quality data of robotic interaction at scale remains an open challenge. In contrast, “in-the-wild” videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. Unlocking this data for learning robotic reward functions could lead to more scalable training of robots for numerous tasks and would be an important step towards generalist robots. In this work, we propose a simple approach, Domain-agnostic Video Discriminator (DVD), for learning multitask reward functions which can generalize by virtue of learning from a small amount of robot data with a broad dataset of human videos. We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.
Learning Task Informed Abstractions
Authors: Anonymous
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL)

Current model-based reinforcement learning methods struggle when operating from complex visual scenes due to their inability to prioritize task-relevant features. To mitigate this problem, we propose learning Task Informed Abstractions (TIA) that separates reward-correlated visual features from background distractions. For learning TIA, we introduce the formalism of Task Informed MDP (TiMDP), which is realized by training two models that learn visual features via cooperative reconstruction, but one model is adversarially dissociated from the reward signal. Empirical evaluation shows that TIA leads to significant performance gains over state-of-the-art methods on many visual control tasks where natural and unconstrained visual distractions pose a formidable challenge.
Resolving Causal Confusion in Reinforcement Learning via Robust Exploration
Authors: Clare Lyle, Amy Zhang, Minqi Jiang, Joelle Pineau, Yarin Gal
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

A reinforcement learning agent must distinguish between spurious correlations and causal relationships in its environment in order to robustly achieve its goals. Contrary to popular belief, such cases of causal confusion {\em can} occur in online reinforcement learning (RL) settings. We demonstrate this, and show how causal confusion can lead to catastrophic failure under even mild forms of distribution shift. We formalize the problem of identifying causal structure in a Markov Decision Process, and highlight the central role played by the data collection policy in identifying and avoiding spurious correlations. We find that under insufficient exploration, many RL algorithms, including those with PAC-MDP guarantees, fall prey to causal confusion under insufficient exploration policies. To address this, we present a robust exploration strategy which enables causal hypothesis-testing by interaction with the environment. Our method outperforms existing state-of-the-art approaches at avoiding causal confusion, improving robustness and generalization in a range of tasks.
PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
Authors: Anonymous
Date: 2021-03-09 00:00:00
Proceeding: ICLR Workshop SSL-RL 2021 5
Tasks: Autonomous Driving, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We study reinforcement learning (RL) with no-reward demonstrations, a setting in which an RL agent has access to additional data from the interaction of other agents with the same environment.
However, it has no access to the rewards or goals of these agents, and their objectives and levels of expertise may vary widely. These assumptions are common in multi-agent settings, such as autonomous driving. To effectively use this data, we turn to the framework of successor features.
This allows us to disentangle shared features and dynamics of the environment from agent-specific rewards and policies. We propose a multi-task inverse reinforcement learning (IRL) algorithm, called \emph{inverse temporal difference learning} (ITD), that learns shared state features, alongside per-agent successor features and preference vectors, purely from demonstrations without reward labels.
We further show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called $\Psi \Phi$-Learning (pronounced `Sci-Fi'). We provide empirical evidence for the effectiveness of $\Psi \Phi$-Learning as a method for improving RL, IRL, imitation, and few-shot transfer, and derive worst-case bounds for its performance in zero-shot transfer to new tasks.
Prototypical Bregman Networks
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Computational Efficiency, Few-Shot Learning, Metric Learning, Structured Prediction

In this work, we approach one-shot and few-shot learning problems as methods for finding good prototypes for each class, where these prototypes are generalizable to new data samples and classes. We propose a metric learner that learns a Bregman divergence by learning its underlying convex function. Bregman divergences are a good candidate for this framework given they are the only class of divergences with the property that the best representative of a set of points is given by its mean. We propose a flexible extension to prototypical networks to enable joint learning of the embedding and the divergence, while preserving computational efficiency. Our preliminary results are comparable with the prior work on the Omniglot and Mini-ImageNet datasets, two standard benchmarks for one-shot and few-shot learning. We argue that our model can be used for other tasks that involve metric learning or tasks that require approximate convexity such as structured prediction and data completion.
Multi-Task Learning via Task Multi-Clustering
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Clustering, Mixture-of-Experts, Multi-Task Learning

Multi-task learning has the potential to facilitate learning of shared representations between tasks, leading to better task performance. Some sets of tasks are related, and can share many features that are useful latent representations for these tasks. Other sets of tasks are less related, possibly sharing some features, but also competing on the representational resources of shared parameters. We propose to discover how to share parameters between related tasks and split parameters between conflicting tasks, by learning a multi-clustering of the tasks. We present a mixture-of-experts model, where each cluster is an expert that extracts a feature vector from the input, and each task belongs to a set of clusters whose experts it can mix. In experiments on the CIFAR-100 MTL domain, multi-clustering outperforms a model that mixes all experts in accuracy and computation time. The results suggest that the performance of our method is robust to regularization that increases the model's sparsity when sufficient data is available, and can benefit from sparser models as data becomes scarcer.
Learning Cancer Outcomes from Heterogeneous Genomic Data Sources: An Adversarial Multi-task Learning Approach
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Multi-Task Learning, Representation Learning

Translating the high-dimensional data generated by genomic platforms into reliable predictions of clinical outcomes remains a critical challenge in realizing the promise of genomic medicine largely due to small number of independent samples. We show that neural networks can be trained to predict clinical outcomes using heterogeneous genomic data sources via multi-task learning and adversarial representation learning, allowing one to combine multiple cohorts and outcomes in training. Experiments demonstrate that the proposed method helps mitigate data scarcity and outcome censorship in cancer genomics learning problems.
Continual adaptation for efficient machine communication
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Continual Learning, Language Modeling, Language Modelling

To communicate with new partners in new contexts, humans rapidly form new linguistic conventions. Recent language models trained with deep neural networks are able to comprehend and produce the existing conventions present in their training data, but are not able to flexibly and interactively adapt those conventions on the fly as humans do. We introduce a repeated reference task as a benchmark for models of adaptation in communication and propose a regularized continual learning framework that allows an artificial agent initialized with a generic language model to more accurately and efficiently understand their partner over time. We evaluate this framework through simulations on COCO and in real-time reference game experiments with human partners.
Differentiable Hebbian Plasticity for Continual Learning
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Continual Learning, Permuted-MNIST, Split-MNIST

Catastrophic forgetting poses a grand challenge for continual learning systems, which prevents neural networks from protecting old knowledge while learning new tasks sequentially. We propose a Differentiable Hebbian Plasticity (DHP) Softmax layer which adds a fast learning plastic component to the slow weights of the softmax output layer. The DHP Softmax behaves as a compressed episodic memory that reactivates existing memory traces, while creating new ones. We demonstrate the flexibility of our model by combining it with existing well-known consolidation methods to prevent catastrophic forgetting. We evaluate our approach on the Permuted MNIST and Split MNIST benchmarks, and introduce Imbalanced Permuted MNIST — a dataset that combines the challenges of class imbalance and concept drift. Our model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.
Improving Relevance Prediction with Transfer Learning in Large-scale Retrieval Systems
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Retrieval, Transfer Learning

Machine learned large-scale retrieval systems require a large amount of training data representing query-item relevance. However, collecting users' explicit feedback is costly. In this paper, we propose to leverage user logs and implicit feedback as auxiliary objectives to improve relevance modeling in retrieval systems. Specifically, we adopt a two-tower neural net architecture to model query-item relevance given both collaborative and content information. By introducing auxiliary tasks trained with much richer implicit user feedback data, we improve the quality and resolution for the learned representations of queries and items. Applying these learned representations to an industrial retrieval system has delivered significant improvements.
Learning to learn to communicate
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Deep Reinforcement Learning, Meta-Learning

How can we teach artificial agents to use human language flexibly to solve problems in a real-world environment? We have one example in nature of agents being able to solve this problem: human babies eventually learn to use human language to solve problems, and they are taught with an adult human-in-the-loop. Unfortunately, current machine learning methods (e.g. from deep reinforcement learning) are too data inefficient to learn a language in this way (3). An outstanding goal is finding an algorithm with a suitable ‘language learning prior’ that allows it to learn human language, while minimizing the number of required human interactions.

In this paper, we propose to learn such a prior in simulation, leveraging the increasing amount of available compute for machine learning experiments (1). We call our approach Learning to Learn to Communicate (L2C). Specifically, in L2C we train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol. Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations unseen during training, including populations of humans. To show the promise of the L2C framework, we conduct some preliminary experiments in a Lewis signaling game (4), where we show that agents
trained with L2C are able to learn a simple form of human language (represented by a hand-coded compositional language) in fewer iterations than randomly initialized agents.