pwc-archive/papers-with-abstracts huggingface dataset:
Hierarchical reinforcement learning for efficent exploration and transfer
Authors: Lorenzo Steccanella, Simone Totaro, Damien Allonsius, Anders Jonsson
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Hierarchical Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Sparse-reward domains are challenging for reinforcement learning algorithms since significant exploration is needed before encountering reward for the first time. Hierarchical reinforcement learning can facilitate exploration by reducing the number of decisions necessary before obtaining a reward. In this paper, we present a novel hierarchical reinforcement learning framework based on the compression of an invariant state space that is common to a range of tasks. The algorithm introduces subtasks which consist in moving between the state partitions induced by the compression. Results indicate that the algorithm can successfully solve complex sparse-reward domains, and transfer knowledge to solve new, previously unseen tasks more quickly.
Meta-Learning GNN Initializations for Low-Resource Molecular Property Prediction
Authors: Cuong Q. Nguyen, Constantine Kreatsoulas, Kim M. Branson
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Drug Discovery, Few-Shot Learning, Meta-Learning, Molecular Property Prediction, Property Prediction

Building $\textit{in silico}$ models to predict chemical properties and activities is a crucial step in drug discovery. However, limited labeled data often hinders the application of deep learning in this setting. Meanwhile advances in meta-learning have enabled state-of-the-art performances in few-shot learning benchmarks, naturally prompting the question: Can meta-learning improve deep learning performance in low-resource drug discovery projects? In this work, we assess the transferability of graph neural networks initializations learned by the Model-Agnostic Meta-Learning (MAML) algorithm - and its variants FO-MAML and ANIL - for chemical properties and activities tasks. Using the ChEMBL20 dataset to emulate low-resource settings, our benchmark shows that meta-initializations perform comparably to or outperform multi-task pre-training baselines on 16 out of 20 in-distribution tasks and on all out-of-distribution tasks, providing an average improvement in AUPRC of 11.2% and 26.9% respectively. Finally, we observe that meta-initializations consistently result in the best performing models across fine-tuning sets with $k \in \{16, 32, 64, 128, 256\}$ instances.
Generalizing Curricula for Reinforcement Learning
Authors: Sanmit Narvekar, Peter Stone
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Curriculum learning for reinforcement learning (RL) is an active area of research that seeks to speed up training of RL agents on a target task by first training them through a series of progressively more challenging source tasks. Each task in this sequence builds upon skills learned in previous tasks to gradually develop the repertoire needed to solve the final task. Over the past few years, many automated methods to develop curricula have been developed. However, they all have one key limitation: the curriculum must be regenerated from scratch for each new agent or task encountered. In many cases, this generation process can be very expensive. However, there is structure that can be exploited between tasks and agents, such that knowledge gained developing a curriculum for one task can be reused to speed up creating a curriculum for a new task. In this paper, we present a method to generalize a curriculum learned for one set of tasks to a novel set of unseen tasks.
A Policy Gradient Method for Task-Agnostic Exploration
Authors: Mirco Mutti, Lorenzo Pratissoli, Marcello Restelli
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: continuous-control, Continuous Control

In a reward-free environment, what is a suitable intrinsic objective for an agent to pursue so that it can learn an optimal task-agnostic exploration policy? In this paper, we argue that the entropy of the state distribution induced by limited-horizon trajectories is a sensible target. Especially, we present a novel and practical policy-search algorithm, Maximum Entropy POLicy optimization (MEPOL), to learn a policy that maximizes a non-parametric, $k$-nearest neighbors estimate of the state distribution entropy. In contrast to known methods, MEPOL is completely model-free as it requires neither to estimate the state distribution of any policy nor to model transition dynamics. Then, we empirically show that MEPOL allows learning a maximum-entropy exploration policy in high-dimensional, continuous-control domains, and how this policy facilitates learning a variety of meaningful reward-based tasks downstream.
Combining Variational Continual Learning with FiLM Layers
Authors: Noel Loo, Siddharth Swaroop, Richard E Turner
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Continual Learning

The standard architecture for continual learning is a multi-headed neural network, which has shared body parameters and task-specific heads. Features for each task are generated in the same way. This could be too restrictive, particularly when tasks are very distinct. We propose combining FiLM layers, a flexible way to enable task-specific feature modulation in CNNs, with an existing algorithm, Variational Continual Learning (VCL). We show that this addition consistently improves performance, particularly when tasks are more varied. Furthermore, we demonstrate how FiLM Layers can mitigate VCL's tendency to over-prune and help it use more model capacity. Finally, we find that FiLM Layers perform feature modulation as opposed to gating, making them more flexible than binary mask based approaches.
Meta-Learning for Recalibration of EMG-Based Upper Limb Prostheses
Authors: Krsto Proroković, Michael Wand, Jürgen Schmidhuber
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Meta-Learning

An EMG-based upper limb prosthesis relies on a statistical pattern recognition system to map the EMG signal of residual forearm muscles into the appropriate hand movements. As the EMG signal changes each time the user puts the prosthesis on, an efficient method for prosthesis recalibration is needed. Here we show that meta-learning is a promising approach for achieving this aim. Furthermore, we show that meta-leaning can be used to recalibrate the prosthesis even when the examples of some movement types are missing in the target session.
Logical Composition in Lifelong Reinforcement Learning
Authors: Geraud Nangue Tasse, Steven James, Benjamin Rosman
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Lifelong learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The ability to produce novel behaviours from existing skills is an important property of lifelong learning agents. We build on recent work which formalises a Boolean algebra over the space of tasks and value functions, and show how this can be leveraged to tackle the ifelong learning problem. We propose an algorithm that determines whether a new task can be immediately solved using an agent’s existing abilities, or whether the task should be learned from scratch. We verify our approach in the Four Rooms domain, where an agent learns a set of skills throughout its lifetime, and then composes them to solve a combinatorially large number of new tasks in a zeroshot manner.
Chaotic Continual Learning
Authors: Touraj Laleh, Mojtaba Faramarzi, Irina Rish, Sarath Chandar
Date: 2020-06-14 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Continual Learning

Training a deep neural network requires the model to go over training data for several epochs and update network parameters. In continual learning, this process results in catastrophic forgetting which is one of the core issues of this domain. Most proposed approaches for this issue try to compensate for the effects of parameter updates in the batch incremental setup in which the training model visits a lot of samples for several epochs. However, it is not realistic to expect training data will always be fed to model in a batch incremental setup. This paper proposes a chaotic stream learner that mimics the chaotic behavior of biological neurons and does not updates network parameters. In addition, it can work with fewer samples compared to deep learning models on stream learning setup. Our experiments on MNIST, CIFAR10, and Omniglot show that the chaotic stream learner has less catastrophic forgetting by its nature in comparison to a CNN model in continual learning.
Skill Discovery for Exploration and Planning using Deep Skill Graphs
Authors: Akhil Bagaria, Jason Crowley, Jing Wei Nicholas Lim, George Konidaris
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: continuous-control, Continuous Control, Decision Making

Temporal abstraction provides an opportunity to drastically lower the decision making burden facing reinforcement learning agents  in rich sensorimotor spaces. Well constructed hierarchies induce state and action abstractions that can reduce large continuous MDPs into small discrete ones, in which planning with a learned model is feasible. We propose a novel algorithm, Deep Skill Graphs, for acquiring such a minimal representation of an environment. Our algorithm seamlessly interleaves discovering skills and planning using them to gain unsupervised mastery over ever increasing portions of the state-space. The constructed skill graph can be used to drive the agent to novel goals at test time, requiring little-to-no additional learning. We test our algorithm on a series of continuous control tasks where it outperforms baseline flat and hierarchical RL methods alike.
Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition and Selective Transfer
Authors: Aswin Raghavan, Jesse Hostetler, Indranil Sur, Abrar Rahman, Ajay Divakaran
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Continual Learning, Lifelong learning, Starcraft, Transfer Learning

We introduce the eigentask framework for lifelong learning. An eigentask is a pairing of a skill that solves a set of related tasks with a generative model that can sample from the skill's input space. The model extends generative replay approaches, which have mainly been used to avoid catastrophic forgetting, to also address other lifelong learning goals such as forward knowledge transfer. We propose a wake-sleep cycle of alternating task learning and knowledge consolidation for learning in our framework, and instantiate it for lifelong supervised learning and lifelong RL. We achieve performance comparable to the state-of-the-art in supervised continual learning benchmarks, and show encouraging preliminary results for forward knowledge transfer in a lifelong RL application in the video game Starcraft~2.
Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning
Authors: Evan Zheran Liu, aditi raghunathan, Percy Liang, Chelsea Finn
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Meta Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL), Visual Navigation

We seek to efficiently learn by leveraging shared structure between different tasks and environments. For example, cooking is similar in different kitchens, even though the ingredients may change location. In principle, meta-reinforcement learning approaches can exploit this shared structure, but in practice, they fail to adapt to new environments when adaptation requires targeted exploration (e.g., exploring the cabinets to find ingredients in a new kitchen). We show that existing approaches fail due to a chicken-and-egg problem: learning what to explore requires knowing what information is critical for solving the task, but learning to solve the task requires already gathering this information via exploration. For example, exploring to find the ingredients only helps a robot prepare a meal if it already knows how to cook, but the robot can only learn to cook if it already knows where the ingredients are. To address this, we propose a new exploration objective (DREAM), based on identifying key information in the environment, independent of this information will exactly be used solve the task. By decoupling exploration from task execution, DREAM explores and consequently adapts to new environments requiring no reward signal when the task is specified via an instruction. Empirically, DREAM scales to more complex problems, such as sparse-reward 3D visual navigation, while existing approaches fail from insufficient exploration.
Towards an Unsupervised Method for Model Selection in Few-Shot Learning
Authors: Simon Guiroy, Vikas Verma, Christopher Pal
Date: 2020-06-15 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Few-Shot Learning, Meta-Learning, Model Selection

The study of generalization of neural networks in gradient-based meta-learning has recently great research interest. Previous work on the study of the objective landscapes within the scope of few-shot classification empirically demonstrated that generalization to new tasks might be linked to the average inner product between their respective gradients vectors (Guiroy et al., 2019). Following that work, we study the effect that meta-training has on the learned space of representation of the network. Notably, we demonstrate that the global similarity in the space of representation, measured by the average inner product between the embeddings of meta-test examples, also correlates to generalization. Based on these observations, we propose a novel model-selection criteria for gradient-based meta-learning and experimentally validate its effectiveness.
Bridging Worlds in Reinforcement Learning with Model-Advantage
Authors: Nirbhay Modhe, Harish K Kamath, Dhruv Batra, Ashwin Kalyan
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Despite the breakthroughs achieved by Reinforcement Learning (RL) in recent years, RL agents often fail to perform well in unseen environments. This inability to generalize to new environments prevents their deployment in the real world. To help measure this gap in performance, we introduce model-advantage - a quantity similar to the well-known (policy) advantage function. First, we show relationships between the proposed model-advantage and generalization in RL — using which we provide guarantees on the gap in performance of an agent in new environments. Further, we conduct toy experiments to show that even a sub-optimal policy (learnt with minimal interactions with the target environment) can help predict if a training environment (say, a simulator) helps learn policies that generalize. We then show connections with Model Based RL.
Active Online Domain Adaptation
Authors: Yining Chen, Haipeng Luo, Tengyu Ma, Chicheng Zhang
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Domain Adaptation, Online Domain Adaptation, regression

Online machine learning systems need to adapt to domain shifts. Meanwhile, acquiring label at every timestep is expensive. We propose a surprisingly simple algorithm that adaptively balances its regret and its number of label queries in settings where the data streams are from a mixture of hidden domains. For online linear regression with oblivious adversaries, we provide a tight tradeoff that depends on the durations and dimensionalities of the hidden domains. Our algorithm can adaptively deal with interleaving spans of inputs from different domains. We also generalize our results to non-linear regression for hypothesis classes with bounded eluder dimension and adaptive adversaries. Experiments on synthetic and realistic datasets demonstrate that our algorithm achieves lower regret than uniform queries and greedy queries with equal labeling budget.
Importance Weighting with a Adversarial Network for Large-Scale Sleep Staging
Authors: Samaneh Nasiri, Gari D. Clifford
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: EEG, Electroencephalogram (EEG), Sleep Staging, Transfer Learning

To develop a generalized automated sleep staging method based on the gold standard modality, electroencephalograms (EEGs), requires a large and accurately labeled training and test set acquired from different individuals with diverse demographics and medical conditions. However, data in the training set may exhibit changes in the EEG patterns that are very different from the data in the test set, due to inherent inter-subject variability, electrode misplacement, and the variability of medication use/response. Training an algorithm on such data without accounting for this diversity can lead to underperformance and a lack of generalizability on novel data. Previous methods have attempted to address this by developing robust representations across all individuals in the dataset using deep transfer learning approaches. However, not all parts of the training data are as relevant as others to the test data. Forcing the alignment of these nontransferable data with the transferable data may lead to a negative impact on the overall performance. This work jointly learns patient-invariant representations and weights features (spectrogram coefficients) to enhance the contribution of relevant features in the final model and decrease the impact of irrelevant features using an unsupervised approach. The proposed method leverages transferable and discriminable knowledge from the training set to the test set. Using a large public database of 42,560 hours of EEG, recorded from 5,793 from Sleep Heart Health Study, we demonstrate that adversarially learning a network with an importance weighting scheme significantly boosts performance compared to state-of-the-art deep learning approaches in the cross-subject scenario. The proposed method improves, on average, accuracy from 0.81 to 0.94, precision from 0.81 to 0.82, and sensitivity from 0.74 to 0.85.
Multilayer Neuromodulated Architectures for Memory-Constrained Online Continual Learning
Authors: Sandeep Madireddy, Angel Yanguas-Gil, Prasanna Balaprakash
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Bayesian Optimization, class-incremental learning, Class Incremental Learning, Continual Learning, Edge-computing, Incremental Learning, Permuted-MNIST, Split-MNIST

We focus on the problem of how to achieve online continual learning under
memory-constrained conditions where the input data may not be known \emph{a
priori}. These constraints are relevant in edge computing scenarios. We have
developed an architecture where input processing over data streams and online
learning are integrated in a single recurrent network architecture. This
allows us to cast metalearning optimization as a mixed-integer optimization
problem, where different synaptic plasticity algorithms and feature extraction
layers can be swapped out and their hyperparameters are optimized to identify
optimal architectures for different sets of tasks. We utilize a Bayesian
optimization method to search over a design space that spans multiple learning
algorithms, their specific hyperparameters, and feature extraction layers. 
We demonstrate our approach for online non-incremental and class-incremental
learning tasks. Our optimization algorithm finds configurations that achieve
superior  continual learning performance on Split-MNIST and Permuted-MNIST
data as compared with other memory-constrained learning approaches, and it
matches  that of the state-of-the-art memory replay-based approaches without 
explicit data storage and replay. Our approach allows us to  explore the
transferability of optimal learning conditions to tasks and datasets that have
not been previously seen.  We demonstrate that the accuracy  of our transfer
metalearning across datasets can be largely explained through a transfer
coefficient that can be based on metrics of dimensionality  and distance
between datasets.
Learning Intrinsically Motivated Options to Stimulate Policy Exploration
Authors: Louis Bagot, Kevin Mets, Steven Latré
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: reinforcement-learning, Reinforcement Learning (RL)

A Reinforcement Learning (RL) agent needs to find an optimal sequence of actions in order to maximize rewards. This requires consistent exploration of states and action sequences to ensure the policy found is optimal.
One way to motivate exploration is through intrinsic rewards, i.e. agent-induced rewards to guide itself towards interesting behaviors.
We propose to learn from such intrinsic rewards through exploration options, i.e. additional temporally-extended actions to call separate policies (or "Explorer" agents) associated to an intrinsic reward. We show that this method has several key advantages over the usual method of weighted sum of rewards, mainly task-transfer abilities and scalability to multiple reward functions.
Lifelong Learning of Factored Policies via Policy Gradients
Authors: Jorge A Mendez, Eric Eaton
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: continuous-control, Continuous Control, Lifelong learning, Policy Gradient Methods

Policy gradient methods have shown success in learning continuous control policies for high-dimensional dynamical systems. A major downside of such methods is the amount of exploration they require before yielding high-performing policies. In a lifelong learning setting, in which an agent is faced with multiple consecutive tasks over its lifetime, reusing information from previously seen tasks can substantially accelerate the learning of new tasks. We provide a novel method for lifelong policy gradient learning that trains lifelong function approximators directly via policy gradients, allowing the agent to benefit from accumulated knowledge throughout the entire training process. We show empirically that our algorithm learns faster and converges to better policies than single-task and lifelong learning baselines, and completely avoids catastrophic forgetting on a variety of challenging domains.
Taming the Herd: Multi-Modal Meta-Learning with a Population of Agents
Authors: Robert Müller, Jack Parker-Holder, Aldo Pacchiano
Date: 2020-06-12 00:00:00
Proceeding: ICML Workshop LifelongML 2020 7
Tasks: Meta-Learning

Meta-learning is a paradigm whereby an agent is trained with the specific goal of fast adaptation. With promising recent results, it is an effective way to use prior knowledge for future learning. However, most of the prominent successes focus entirely on adapting to similar tasks, from a unimodal distribution. This drastically differs from the real world, where problems may be more diverse. In this paper we address this issue, and provide a simple approach to solve it, which we call Taming the Herd. The Herd refers to a population of agents, each specializing on a subset (or mode) of the task distribution. At test time, we automatically allocate the appropriate member of the Herd and thus perform comparably with an oracle trained solely on those tasks. We apply our approach to both MAML and PEARL, and demonstrate its efficacy on a simple yet challenging multi-modal task distribution.