pwc-archive/papers-with-abstracts huggingface dataset:
Tools for Data-driven Modeling of Within-Hand Manipulation with Underactuated Adaptive Hands
Authors: Avishai Sintov, Andrew Kimmel, Bowen Wen, Abdeslam Boularias, Kostas Bekris
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: Model-based Reinforcement Learning, Reinforcement Learning (RL)

Precise in-hand manipulation is an important skill for a robot to perform tasks in human environments. Practical robotic hands must be low-cost, easy to control and capable. 3D-printed underactuated adaptive hands provide such properties as they are cheap to fabricate and adapt to objects of uncertain geometry with stable grasps. Challenges still remain, however, before such hands can attain human-like performance due to complex dynamics and contacts. In particular, useful models for planning, control or model-based reinforcement learning are still lacking. Recently, data-driven approaches for such models have shown promise. This work provides the first large public dataset of real within-hand manipulation that facilitates building such models, along with baseline data-driven modeling results. Furthermore, it contributes ROS-based physics-engine model of such hands for independent data collection, experimentation and sim-to-reality transfer work.
Feynman-Kac Neural Network Architectures for Stochastic Control Using Second-Order FBSDE Theory
Authors: Marcus Pereira, Ziyi Wang, Tianrong Chen, Emily Reed, Evangelos Theodorou
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: LEMMA

We present a deep recurrent neural network architecture to solve a class of stochastic optimal control problems described by fully nonlinear Hamilton Jacobi Bellman partial differential equations. Such PDEs arise when considering stochastic dynamics characterized by uncertainties that are additive, state dependent, and control multiplicative. Stochastic models with these characteristics are important in computational neuroscience, biology, finance, and aerospace systems and provide a more accurate representation of actuation than models with only additive uncertainty. Previous literature has established the inadequacy of the linear HJB theory for such problems, so instead, methods relying on the generalized version of the Feynman-Kac lemma have been proposed resulting in a system of second-order Forward-Backward SDEs. However, so far, these methods suffer from compounding errors resulting in lack of scalability. In this paper, we propose a deep learning based algorithm that leverages the second-order FBSDE representation and LSTM-based recurrent neural networks to not only solve such stochastic optimal control problems but also overcome the problems faced by traditional approaches, including scalability. The resulting control algorithm is tested on a high-dimensional linear system and three nonlinear systems from robotics and biomechanics in simulation to demonstrate feasibility and out-performance against previous methods.
Toward fusion plasma scenario planning for NSTX-U using machine-learning-accelerated models
Authors: Mark Boyer
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: BIG-bench Machine Learning

One of the most promising devices for realizing power production through nuclear fusion is the tokamak. To maximize performance, it is preferable that tokamak reactors achieve advanced operating scenarios characterized by good plasma confinement, improved magnetohydrodynamic stability, and a largely non-inductively driven plasma current. Such scenarios could enable steady-state reactor operation with high fusion gain --- the ratio of produced fusion power to the external power provided through the plasma boundary. Precise and robust control of the evolution of the plasma boundary shape as well as the spatial distribution of the plasma current, density, temperature, and rotation will be essential to achieving and maintaining such scenarios. The complexity of the evolution of tokamak plasmas, arising due to nonlinearities and coupling between various parameters, motivates the use of model-based control algorithms that can account for the system dynamics. In this work, a learning-based accelerated model trained on data from the National Spherical Torus Experiment Upgrade (NSTX-U) is employed to develop planning and control strategies for regulating the density and temperature profile evolution around desired trajectories. The proposed model combines empirical scaling laws developed across multiple devices with neural networks trained on empirical data from NSTX-U and a database of first-principles-based computationally intensive simulations. The reduced execution time of the accelerated model will enable practical application of optimization algorithms and reinforcement learning approaches for scenario planning and control development. An initial demonstration of applying optimization approaches to the learning-based model is presented, including a strategy for mitigating the effect of leaving the finite validity range of the accelerated model. The approach shows promise for actuator planning between experiments and in real-time.
Localized Learning of Robust Controllers for Networked Systems with Dynamic Topology
Authors: SooJean Han
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6

This paper addresses the problem of controller synthesis for networked systems with dynamic and unknown topology. Such networked systems arise in applications such as the power grid when lines between consumers are downed due to natural causes, or in multi-agent robotic networks and embedded sensor networks, both of which tend to be structurally complex and prone to high faults. We develop a robust and adaptive system-level controller synthesis approach to this problem. The algorithm is described sequentially in three settings: centralized, localized, and iterative localized for multiple modifications over time. In particular, the iterative localized scheme is designed around networks which switch between topological configurations arranged in a finite-state Markov Chain; the subsystems additionally perform consensus to estimate its transition probabilities. The advantages of our method are twofold. First, as a result of the localized implementation of the system-level approach, our controller can be extended to large-scale networks. Second, achieving exact consensus or learning precisely where the network structure has changed is not necessary to stabilize the overall network due to the robust nature of our controller. To demonstrate performance, we simulate the iterative localized algorithm for a simple centered hexagon network with 4 different topological states.
Constrained Upper Confidence Reinforcement Learning with Known Dynamics
Authors: Liyuan Zheng, Lillian Ratliff
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Constrained Markov Decision Processes are a class of stochastic decision problems in which the decision maker must select a policy that satisfies auxiliary cost constraints. This paper extends upper confidence reinforcement learning for settings in which the reward function and the  constraints, described by cost functions, are unknown a priori but the transition kernel is known. Such a setting is well-motivated by a number of applications including exploration of unknown, potentially unsafe, environments. We present an algorithm C-UCRL and show that it achieves sub-linear regret with respect to the reward while satisfying the constraints even while learning with high probability. An illustrative example is provided.
Learning-based Stochastic Model Predictive Control with State-Dependent Uncertainty
Authors: Angelo Domenico Bonzanini, Ali Mesbah
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: Model Predictive Control

The increasing complexity of modern engineering systems can introduce a great deal of uncertainty in our knowledge of system dynamics, which can, in turn, pose a major challenge to safe model-based control. This paper presents a learning-based stochastic model predictive control (LB-SMPC) strategy with chance constraints for tracking. The LB-SMPC strategy systematically handles mismatch between the actual system dynamics and a system model via a state-dependent uncertainty term that is intended to correct model predictions at each sampling time. A chance constraint handling method is presented to ensure state constraint satisfaction to a desired level for the case of state-dependent model uncertainty. Closed-loop simulations demonstrate the usefulness of LB-SMPC for control of a safety-critical plasma system for processing of (bio)materials with hard-to-model and time-varying dynamics.
Bayesian Learning with Adaptive Load Allocation Strategies
Authors: Manxi Wu, Saurabh Amin, Asuman Ozdaglar
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6

We study a Bayesian learning dynamics induced by agents who repeatedly allocate loads on a set of resources based on their belief of an unknown parameter that affects the cost distributions of resources. In each step, belief update is performed according to Bayes' rule using the agents' current load and a realization of costs on resources that they utilized. Then, agents choose a new load using an adaptive strategy update rule that accounts for their preferred allocation based on the updated belief. We prove that beliefs and loads generated by this learning dynamics converge almost surely. The convergent belief accurately estimates cost distributions of resources that are utilized by the convergent load. We establish conditions on the initial load and strategy updates under which the cost estimation is accurate on all resources. These results apply to Bayesian learning in congestion games with unknown latency functions. Particularly, we provide conditions under which the load converges to an equilibrium or socially optimal load with complete information of cost parameter. We also design an adaptive tolling mechanism that eventually induces the socially optimal outcome.
Lambda-Policy Iteration with Randomization for Contractive Models with Infinite Policies: Well-Posedness and Convergence
Authors: Yuchao Li, Karl Henrik Johansson, Jonas Mårtensson
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6

Abstract dynamic programming models are used to analyze $\lambda$-policy iteration with randomization algorithms. Particularly, contractive models with infinite policies are considered and it is shown that well-posedness of the $\lambda$-operator plays a central role in the algorithm. The operator is known to be well-posed for problems with finite states, but our analysis shows that it is also well-defined for the contractive models with infinite states studied. Similarly, the algorithm we analyze is known to converge for problems with finite policies, but we identify the conditions required to guarantee convergence with probability one when the policy space is infinite regardless of the number of states. Guided by the analysis, we exemplify a data-driven approximated implementation of the algorithm for estimation of optimal costs of constrained linear and nonlinear control problems. Numerical results indicate potentials of this method in practice.
Learning the model-free linear quadratic regulator via random search
Authors: Hesameddin Mohammadi, Mihailo R. Jovanovic', Mahdi Soltanolkotabi
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: Reinforcement Learning (RL)

Model-free reinforcement learning attempts to find an optimal control action for an unknown dynamical system by directly searching over the parameter space of controllers. The convergence behavior and statistical properties of these approaches are often poorly understood because of the nonconvex nature of the underlying optimization problems as well as the lack of exact gradient computation. In this paper, we examine the standard infinite-horizon linear quadratic regulator problem for continuous-time systems with unknown state-space parameters. We provide theoretical bounds on the convergence rate and sample complexity of a random search method. Our results demonstrate that the required simulation time for achieving $\epsilon$-accuracy in a model-free setup and the total number of function evaluations are both of $O (\log \, (1/\epsilon) )$.
Learning supported Model Predictive Control for Tracking of Periodic References
Authors: Janine Matschek, Rolf Findeisen
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: Gaussian Processes, Model Predictive Control, Motion Compensation

Increased autonomy of controllers in tasks with uncertainties stemming from the interaction with the environment can be achieved by incorporation of learning. Examples are control tasks where the system should follow a reference which depends on measurement data from surrounding systems as e.g. humans or other control systems. We propose a learning strategy for Gaussian processes to model, filter and predict references for control systems under model predictive control. 
Hereby constraints in the learning are included to achieve safety guarantees as trackability and recursive feasibility. An illustrative simulation example for motion compensation is given which shows performance improvements of combined constrained learning and predictive control besides the provided guarantees.
Riccati updates for online linear quadratic control
Authors: Mohammad Akbari, Bahman Gharesifard, Tamas Linder
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6

We study an online setting of the linear quadratic Gaussian optimal control problem on a sequence of cost functions, where similar to classical online optimization, the future decisions are made by only knowing the cost in hindsight. We introduce a modified online Riccati update that under some boundedness assumptions, leads to logarithmic regret bounds, improving the best known square-root bound. In particular, for the scalar case we achieve the logarithmic regret without any boundedness assumption. As opposed to earlier work, proposed method does not rely on solving semi-definite programs at each stage.
Parameter Optimization for Learning-based Control of Control-Affine Systems
Authors: Armin Lederer, Alexandre Capone, Sandra Hirche
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: regression

Supervised machine learning is often applied to identify system dynamics where first principle methods fail. When combining learning with control methods, probabilistic regression is typically applied to increase robustness against learning errors and analyze the stability of the closed-loop system. Although this approach allows to formulate performance guarantees for many control techniques, the obtained bounds are usually conservative, and cannot be employed for efficient control parameter tuning. Therefore, we reformulate the parameter tuning problem using robust optimization with performance constraints based on Lyapunov theory. By relaxing the problem through scenario optimization we derive a provably optimal method for control parameter tuning. We demonstrate its flexibility and efficiency on parameter tuning problems for a feedback linearizing and a computed torque controller.
Bayesian model predictive control: Efficient model exploration and regret bounds using posterior sampling
Authors: Kim Peter Wabersich, Melanie Zeilinger
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: model, Model Predictive Control

Tight performance specifications in combination with operational constraints make model predictive control (MPC) the method of choice in various industries. As the performance of an MPC controller depends on a sufficiently accurate objective and prediction model of the process, a significant effort in the MPC design procedure is dedicated to modeling and identification. Driven by the increasing amount of available system data and advances in the field of machine learning, data-driven MPC techniques have been developed to facilitate the MPC controller design. While these methods are able to leverage available data, they typically do not provide principled mechanisms to automatically trade off exploitation of available data and exploration to improve and update the objective and prediction model. To this end, we present a learning-based MPC formulation using posterior sampling techniques, which provides finite-time regret bounds on the learning performance while being simple to implement using off-the-shelf MPC software and algorithms. The performance analysis of the method is based on posterior sampling theory and its practical efficiency is illustrated using a numerical example of a highly nonlinear dynamical car-trailer system.
Universal Simulation of Stable Dynamical Systems by Recurrent Neural Nets
Authors: Joshua Hanson, Maxim Raginsky
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6

It is well-known that continuous-time recurrent neural nets are universal approximators for continuous-time dynamical systems. However, existing results provide approximation guarantees only for finite-time trajectories. In this work, we show that infinite-time trajectories generated by dynamical systems that are stable in a certain sense can be reproduced arbitrarily accurately by recurrent neural nets. For a subclass of these stable systems, we provide quantitative estimates on the sufficient number of neurons needed to achieve a specified error tolerance.
Constraint Management for Batch Processes Using Iterative Learning Control and Reference Governors
Authors: Aidan Laracy, Hamid Ossareh
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: Management

 This paper provides a novel combination of Reference Governors (RG) and Iterative Learning Control (ILC) to address the issue of simultaneous learning and constraint management in systems that perform a task repeatedly. The proposed control strategy leverages the measured output from the previous iterations to improve tracking, while guaranteeing constraint satisfaction during the learning process. To achieve this, the system is modeled by a linear system with polytopic uncertainties. An RG solution based on a robust Maximal Admissable Set (MAS) is proposed that endows the ILC algorithm with constraint management capabilities. An update law on the MAS is proposed to further improve performance.
Data-driven Identification of Approximate Passive Linear Models for Nonlinear Systems
Authors: S Sivaranjani, Etika Agarwal, Vijay Gupta
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6

In model-based approaches to learning for controller design, it is important to first identify a system model from input-output data. Assume that we have access to some information about the true system satisfying a structural property that makes it easy to design a controller and obtain a desired performance or stability guarantee on the closed loop system. Can we identify a system model that satisfies this property? In this paper, we consider the property to be that of passivity, that can be used to ensure stability with a learned controller. We present an algorithm to learn a passive linear model of a unknown passive nonlinear system from time domain input-output data. We first learn an approximate linear model of the nonlinear system using standard regression techniques. We then perturb the system matrices of the linear model to enforce passivity. We show that the perturbation can be chosen to ensure that the linear model closely approximates the dynamical behavior of the nonlinear system. Further, we provide an analytical relationship between the size of the perturbation and the radius in which the passivity of the linear model guarantees local passivity of the unknown nonlinear system.
Black-box continuous-time transfer function estimation with stability guarantees: a kernel-based approach
Authors: Mirko Mazzoleni, Matteo Scandella, Simone Formentin, Fabio Previdi
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: Model Selection

Continuous-time parametric models of dynamical systems are usually preferred given their physical interpretation. When there is a lack of prior physical knowledge, the user is faced with the model selection issue. In this paper, we propose a non-parametric approach to estimate a continuous-time stable linear model from data, while automatically selecting a proper structure of the transfer function and guaranteeing to preserve the system stability properties. Results show how the proposed approach outperforms the state of the art.
Scalable Reinforcement Learning of Localized Policies for Multi-Agent Networked Systems
Authors: Guannan Qu, Adam Wierman, Na Li
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: reinforcement-learning, Reinforcement Learning (RL)

We study reinforcement learning (RL) in a setting with a network of agents whose states and actions interact in a local manner where the objective is to find localized policies such that the (discounted) global reward is maximized. A fundamental challenge in this setting is that the state-action space size scales exponentially in the number of agents, rendering the problem intractable for large networks. In this paper, we propose a Scalable Actor Critic (SAC) framework that exploits the network structure and finds a localized policy that is an $O(\rho^\kappa)$-approximation of a stationary point of the objective for some $\rho\in(0,1)$, with complexity that scales with the local state-action space size of the largest $\kappa$-hop neighborhood of the network.
Learning Navigation Costs from Demonstrations with Semantic Observations
Authors: Tianyu Wang, Vikas Dhiman, Nikolay Atanasov
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: Autonomous Driving, Motion Planning, Robot Navigation

This paper focuses on inverse reinforcement learning (IRL) for autonomous robot navigation using semantic observations. The objective is to infer a cost function that explains demonstrated behavior while relying only on the expert’s observations and state-control trajectory. We develop a map encoder, which infers semantic class probabilities from the observation sequence, and a cost encoder, defined as deep neural network over the semantic features. Since the expert cost is not directly ob-servable, the representation parameters can only be optimized by differentiating the error between demonstrated controls and a control policy computed from the cost estimate. The error is optimized using a closed-form subgradient computed only over a subset of promising states via a motion planning algorithm. We show that our approach learns to follow traffic rules in the autonomous driving CARLA simulator by relying on semantic observations of cars, sidewalks and road lanes.
Counterfactual Programming for Optimal Control
Authors: Luiz F.O. Chamon, Santiago Paternain, Alejandro Ribeiro
Date: 2020-06-08 00:00:00
Proceeding: L4DC 2020 6
Tasks: counterfactual

In recent years, considerable work has been done to tackle the issue of designing control laws based on observations to allow unknown dynamical systems to perform pre-specified tasks. At least as important for autonomy, however, is the issue of learning which tasks can be performed in the first place. This is particularly critical in situations where multiple~(possibly conflicting) tasks and requirements are demanded from the agent, resulting in infeasible specifications. Such situations arise due to over-specification or dynamic operating conditions and are only aggravated when the dynamical system model is learned through simulations. Often, these issues are tackled using regularization and penalties tuned based on application-specific expert knowledge. Nevertheless, this solution becomes impractical for large-scale systems, unknown operating conditions, and/or in online settings where expert input would be needed during the system operation. Instead, this work enables agents to autonomously pose, tune, and solve optimal control problems by compromising between performance and specification costs. Leveraging duality theory, it puts forward a counterfactual optimization algorithm that directly determines the specification trade-off while solving the optimal control problem.