Original: M ETA -A CTIVE L EARNING IN P ROBABILISTICALLYS AFE O PTIMIZATION
Revised: M ETA -A CTIVE L EARNING IN P ROBABILISTICALLYS AFE O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning to control a safety-critical system with latent dynamics (e.g. for deep brain stimulation) requires judiciously taking calculated risks to gain information
Revised: Learning to control a safety-critical system with latent dynamics (e.g. for deep brain stimulation) requires judiciously taking calculated risks to gain information

Original: We present a probabilistically-safe, meta-active learning approach to efﬁciently learn system dynamics and optimal conﬁgurations.
Revised: We present a probabilistically-safe, meta-active learning approach to efﬁciently learn system dynamics and optimal conﬁgurations.

Original: The key to our approach is a novel integration of meta-learning and chance-constrained optimization in which we 1) meta-learn an LSTM-based embedding of the active learning sample history, 2) encode a deep learning-based acquisition function with this embedding into a mixed-integer linear program (MILP) and 3) solve the MILP to ﬁnd the optimal action trajectory, trading off the predicted information gain from the acquisition function and the likelihood of safe control.
Revised: The key to our approach is a novel integration of meta-learning and chance-constrained optimization in which we 1) meta-learn an LSTM-based embedding of the active learning sample history, 2) encode a deep learning-based acquisition function with this embedding into a mixed-integer linear program (MILP) and 3) solve the MILP to ﬁnd the optimal action trajectory, trading off the predicted information gain from the acquisition function and the likelihood of safe control.

Original: We set a new state-of-the-art in active learning to control a high-dimensional system with latent dynamics, achieving a 46% increase in information gain and a 20% speedup in computation time.
Revised: We set a new state-of-the-art in active learning to control a high-dimensional system with latent dynamics, achieving a 46% increase in information gain and a 20% speedup in computation time.

Original: We then outperform baseline methods in learning the optimal parameter settings for deep brain stimulation in rats to enhance the rats’ performance on a cognitive task while safely avoiding unwanted side effects (i e., triggering seizures).
Revised: We then outperform baseline methods in learning the optimal parameter settings for deep brain stimulation in rats to enhance the rats’ performance on a cognitive task while safely avoiding unwanted side effects (i e., triggering seizures).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Safe and efﬁcient control of a novel systems with latent dynamics is an important objective in domains from healthcare to robotics.
Revised: Safe and efﬁcient control of a novel systems with latent dynamics is an important objective in domains from healthcare to robotics.

Original: In healthcare, deep brain stimulation devices implanted in the brain can improve memory deﬁcits in patients with Alzheimers (Posporelis et al., 2018) and responsive neurostimulators (RNS) can counter epileptiform activity to mitigate seizures.
Revised: In healthcare, deep brain stimulation devices implanted in the brain can improve memory deﬁcits in patients with Alzheimers (Posporelis et al., 2018) and responsive neurostimulators (RNS) can counter epileptiform activity to mitigate seizures.

Original: Yet, the surgeon’s trial-and-error process of ﬁnding effective RNS parameters for each patient is time-consuming and risky, with poor device settings possibly damaging the brain.
Revised: Yet, the surgeon’s trial-and-error process of ﬁnding effective RNS parameters for each patient is time-consuming and risky, with poor device settings possibly damaging the brain.

Original: Researchers studying active learning and Bayesian optimization have sought to develop algorithms to efﬁciently and safely learn a systems’ dynamics, e.g. learning a brain’s dynamics for RNS conﬁguration (Ashmaig et al., 2018;
Revised: Researchers studying active learning and Bayesian optimization have sought to develop algorithms to efﬁciently and safely learn a systems’ dynamics, e.g. learning a brain’s dynamics for RNS conﬁguration (Ashmaig et al., 2018;

Original: Sui et al., 2018).
Revised: Sui et al., 2018).

Original: However, because these algorithms fail to scale up to higher-dimensional state-action spaces, researchers utilize only simple voltage and frequency controls rather than all 32 channels of the RNS waveform (Ashmaig et al., 2018).
Revised: However, because these algorithms fail to scale up to higher-dimensional state-action spaces, researchers utilize only simple voltage and frequency controls rather than all 32 channels of the RNS waveform (Ashmaig et al., 2018).

Original: Similarly, tasks in robotics, e.g. learning the dynamics of novel robotic systems (e.g., an autopilot learning to ﬂy a damaged aircraft), require active learning methods that succeed in higher-dimensional domains.
Revised: Similarly, tasks in robotics, e.g. learning the dynamics of novel robotic systems (e.g., an autopilot learning to ﬂy a damaged aircraft), require active learning methods that succeed in higher-dimensional domains.

Original: In this paper, we develop a probabilistically-safe, meta-active learning approach to tackle these challenging tasks to efﬁciently learn system dynamics and optimal conﬁgurations.
Revised: In this paper, we develop a probabilistically-safe, meta-active learning approach to tackle these challenging tasks to efﬁciently learn system dynamics and optimal conﬁgurations.

Original: We draw inspiration from recent contributions in meta-learning (Finn et al., 2017;
Revised: We draw inspiration from recent contributions in meta-learning (Finn et al., 2017;

Original: Nagabandi et al., 2019; Wang et al., 2016; Andrychowicz et al., 2016) that seek to leverage a distribution over training tasks to optimize the parameters of a neural network for efﬁcient, online adaptation.
Revised: Nagabandi et al., 2019; Wang et al., 2016; Andrychowicz et al., 2016) that seek to leverage a distribution over training tasks to optimize the parameters of a neural network for efﬁcient, online adaptation.

Original: Researchers have previously investigated meta-learning for active learning, e.g. learning a Bayesian prior over a Gaussian
Revised: Researchers have previously investigated meta-learning for active learning, e.g. learning a Bayesian prior over a Gaussian

Original: Process (Wang et al., 2018a) for learning an acquisition function.
Revised: Process (Wang et al., 2018b) for learning an acquisition function.

Original: However, these approaches do not consider the important problem of safely and actively learning to control a system with altered dynamics, which is a requirement for safety-critical robotic applications.
Revised: However, these approaches do not consider the important problem of safely and actively learning to control a system with altered dynamics, which is a requirement for safety-critical robotic applications.

Original: Furthermore, as we show in
Revised: Furthermore, as we show in

Original: Section 5, on challenging control tasks for healthcare and robotics, the performance of prior active learning approaches (Kirsch et al., 2019; Hastie et al., 2017) leaves much to be desired.
Revised: Section 5, on challenging control tasks for healthcare and robotics, the performance of prior active learning approaches (Kirsch et al., 2019; Hastie et al., 2017) leaves much to be desired.

Original: We seek to overcome these key limitations of prior work by harnessing the power of meta-learning for active learning in a chance-constrained optimization framework for safe, online adaptation by encoding a learned representation of sample history.
Revised: We seek to overcome these key limitations of prior work by harnessing the power of meta-learning for active learning in a chance-constrained optimization framework for safe, online adaptation by encoding a learned representation of sample history.

Original: Instead of hand-engineering an acquisition function for our speciﬁc domains, our approach employs a data-drive, meta-learning approach, which results in better performance than prior approaches, as shown in Section 5.
Revised: Instead of hand-engineering an acquisition function for our speciﬁc domains, our approach employs a data-drive, meta-learning approach, which results in better performance than prior approaches, as shown in Section 5.

Original: Furthermore, our approach has the unique ability to impose analytical safety constraints over a sample trajectory.
Revised: Furthermore, our approach has the unique ability to impose analytical safety constraints over a sample trajectory.

Original: Contributions – We develop a probabilistically safe, meta-learning approach for active learning (”meta-active learning”) that sets a new state-of-the-art.
Revised: Contributions – We develop a probabilistically safe, meta-learning approach for active learning (”meta-active learning”) that sets a new state-of-the-art.

Original: Our acquisition function (i.e., the function that predicts the expected information gain of a data point) is meta-learned ofﬂine, allowing the policy to beneﬁt from past experience and provide a more robust measure of the value of an action.
Revised: Our acquisition function (i.e., the function that predicts the expected information gain of a data point) is meta-learned ofﬂine, allowing the policy to beneﬁt from past experience and provide a more robust measure of the value of an action.

Original: The key to our approach is a novel interweaving of our deep, meta-learned acquisition function as a Long-Short Term Memory Network (Gers et al., 1999) (LSTM) within a chance-constrained, mixed-integer linear program (MILP)
Revised: The key to our approach is a novel interweaving of our deep, meta-learned acquisition function as a Long-Short Term Memory Network (Gers et al., 1999) (LSTM) within a chance-constrained, mixed-integer linear program (MILP)

Original: (Schrijver, 1998).
Revised: (Schrijver, 1998).

Original: By encoding the LSTM’s linear, piece-wise output layers into the MILP, we directly optimize an action trajectory that best ensures the safety of the system while also maximizing the information learned about the system.
Revised: By encoding the LSTM’s linear, piece-wise output layers into the MILP, we directly optimize an action trajectory that best ensures the safety of the system while also maximizing the information learned about the system.

Revised: In this paper, we describe our novel architecture which uniquely combines the power of a learned acquisition function with chance-constrained optimization and evaluate its performance against state-of-the-art baselines in several relevant domains.

Revised: To the best of our knowledge, this is the only architecture which meta-learns an acquisition function for optimization tasks and is capable of embedding this acquisition function in a chance-constrained linear program to guarantee a minimum level of safe operation.

Original: The contributions of this paper are as follows:
Revised: The contributions of this paper are as follows

Original: Meta-active learning for autonomously synthesizing an acquisition function to efﬁciently infer altered or unknown system dynamics and optimize system parameters.
Revised: Meta-active learning for autonomously synthesizing an acquisition function to efﬁciently infer altered or unknown system dynamics and optimize system parameters.

Original: Probabilistically-safe control combined with an active-learning framework through the integration of our deep learning-based acquisition function and integer linear programming.
Revised: Probabilistically-safe control combined with an active-learning framework through the integration of our deep learning-based acquisition function and integer linear programming.

Original: State-of-the art results for safe, active learning.
Revised: State-of-the art results for safe, active learning.

Original: We achieve a 46% increase in information gain in a high-dimensional environment of controlling a damaged aircraft, and we achieve a 58% increase in information gain in our deep brain stimulation against our baselines.
Revised: We achieve a 46% increase in information gain in a high-dimensional environment of controlling a damaged aircraft, and we achieve a 58% increase in information gain in our deep brain stimulation against our baselines.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we review the foundations of our work in active, meta-, and reinforcement learning.
Revised: In this section, we review the foundations of our work in active, meta-, and reinforcement learning.

Original: Active Learning – Labelled training data is often difﬁcult to obtain due either to tight time constraints or lack of expert resources.
Revised: Active Learning – Labelled training data is often difﬁcult to obtain due either to tight time constraints or lack of expert resources.

Original: Active learning attempts to address this problem by utilizing an “acquisition function” to quantify the amount of information an unlabelled training sample, x ∈ D U
Revised: Active learning attempts to address this problem by utilizing an “acquisition function” to quantify the amount of information an unlabelled training sample, x ∈ D U

Original: = (cid:104) x
Revised: = (cid:104) x

Original: i (cid:105) ni =1 , would provide a base learner, ˆ T ψ , if that sample were given a label, y and added to a labeled dataset, D L
Revised: i (cid:105) ni =1 , would provide a base learner, ˆ T ψ , if that sample were given a label, y and added to a labeled dataset, D L

Original: = (cid:104) x j , y j (cid:105) mj
Revised: = (cid:104) x j , y j (cid:105) mj

Original: =1 , i.e., D L ← D L ∪ (cid:104) x, y (cid:105) .
Revised: =1 , i.e., D L ← D L ∪ (cid:104) x, y (cid:105) .

Original: The active learning algorithm queries its acquisition function, H ( D U , D L , T ψ ) , to select which x ∈ D
Revised: The active learning algorithm queries its acquisition function, H ( D U , D L , T ψ ) , to select which x ∈ D

Original: U should be labeled and added to D L ; then, a label is queried (e.g., by taking an action in an environment and observing the effect) for x , and the new labeled sample is added to D L (Muslea et al., 2006; Pang et al., 2018).
Revised: U should be labeled and added to D L ; then, a label is queried (e.g., by taking an action in an environment and observing the effect) for x , and the new labeled sample is added to D L (Muslea et al., 2006; Pang et al., 2018).

Original: Meta-Learning – Meta-learning approaches attempt to learn a method to quickly adapt to new tasks online.
Revised: Meta-Learning – Meta-learning approaches attempt to learn a method to quickly adapt to new tasks online.

Original: In contrast to active learning, meta-learning attempts to learn a skill or learning method, e.g. learning an active learning function, which can be transferable to novel tasks or scenarios.
Revised: In contrast to active learning, meta-learning attempts to learn a skill or learning method, e.g. learning an active learning function, which can be transferable to novel tasks or scenarios.

Original: These tasks or skills are trained ofﬂine, and a common assumption is that the tasks selected at test time are drawn from the same distribution used for training (Hospedales et al., 2020).
Revised: These tasks or skills are trained ofﬂine, and a common assumption is that the tasks selected at test time are drawn from the same distribution used for training (Hospedales et al., 2020).

Original: Reinforcement Learning and Q-Learning – A Markov decision process (MDP) is a stochastic control process for decision making and can be deﬁned by the 5-tuple (cid:104)X , U , T , R , γ (cid:105) .
Revised: Reinforcement Learning and Q-Learning – A Markov decision process (MDP) is a stochastic control process for decision making and can be deﬁned by the 5-tuple (cid:104)X , U , T , R , γ (cid:105) .

Original: X represents the set of states and U the set of actions.
Revised: X represents the set of states and U the set of actions.

Original: T :
Revised: T :

Original: X × U × X (cid:48)
Revised: X × U × X (cid:48)

Original: → [0 , 1] is the transition function that returns the probability of transitioning to state x (cid:48) from state x applying action, u .
Revised: → [0 , 1] is the transition function that returns the probability of transitioning to state x (cid:48) from state x applying action, u .

Original: R :
Revised: R :

Original: X × U
Revised: X × U

Original: → R is a reward function that maps a state and action to a reward, and γ weights the discounting of future rewards.
Revised: → R is a reward function that maps a state and action to a reward, and γ weights the discounting of future rewards.

Original: Reinforcement learning seeks to synthesize a policy, π : X → U , mapping states to actions to maximize the future expected reward.
Revised: Reinforcement learning seeks to synthesize a policy, π : X → U , mapping states to actions to maximize the future expected reward.

Original: When π is the optimal policy, π ∗ , the following Bellman condition holds: Q π 
Revised: When π is the optimal policy, π ∗ , the following Bellman condition holds: Q π ∗ ( x, u ) :=

Original: [Equation]
Revised: E x (cid:48) ∼ T (cid:2) R ( x, u ) + γQ π ∗ ( x (cid:48) , π ∗ ( x )) (cid:3) (Sutton & Barto, 2018).

Original: 2. P ROBLEM S ET - UP
Revised: 2. P ROBLEM S ET - UP

Original: Our work is at the unique nexus of active learning, meta-learning and deep reinforcement learning with the objective of learning the Q-function as an acquisition function to describe the expected future information gained when taking action, u , in state, x , given a set of previously experienced states and actions.
Revised: Our work is at the unique nexus of active learning, meta-learning and deep reinforcement learning with the objective of learning the Q-function as an acquisition function to describe the expected future information gained when taking action, u , in state, x , given a set of previously experienced states and actions.

Original: We deﬁne information gain as the percent decrease in the error of the objective (e.g., decrease in model error).
Revised: We deﬁne information gain as the percent decrease in the error of the objective (e.g., decrease in model error).

Original: A formal deﬁnition is provided in the Appendix.
Revised: A formal deﬁnition is provided in the Appendix.

Original: The Q-function is trained via a meta-learning strategy in which an agent interacts in environments sampled from a distribution of different scenarios to distill the most effective acquisition function for active learning of system dynamics.
Revised: The Q-function is trained via a meta-learning strategy in which an agent interacts in environments sampled from a distribution of different scenarios to distill the most effective acquisition function for active learning of system dynamics.

Original: In our context, the state, X , of the active learning system is given by X : (cid:104) D U , D L , ˆ T ψ (cid:105) , where D U consists of all possible state-action pairs, D L is the set of state-action pairs that the agent has already experienced, and ˆ T ψ is a neural network function approximator of the transition dynamics, which is parameterized by ψ and updated online as samples are collected.
Revised: In our context, the state, X , of the active learning system is given by X : (cid:104) D U , D L , ˆ T ψ (cid:105) , where D U consists of all possible state-action pairs, D L is the set of state-action pairs that the agent has already experienced, and ˆ T ψ is a neural network function approximator of the transition dynamics, which is parameterized by ψ and updated online as samples are collected.

Original: We note that this state, X , is distinct from the state, X , of the robotic (or other) system we are controlling.
Revised: We note that this state, X , is distinct from the state, X , of the robotic (or other) system we are controlling.

Original: The reward, R ( i ) , is proportional to the reduction of the mean squared error (MSE) loss of ˆ T ψ at time step, i .
Revised: The reward, R ( i ) , is proportional to the reduction of the mean squared error (MSE) loss of ˆ T ψ at time step, i .

Original:  S AFE M ETA -L EARNING A RCHITECTURE
Revised:  S AFE M ETA -L EARNING A RCHITECTURE

Original: Several key components are vital for learning about an unknown system in a timely manner.
Revised: Several key components are vital for learning about an unknown system in a timely manner.

Original: First, an encoded representation of the context of the new dynamics is important for determining where exploration should be focused and which actions may be beneﬁcial for gaining information of unknown dynamics.
Revised: First, an encoded representation of the context of the new dynamics is important for determining where exploration should be focused and which actions may be beneﬁcial for gaining information of unknown dynamics.

Original: Second, a range of prior experiences in active learning should be leveraged to best inform which actions elicit the most information in a novel context within a distribution of tasks.
Revised: Second, a range of prior experiences in active learning should be leveraged to best inform which actions elicit the most information in a novel context within a distribution of tasks.

Original: We seek to develop a framework with these key attributes to enable sample-efﬁcient and computationally light-weight active learning.
Revised: We seek to develop a framework with these key attributes to enable sample-efﬁcient and computationally light-weight active learning.

Original: An overview of our system is shown in Fig.
Revised: An overview of our system is shown in Fig.

Original: 3. M ETA -L EARNING A LGORITHM
Revised: 3. M ETA -L EARNING A LGORITHM

Original: To infer the Q-function for an action (i.e. the acquisition function), we meta-learn over a distribution of altered dynamics as described in Algorithm 1.
Revised: To infer the Q-function for an action (i.e. the acquisition function), we meta-learn over a distribution of altered dynamics as described in Algorithm 1.

Original: For each episode, we sample from this distribution of altered dynamics and limit each episode to the number of time steps, M , tuned to collect enough data to accurately learn our approximate dynamics, ˆ T ψ , as a neural network with parameters, ψ .
Revised: For each episode, we sample from this distribution of altered dynamics and limit each episode to the number of time steps, M , tuned to collect enough data to accurately learn our approximate dynamics, ˆ T ψ , as a neural network with parameters, ψ .

Original: We utilize Q-learning to infer the Q-function which is represented by a DQN.
Revised: We utilize Q-learning to infer the Q-function which is represented by a DQN.

Original: In our training scheme we search over the action space via a MILP, as described in Section 3.2 and select the action set, (cid:126) U
Revised: In our training scheme we search over the action space via a MILP, as described in Section 3.2 and select the action set, (cid:126) U

Original: ( t : T ) , which maximizes the Q-function while satisfying safety constraints.
Revised: ( t : T ) , which maximizes the Q-function while satisfying safety constraints.

Original: The acquisition Q-function, Q θ , as described in Eq.
Revised: The acquisition Q-function, Q θ , as described in Eq.

Original: 1 is trained via Deep Q-Learning (Ganger et al., 2016) with target network, Q φ .
Revised: 1 is trained via Deep Q-Learning (Ganger et al., 2016) with target network, Q φ .

Original: During training, our Deep Q-Learning framework is augmented to account for safety .
Revised: During training, our Deep Q-Learning framework is augmented to account for safety .

Original: The learned acquisition function, Q θ , is utilized by our policy (Eq. 1), which is solved via a safety-constrained MILP solver.
Revised: The learned acquisition function, Q θ , is utilized by our policy (Eq. 1), which is solved via a safety-constrained MILP solver.

Original: The reward, R ( t ) , for taking a set of actions in a given state is deﬁned as the percent decrease in the MSE error of the model, ˆ T ψ .
Revised: The reward, R ( t ) , for taking a set of actions in a given state is deﬁned as the percent decrease in the MSE error of the model, ˆ T ψ .

Original: This Bellman loss is backpropagated through the Q-value linear and (ReLU) output layers through the LSTM encoding layers.
Revised: This Bellman loss is backpropagated through the Q-value linear and (ReLU) output layers through the LSTM encoding layers.

Original: π ( (cid:126)Z ( t +1) ) is the set of actions, (cid:126) U
Revised: π ( (cid:126)Z ( t +1) ) is the set of actions, (cid:126) U

Original: ( t +1) , determined by maximizing Eq. 1, which we describe in Section 3.2.
Revised: ( t +1) , determined by maximizing Eq. 1, which we describe in Section 3.2.

Original: The dynamics model, ˆ T ψ , is retrained with each new set of state-action pairs.
Revised: The dynamics model, ˆ T ψ , is retrained with each new set of state-action pairs.

Original: [Table] [Table] [Table] [Table]
Revised: 1) ) , (cid:126)Z ( t +1) (cid:17) 16:

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: Update Q θ according to (cid:16) y ( t ) − Q θ ( (cid:126) U ( t ) , (cid:126)Z ( t ) )

Original: [Table] [Table] [Table] [Table]
Revised: (cid:17) 17:

Original: y

Original: R ( t (cid:16) (cid:126)Z (cid:126)Z
Revised: Q φ ← τQ θ + τ (1 − Q φ ) 18: end for 19: end for ;

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: θ + τ (1 − Q φ ) 18: end for 19: end for ;

Original: 1: Draw test example from dis- tribution 2: Initialize ˆ T ψ 3: for i=1 to M do 4:
Revised: 1: Draw test example from dis- tribution 2: Initialize ˆ T ψ 3: for i=1 to M do 4:

Original: Do forward pass through encoder to get (cid:126)Z ( i ) 5:
Revised: Do forward pass through encoder to get (cid:126)Z ( i ) 5:

Original: Select actions, U ( i ) , according to Eq.6:
Revised: Select actions, U ( i ) , according to Eq.6:

Original: Execute actions, U ( i ) ; observe states X ( i +1) and reward R ( i ) 7:
Revised: Execute actions, U ( i ) ; observe states X ( i +1) and reward R ( i ) 7:

Original: Retrain ˆ T ψ on X (0: i +1) ,
Revised: Retrain ˆ T ψ on X (0: i +1) ,

Original: U (0: i +1) 8: end for
Revised: U (0: i +1) 8: end for

Original: 3. M IXED -I NTEGER L INEAR P ROGRAM
Revised: 3. M IXED -I NTEGER L INEAR P ROGRAM

Original: Our objective (Eq. 1) is to maximize both the probability of the system remaining in a safe conﬁguration and the information gained along the  trajectory  from [ t, t + T ) . 
Revised: Our objective (Eq. 1) is to maximize both the probability of the system remaining in a safe conﬁguration and the information gained along the ﬁnite trajectory horizon from [ t, t + T ) . [Equation] argmax [Equation]

Revised: Q

Original: [Equation]
Revised: [Equation]

Original: Q φ ( (cid:126) U ( t : T ) , (cid:126)Z ) describes the expected information gained along the trajectory when the set of actions (cid:126) U ( t :  T ) is taken in the context of the embedding (cid:126)Z , and ¯ Q θ ( · , (cid:126)Z ) is the expected Q-value in state (cid:126)Z , which we discuss further in Section 3.3.
Revised: ( (cid:126) U ( t [Equation] t + T ) , (cid:126)Z ) describes the expected information gained along the trajectory when the set of actions (cid:126) U ( t : t + T ) is taken in the context of the embedding (cid:126)Z , and ¯ Q θ ( · , (cid:126)Z ) is the expected Q-value in state (cid:126)Z , which we discuss further in Section 3.3.

Original: λ 1 is a hyper-parameter that can be tuned to adjust the tradeoff between safety and information gain.
Revised: λ 1 is a hyper-parameter that can be tuned to adjust the tradeoff between safety and information gain.

Revised: Pr (cid:110) (cid:13)(cid:13) (cid:126)x ( t + T ) − (cid:126)x r (cid:13)(cid:13) 1 ≤ r (cid:111) deﬁnes the probability of remaining within the hyper-ellipsoid of safety.

Original: We derive a linearization of this equation based on an assumption of Gaussian dynamics which can then be solved via MILP.
Revised: We derive a linearization of this equation based on an assumption of Gaussian dynamics which can then be solved via MILP.

Original: Deﬁnition of Safety - We deﬁne a hyper-ellipsoid of safety, parameterized by the safe state x r and radius (cid:126)r encompassing all known safe system states.
Revised: Deﬁnition of Safety - We deﬁne a hyper-ellipsoid of safety, parameterized by the safe state x r and radius (cid:126)r encompassing all known safe system states.

Original: In the case of an aircraft, (cid:126)x r would be straight and level ﬂight and the hyper-ellipsoid would include deviations considered safe.
Revised: In the case of an aircraft, (cid:126)x r would be straight and level ﬂight and the hyper-ellipsoid would include deviations considered safe.

Original: Additionally, we assume that our model error comes from a Gaussian distribution with known mean and variance.
Revised: Additionally, we assume that our model error comes from a Gaussian distribution with known mean and variance.

Original: This assumption allows us to impose safety constraints which can either be enforced with a probability of one, or this requirement can be relaxed to increase the potential for information gain.
Revised: This assumption allows us to impose safety constraints which can either be enforced with a probability of one, or this requirement can be relaxed to increase the potential for information gain.

Original: By assuming that our model error originates from a Gaussian distribution, we can linearize our probability constraints described in Eq.
Revised: By assuming that our model error originates from a Gaussian distribution, we can linearize our probability constraints described in Eq.

Original: 9 to include in the MILP (See Appendix for full derivation).
Revised: 9 to include in the MILP (See Appendix for full derivation).

Original: Here, Φ − 1 is the inverse cumulative distribution function for the standard normal distribution and 1 − (cid:15)
Revised: Here, Φ − 1 is the inverse cumulative distribution function for the standard normal distribution and 1 − (cid:15)

Original: d denotes the probability level.
Revised: d denotes the probability level.

Original: σ represents the uncertainty in the dynamics network parameters and ¯ a and ¯ b are the point estimates of the dynamics.
Revised: σ represents the uncertainty in the dynamics network parameters.

Original: d represents the d th row and j  the columns.
Revised: d represents the d th row and j represents the j th column. ∆ ( t : t + T ) d [Equation]

Revised: the evolution of a state (See Appendix A.4.1 for more detail).

Original: We compute the uncertainty, σ , of our network via bootstrapping (Efron, 2020; Franke & Nuemann, 1998), which we describe furhter in the Supplementary.
Revised: We compute the uncertainty, σ , of our network via bootstrapping (Efron, 2020; Franke & Nuemann, 1998), which we describe further in the Appendix.

Original: We guarantee a minimum probability of safety by selecting the minimum risk value, (cid:15) d , to be at or above our minimum desired probability of safety.
Revised: We guarantee a minimum probability of safety by selecting the minimum risk value, (cid:15) d , to be at or above our minimum desired probability of safety.

Original: This procedure ensure that our algorithm must select a safety constraint that exceeds this minimum safety requirement.
Revised: This procedure ensure that our algorithm must select a safety constraint that exceeds this minimum safety requirement.

Original: Note: the hyper-ellipse, when linearized, is a hyper-rectangle.
Revised: Note: the hyper-ellipse, when linearized, is a hyper-rectangle.

Original: [Equation]
Revised: [Table]

Original:  Our policy takes a set of information rich actions at time t , potentially out of the d-dimensional hyper-ellipsoid of safety, guaranteeing with probability 1 − (cid:15) that the system will return to a safe state after T time steps.
Revised: 1 < r d , ∀ d (2) Our policy takes a set of information rich actions at time t , potentially out of the d-dimensional hyper-ellipsoid of safety, guaranteeing with probability 1 − (cid:15) that the system will return to a safe state after T time steps.

Original: Thus, our T -step propagation allows the system to deviate from the safe region, if desired, to focus on actively learning so long as it can return to the safe region with high conﬁdence.
Revised: Thus, our T -step propagation allows the system to deviate from the safe region, if desired, to focus on actively learning so long as it can return to the safe region with high conﬁdence.

Original: 3. A N LSTM Q-F UNCTION AS A L INEAR P ROGRAM
Revised: 3. A N LSTM Q-F UNCTION AS A L INEAR P ROGRAM

Original: We leverage an LSTM as a function approximator for Q-learning.
Revised: We leverage an LSTM as a function approximator for Q-learning.

Original: To blend our Q-function with the MILP, we pass the LSTM output embedding, which serves as a learned representation of the sample history, through a fully connected layer with ReLU activations.
Revised: To blend our Q-function with the MILP, we pass the LSTM output embedding, which serves as a learned representation of the sample history, through a fully connected layer with ReLU activations.

Original: This design enables us to backprop the Bellman residual through the output layers encoded in the MILP all the way to the LSTM inputs.
Revised: This design enables us to backprop the Bellman residual through the output layers encoded in the MILP all the way to the LSTM inputs.

Original: Thus we can leverage the power of an LSTM and mathematical programming in a seamless framework.
Revised: Thus we can leverage the power of an LSTM and mathematical programming in a seamless framework.

Original: 2 Given our meta-learned acquisition function and chance-constrained optimization framework, we can now perform probabilistically-safe active learning.
Revised: 2 Given our meta-learned acquisition function and chance-constrained optimization framework, we can now perform probabilistically-safe active learning.

Original: Algorithmdescribes how we perform our online, safe and active learning.
Revised: Algorithmdescribes how we perform our online, safe and active learning.

Original: Intuitively, our algorithm initializes a new dynamics model to represent the unknown or altered dynamics, and we iteratively sample information rich, safe actions via our MILP policy, update our dynamics model, and repeat.
Revised: Intuitively, our algorithm initializes a new dynamics model to represent the unknown or altered dynamics, and we iteratively sample information rich, safe actions via our MILP policy, update our dynamics model, and repeat.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: We design two experiments to validate our approach’s ability to safely and actively learn altered or unknown dynamics in real time.
Revised: We design two experiments to validate our approach’s ability to safely and actively learn altered or unknown dynamics in real time.

Original: We compare our approach to several baseline approaches and demonstrate that our approach is superior in terms of both information gain and computation time.
Revised: We compare our approach to several baseline approaches and demonstrate that our approach is superior in terms of both information gain and computation time.

Original: More details on the experimental domains can be found in the Appendix.
Revised: More details on the experimental domains can be found in the Appendix.

Original: 4. A SYNCHRONOUS D ISTRIBUTED M ICROELECTRODE T HETA
Revised: 4. A SYNCHRONOUS D ISTRIBUTED M ICROELECTRODE T HETA

Original: Asynchronous distributed microelectrode theta stimulation (ADMETS) is a cutting edge deep brain stimulation approach to treating seizure conditions that cannot be controlled via pharmacological methods.
Revised: Asynchronous distributed microelectrode theta stimulation (ADMETS) is a cutting edge deep brain stimulation approach to treating seizure conditions that cannot be controlled via pharmacological methods.

Original: In ADMETS, a neuro-stimulator is implanted in the brain to deliver continuous electrical pulses to reduce seizures.
Revised: In ADMETS, a neuro-stimulator is implanted in the brain to deliver continuous electrical pulses to reduce seizures.

Original: However, there is no clear mapping from parameter values to reduction in seizures that applies to all patients, as the optimal parameter settings can depend on the placement of the device, the anatomy of an individual’s brain and other confounding factors.
Revised: However, there is no clear mapping from parameter values to reduction in seizures that applies to all patients, as the optimal parameter settings can depend on the placement of the device, the anatomy of an individual’s brain and other confounding factors.

Original: Further, a latent subset of parameters can cause negative side-effects.
Revised: Further, a latent subset of parameters can cause negative side-effects.

Revised: [Figure]

Original: As a surrogate task for evaluating how well an algorithm can ﬁnd the optimal paremeters of an
Revised: As a surrogate task for evaluating how well an algorithm can ﬁnd the optimal paremeters of an

Original: ADMETS device for in vivo epilepsy therapy, researchers have proposed leveraging experimental data collected from parameter sweeps of ADMETS devices in rats.
Revised: ADMETS device for in vivo epilepsy therapy, researchers have proposed leveraging experimental data collected from parameter sweeps of ADMETS devices in rats.

Original: In keeping with Ashmaig et al.
Revised: In keeping with Ashmaig et al.

Original: (2018), we create simulation environments for six rats where, at each ADMETS parameter setting, the cognitive function of a rat was tested (i.e., the rats ability to recall where a treat was located in a maze), as measured by a “memory score.”
Revised: (2018), we create simulation environments for six rats where, at each ADMETS parameter setting, the cognitive function of a rat was tested (i.e., the rats ability to recall where a treat was located in a maze), as measured by a “memory score.”

Original: The task for safe, active learning task is to determine[Figure] the ADMETS parameters (i.e., signal amplitude) in the simulation environments that maximize each rat’s memory scores without causing unwanted side effects (e.g., seizures), which can arise when the memory score drops below zero.
Revised: The task for safe, active learning task is to determine the ADMETS parameters (i.e., signal amplitude) in the simulation environments that maximize each rat’s memory scores without causing unwanted side effects (e.g., seizures), which can arise when the memory score drops below zero.

Original: The reward signal utilized by our meta-learner is the percent decrease in error between the predicted and actual optimal parameters.
Revised: The reward signal utilized by our meta-learner is the percent decrease in error between the predicted and actual optimal parameters.

Original: 4. H IGH - DIMENSIONAL D OMAIN (R ECOVERING D YNAMICS OF A D AMAGED A IRCRAFT )
Revised: 4. H IGH - DIMENSIONAL D OMAIN (R ECOVERING D YNAMICS OF A D AMAGED A IRCRAFT )

Original: Active learning algorithms can be ineffective in high-dimensional domains.
Revised: Active learning algorithms can be ineffective in high-dimensional domains.

Original: As such, we seek to stress-test our algorithms in just such a domain: Learning the nonlinear dynamics of a damaged aircraft online before the system enters an unrecoverable conﬁguration (e.g., a spin or crashing).
Revised: As such, we seek to stress-test our algorithms in just such a domain: Learning the nonlinear dynamics of a damaged aircraft online before the system enters an unrecoverable conﬁguration (e.g., a spin or crashing).

Original: We base our simulation on theoretical damage models from prior work describing the full equations of motion (Watkiss, 1994; Zhang et al., 2017; Ouellette, 2010) within the Flightgear virtual environment.
Revised: We base our simulation on theoretical damage models from prior work describing the full equations of motion (Watkiss, 1994; Zhang et al., 2017; Ouellette, 2010) within the Flightgear virtual environment.

Original: The objective of this domain is to learn the difference between the altered dynamics that result from the damage and to maintain safe ﬂight.
Revised: The objective of this domain is to learn the difference between the altered dynamics that result from the damage and to maintain safe ﬂight.

Original: We consider safe ﬂight to be our designated safe state, (cid:126)x r .
Revised: We consider safe ﬂight to be our designated safe state, (cid:126)x r .

Original: The aircraft takes an information rich action potentially resulting in a deviation outside of the d-dimensional hyper-ellipsoid of safety.
Revised: The aircraft takes an information rich action potentially resulting in a deviation outside of the d-dimensional hyper-ellipsoid of safety.

Original: The next action is constrained to guarantee that the plane returns to the hyper-ellipsoid of safety with probability 1 − (cid:15) via action u ( t +1) .
Revised: The next action is constrained to guarantee that the plane returns to the hyper-ellipsoid of safety with probability 1 − (cid:15) via action u ( t +1) .

Original: 4. B ASELINE C OMPARISONS
Revised: 4. B ASELINE C OMPARISONS

Original: We evaluate against the following baselines in active learning and Bayesian optimization.
Revised: We evaluate against the following baselines in active learning and Bayesian optimization.

Original: The acquisition function baselines, Epistemic Uncertainty (Hastie et al., 2017) and Maximizing Diversity (Schrum & Gombolay, 2020) are embedded in our safety framework, therefore providing a head-tohead comparison between our meta-learned acquisition function and these active learning heuristics.
Revised: The acquisition function baselines, Epistemic Uncertainty (Hastie et al., 2017) and Maximizing Diversity (Schrum & Gombolay, 2020) are embedded in our safety framework, therefore providing a head-tohead comparison between our meta-learned acquisition function and these active learning heuristics.

Original: • Epistemic Uncertainty (Hastie et al., 2017) -
Revised: • Epistemic Uncertainty (Hastie et al., 2017) -

Original: This active learning metric selects the action that maximizes uncertainty.
Revised: This active learning metric selects the action that maximizes uncertainty.

Original: • Maximizing Diversity (Schrum & Gombolay, 2020) -
Revised: • Maximizing Diversity (Schrum & Gombolay, 2020) -

Original: This acquisition function selects actions which maximize the difference between previously seen states and actions.
Revised: This acquisition function selects actions which maximize the difference between previously seen states and actions.

Original: • Bayesian Optimization (BaO) (Ashmaig et al., 2018) -
Revised: • Bayesian Optimization (BaO) (Ashmaig et al., 2018) -

Original: This algorithm was developed for the ADMETS domain (Section 4.1) and is based upon a Gaussian Process model.
Revised: This algorithm was developed for the ADMETS domain (Section 4.1) and is based upon a Gaussian Process model.

Original: • Meta Bayesian Optimization (Meta BO) (Wang et al., 2018b) -
Revised: • Meta Bayesian Optimization (Meta BO) (Wang et al., 2018a) -

Original: This approach meta-learns a Gaussian process prior ofﬂine over previously sampled data.
Revised: This approach meta-learns a Gaussian process prior ofﬂine over previously sampled data.

Original: • Learning Active Learning (LAL) (Konyushkova et al., 2017) -
Revised: • Learning Active Learning (LAL) (Konyushkova et al., 2017) -

Original: This approach meta-learns an acquisition function leveraging hand-engineered features.
Revised: This approach meta-learns an acquisition function leveraging hand-engineered features.

Original: For our ADMETS domain, we benchmark against (Ashmaig et al., 2018; Konyushkova et al., 2017; Wang et al., 2018b) as those baselines are designed or easily adapted to optimization tasks. For our high-dimensional domain, we benchmark against (Ashmaig et al., 2018; Hastie et al., 2017; Konyushkova et al., 2017; Schrum & Gombolay, 2020) as those baselines are designed or easily adapted to system identiﬁcation tasks.
Revised: We benchmark against each of the above algorithms in both the ADMETS and high-dimensional domains.

Original: We do not compare to MAML (Finn et al., 2017) or Nagabandi et al.
Revised: We do not compare to MAML (Finn et al., 2017) or Nagabandi et al.

Original: (2019) as these algorithms have no notion of safety or active learning.
Revised: (2019) as these algorithms have no notion of safety or active learning.

Original:  R ESULTS
Revised:  R ESULTS

Original: We empirically validate that  our meta-active learning approach outperforms baselines across both the ADMETS and high-dimensional domains in terms of its ability to actively learn latent parameters and its ability to safely perform these tasks.
Revised: We empirically validate that in the our meta-active learning approach outperforms baselines across both the ADMETS and high-dimensional domains in terms of its ability to actively learn latent parameters and its ability to safely perform these tasks.

Original: In contrast to our approach, even though Bao, Meta BO, and LAL do not consider safety, we are able to outperform those baselines across all metrics.
Revised: In contrast to our approach, even though BaO, Meta BO, and LAL do not consider safety, we are able to outperform those baselines across all metrics.

Original: [Figure]
Revised: [Figure]

Original: Results fromboth the ADMETS and the high domains empirically validate that our algorithm more efﬁciently and safely learns the optimal control parameters (Fig. 4) and system dynamics (Fig. 5).
Revised: from domains empirically validate that our algorithm more efﬁciently and safely learns the optimal control parameters (Fig. 4) and system dynamics (Fig. 5).

Original: In Fig.
Revised: In Fig.

Original: 4b-5b, we report the mean (standard deviation) for each measure and baseline where possible.
Revised: 4b-5b, we report the mean (standard deviation) for each measure and baseline where possible.

Original: In the ADMETS domain, our model selects an action which results in a 58% higher information gain than BaO and 87% higher information gain than Meta BO on average.
Revised: In the ADMETS domain, our model selects an action which results in a 58% higher information gain than BaO and 87% higher information gain than Meta BO on average.

Revised: When compared to the state-of-the-art active learning baselines, our method performs 41% better than Maximizing Diversity and 98% better than Uncertainty in terms of information gain.

Original: In the high-dimensional domain, we achieve a 49% improvement over Hastie et al.
Revised: In the high-dimensional domain, we achieve a 49% improvement over Hastie et al.

Original: (2017) and a 46% improvement over both Schrum & Gombolay (2020) and Konyushkova et al.
Revised: (2017) and a 46% improvement over both Schrum & Gombolay (2020) and Konyushkova et al.

Original: We point out that our approach outperforms active learning heuristics (i.e., Maximizing Diversity and
Revised: We point out that our approach outperforms active learning heuristics (i.e., Maximizing Diversity and

Original: Epistemic Uncertainty) when run inside our chance-constrained framework for a direct comparison.
Revised: Epistemic Uncertainty) when run inside our chance-constrained framework for a direct comparison.

Original: This resultvalidates meta-learning an acquisition function is a necessary and beneﬁcial component of our framework.
Revised: This result meta-learning an acquisition function is a necessary and beneﬁcial component of our framework.

Original: Computation Time – We demonstrate that our method also outperforms previous work in terms of computation time.
Revised: Computation Time – We demonstrate that our method also outperforms previous work in terms of computation time.

Original: Across both domains, our approach not only achieves a more efﬁcient reduction in model error and improvement in information gain, we are also faster than all baselines in the more challenging, high-dimensional (Fig. 5c) environment.
Revised: Across both domains, our approach not only achieves a more efﬁcient reduction in model error and improvement in information gain, we are also faster than all baselines in the more challenging, high-dimensional (Fig. 5c) environment.

Original: In the lower-dimensional, ADMETS  environment (Fig. 4c), BaO has a slight advantage in computation time, but our algorithm trades the time for a 58% information gain over BaO. Additionally, we are 68x faster than LAL and 61x faster than Meta BO, the two other meta-learning approaches we benchmark against. 
Revised: In the lower-dimensional, ADMETS [Figure] environment (Fig. 4c), BaO has a slight advantage in computation time, but our algorithm trades the time for a 58% information gain over BaO. Additionally, we are 68x faster than LAL and 61x faster than Meta BO, the two other meta-learning approaches we benchmark against. [Figure]

Original: Safety – We empirically validate that, for the information results we report in Fig.
Revised: Safety – In the high-dimensional domain, we empirically validate that, for the information results we report in Fig.

Original: 5 we can achieve an 87% probability the aircraft will return to the safe region 5a.
Revised: 5 we can achieve an 87% probability the aircraft will return to the safe region 5a.

Original: As shown in Fig 5b, only Maximizing Diversity (85% safe) and Epistemic Uncertainty (78% safe) allow for safety constraints to be imposed.
Revised: As shown in Fig 5b, only Maximizing Diversity (85% safe) and Epistemic Uncertainty (78% safe) allow for safety constraints to be imposed.

Original: Our algorithm thus outperforms the baselines in safety as well as information gain.
Revised: Our algorithm thus outperforms the baselines in safety as well as information gain.

Original: When we maximize for safety, our algorithm is able to achieve a 99.9% safe return to the hyper-ellipsoid even without a ground-truth dynamics model.
Revised: When we maximize for safety (i.e., λ = 0 ), our algorithm is able to achieve a 99.9% safe return to the hyper-ellipsoid even without a ground-truth dynamics model.

Original: In the ADMETS domain, we achieve can increase active learning safety  to 97% that seizures will not occur while only losing 15% information gain compared to when no safety constraints are imposed.
Revised: In the ADMETS domain, we ﬁnd that our algorithm achieves a 2.3% higher guarantee of safety compared to Maximizing Diversity. Our baseline Uncertainty achieves an equivalent safety guarantee to our algorithm, yet our algorithm achieves a 98% greater information gain comparatively, placing it at the Pareto front in terms of safety and information gain.

Original: We outperform LAL, Bao, and MetaBo in information gain and computation time under our safety constraints even those these algorithms are unconstrained.
Revised: We outperform LAL, BaO, and MetaBo in information gain and computation time under our safety constraints even though these algorithms are unconstrained.

Original: [Figure] in batches
Revised: We show in Fig 6 the results of our algorithm in both domains when maximizing only safety and removing the active learning component from our objective.

Original: , enabling the to deviate from temporarily
Revised: Without the meta-learned active learning function, our algorithm does not explore as efﬁciently and therefore achieves a lower model accuracy at each time step, demonstrating that our meta-learned acquisition function is an important component of our architecture.

Original: 5. D ISCUSSION
Revised: 5. D ISCUSSION

Original: Through our empirical investigation, we have demonstrated that our meta-learned acquisition function operating within a chance-constrained optimization framework outperforms prior work in active learning, meta-learning, and Bayesian optimization (Hastie et al., 2017; Schrum & Gombolay, 2020;
Revised: Through our empirical investigation, we have demonstrated that our meta-learned acquisition function operating within a chance-constrained optimization framework outperforms prior work in active learning, meta-learning, and Bayesian optimization (Hastie et al., 2017; Schrum & Gombolay, 2020;

Original: Ashmaig et al., 2018; Konyushkova et al., 2017; Wang et al., 2018b).
Revised: Ashmaig et al., 2018; Konyushkova et al., 2017; Wang et al., 2018a).

Original: Speciﬁcally, we are able to simultaneously achieve an improvement in information gain via increased sample efﬁciency and decreased computation time.
Revised: Speciﬁcally, we are able to simultaneously achieve an improvement in information gain via increased sample efﬁciency and decreased computation time.

Original: We achieve a 46% increase in information gain while still achieving a 20% speedup in computation compared to active learning baselines and 60x faster computation time compared to our meta learning baseline.
Revised: We achieve a 46% increase in information gain while still achieving a 20% speedup in computation compared to active learning baselines and 60x faster computation time compared to our meta learning baseline.

Original: Our novel, deep learning architecture, demonstrates a unique ability to leverage the power of feature learning across time-series data within a LSTM neural network and the utility of deep Q-learning, within mathematical optimization with chance constraints to explicitly tradeoff safety and active learning.
Revised: Our novel, deep learning architecture, demonstrates a unique ability to leverage the power of feature learning across time-series data within a LSTM neural network and the utility of deep Q-learning, within mathematical optimization with chance constraints to explicitly tradeoff safety and active learning.

Original: Our approach allows for minimum safety guarantees while also maximizing the information gained based on a learned representation of sample history.
Revised: Our approach allows for minimum safety guarantees while also maximizing the information gained based on a learned representation of sample history.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work lies at the crossroads of active learning, meta-learning and safe learning.
Revised: Our work lies at the crossroads of active learning, meta-learning and safe learning.

Original: We discuss the contributions of our work and why our approach is novel in light of prior work.
Revised: We discuss the contributions of our work and why our approach is novel in light of prior work.

Original: Active Learning - Active learning acquisition functions provide heuristics to selecting the candidate unlabeled training data sample that, if the label were known, would provide the most information to the model being learned (Burbidge et al., 2007;
Revised: Active Learning - Active learning acquisition functions provide heuristics to selecting the candidate unlabeled training data sample that, if the label were known, would provide the most information to the model being learned (Burbidge et al., 2007;

Original: Hasenjager & Ritter, 1998;
Revised: Hasenjager & Ritter, 1998;

Original: Cai et al., 2017; Hastie et al., 2017).
Revised: Cai et al., 2017; Hastie et al., 2017).

Original: Common approaches include Gaussian Process-based methods Snoek et al. 
Revised: For example, in Hastie et al. (2017) the action is selected that the learner is least certain about. In work by Ashmaig et al. (2018), the authors utilize the acquisition function Expected

Revised: Improvement (EI) to balance exploration versus exploitation to determine the optimal stimulation parameters in ADMETS.

Original: Prior literature has also investigated on-the-ﬂy active learning and meta-active learning (Bachman et al., 2016; Konyushkova et al., 2017). 
Revised: Prior literature has also investigated on-the-ﬂy active learning and meta-active learning (Bachman et al., 2016; Konyushkova et al., 2017). Konyushkova et al.

Original: Work by Geifman & El-Yaniv (2018) attempts to actively learn the neural network architecture that is most appropriate for  a given task, e.g. active learning.
Revised: (2017) describes the algorithm Learning Active Learning (LAL). The authors present a meta-learning method for learning an acquisition function in which a regressor is trained to predict the reduction in model error of candidate samples via hand engineered features.

Original: Kirsch et al.
Revised: Wang et al.

Original: (2019) consider a Bayesian batch active learning and scores samples based  on mutual information.
Revised: (2018a) alternatively considers a Gaussian Process based method to meta-train an acquisition function on a distribution of tasks. They show that their method is capable of extracting structural properties of the objective function for improved data-efﬁciency. Work by Geifman & El-Yaniv (2018) attempts to actively learn the neural network architecture that is most appropriate for a given task, e.g. active learning.

Original: Pang et al.
Revised: Pang et al.

Original: (2018)  proposed a method to learn an acquisition function that generalizes to a variety of classiﬁcation tasks.
Revised: (2018) additionally proposed a method to learn an acquisition function that generalizes to a variety of classiﬁcation tasks.

Original: Yet, this work has only been demonstrated for classiﬁcation.
Revised: Yet, this work has only been demonstrated for classiﬁcation.

Original: Meta-Learning for Dynamics - Prior work has attempted to address the problem of learning altered dynamics via a meta-learning (Clavera et al., 2009).
Revised: Meta-Learning for Dynamics - Prior work has attempted to address the problem of learning altered dynamics via a meta-learning (Clavera et al., 2009).

Original: Belkhale et al.
Revised: Belkhale et al.

Original: (2020) investigate a meta-learning approach to learn the altered dynamics of an aircraft carrying a payload; the authors train a neural network on prior data to predict environmental and task factors to inform how to adapt to new payloads.
Revised: (2020) investigate a meta-learning approach to learn the altered dynamics of an aircraft carrying a payload; the authors train a neural network on prior data to predict environmental and task factors to inform how to adapt to new payloads.

Original: Finn et al.
Revised: Finn et al.

Original: (2018) present a meta-learning approach to quickly learning a control policy.
Revised: (2018) present a meta-learning approach to quickly learning a control policy.

Original: In this approach, a distribution over prior model parameters that are most conducive to learningof the new dynamics is meta-learned ofﬂine.
Revised: In this approach, a distribution over prior model parameters that are most conducive to learning the new dynamics is meta-learned ofﬂine.

Original: While this approach provide fast policies for learning new dynamics, it does not explicitly reason about sample efﬁciency or safety.
Revised: While this approach provides fast policies for learning new dynamics, it does not explicitly reason about sample efﬁciency or safety.

Original: Safe Learning - Prior work has investigated safe learning in the context of safe Bayesian optimization and safe reinforcement learning.
Revised: Safe Learning - Prior work has investigated safe learning in the context of safe Bayesian optimization and safe reinforcement learning.

Original: For example, Sui et al.
Revised: For example, Sui et al.

Original: (2015) develop the algorithm SafeOpt which balances exploration and exploitation to learn an unknown function; however, this approach has signiﬁcant limiting assumptions about the underlying nature of the task.
Revised: (2015) develop the algorithm SafeOpt which balances exploration and exploitation to learn an unknown function; however, this approach has signiﬁcant limiting assumptions about the underlying nature of the task.

Original: Turchetta et al.
Revised: Turchetta et al.

Original: (2016) address the problem of safely exploring an MDP by deﬁning an a priori unknown safety constraint updated during exploration, and Zimmer et al.
Revised: (2016) address the problem of safely exploring an MDP by deﬁning an a priori unknown safety constraint updated during exploration, and Zimmer et al.

Original: (2018) utilizes a Gaussian process for safely learning time series data.
Revised: (2018) utilizes a Gaussian process for safely learning time series data.

Original: However, these approaches do not incorporate knowledge from prior data to increase sample efﬁciency, limiting their ability to choose the optimal action.
Revised: However, these approaches do not incorporate knowledge from prior data to increase sample efﬁciency, limiting their ability to choose the optimal action.

Revised: Schrum & Gombolay (2020) attempt to overcome this problem by employing a novel acquisition function, Maximizing Diversity, which is utilized to quickly learn altered system dynamics in a chance constrained framework.

Revised: Yet, the hand engineered acquisition function limits the capabilities of this approach.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we demonstrate a state of the art meta-learning approach to active learning for control.
Revised: In this paper we demonstrate a state of the art meta-learning approach to active learning for control.

Original: By encoding the context of the dynamics via an LSTM and learning a Q-function – which we encode into a mixed-integer optimization framework – that predicts the expected information gain of taking a given action, our framework is able to efﬁciently and safely learn the nature of the altered dynamics.
Revised: By encoding the context of the dynamics via an LSTM and learning a Q-function – which we encode into a mixed-integer optimization framework – that predicts the expected information gain of taking a given action, our framework is able to efﬁciently and safely learn the nature of the altered dynamics.

Original: We compare our approach to baseline acquisition functions and demonstrate that ours is superior in both computation time and information gain, achieving a 46% increase in information gain while still achieving a 20% speedup in computation time compared to state-of-the-art acquisition functions and more than a 58% higher information compared to Bayesian approaches.
Revised: We compare our approach to baseline acquisition functions and demonstrate that ours is superior in both computation time and information gain, achieving a 46% increase in information gain while still achieving a 20% speedup in computation time compared to state-of-the-art acquisition functions and more than a 58% higher information compared to Bayesian approaches.

Original: A A PPENDIX A. D ETAILS ON D OMAINS
Revised: A A PPENDIX A. D ETAILS ON D OMAINS

Original: Here we describe in more detail the input and output spaces of the ADMETS and Aircraft domains as well as other details relevant to implementation.
Revised: Here we describe in more detail the input and output spaces of the ADMETS and high-dimensional domains as well as other details relevant to implementation.

Original: A.1. A IRCRAFT
Revised: A.1. A IRCRAFT

Original: Our simulation consists of ten states, i.e., forward velocity ( u ), vertical velocity ( w ), pitch rate ( q ), pitch angle ( θ ), sideslip angle ( β ), roll rate ( p ), yaw rate ( r ), roll angle ( ψ ), yaw angle ( φ ), and altitude ( Z ).
Revised: Our simulation consists of ten states, i.e., forward velocity ( u ), vertical velocity ( w ), pitch rate ( q ), pitch angle ( θ ), sideslip angle ( β ), roll rate ( p ), yaw rate ( r ), roll angle ( ψ ), yaw angle ( φ ), and altitude ( Z ).

Original: The control inputs are elevator ( ∆ e ), thrust ( ∆ ( t ) ), aileron ( ∆ a ), and rudder ( ∆ r ).
Revised: The control inputs are elevator ( ∆ e ), thrust ( ∆ ( t ) ), aileron ( ∆ a ), and rudder ( ∆ r ).

Original: Our sampling rate is 20 Hz.
Revised: Our sampling rate is 20 Hz.

Original: A.1. ADMETS
Revised: A.1. ADMETS

Original: The input parameter space is one-dimensional and consists of the voltage amplitude ( v a ).
Revised: The input parameter space is one-dimensional and consists of the voltage amplitude ( v a ).

Original: The output space is a score quantifying memory, referred to as the discrimination score ( d s ).
Revised: The output space is a score quantifying memory, referred to as the discrimination score ( d s ).

Original: The objective of this domain is to maximize discrimination area, which is deﬁned as d a :=
Revised: The objective of this domain is to maximize discrimination area, which is deﬁned as d a :=

Original: d s ∗ v a .
Revised: d s ∗ v a .

Original: More detail on this domain and the data on which the simulation is based can be found at Ashmaig et al.
Revised: More detail on this domain and the data on which the simulation is based can be found at Ashmaig et al.

Original: A. D EFINITION OF I NFORMATION G AIN
Revised: A. D EFINITION OF I NFORMATION G AIN

Original: We deﬁne information gain, I , at time step t as the percent decrease in the error of the objective as described in Eq.
Revised: We deﬁne information gain, I , at time step t as the percent decrease in the error of the objective as described in Eq.

Original: e ( t ) is the error of the objective at time step t .
Revised: e ( t ) is the error of the objective at time step t .

Original: In the aircraft domain, e ( t ) is deﬁned as thethe mean squared error of the model ( e ( t ) = 1 N (cid:80)
Revised: In the high-dimensional domain, e ( t ) is deﬁned as the mean squared error of the model, e ( t ) = 1 N (cid:80)

Original: Ni − 1 ( x ( i ) − ˆ x
Revised: Ni − 1 ( x ( i ) − ˆ x

Original: ( i ) )2 ) and in the
Revised: ( i ) )

Revised: x is the ground truth state and ˆ x is the state predicted by the model.

Original:  ADMETS domain e ( t ) is deﬁned as the L 1 norm of the predicted optimal parameter and the ground truth optimal parameter ( e ( t ) =
Revised: In the ADMETS domain e ( t ) is deﬁned as the Lnorm of the predicted optimal parameter and the ground truth optimal parameter ( e ( t ) =

Original: || d a − ˆ d a || 1 ).
Revised: || d a − ˆ d a || 1 ).

Original: [Equation] [Equation] B
Revised: During ofﬂine training, the ground truth can be obtained from the known model.

Original: [Equation]
Revised: [Equation]

Original:  A CQUISITION F UNCTIONS A.3. M AXIMIZING D IVERSITY
Revised: A. B ASELINE A CQUISITION F UNCTIONS A.3. M AXIMIZING D IVERSITY

Original: We compare our method to the acquisition function, maximizing diversity, presented by (Schrum &
Revised: We compare our method to the acquisition function, maximizing diversity, presented by (Schrum &

Original: Gombolay, 2020).
Revised: Gombolay, 2020).

Original: Here, u ( i ) and x ( i ) are states and actions that the robot has previously experienced, and f is the current dynamics model.
Revised: Here, u ( i ) and x ( i ) are states and actions that the robot has previously experienced, and f is the current dynamics model.

Original: [Equation]
Revised: [Equation]

Original: A.3. M AXIMIZING U NCERTAINTY
Revised: A.3. M AXIMIZING U NCERTAINTY

Original: This active learning metric described by (Hastie et al., 2017) quantiﬁes the uncertainty in the output of the model for each training example as described in Eq.
Revised: This active learning metric described by (Hastie et al., 2017) quantiﬁes the uncertainty in the output of the model for each training example as described in Eq.

Original: Here, ˆ T z ( u )) is the z th dynamics model and ¯ T is the average across models z .
Revised: Here, ˆ T z ( u )) is the z th dynamics model and ¯ T is the average across models z .

Original: [Equation]
Revised: [Equation]

Original: A. A DDITIONAL D ETAILS ON M IXED -I NTEGER L INEAR P ROGRAM F ORMULATION
Revised: A. A DDITIONAL D ETAILS ON M IXED -I NTEGER L INEAR P ROGRAM F ORMULATION

Original: In this section we provide additional details on the linearization of our probability constraints and objective function for integration into a linear programming formulation.
Revised: In this section we provide additional details on the linearization of our probability constraints and objective function for integration into a linear programming formulation.

Original: A.4. D YNAMICS M ODEL R EPRESENTATION
Revised: A.4. D YNAMICS M ODEL R EPRESENTATION

Original: In practice, in the aircraft domain we ﬁnd that ˆ T ψ can be represented as a single-layer perceptron (i.e. linear regression) which advantageously is computationally efﬁcient in domains that require fast computation times.
Revised: In practice, in the high-dimensional domain we ﬁnd that ˆ T ψ can be represented as a single-layer perceptron (i.e. linear regression) which advantageously is computationally efﬁcient in domains that require fast computation times.

Original: We adopt a multi-layer perceptron with ReLU activations in the ADMETS domain.
Revised: We adopt a multi-layer perceptron with ReLU activations in the ADMETS domain.

Original: Our inferred dynamics therefore evolve according Γ .
Revised: Our inferred dynamics therefore evolve according Γ .

Original: A describes the evolution of a state with no input, B the change in the state due to an action at time t, (cid:126)u ( t ) , and I is the identity matrix.[Equation] [Equation]
Revised: A describes the evolution of a state with no input, B the change in the state due to an action at time t, (cid:126)u ( t ) , and I is the identity matrix.

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Revised: AB

Revised: B

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: A.4.
Revised: AB B

Revised: [Equation]

Revised: [Equation]

Revised: B   (8)

Original:  L INEARIZATION OF P ROBABILITY C ONSTRAINTS
Revised: A.4. L INEARIZATION OF P ROBABILITY C ONSTRAINTS

Original: To include our safety constraints in a mixed-integer linear programming formulation, we remove the non-linearities via conservative assumptions and other techniques.
Revised: To include our safety constraints in a mixed-integer linear programming formulation, we remove the non-linearities via conservative assumptions and other techniques.

Original: Our safety constraints are deﬁned in Eq.
Revised: Our safety constraints are deﬁned in Eq.

Original: 9 and the dynamics evolve according to equations 6-8.
Revised: 9 and the dynamics evolve according to equations 6-8.

Original: d represents the d th row andd i and j represent the columns.
Revised: d represents the d th row and j represents the columns.

Original: [Equation]
Revised: [Equation]

Original: The following conservative assumption is made to linearize the sum of squares in Eq. 9.
Revised: The following conservative assumption is made to linearize the sum of squares in Eq. 9.

Original: [Equation]
Revised: [Equation]

Original: We utilize the binary decision variable, δ (cid:15) ∈ { 0 , 1 } as a probability selector variable to linearize the absolute value in Eq. 9.
Revised: We utilize the binary decision variable, δ (cid:15) ∈ { 0 , 1 } as a probability selector variable to linearize the absolute value in Eq. 9.

Original: M represents a large positive number and ¯Γ is the point estimate of the dynamics.
Revised: M represents a large positive number and ¯Γ is the point estimate of the dynamics.

Original: This linearization technique combined with the conservative assumption in Eq. 10 results in the following linear equations (Eq. 11-13) which can be integrated into a mixed-integer linear programming formulation.
Revised: This linearization technique combined with the conservative assumption in Eq.results in the following linear equations (Eq. 11-13) which can be integrated into a mixed-integer linear programming formulation.

Original: [Equation]
Revised: E is the set of “probability levels”, e.g., E = { 0 . 8 , ... } where min E deﬁnes the minimum enforced probability of safety.

Original: [Equation]
Revised: [Equation]

Original: A.4. V ARIANCE E STIMATION
Revised: A.4. V ARIANCE E STIMATION

Original: We compute uncertainty of our network via bootstrapping.
Revised: We compute uncertainty of our network via bootstrapping.

Original: We follow the method proposed in Efron (2020) and randomly redraw bootstrap training samples with replacement from our set of training data.
Revised: We follow the method proposed in Efron (2020) and randomly redraw bootstrap training samples with replacement from our set of training data.

Original: This technique has been veriﬁed by Franke & Nuemann (1998) to be an effective method for approximating the uncertainty of neural networks.
Revised: This technique has been veriﬁed by Franke & Nuemann (1998) to be an effective method for approximating the uncertainty of neural networks.

Original: The components of σ are calculated according to
Revised: The components of σ are calculated according to

Original: Eq.
Revised: Eq.

Original: Here ¯ x is the average of the bootstrapped networks and x ( b ) d,d (cid:48) a single bootstrapped network.
Revised: Here ¯ x is the average of the bootstrapped networks and x ( b ) d,j a single bootstrapped network.

Original: B is the number of bootstrapped networks.
Revised: B is the number of bootstrapped networks.

Original: [Equation]
Revised: [Equation]

Original: A.4. L INEARIZATION OF Q-F UNCTION
Revised: L INEARIZATION OF Q-F UNCTION

Original: Our Q-function includes a non-linear relu activation function which is linearized to be included in the mixed-integer linear programming formulation.
Revised: Our Q-function includes a non-linear relu activation function which is linearized to be included in the mixed-integer linear programming formulation.

Original: Equations 15-17 deﬁne the equations for a neural network with relu activation.
Revised: Equations 15-17 deﬁne the equations for a neural network with relu activation.

Original: ξ = (cid:104)(cid:2) U
Revised: ξ = (cid:104)(cid:2) U

Original: ( t : T ) (cid:3) (cid:124) , (cid:104) (cid:126)Z (cid:105) (cid:124) (cid:105) is the input to the Q-function and ω ( l ) j,i is the connection between neurons j and i between layers l and l + 1 .
Revised: ( t : T ) (cid:3) (cid:124) , (cid:104) (cid:126)Z (cid:105) (cid:124) (cid:105) is the input to the Q-function and ω ( l ) j,i is the connection between neurons j and i between layers l and l + 1 .

Original: [Equation]
Revised: [Equation]

Original: This formulation is linearized in Eq.
Revised: This formulation is linearized in Eq.

Original: k i ∈ { 0 , 1 } is a binary indicator variable and M represents a large positive number.
Revised: k i ∈ { 0 , 1 } is a binary indicator variable and M represents a large positive number.

Original: [Equation]
Revised: [Equation]

Original: A.4. L INEARIZATION OF “ RESETTING ” TERM
Revised: A.4. L INEARIZATION OF “ RESETTING ” TERM

Original: In practice, we ﬁnd that taking a ﬁnal, “resetting” action at time t + T by adding z 3 = (cid:13)(cid:13) (cid:126)x ( T ) − (cid:126)x r (cid:13)(cid:13) to minimize the distance between the aircraft’s state and (cid:126)x r helps to ensure the aircraft does not loiter along the boundary of safe operation until where a random perturbation could result in failure the aircraft.
Revised: In practice, we ﬁnd that taking a ﬁnal, “resetting” action at time t + T by adding z 3 = (cid:13)(cid:13) (cid:126)x ( T ) − (cid:126)x r (cid:13)(cid:13) to minimize the distance between the aircraft’s state and (cid:126)x r helps to ensure the aircraft does not loiter along the boundary of safe operation until where a random perturbation could result in failure the aircraft.

Original: We linearize the resetting term in our objective function, i.e. the difference between our designated safe state, (cid:126)x r and the ﬁnal state x T by maximizing − ( z + + z − ) subject to the constraint in Eq.
Revised: We linearize the resetting term in our objective function, i.e. the difference between our designated safe state, (cid:126)x r and the ﬁnal state x T by maximizing − ( z + + z − ) subject to the constraint in Eq.

Original: z + and z − are both positive continuous variables.
Revised: z + and z − are both positive continuous variables.

Original: [Equation]
Revised: [Equation]

Original: A.4. L INEARIZED O BJECTIVE
Revised: A.4. L INEARIZED O BJECTIVE

Original: The resultant linearized objective is deﬁned in Eq.
Revised: The resultant linearized objective is deﬁned in Eq.

Original: In the ADMETS domain λ 3 is set to 0.
Revised: In the ADMETS domain λ 3 is set to 0.

Original: [Equation]
Revised: [Equation]

Original: B S AFETY B.0. RNS D OMAIN
Revised: B S AFETY B.0. ADMETS D OMAIN

Original: The baselines used in the RNS domain did not have built in safety guarantees.
Revised: The baselines used in the ADMETS domain did not have built in safety guarantees.

Original: Therefore, when comparing against these baselines, we removed the safety constraints in our algorithm to make the comparison fair.
Revised: Therefore, when comparing against these baselines, we removed the safety constraints in our algorithm to make the comparison fair.

Original: B.0. A IRCRAFT D OMAIN
Revised: B.0. H IGH -D IMENSIONAL D OMAIN

Original: Safety constraints are necessary in the aircraft domain because without these constraints, the aircraft would quickly go into an unrecoverable conﬁguration.
Revised: Safety constraints are necessary in the high-dimensional domain because without these constraints, the aircraft would quickly go into an unrecoverable conﬁguration.

Original: The epistemic uncertainty Hastie et al.
Revised: The epistemic uncertainty Hastie et al.

Original: and
Revised: and

Original: diversity Schrum & Gombolay (2020)
Revised: diversity Schrum & Gombolay (2020)

Original: baselines can be integrated into our safety framework by linearizing the acquisition functions.
Revised: baselines can be integrated into our safety framework by linearizing the acquisition functions.

Original: Therefore, in our comparison of our algorithm versus the baselines, we enforced the safety constraints.
Revised: Therefore, in our comparison of our algorithm versus the baselines, we enforced the safety constraints.

Original: LAL Konyushkova et al.
Revised: LAL Konyushkova et al.

Original: (2017), however, is not an inherently safe method of active learning.
Revised: (2017), however, is not an inherently safe method of active learning.

Original: To fairly compare this baseline to our method in the aircraft domain, we simulate the possible actions that can be selected by LAL and discard those that are not safe (i.e., the actions that take the aircraft out of the cylinder of safety).
Revised: To fairly compare this baseline to our method in the high-dimensional domain, we simulate the possible actions that can be selected by LAL and discard those that are not safe (i.e., the actions that take the aircraft out of the cylinder of safety).

Original: Therefore, LAL can only select an action considered safe by our deﬁnition of safety.
Revised: Therefore, LAL can only select an action considered safe by our deﬁnition of safety.

Original: B. S ENSITIVITY A NALYSIS
Revised: B. S ENSITIVITY A NALYSIS

Original: To robustly evaluate our method compared to the baselines, we vary the hyperparameters of the approach by (Schrum & Gombolay, 2020) for maximizing diversity and our meta-learned function to show that our function is robust and is still superior despite changes in hyperparameters.
Revised: To robustly evaluate our method compared to the baselines, we vary the hyperparameters of the approach by (Schrum & Gombolay, 2020) for maximizing diversity and our meta-learned function to show that our function is robust and is still superior despite changes in hyperparameters.

Original: The results of this hyperparameter sweep are shown in Table 1.
Revised: The results of this hyperparameter sweep are shown in Table 1.

Original: The hyper-parameter we vary for maximizing diversity is the number of previous training samples that we compare to.
Revised: The hyper-parameter we vary for maximizing diversity is the number of previous training samples that we compare to.

Original: The information gain increases as the number of samples increases up to a point at which the selected sample tends to converge to the mean of the previously collected samples, causing the information gain to fall.
Revised: The information gain increases as the number of samples increases up to a point at which the selected sample tends to converge to the mean of the previously collected samples, causing the information gain to fall.

Original: We vary the number of hidden neurons in our Q-function as the hyper-parameter of interest as it governs the trade-off between computational speed and function approximation power.
Revised: We vary the number of hidden neurons in our Q-function as the hyper-parameter of interest as it governs the trade-off between computational speed and function approximation power.

Original: In our approach, the addition of a hidden neuron adds an additional integer variable, resulting in an increase in computation time as demonstrated in Table 1.
Revised: In our approach, the addition of a hidden neuron adds an additional integer variable, resulting in an increase in computation time as demonstrated in Table 1.

Original: However, an addition of 25 neurons only increase the computation time by 3.7% and provides a 50% increase in information gain.
Revised: However, an addition of 25 neurons only increase the computation time by 3.7% and provides a 50% increase in information gain.

Original: In comparison, Schrum & Gombolay (2020)’s approach results in increase in information gain 4% while trading off a 6% loss in computation time.
Revised: In comparison, Schrum & Gombolay (2020)’s approach results in increase in information gain 4% while trading off a 6% loss in computation time.

Original: Likewise, increasing the number of bootstrapped models in Hastie et al.
Revised: Likewise, increasing the number of bootstrapped models in Hastie et al.

Original: (2017)’s approach results in a 35% increase in information gain and a 37.5% increase in computation time.
Revised: (2017)’s approach results in a 35% increase in information gain and a 37.5% increase in computation time.

Original: Therefore, in our approach we are able to gain more information without large increases in computation time.
Revised: Therefore, in our approach we are able to gain more information without large increases in computation time.

Original: [Table]
Revised: [Table]

Original: and (Hastie et al., 2017).

Original: We vary the number of previous samples in the diversity maximization problem from one to ﬁve and the number of bootstrapped models from two to four in maximizing uncertainty heuristic.

Original: We vary the number of hidden neurons in our meta-learned Q-function.

Original: We bold the setting of our algorithm that outperforms our baselines across all hyperparameter settings tested.

Original: B. A DDITIONAL R ESULTS
Revised: B. A DDITIONAL R ESULTS

Original: We present results for each damage condition in the aircraft domain comparing our approach to the baseline acquisition functions diversity Schrum & Gombolay (2020) and epistemic uncertainty Hastie et al.
Revised: We present results for each damage condition in the high-dimensional domain comparing our approach to the baseline acquisition functions diversity Schrum & Gombolay (2020) and epistemic uncertainty Hastie et al.

Original: Our approach outperforms the baselines for each damage condition.
Revised: Our approach outperforms the baselines for each damage condition.

Original: [Table]
Revised: [Table]

Original: B. H YPERPARAMETERS
Revised: B. H YPERPARAMETERS

Original: The hyperparameters employed for each domain are listed in the Table 3.
Revised: The hyperparameters employed for each domain are listed in the Table 3.

Original: We show the learning rates for each domain which were determined via experimentation to be effective values.
Revised: We show the learning rates for each domain which were determined via experimentation to be effective values.

Original: The size of the
Revised: The size of the

Original: LSTM hidden layer and each of the two layers of the Q-function is also presented.
Revised: LSTM hidden layer and each of the two layers of the Q-function is also presented.

Original: τ is the soft update coefﬁcient for updating the target network.
Revised: τ is the soft update coefﬁcient for updating the target network.

Original: [Table]
Revised: [Table]