Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
Abstract: Learning complex robot behaviors through interaction requires structured exploration.
Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective.
This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards.
We combine latent world models with value function estimation to predict inﬁnite-horizon returns and recover associated uncertainty via ensembling.
The policy is then trained on an upper conﬁdence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance.
We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efﬁciency in comparison to state-of-the-art and other exploration objectives.
In sparse and hard to explore environments we achieve an average improvement of over 30%.
Introduction
[Figure]
The ability to learn complex behaviors through interaction will enable the autonomous deployment of various robotic systems in the real world.
Reinforcement learning (RL) provides a key framework for realizing these capabilities, but efﬁciency of the learning process remains a prevalent concern.
Real-life applications yield complex planning problems due to high-dimensional environment states, which are further exacerbated by the agent’s continuous action space.
For RL to enable real-world autonomy, it therefore becomes crucial to determine efﬁcient representations of the underlying planning problem, while formulating interaction strategies capable of exploring these representations efﬁciently.
In traditional controls, planning problems are commonly formulated based on the underlying statespace representation.
This may inhibit efﬁcient learning when the environment states are highdimensional or their dynamics are susceptible to non-smooth events such as singularities and discontinuities [1, 2, 3].
It may then be desirable for the agent to abstract a latent representation that facilitates efﬁcient learning [4, 5, 6].
The latent representation may then be leveraged either in a model-free or model-based setting.
Model-free techniques estimate state-values directly from observed data to distill a policy mapping.
Model-based techniques learn an explicit representation of the environment that is leveraged in generating ﬁctitious interactions and enable policy learning in imagination [7].
While the former reduces potential sources of bias, the latter offers a structured representation encoding deeper insights into underlying environment behavior.
The agent should leverage the chosen representation to efﬁciently identify and explore informative interactions.
For example, an engineer would ideally only provide sparse high-level feedback to a robot and the system should learn without getting distracted by environment behavior irrelevant to task completion.
This requires models that can disentangle behavioral surprise from expected performance and motivate the agent to investigate the latter.
Figure 1 provides a one-dimensional example of a value mapping.
The true function and its samples are visualized in red with the true maximum denoted by the green dot.
Relying only on the predicted mean can bias policy learning towards local optima (orange dot; Sutton and Barto [8]), while added stochasticity can waste samples on uninformative interactions.
Auxiliary information-gain objectives integrate predicted uncertainty, however, uncertain environment behavior does not equate to potential for improvement (pink dot).
It is desirable to focus exploration on interactions that harbor potential for improving overall performance.
Combining mean performance estimates with their uncertainty into an upper conﬁdence bound (UCB) objective provides a concise method for this (blue dot; Auer et al.
[9], Krause and Ong [10]).
The uncertainty can be explicitly represented by an ensemble of hypothesis on environment behavior [11, 12].
Figure 1 demonstrates this selective uncertainty reduction by providing forward predictions of a model ensemble on two motion patterns for a Walker.
The expected high-reward walking behavior has been sufﬁciently explored and the models strongly agree, while little effort has been extended to reduce uncertainty and learn the speciﬁc details of the expected low-reward falling behavior.
This paper demonstrates that exploring interactions through imagined positive futures can yield information-dense sampling and data-efﬁcient learning.
We present latent optimistic value exploration (LOVE), an algorithm that leverages optimism in the face of uncertain long-term rewards in guiding exploration.
Potential futures are imagined by an ensemble of latent variable models and their predicted inﬁnite-horizon performance is obtained in combination with associated value function estimates.
Training on a UCB objective over imagined futures yields a policy that behaves inherently optimistic and focuses the robot learner on interactions with the potential to improve performance under its current world model.
This provides a concise, differentiable framework for driving deep exploration while not relying on stochasticity.
LOVE therefore provides an agent with
• an uncertainty-aware model for diverse hypotheses on trajectory evolution and performance, • intrinsic exploration without reward feedback by active querying of uncertain long-term returns, • targeted exploration guided by improvement potential, ignoring uncertainty tangential to the task
We present empirical results on challenging visual robot control tasks that highlight the necessity for deep exploration in scenarios with sparse rewards, where we achieve on average more than 30% improvement over state-of-the-art and an exploration baseline.
We further demonstrate 15% average improvement on benchmarking environments from the DeepMind Control Suite [13].
We compare to the recent agents Dreamer [7], a curiosity-based agent inspired by Sekar et al.
[14], and DrQ [15].
Related work
Problem representation Model-free approaches learn a policy by directly estimating performance from interaction data.
While their asymptotic performance previously came at the cost of sample complexity [16, 17, 18], recent advances in representation learning through contrastive methods and data augmentation have improved their efﬁciency [19, 20, 15].
However, their implicit representation of the world can make generalization of learned behaviors under changing task speciﬁcations difﬁcult.
Model-based techniques leverage a structured representation of their environment that enables them to imagine potential interactions.
The nature of the problem hereby dictates model complexity, ranging from linear [21, 22], over Gaussian process models [23, 24] to deep neural networks [25, 26].
In high-dimensional environments, latent variable models can provide concise representations that improve efﬁciency of the learning process [27, 4, 6, 7].
Planning interactions Model-based approaches leverage their representation of the world in predicting the performance of action sequences.
The agent may then either solve a model predictive control-style optimization [28, 25, 29] or train a policy in simulation [30, 26].
The resulting ﬁnitehorizon formulations can be extended by value function estimates to approximate an inﬁnite-horizon planning problem [31, 7, 32].
When considering learned models, ensembling the model predictions may further be leveraged in debiasing the actor during training [30, 25, 26, 32]
Directed exploration Directed exploration can improve over random exploration by focusing on information-dense interactions [36].
These methods are commonly driven by uncertainty estimates.
Information gain techniques deﬁne an auxiliary objective that encourages exploration of unexpected environment behavior or model disagreement and have been applied in discrete [37, 38, 39] and continuous actions spaces [40, 41, 42].
Improving knowledge of the dynamics yields general purpose models, while the agent may explore uncertain interactions tangential to a speciﬁc task objective.
Alternatively, exploration can be driven by uncertainty over performance as encoded by value functions [11, 43, 44, 45, 46], multi-step imagined returns [47, 48] or their combination [31, 1, 32].
Model-ensemble agents Related work on ensemble agents has demonstrated impressive results.
We note key differences to our approach.
ME-TRPO [30] leverages a dynamics ensemble to debias policy optimization on ﬁnite-horizon returns under a known reward function and random exploration.
MAX [49] and Amos et al.
[50] explore via ﬁnite-horizon uncertainty in a dynamics ensemble.
Plan2Explore [14] combines this idea with Dreamer [7], learning a model that enables adaptation to multiple downstream tasks.
RP1 [51] explores in reward space via ﬁnite-horizon returns, but assumes access to the nominal reward function and full proprioceptive feedback.
Seyde et al.
[32] leverage full proprioception and embed optimism into the value function, which prohibits adjustment of the exploration trade-off during learning and limits transferability.
Concurrent work by Rafailov et al.
[52] penalizes uncertainty over latent states for learning robust visual control policies via ofﬂine RL.
Exploring uncertain dynamics samples interactions orthogonal to task completion and ﬁnite-horizon objectives limit exploration locally, while full-observability and access to the reward function are strong assumptions.
We learn latent dynamics, reward and value functions under partial observability to explore uncertain inﬁnite-horizon returns.
This enables backpropagation through imagined trajectories to recover analytic policy gradients, as well as guided deep exploration based on expected potential for long-term improvement.
Preliminaries 3.1 Problem formulation
We formulate the underlying optimization problem as a partially observable Markov decision process (POMDP) deﬁned by the tuple {X , A , T, R, Ω , O , γ } , where X , A , O denote the state, action and observation space, respectively, T :
X × A → X signiﬁes the transition mapping, R :
X ×
A → R the reward mapping, Ω: X → O the observation mapping, and γ ∈ [0 , 1) is the discount factor.
We deﬁne x t and a t to be the state and action at time t , respectively, and use the notation r t =
R ( x t , a t ) .
Let π φ ( a t | o t ) denote a policy parameterized by φ and deﬁne the discounted inﬁnite horizon return
G t = (cid:80) ∞ τ
= t γ τ − t R ( x τ , a τ ) , where
x t +1 ∼ T ( x t +1 | x t , a t ) and a t ∼ π φ ( a t | o t ) .
The goal is then to learn the optimal policy maximizing G t under unknown nominal dynamics and reward mappings.
3.2 Planning from pixels
We build on the world model introduced in Hafner et al.
[29] and reﬁned in Hafner et al.
Highdimensional image observations are ﬁrst embedded into a low-dimensional latent space using a neural network encoder.
A recurrent state space model (RSSM) then provides probabilistic transitions and deﬁnes the model state s .
Together, the encoder and RSSM deﬁne the representation model.
The agent therefore abstracts observation o t of environment state x t into model state s t , which is leveraged for planning.
Consistency of the learned representations is enforced by minimizing the reconstruction error of a decoder network in the observation model and the ability to predict rewards of the reward model.
For details, we refer the reader to Hafner et al.
[7], and provide their deﬁnitions
Representation model:
[Equation]
Transition
[Equation] [Equation] [Equation] Reward
model:
[Equation]
where p and q denote distributions in latent space, with θ as their joint parameterization.
The action model π φ ( a t | s t ) is then trained to optimize the predicted return of imagined world model rollouts.
The world model is only rolled-out over a ﬁnite horizon H , but complemented by predictions from the value model v ψ ( s t ) at the terminal state s t + H to approximate the inﬁnite horizon return.
The policy and value function are trained jointly using policy iteration on the objective functions
[Equation]
respectively.
Here, V λ ( s τ ) represents an exponentially recency-weighted average of the k -step value estimates V kN ( s τ ) along the trajectory to stabilize the learning [8] as further speciﬁed in Appendix K.
Uncertainty-guided latent exploration
The world model introduced in Section 3.2 can be leveraged in generating ﬁctitious interactions for the policy to train on.
However, the learned model will exhibit bias in uncertain regions where insufﬁcient samples are available.
Training on imagined model rollouts then propagates simulation bias into the policy.
Here, we address model bias by ensembling our belief over environment behavior.
We can leverage the underlying epistemic uncertainty in formulating a UCB objective for policy learning that focuses exploration on regions with high predicted potential for improvement.
4.1 Model learning with uncertainty estimation
The model parameters are only weakly constrained in regions where interaction data is scarce and random inﬂuences strongly impact prediction performance.
In order to prevent the agent from learning to exploit these model mismatches, we consider predictions from an ensemble.
Individual predictions will align in regions of high data support and diverge in regions of low support.
The ensemble mean then serves as a debiased estimator of environment behavior, while the epistemic uncertainty is approximated via model disagreement [12].
We consider an ensemble of M latent-space particles.
Each particle is represented by a unique pairing of a transition, reward and value model to yield,
Particles: { ( q θ i ( s t | s t − 1 , a t − 1 ) , q θ i ( r t | s
[Equation]
The encoder and decoder remain shared to ensure that all particles leverage the same latent space, while the internal transition dynamics retain the ability of expressing distinct hypothesis over environment behavior.
For particle i , we deﬁne the predicted inﬁnite-horizon trajectory return as
[Equation]
where V λ,i ( s τ ) is computed via Eq.
(8) with the particle’s individual transition, reward and value models.
Distinctness of the particles is encouraged by varying the initial network weights between ensemble members and shufﬂing the batch order during training.
Predicted trajectory returns with corresponding uncertainty estimates are then obtained by considering the ensemble mean and variance,
[Equation]
4.2 Policy learning with directed exploration
The policy learning objective in Eq.
(2) could be replaced by the ensemble mean in Eq.
This would reduce model bias in the policy, but require an auxiliary objective to ensure sufﬁcient exploration.
We consider exploration to be desirable when it reduces uncertainty over realizable task performance.
The trajectory return variance in Eq.
(6) encodes uncertainty over long-term performance of actions.
In combination with the expected mean, we recover estimated performance bounds.
During data acquisition, we explicitly leverage the epistemic uncertainty in identifying interactions with potential for improvement and deﬁne the acquisition policy objective via an upper conﬁdence bound (UCB) as
[Equation]
where the scalar variable β quantiﬁes the exploration-exploitation trade-off.
For β < 0 we recover a safe-interaction objective, while β > 0 translates to an inherent optimism that uncertainty harbors potential for improvement.
Here, we learn an optimistic policy π φ aq that is intrinsically capable of deep exploration and focuses interactions on regions with high information-density.
Furthermore, in the absence of dense reward signals, the acquisition policy can leverage prediction uncertainty in driving exploration.
This behavior is not limited to the preview window, as the value function ensemble projects long-term uncertainty into the ﬁnite-horizon model rollouts.
While training in imagination, we leverage the optimistic policy to update our belief in regions that the acquisition policy intends to visit.
In parallel, we train an evaluation policy π φ ev that aims to select the optimal actions under our current belief.
The evaluation policy optimizes the expected mean return ( β = 0 ).
4.3 Latent optimistic value exploration (LOVE)
In the following, we provide a high-level description of the proposed algorithm, LOVE, together with implementation details of the overall training process and the associated pseudo-code in Algorithm 1.
Initialize: random parameters { θ i , ψ i , φ aq , φ ev } for episode in episodes do for timestep t = 1 to T do a
[Table]
[Table]
[Table]
[Table]
[Table]
) Observe environment transition and add to D for trainstep s = 1 to S do for particle i = 1 to M do Sample sequence batch
[Table]
, i )
Representation learning of θ i on { ( [Table] Regression
[Table] [Table] Sample sequences { ( [Table]
[Table] [Table] [Table]
for particle
[Table]
to M do
[Table]
Summary LOVE leverages an ensemble of latent variable models in combination with value function estimates to predict inﬁnitehorizon trajectory performance and associated uncertainty.
The acquisition policy is trained on a UCB objective to imagine positive futures and focus exploration on interactions with high predicted potential for longterm improvement.
The ensemble members are constrained to operate over the same latent space to encourage learning of abstract representations conducive to the objective, while ensuring prediction consistency.
Implementation The algorithm proceeds in two alternating phases.
In the online phase, the agent leverages its acquisition policy to explore interactions optimistically and resulting transitions are appended to memory D .
In the ofﬂine phase, the agent ﬁrst updates its belief about environment behavior and then adjusts its policy accordingly.
The representation learning step extends the procedure introduced in Hafner et al.
[7] to ensemble learning and groups each model with a unique value function estimator into a particle.
The batch order is varied between particles during training to ensure variability in the gradient updates and to prevent collapse into a single mode.
The policy learning step combines particle predictions to generate value targets in Eq.
(4) by simulating ensemble rollouts from the same initial conditions.
The trajectory return statistics of Eq.
(6) and Eq.
(5) are combined into the UCB objective of Eq.
(7), which the acquisition policy optimizes, while the evaluation policy optimizes the mean return in Eq.
Experiments
We evaluate LOVE on continuous visual control tasks.
First, we showcase intrinsic exploration in the absence of reward signals.
Then, we illustrate interpretable uncertainty estimation of long-term returns with sparse reward signals.
Finally, we benchmark performance on tasks from the DeepMind
Control Suite [13].
We use a single set of hyperparameters throughout, as detailed in
Appendix A. 5.1 Intrinsic exploration without reward feedback
In the absence of reward feedback an agent relies on intrinsic motivation to explore.
LOVE enables intrinsic exploration by considering uncertainty over inﬁnite-horizon returns.
Without informative mean performance estimates, optimism acts as a proxy to guide exploration (see Eq. 7).
We consider a reward-free bug trap environment.
The agent starts inside the trap and can exit through a narrow passage.
There is no reward feedback to guide planning and escapes need to arise implicitly through exploration.
The agent is represented as a pointmass under continuous actuation-limited acceleration control and observes a top-down view of the environment.
The ratio of agent diameter to passage width is 0 .
9 and collisions are modelled as inelastic with the coefﬁcient of restitution set to 0 .
The relative size constrains the set of policies allowing for entry into the passage, while inelastic
We compare performance of LOVE to both Dreamer and LVE, a variation that ablates on LOVE’s optimism ( β = 0 ).
Two variations of the environment are run on 3 random seeds for 100 k timesteps.
Exemplary occupancy maps are provided in Figure 2 (left).
We note that LOVE escapes the bug trap in both scenarios to explore a much larger fraction of the state space than either LVE or Dreamer.
This can be attributed to LOVE’s ability to envision uncertain rewards beyond the preview horizon, which optimistically drives deep exploration in the absence of reward signals (row 1).
Removing long-term optimism by only guiding interactions through mean performance estimates can lead to prediction collapse and the agent assuming the absence of reward without explicitly querying the environment for conﬁrmation (row 2).
We observe similar behavior for the Dreamer agent, which employs a single latent model and leverages random noise for exploration (row 3).
The total escape count conﬁrms this insight (column 3), where the remaining occupancy maps are provided in Appendix E.
We further highlight the differences in intrinsic exploration between LOVE and LVE on a reward-free maze exploration task.
The environment follows the same physics as the bug trap and the ratio of agent diameter to corridor width is 0 .
The occupancy maps in Figure 3 exemplify how optimism over returns can set its own exploration goals in domains without reward feedback.
These observations further underline the potential of combining model ensembling for policy debiasing with optimism over inﬁnite-horizon returns in guiding deep exploration.
5.2 Targeted exploration with sparse rewards
Efﬁcient planning under sparse reward feedback requires long-term coordination.
LOVE focuses on interactions with high potential for improving inﬁnite-horizon performance by trading-off predicted returns with their uncertainty.
The agent then only reduces uncertainty when it aligns with successful task completion.
We consider a sparse reward pendulum task, as its functional representations are readily interpretable.
The pendulum is always initialized in a downward conﬁguration and only receives reward at the top.
The agent can therefore not rely on random initialization to observe initial reward signals and needs to actively explore the upright conﬁguration.
Figure 2 (right) visualizes the mean and variance of the value ensemble during training.
We generate predictions in state-space by providing 5 contextual images for velocity estimation.
Initially, the agent builds a world model within which it learns to quantify uncertainty (Ep. 10).
The agent then leverages the model to explore the top.
This yields a spinning behavior and reward propagates into the value mean, while uncertainty is allocated for trajectories that could yield stabilization at the top (Ep. 50).
The agent queries these promising motions and reﬁnes its behavior to reach the upright, reducing associated uncertainty (Ep. 100).
The UCB objective allows the agent to ignore remaining uncertainty tangential to successful task completion (circle).
LOVE therefore guides exploration via potential for long-term improvement while offering interpretable uncertainty estimates.
5.3 Performance with mixed sparse and dense rewards
The previous sections highlighted LOVE’s intrinsic motivation to explore potential for long-term improvement based on its uncertainty estimation over returns.
In the following, we compare performance on a variety of visual control tasks from the DeepMind Control Suite and sparsiﬁed variations.
The environments feature (64 , 64) image observations, continuous actions, and dense or sparse rewards.
The sparse Cheetah Run, Walker Run and Walker Walk tasks are generated by zeroing rewards below a threshold ( 0 . 25 , 0 . 25 and 0 . 7 , respectively; rescaled to [0, 1]).
We use the same set of parameters throughout all experiments mirroring those of Dreamer.
However, we do not use random exploration noise.
We furthermore use an ensemble of M = 5 latent models, an initial UCB trade-off of β = 0 . 0 , an episodic growth of δ = 10 − 3 , and alter the policy learning rate and training steps to account for the more complex UCB objective.
The changes to the policy learning rate and training steps are also applied to Dreamer to yield ∆ Dreamer, as we found this to improve performance (see
Appendix G).
We furthermore introduce an exploration baseline, ∆ Dreamer + Curious, inspired by the curiosity bonus in Sekar et al.
[14] over an ensemble of RSSM dynamics.
Table 1 provides benchmarking performance on 5 seeds over 300 episodes (see also Appendix F).
On the sparsiﬁed environments, LOVE achieves 40% average improvement over the state-of-the-art ∆ Dreamer agent and 30% over both the exploration baseline ∆ Dreamer + Curious and ensembling baseline LVE.
On the other environments, LOVE improves by 26% , 37% , and 9% , respectively.
This suggests that LOVE’s combination of latent model ensembling with directed exploration can aid in identifying interactions conducive to task completion.
Ensembling reduces propagation of model-speciﬁc biases into the policy.
Optimistic exploration of uncertain long-term returns focuses sampling on regions with promising performance estimates while ignoring uncertain regions that are tangential to task completion.
This is an advantage of formulating the exploration objective in terms of uncertain rewards and not uncertain dynamics, as shown by the comparison to ∆ Dreamer + Curious.
We ablate the performance of LOVE on the UCB trade-off parameter β and compare against LVE ( β = 0 ).
We note that while LOVE outperforms LVE on the majority of tasks, both reach similar terminal performance in several instances.
However, LOVE provides clear improvements on the fully-sparse Cartpole Swingup and the partially-sparse Hopper Hop tasks, as well as the sparse locomotion tasks (Table 1, top).
These environments initialize the agent in conﬁgurations that provide no reward feedback, thereby forcing the agent to actively explore.
LOVE leverages uncertainty-guided exploration and gains an advantage under these conditions.
Similarly, this can explain performance on the Pendulum Swingup task.
While the task only provides sparse reward feedback, random environment initializations offer sufﬁcient visitation to the non-zero reward states, removing the need for active exploration.
LOVE also improves performance on the dense Walker tasks.
These environments require learning of stable locomotion patterns under complex dynamics, where directed exploration efﬁciently identiﬁes tweaks to the gait.
Similar to the bug trap environment of Section 5.1,
[Table]
[-and standard deviation on seeds after 3 × 
-][Table]
LOVE improves sample efﬁciency particularly under sparse reward feedback.
Temporally-extended optimism helps LOVE in actively exploring uncertain returns, providing an advantage over LVE.
Formulating intrinsic motivation in reward-space enables LOVE to identify uncertain interactions conducive to solving the task, providing an advantage over the curiosity baseline ∆ Dreamer + Curious.
[Table]
we observe that optimistic exploration is especially favoured by objectives that provide sparse reward feedback, while enabling efﬁcient discovery of tweaks to motion patterns under complex dynamics.
We additionally compare to Data Regularized Q (DrQ), a concurrent model-free approach.
DrQ updates its actor-critic models online giving it an advantage over LOVE and Dreamer.
LOVE performs favourably on the majority of environments with signiﬁcant differences on the sparse tasks (Pendulum,
Cartpole Sparse) and the locomotion tasks (Hopper, Walker).
DrQ outperforms LOVE on Finger Spin and Cartpole with dense rewards.
On these tasks, learning an explicit world model may actually be detrimental to attaining performance quickly.
The former task features high-frequency behaviors that may induce aliasing, while the latter task allows the agent to leave the frame yielding transitions with no visual feedback.
We additionally provide converged performance results for pixel-based D4PG [53] and proprioception-based A3C [54] at 10 8 environment steps to put the results into perspective.
Conclusion
We propose Latent Optimistic Value Exploration (LOVE), a model-based reinforcement learning algorithm that leverages long-term optimism to guide exploration for continuous visual control.
LOVE leverages ﬁnite-horizon rollouts of a latent model ensemble in combination with value function estimates to predict long-term performance of candidate action sequences.
The ensemble predictions are then combined into an upper conﬁdence bound objective for policy learning.
Training on this objective yields a policy that optimistically explores interactions that have the potential of improving task performance while ignoring uncertain interactions tangential to task completion.
These aspects are particularly important for robot control in complex environments, where the engineer aims to provide sparse high-level feedback without intensive reward engineering and the system should efﬁciently learn without getting distracted by irrelevant environment behavior.
To this end, we evaluate LOVE regarding its exploration capabilities and performance on a variety of tasks.
In the absence of reward signals, LOVE demonstrates an intrinsic motivation to explore interactions based on their information density.
Empirical results on DeepMind Control Suite tasks showcase LOVE’s competitive performance and ability to focus exploration on interactions conducive to task completion, particularly when reward signals are sparse.
Lifting intrinsic motivation into reward-space is then preferred over dynamics-space, as shown by comparison to a curiosity baseline.
Furthermore, LOVE demonstrates improved sample efﬁciency over the recent model-based Dreamer agent and competitiveness with the model-free DrQ agent.
Acknowledgements
This work was supported in part by the Ofﬁce of Naval Research (ONR)
Grant N00014-18-12830, Qualcomm and Toyota Research Institute (TRI).
This article solely reﬂects the opinions and conclusions of its authors and not TRI, Toyota, or any other entity.
We thank them for their support.
The authors further would like to thank Lucas Liebenwein for assistance with cluster deployment, and acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing
HPC resources.
We would also like to thank the reviewers and program chairs for their helpful feedback and suggestions for improvement