Adversarially Robust Imitation Learning
Abstract: Modern imitation learning (IL) utilizes deep neural networks (DNNs) as function approximators to mimic the policy of the expert demonstrations.
However,
DNNs can be easily fooled by subtle noise added to the input, which is even non-detectable by humans.
This makes the learned agent vulnerable to attacks, especially in IL where agents can struggle to recover from the errors.
In such light, we propose a sound Adversarially Robust Imitation Learning (ARIL) method.
In our setting, an agent and an adversary are trained alternatively.
The former with adversarially attacked input at each timestep mimics the behavior of an online expert and the latter learns to add perturbations on the states by forcing the learned agent to fail on choosing the right decisions.
We theoretically prove that ARIL can achieve adversarial robustness and evaluate ARIL on multiple benchmarks from DM Control Suite.
The result reveals that our method (ARIL) achieves better robustness compare with other imitation learning methods under both sensory attack and physical attack.
Introduction
Imitation learning is a powerful and practical alternative to reinforcement learning for learning sequential decision-making policies.
Together with the computational advancement brought by deep learning, imitation learning has shown great success in autonomous driving [1, 2], robot control [3], game AI [4] and motion capture [5].
Despite achieving human-level performance on many tasks, the existence of adversarial examples [6] in DNNs and many successful attacks to sequential decisionmaking policies [7, 8, 9, 10] motivate us to study robust imitation algorithms.
The observations by the agent unavoidably contain uncertainty that naturally originates from sensor errors or equipment inaccuracy.
A policy not robust to such uncertainty can lead to catastrophic failures (e g., a small piece of white scotch tape on a trafﬁc sign could cause an improper left turn, which will further lead to miles of unnecessary travel and even crush of the vehicle in the worst case).
Therefore, studying the fundamental principles of robust imitation learning is crucial.
One effective approach to induce robustness is domain randomization [11], a method where a designer with expertise identiﬁes the components of the model that they are uncertain about.
A set of training environments is constructed where the uncertain components are randomized, ensuring that the agent is robust on average to this set.
However, this requires careful parametrization of the uncertainty set as well as hand-designing of the environments.
A more easily automated approach is to formulate the problem as a zero-sum game and learn an adversary that perturbs the transition dynamics [12, 13, 14].
If a Nash equilibrium of this problem is found, then that equilibrium provides a worst-case performance bound under the speciﬁed set of perturbations.
Speciﬁcally, we can search over adversary policies f and agent policies π , so that min π ∈ Π max f ∈F J ( π, f ) , where J ( π, f ) is the expected cumulative reward for the attacker policy.
Unfortunately, in most imitation learning settings, quantitative reward from the environment is unavailable.
In this work, we propose a theoretically grounded framework of adversarially robust imitation learning (ARIL), where we build a surrogate loss to measure the distance between the actions generated from π and the actions from the expert.
We consider two ways of attack: sensory attack and physical attack.
In the sensory attack setting, the adversary only modiﬁes the inputs to π (e.g., the underlying true state of the environment is untouched as s t ), while in the physical attack setting, the adversary directly modiﬁes the actual physical state, i.e., it changes the underlying state to s (cid:48) t .
We consider attacking under interactive imitation learning (IL) setting where the expert is available for queries at any time step [15].
We evaluate our method in DM Control Suite [16], (e.g. HalfCheetah, Swimmer,
Ant), under both sensory attack and physical attack settings.
Our method signiﬁcantly improves the robustness under adversarial attacks over the baseline methods.
To summarize, this paper makes four contributions: (1) We study the theory and practice of Adversarially Robust Imitation Learning (ARIL).
(2) We systematically deﬁne sensory attack and physical attack.
(3) We prove that ARIL can achieve the adversarial robustness under realizability assumption, robust expert assumption, and optimal adversary assumption.
(4) We show the efﬁciency of our method in both attack stage and defense stage on multiple DM Control Suite benchmarks in comparison with other adversarial attack methods and other robust imitation learning methods.
Related Work
Adversarial Attack Adversarial attacks can be divided into white-box attacks and black-box attacks.
In white-box attacks, the attack has access to the structure, algorithm, and system parameters of the target model.
In black-box attacks, the attack cannot obtain information from the target model.
Instead, it can only interact with the target model by querying with the input and observing the output.
In our setting, we consider white-box attacks.
Szegedy et al.
[17] ﬁrst pointed out the intriguing property of neural networks and proposed a method called Large-BFGS to ﬁnd adversarial samples.
Goodfellow et al.
[18] proposed a Fast Gradient Sign method (FGSM) to generate small perturbations to fool the neural networks by calculating the gradient of the loss function concerning the inputs.
Kurakin et al.
[19] proposed Targeted FGSM to generate adversarial samples towards a target class and Iterative FGSM to apply FGSM multiple steps with small step sizes.
Tramer et al.
[20] proposed a randomized single-step attack to escape the non-smooth vicinity of the data point.
Kurakin et al.
[19] apply adversarial training on large-scale datasets.
Huang et al.
[7] showed that adversarial examples generated from FGSM could degrade test-time performance of a well-trained NN policy in RL.
Sun et al.
[21] studied the safe sequential decision-making problem under the setting of adversarial contextual bandits which is a one-step RL problem.
Lin et al.
[9] considered strategically-timed attack where the attacker picks speciﬁc time steps to attack.
Han et al.
[22] considered attacking an RL algorithm during training time.
All these previous works mainly focus on applying attacks developed from the supervised learning literature to RL setting, while in this work, we focus on learning adversarial attacks in imitation learning.
Robust Imitation Learning Imitation learning aims to learn sequential decision-making policies from expert demonstrations.
However, the agent can be vulnerable to cascading failures when the trajectory diverges from the demonstrations [23].
Laskey et al.
[24] proposes to inject noise into the supervisor’s demonstration policies, which forces the supervisor to demonstrate how to recover from errors.
Wang et al.
[25] utilizes VAEs [26] and GAIL [27] to learn semantic policy embeddings in imitation learning, which learns robust diverse gaits with limited demonstrations.
For robust imitation learning with different-quality demonstrations, Tangkaratt et al.
[28] makes use of a probabilistic graphical model to estimate the quality of demonstrations.
Brown et al.
[29] proposes a safe and robust Bayesian reward learning through preferences over imitating demonstrations.
Shin et al.
[30] performs the derivative-free random search optimization method with linear policies.
In our work, we focus on the robust imitation learning against adversary attacks, which can cause catastrophic failure to DNN-based agents with trivial perturbations.
Preliminaries
Before we delve into the details of ARIL, we ﬁrst outline our terminology.
In the rest of the paper, we focus on ﬁnite-horizon Markov Decision Process (MDP), which is deﬁned as ( S , A , H, P, r ) where
S is the state space, A is the action space, r is a reward function, P is the transition and H ∈ N + is the ﬁnite horizon.
A policy π : S → ∆( A ) is a mapping that maps from states to distributions over action space, i.e., π ( a | s ) is the probability of picking a given s .
3.1 Sensory Attack vs Physical Attack
For any pair of policy and attack ( π, f ) , we can treat the composition of them as a policy: π ◦ f :
S → A , i.e., a ∼ ( π ◦ f )( s ) is equivalent to ﬁrst generate stochastic adversarial example s (cid:48) ∼ f ( ·| s ) , and then sample action accordingly a ∼ π ( ·| s (cid:48) ) .
Denote d π as the average state distribution induced by policy π .
As π ◦ f can be understood as a policy, d π ◦ f will be the state distribution induced by π ◦ f .
We ﬁrst introduce the sensory attack setting where the adversary can only modify the sensory inputs of the RL agent, but not the underlying physical state.
Denote a trajectory as τ = { s 1 , s (cid:48) 1 , a [Equation]
, ..., s H , s (cid:48) H , a H } generated from the composite policy π ◦ f , the likelihood of such trajectory is ρ π ◦
[Equation]
[Equation]
where we denote P ( s 1 )
=
P ( s 1 | s 0 , a 0 ) as the initial state distribution.
Note that here adversary modiﬁes state s t to s (cid:48) t and then the policy π only sees s (cid:48) t .
For physical attack setting, the adversary will have the ability to modify the underlying physical states.
Again, we denote a trajectory as τ = { s 1 , s (cid:48) 1 , a 1 , ..., s H , s (cid:48) H , a H } generated from the composite policy π ◦ f , the likelihood of such trajectory now is:
[Equation]
Note the difference between Eq. 2 and Eq. 1, where transition probability is conditioned on attacked state.
In physical attack (Eq. 2), the adversary modiﬁes the state s to s (cid:48) , and then the environment actually generates the next state based on s (cid:48) , as the underlying physical state is now s (cid:48) .
The sensory attack is similar to attacks considered in today’s supervised learning setting, where the physical attack is similar to the robust control setting (e.g., a min-max formulation of H ∞ optimal control [31] under physical disturbances).
Note that under both settings, the attacks are sequential: the adversary can affect the learner’s decision via corrupting the state s , which will, in turn, affect the future states.
In supervised learning, an attack on a given sample will not affect the future samples, since samples are i.i.d in supervised learning.
3.2 Expert Robustness
For the expert π e , we have:
[Equation]
In practice consider the setting where the expert π e is a human driver who will not be affected by adversarial examples with small perturbations (e.g., human can easily tell a stop sign even with a sticker on it).
We later will relax the notion of expert robustness to include the possibility that the expert will be affected by adversarial attacks.
We consider interactive imitation learning setting proposed in [32] where the expert π e is always available during training time.
A learner can query the expert at any state s during training to get feedback π e ( s ) .
Adversarially Robust Imitation Learning
The goal of adversarial robust imitation learning (ARIL) is to learn a policy that can achieve good performance under all possible attacks:
[Equation]
where J ( π, f ) is the expected total cost of executing policy π with examples corrupted by attack f :
[Equation]
[Equation] We consider a loss function (cid:96) ( a, a (cid:48) ) that measures the difference between two actions a and a (cid:48) .
For instance, (cid:96) ( a, a (cid:48) ) could be zero-one loss if A is discrete, or l2 distance between two actions if A is continuous.
If we can control the long-term prediction error of the learned policy ˆ π under the worst possible adversary
[Equation]
then we can upper bound the performance of ˆ π at test time with the existence of an arbitrary adversary f ∈ F .
To see that, let us consider the setting where A is discrete and (cid:96) ( a, a (cid:48) ) is the zero-one loss:
[Equation]
Here, Q e denotes the state-action value of the expert policy and Q max denotes the upper bound of
Q e .
Namely, if δ is small, the learned policy ˆ π can achieve similar performance as the expert, even under an adversarial attack f during the execution of ˆ π .
This justiﬁes the reason for ﬁnding ˆ π that minimizes the objective function in Eq.
Let us deﬁne the objective as follows:
[Equation]
We perform an iteratively alternating update for π and f .
4.1 Update π with ﬁxed f
At iteration t , given the current f t , and π t , we want to update π t to π t +1 .
We consider the loss function with respect to π at round t :
[Equation]
which is a classiﬁcation loss where features are corrupted sampled s ∼ ˜
d π t ◦ f t , and labels are computed from expert query π e ( s ) .
Note that for discrete action space, the above loss function is simply a classiﬁcation loss of the classiﬁer π , with features generated from the distribution ˜ d
π t ◦ f t , and the ground truth label is generated from π e .
We perform no-regret update on (cid:96) t ( π ) to generate π t +1 .
[Equation]
Note that the above optimization can be understood as a classiﬁcation task where the features are sampled from the mixture of distributions { ˜ d π i ◦ f i } ti =1 with equal weight 1 /t , and the corresponding labels are generated based on π e ◦ f i for a data point sampled from ˜ d
π i ◦ f i , ∀ i ∈
[ t ] .
In practice, we need to use samples to replace expectations, which leads us to a Data Aggregation style update (DAgger [32]) for π .
4.2 Update f with ﬁxed π
At iteration t , given the current f t and π t , we want to update f t to f t +1 .
We consider the objective function with respect to f at round t as:
[Equation]
We want to ﬁnd f that maximize the above objective function, namely, increasing the classiﬁcation loss of the current policy π t as much as possible.
Note that different from the loss function for updating π , here the expectation is deﬁned with respect to the ﬁxed π t and f .
If π t is a stochastic
1: Input: MDP M , policy class Π , expert policy π ∗ , set of adversaries F , a supervised learning
2: Initialize a dataset D = ∅ 3: Initialize a policy π 0 ∈ Π 4: for t = 0 to N do 5: f t = arg max f ∈F E s ∼ d πt ◦ f
[ (cid:96) ( π t ( f ( s )) , π e ( s ))] (RL Step) 6:
Execute π t ◦ f t to generate a set of non-corrupted states { s i } ni = 7:
Query expert during execution to get labels { π e ( s i ) } ni = 8:
Data Aggregation: D
= D + { f t ( s i ) , π e
( s i ) } ni = 9: π t +1 =
L EARNER ( D ) (Supervised Learning, e.g., Classiﬁcation if A is discrete ) 10: end for policy, and f is parameterized as f ( s ; θ ) , then one can use policy gradient technique to compute the ∇ θ ˜ (cid:96) t ( f ) as follows:
[Equation]
where the second step simply uses chain rule, and in the last step we simply treat π ◦ f together as a policy that maps from states to actions—hence the Q function is deﬁned with respect to π t ◦ f t .
Intuitively, we want to optimize f θ such that it increases the likelihood of generating actions via π t to increase the total “reward"—the long-term prediction error (with respect to π e ) of the policy π t .
Using the above gradient to perform gradient ascent to optimizing f θ is similar to classic policy gradient methods and in practice, one can use existing techniques developed for PG here as well (e.g., natural gradient, actor-critic where the critic is Q π t ◦ f t ).
4.3 Algorithm
We summarize the algorithm ARIL in Alg. 1, where every round, we update the adversary f and the policy π .
Given a policy π t , we aim to ﬁnd the adversary f t from F that increases the long-term prediction error of π t (w/ expert’s prediction) as much as possible.
Once we ﬁnd such adversary f t , we generate data by executing the composite policy π t ◦ f t on the real system to generate states { s i } ni =1 ∼ d π t ◦ f t and labels { π e ( s i ) } ni =1 (this procedure requires interactively querying expert, just as DAgger does [32]).
We then corrupt the data by applying the latest adversary f t at every data point s t , and appendix the new dataset { f t ( s i ) , π e ( s i ) } ni =1 to the aggregated dataset D .
Lastly, we train the new policy using the black box supervised learner on the aggregated dataset D .
The data aggregation technique (DAgger [32]) is used to ensure stability in the process of updating π and corresponds to the speciﬁc no-regret learner FTL as we explained before.
However, any other no-regret learner can be used here as well, such as Online Gradient Descent [33], with the potential beneﬁt of no need to maintain an ever-growing dataset D .
In theory, optimizing f t is challenging as the objective function could potentially be non-convex here.
However, in practice for parameterized adversary f , as we showed in the previous section, we can compute the gradient of f using classic policy gradient technique, which in turn allows us to leverage practically effective techniques such as natural gradient, actor-critic, to optimize f t .
Note that the optimization procedure for f t does not require any expert data.
4.4 Analysis
We consider the following setting.
First, for optimization in f t , we assume we can achieve nearoptimality:
[Equation] [Equation]
with (cid:15) rl ∈ R + .
Namely, we do not assume we can ﬁnd the global optimal adversary at every round.
The (cid:15) here quantiﬁes the performance of the potential RL algorithm in terms of optimizing ˜ (cid:96) t ( f ) with respect to f .
For simplicity, we consider the realizability setting : there exists a policy π ∈ Π , such that for any state s ∈ S , E a ∼ π ( s )
[ a (cid:54) =
π e ( s )] = 0 .
Namely there exists at least one policy in policy class Π that can predict expert’s action at any state s ∈ S .
Relaxing the realizability setting to agnostic setting is straightforward here.
Theorem 4.1.
As T → ∞ , Alg.
1 outputs a policy π
i
[Equation]
∈ { π t } Tt =
such that:
[Equation]
The above theorem indicates that (1) under the realizability assumption (i.e., policy class Π is rich enough to approximate π e well), (2) the assumption that the expert is robust with respect to any attack (i.e., (cid:15) e is deﬁned as the error between π e ( s ) and π e ( f ( s )) , which is negligible), and (3) the assumption that we can near-optimally solve the RL-like optimization problem max f ∈F ˜ (cid:96) t ( f ) at every round t (i.e., (cid:15) rl is small), then Alg.
1 can achieve adversarial robustness.
Please refer to supplementary material for proof.
Experiments
We evaluate our proposed adversary and robust imitation algorithm with baseline methods using DM
Control Suite [16].
We compare our method with FGSM [18].
Considering FGSM is commonly used in supervised learning setting, we apply FGSM to modify state s h such that it hurts the current one-step prediction of π t , i.e. f fgsm ( s h ) =
s h + (cid:15) sign ( ∇ s E a ∼ π t ( s ) ( (cid:96) ( a, π e
( s h )))
| s h ) .
Our algorithm achieves better robustness under both sensory attack and physical attack.
The details of the datasets and other experimental settings are described below.
5.1 ARIL with Sensory Attack
In sensory attack, the adversary only modiﬁes the sensory inputs of the agent, but not the underlying physical state, which can be treated as sensory noise in real scenarios.
Environment Setting We investigate the performance of attacks on three continuous control tasks from OpenAI gym robotic benchmarks [34]: Ant-v2 , HalfCheetah-v2 and Swimmer-v2 .
For the convenience of training and testing, we set the horizon H to 1000 in all the benchmark environments.
We only attack the zero-order state in the robotics environments, i.e. the positions and angles other than velocities (ﬁrst-order state).
The attack is bounded by ± (cid:15) ( (cid:15) is a small positive number), which varies according to the robustness of each environment.
The (cid:15) is chosen to have no more than 40% performance drop with FGSM attack and its values for all experimental setups can be found in supplementary materials.
For comparison, all attack methods are bounded to the same (cid:15) when applying to the same robotics environment.
Training Details Our implementation of ARIL is built on top of rllab [35] and uses Trust Region
Policy Optimization (TRPO) [36].
Throughout the experiments, both policy π and adversary f θ are parameterized with two hidden layers.
The output layer of adversary uses tanh as activation function and then multiples (cid:15) to restrict the output in [ − (cid:15), (cid:15) ] .
The standard deviation of f θ is initialized to 0 .
1 (cid:15) and the attack to state s is sampled from N ( f θ ( s ) , 0 .
1 (cid:15) ) .
We also clip the sampled adversarial noise to [ − (cid:15), (cid:15) ] .
Hyperparameters are selected by grid search and the values for all experimental setups can be found in supplementary materials.
Results We ﬁrst compare our adversary with the baseline Fast Gradient Sign Method (FGSM) [18] and Transferable adversarial perturbations (FGSMˆk) [37]
In both our method and FGSM, the perturbation added to each state is bounded to a small value such that it’s imperceptible as shown in the supplementary material.
From Figure 1 (top), we ﬁrst observe that FGSM attack in imitation learning works well.
By forcing the policy to generate actions that are far away from the expert’s, FGSM attack is effective and successfully decreases the performance of π .
The performance of FGSM and our attacks in all settings can be found in Table 1 (Adversary).
In all tasks, our attack outperforms the FGSM attack.
We experimentally demonstrate that through maximizing the total difference to expert policy, the adversarial attack could signiﬁcantly decrease the total reward.
We also show that naively applying adversarial attack techniques developed for supervised learning to a sequential decision-making agent is not effective since the attack is myopic.
This result proves the efﬁciency of our attacks even without ever accessing to reward signals.
We then compare ARIL with another robust imitation learning method, DART [24].
DART learns to inject an optimized level of noise into the expert demonstrations in imitation learning, which forces the expert to demonstrate how to recover from the errors.
A noise distribution, p ( ξ | π θ ∗ , ψ ) , is injected to the demonstration distribution π θ ∗ ( u | x, ψ ) , where ψ denotes the sufﬁcient statistics that deﬁne the noise distribution.
In our experiments, we set π θ ∗ ( u | x, ψ ) as a Gaussian distribution, N ( π θ ∗ ( x ) , Σ) .
The optimal parameter ˆ ψ is estimated by the anticipated ﬁnal robot’s error and subsequently scaling the current simulated error to this level, denoted as E p ( ξ | π θ ∗ , ˆ ψ ) (cid:80) t (cid:96) ( u t , π π ∗ ( x t )) .
For quantitative comparison, we train an adversary to attack DART using our proposed attacker (ARIL-A).
As shown in Table 1 (bottom), our method signiﬁcantly improves the robustness over DAgger [32] (3rd row and 5th row).
ARIL also achieves better robustness compared with DART (4th row and 5th row) in all tasks under the worst possible adversary.
[Table]
5.2 ARIL with Physical Attack
In physical attack
[Table]
the attacker can also affect the underlying transition dynamics.
Considering the attacker changes the actual state of in the environment [Table] the transition model can be described as s t +1 =
f ( P ( ·| s t , a t )) .
Namely, the actual zero-order states are changed under the attack.
We can treat physical attack as an external perturbation (e.g. a kick on the robot or an unpredicted bump on the road).
Environment Setting Similar to sensory attack, the attack is bounded by ± (cid:15) .
It’s worth mentioning that (cid:15) is different for sensory attack and physical attack even in the same environment.
Since physical attack with the ability to change underlying dynamics is more effective than sensory attack.
The (cid:15) are chosen to have no more than 40% performance drop with FGSM attack and its values for all experimental setups can be found in supplementary materials.
Results We ﬁrst compare our adversary with the FGSM baseline.
As one might expect, the physical attack is more effective than the sensor attack.
Under the physical attack setting, π is more likely to fail.
Our experiments support this viewpoint in the sense that (cid:15) for sensory attack needs to be larger than the one for the physical attack to achieve similar attack performance.
Again, in all tasks, our attack outperforms the FGSM attack.
We also compare ARIL with the DART (Table 2).
This result proves the efﬁciency of our robust imitation algorithm even the underlying transition dynamics are changed.
[Table]
Conclusion
In conclusion, we propose an adversarially robust imitation learning algorithm.
We theoretically prove that under realizability assumption, robust expert assumption, and optimal adversary assumption,
ARIL achieves adversarial robustness.
We evaluate our method on a set of continuous control tasks from OpenAI Gym and show that our method achieve better robustness under both sensory attack and physical attack compare with baselines.
We hope that our works can inspire more studies in robust imitation learning methods when the quantitative reward from the environment is unavailable. Acknowledgements.
The authors would like to thank Wen Sun, Yan Xu, Yunze Man for fruitful discussion and detailed feedback.
This work was supported by the PanGU Young Investigator Award.
A Proof of Theorem 4.
Proof.
Since Alg.
1 runs no-regret online learner to update π on the sequence of loss functions { (cid:96) t ( π ) } , we must have:
[Equation]
Add and subtract (cid:80) Tt =0 (cid:96) t ( π e ) on the left hand side of the above inequality, we have:
[Equation]
Since we operate under the realizability setting, the term inside the parenthesis on the RHS of the above inequality is guaranteed to be less than or equal to zero.
Hence, the above inequality simpliﬁes to:
[Equation]
For (cid:96) t ( π e ) , using the deﬁnition of (cid:96) t , we see:
[Equation]
[Equation]
where we use the expert robustness deﬁnition above (i.e., π e ( f ( s ))
=
π e ( s ) for all s and f ).
Hence, we have:
[Equation]
Now we need to lower bound (cid:96) t ( π ) .
Using the deﬁnition of (cid:96) t again, we have:
[Equation]
where the ﬁrst inequality comes from the assumption that the RL solver returns a (cid:15) rl near-optimal solution.
Combine all the results above together, we get:
[Equation]
Hence as T approaches to ∞ , we have that the long-term prediction loss of the learned policy π i (cid:63) under the worst possible adversarial attack from F is upper bounded by (cid:15) rl —the error introduced from optimizing f t .
Under agnostic setting, it is not guaranteed that there exists a = π e ( s ) , where a ∼ π ( s ) and π ∈ Π for ∀ s ∈ S .
However, it can be assumed that the error between a and π e ( s ) is bounded by a small number (cid:15) a , namely E a ∼ π ( s )
[ a (cid:54) =
π e ( s )] ≤ (cid:15) a .
Hence Equation 4 is modiﬁed as:
[Equation]
[Equation]
Plugging this to Equation 3, we have:
[Equation]
Combining the results with Equation 6, we get
[Equation]
Similarly to the realizability setting, as T approaches to ∞ , the long-term prediction loss of the learned policy π i (cid:63) under the worst possible adversarial attack from F is still upper bounded.
A.1 Training Details
Both attacker and student are trained using Adam optimizer at learning rate of 0 . 001 .
At the start of ARIL algorithm, we initialized the buffer for DAgger algorithm with size 5 M timesteps.
In each attack stage, it takes 1 e 6 timesteps for the attacker to collect the trajectories each time.
Then at each defense stage, we collected 40 trajectories of student under attack and labeled them with expert actions into the DAgger buffer.
[Table]