Imitation from Observation With Bootstrapped Contrastive Learning
Abstract
Imitation from observation (IfO) is a learning paradigm that consists of training autonomous agents in a Markov Decision Process (MDP) by observing expert demonstrations without access to its actions.
These demonstrations could be sequences of environment states or raw visual observations of the environment.
Recent work in IfO has focused on this problem in the case of observations of low-dimensional environment states, however, access to these highly-specific observations is unlikely in practice.
In this paper, we adopt a challenging, but more realistic problem formulation, learning control policies that operate on a learned latent space with access only to visual demonstrations of an expert completing a task.
We present BootIfOL, an IfO algorithm that aims to learn a reward function that takes an agent trajectory and compares it to an expert, providing rewards based on similarity to agent behavior and implicit goal.
We consider this reward function to be a distance metric between trajectories of agent behavior and learn it via contrastive learning.
The contrastive learning objective aims to closely represent expert trajectories and to distance them from non-expert trajectories.
The set of non-expert trajectories used in contrastive learning is made progressively more complex by bootstrapping from roll-outs of the agent learned through RL using the current reward function.
We evaluate our approach on a variety of control tasks showing that we can train effective policies using a limited number of demonstrative trajectories, greatly improving on prior approaches that consider raw observations.
Introduction
Imitation from Observation (IfO) [24; 17] involves learning a policy function for an agent to solve a task using a set of demonstrations of an expert performing the same task.
Distinct from Imitation
Learning (IL) [12; 20; 18; 2], in IfO, the demonstrations are only sequences of observations of the environment by the expert.
Depending on the environment and the information available to the agent, an observation can be a description of the environment at a given time (joint angles, distance between objects, direction vector coordinates, velocity, etc.) provided in the form of a low-dimensional state vector of the environment, or a raw visual of the environment subject to further analysis and processing.
The advantage of the IfO formalization is that it allows more natural agent learning scenarios, where difficult-to-acquire precise action data is not available.
Furthermore, it more accurately approximates the way in which humans imitate experts during learning.
The two main methods in the literature that have been used to train IfO problem agents are adversarial methods and reward learning methods [27].
The adversarial methods [25; 26] adapt ideas from
[Figure]
inverse reinforcement learning [10], proposing a GAN-like architecture [6] where the agent is a trajectory generator associated with a discriminator function that evaluates how similar the state transitions produced are to that of an expert.
These methods have proven their effectiveness in the case of low-dimensional state observations.
When dealing with high-dimensional visuals, [25] uses a small stack (of size 3) of successive images to evaluate fake and real data.
However, when using high-dimensional visual observations instead of low-dimensional state sequences, each video transcribes the variations of the system states without clearly designating the main and critical features that describe the environment.
The fact of reducing the problem to a classification of small sequences limits the capacity of the discriminator to analyze and identifies these features describing optimal actions.
Being able to correctly understand and represent the system state using environment images is essential to evaluate state transitions.
Moreover, this approach limits the understanding of specific behaviors planning their actions over long horizons.
On the other hand, reward learning methods [14; 17; 3] are based on the training of a reward function that will be subsequently used to reward an agent while applying a traditional RL algorithm.
The main interest of such an approach is to learn a reward function that measures the efficiency of the agent actions based on the expert demonstrations.
The different reward learning methods exploit a precise self-supervised learning objective, such as context variation between many executions [17], which allows the model to converge to a meaningful and efficient reward function for training the imitating agent.
In this paper, we present BootIfOL , a novel reward learning IfO algorithm that relies on the representation of agent behaviors in a latent space using raw pixel-based demonstrations, illustrated in Figure 1.
Unlike adversarial approaches, we use the entire observation sequences with a Long
Short-Term Memory (LSTM) neural network [11] to keep track of the agent’s actions and to represent the induced behavior of the agent at each sequence timestep.
This method directly exploits a limited set of visual demonstrations from an expert to train an imitating agent, without having access to the expert policy or actions.
Inspired by Berseth et al.
[1], we also train the reward function progressively with new visual trajectories generated by the agent.
In contrast to [1], our method trains the trajectory encoding function on a set of failure trajectories before the agent training starts.
This helps to ensure the consistency of the reward function from the beginning of the agent training.
We also use
Contrastive Multiview Coding [23] and Dense Predictive Coding [8] methods for self-supervised learning of trajectories.
Like [17; 1], a reward function is trained by interacting with the environment to encourage the imitating agent’s behavior to be similar to the behavior of the expert.
The motivation behind this approach is: (1) To learn and estimate, at each timestep, the system state using the agent’s visual observations (knowing that the real system state is not accessible); (2) To identify the behavior induced by the agent over a certain period of task execution and make sure that this behavior serves the same purpose as the expert.
Our method also demonstrates the interest in a first-round training of the trajectory encoding function in order to provide meaningful rewards to the agent from the beginning of its training, contrary to other similar reward learning methods that bypass this step.
Our method successfully solves a range of tasks in the Deepmind Control Suite [22] and the Meta-world environment [30], approaching episodic rewards of task experts.
𝜃 𝜃 ω ɸ ɸ . . . . . .
Ɣ
Method
Agent training We aim to train an agent to perform a task only using a set of videos D e of an expert performing the same task many times.
Our proposal is based on the trajectory matching objective, which consists of training the agent to have a trajectory similar to that of the expert.
In other words, the only information we seek to extract from the expert’s videos is its behavior.
Our algorithm is divided into two phases, an Alignment Phase and an Interactive Phase .
During the Alignment Phase, we learn a sequence encoding function f ω , which encodes sequence of frames of an agent trajectory, and an associated distance metric between two sequences of two agent trajectories.
We also jointly train an image encoding function g θ .
Note that this image encoding function g θ can also be replaced by a pre-trained encoder.
We discuss the impact of this replacement in Section 3.
In the Alignment
Phase, we only use trajectories coming from two distributions: the distribution of expert’s trajectories
O ∼ p ( D e ) , and a distribution of trajectories O ∼ p
( D a ) generated by a randomly defined policy function.
During the Interactive Phase, we learn the agent policy π using a standard Reinforcement
Learning (RL) algorithm
To obtain the reward we use the learned distance (trained during the
Alignment Phase) between sequence encodings of the expert and the current agent.
Critically, we fine-tune the image and sequence encoding functions with additional visual observations from our online interactions with the environment.
In this work, we use DrQ-v2 [29] RL algorithm to train the agent policy, π , and its associated q-value function, Q , though our method is agnostic to the choice of the RL algorithm.
Specifically, in the Interactive Phase at each new episode of the agent, we sample an expert episode
O e ∼ p
( D e ) that acts as a reference expert policy.
At each agent step, t , we evaluate the distance between o 0: t , the agent’s trajectory until the step t , and o e, 0:
t the expert’s trajectory until t by d ( o 0: t , o e, 0: t ) =
|| f ω ( g θ ( o 0: t ))
− f ω ( g θ ( o e, 0: t )) || .
The reward assigned to the agent at this step is finally r t = − d ( o 0: t , o e, 0: t ) .
The function f ω from the Alignment Phase is trained only on expert trajectories and random trajectories, thus as the policy of the agent improves, the distribution of agent trajectories will change, making the distances produced inconsistent due to the shift of the trajectory distribution.
In order to fix this in the Interactive Phase, we continue to update f ω and g θ on the new trajectories generated by the agent during the training of π .
In Paragraph 3, we discuss the necessity of the Alignment Phase before training the agent.
The full training process is detailed in Algorithm 1.
Image encoding We aim to encode videos (observation trajectories) and thus we need to find encodings of individual frames.
Although a pre-trained image model may be used, a suitable one is not always available or appropriate (e.g. too expensive for the task).
Thus we propose and evaluate learning the image encodings along with the sequence encodings.
The image level objective contains three terms: (1) L triplet which enforces image encodings similarity of adjacent frames, (2) L ae which permits the encoding to be decoded back to an image, and finally (3) we learn a contrastive multiview encoding following [23] based on a Lab color space [4] image representation.
The first term L triplet compares the distance between an anchor image o , a positive image o p temporally close to the anchor, and a negative image o n distant from the anchor image in the sequence:
[Equation]
D e = { O ie } 1 ≤ i ≤ N = { ( o ie, 0 , o ie, 1 , ..., o ie,T ) } 1 ≤ i ≤ N
Initialize f ω , g θ , q γ , d ϕ // Alignment Phase: Training f , g while k ≤ N pretrain do { O ie } 1 ≤ i ≤ n ← sample ( D e ) { O i } 1 ≤ i ≤ n ← π random ( env )
Eval.
L ( θ, γ, ω, ϕ ) with ( { O ie } , { O i } )
Optimize θ , γ , ω , ϕ end //
Interactive Phase: Training π , Q , f , g
Initialize D a = {}
Initialize D = {} , Q , π // Replay buffer, Q-value function, policy while step ≤ N π do
O e ← sample ( D e ) o 0 ← env.reset () for t ∈ 0 , .., T − 1 do a t ← π ( o t ) o t +1 ← env.step ( a t ) r t +1 ← −|| f ( o 0: t +1 )
− f ( o e, 0: t +1 ) ||
Save ( o t , a t , o t +1 , r t +1 ) into D ( o, a, o ′ , r ) ← sample
( D )
Using DrQ-v2 update Q , π with ( o, a, o ′ , r ) endif ( step ≤ N train ) and ( step mod N update = 0) then
Save O = { o 0: t +1 } into D a { O ie } 1 ≤ i ≤ n ← sample ( D e ) { O i } 1 ≤ i ≤ n ← sample ( D a )
Eval.
L ( θ, γ, ω, ϕ ) with ( { O ie } , { O i } )
Optimize θ , γ , ω , ϕ end end where s = g θ ( o ) , s p =
g θ ( o p ) .
The training objective of the auto-encoder is then to minimize the loss function L ae ( θ, γ ) =
|| o − q γ ( g θ ( o ))
|| 2 .
Finally, we incorporate contrastive multiview coding to enrich the visual representations following [23].
Specifically we consider the color space transformation { R, G, B } → { L, ab } where: (1) the component L is the view v 1 , processed by the
CNN g θ 1 , (2) the component ab is the view v 2 processed by g θ 2 .
The final encoding s of an image o is given by s =
g θ ( o ) =
[ g θ 1 ( v 1 ) , g θ 2 ( v 2 )] .
Following the formulation [23], we align the different views using a contrastive learning loss denoted L S .
This process is illustrated in Figure 2.
Overall, the objective function to be minimized for the training of the image encoding function is:
[Equation]
Sequence encoding In order to allow us to construct a reward function, we learn to represent a sequence of frames [7; 8] with a semantically meaningful distance between them.
We use two losses: (1) L Z a standard video representation learning approach Dense Predictive Coding (DPC) [8]; (2) and L O bootstrapped expert behavior loss, which encourages separation between expert and non-expert trajectories.
The loss, L O , lies at the heart of our method and allows the sequence representation to gradually improve as non-expert trajectories improve.
DPC is a self-supervised learning method based on the surrogate task of predicting the next frame using the sequence of previous frames as input.
Let us consider a dataset of videos D
= { O i } 1 ≤ i ≤ N = { o i 0: T } 1 ≤ i ≤ N .
Denote, d ϕ , which from an encoding z t of a sequence of states s 0: t , predicts the next state
ˆ s t +1 of the sequence.
Following [8] we can use this for spatio-temporal representation learning.
We first predict the next state
ˆ s t +1 =
d ϕ ( f ω ( s 1 , . . .
, s t )) and subsequently use this to predict additional K-1 states ˆ s t
+ k = d
ϕ ( f ω ( s 1 , . . .
, s t , ˆ s t +1 , · · · , ˆ s t + k − 1 )) .
We then apply the learning objective to minimize the loss function L Z given by:
[Equation]
where h ( a, b ) = exp( a T b τ ∥ a ∥∥ b ∥ ) and τ is the temperature parameter.
Indeed, the loss function is also a contrastive loss function where the positive pairs are (ˆ s t , s t ) and the negative pairs are (ˆ s t , s t ′ )
t ̸ = t ′ in the same demonstration.
To bring sequences from the same distribution closer together, we use another contrastive loss that applies to the sequence encoding vectors z returned by f ω .
Thus, for a sequence O belonging to one of the two trajectory distributions, we consider a positive sequence O p of the same distribution and k negative { O n,i } 1 ≤ i ≤ k of different distributions.
The objective function to be minimized is L O :
[Equation]
The objective function that we seek to minimize to train the sequence encoding function is
L seq ( θ, ω, ϕ ) =
L Z ( θ, ω, ϕ ) + L O ( θ, ω ) .
Finally, for a triplet ( O, O p , { O n,i } 1 ≤ i ≤ k ) , the frames used to evaluate L frame come from the sequences of the sample, allowing us to compute the objective loss function L ( θ, γ, ω, ϕ ) =
L frame ( θ, γ ) + L seq ( θ, ω, ϕ ) .
Experiments
We evaluate our method using a diverse set of continuous control tasks including 4 tasks (Reacher
Hard, Finger Turn Easy, Hopper Stand, and Walker Run) from the DM Control Suite [22] and 3 tasks (Button Press, Plate Slide, Drawer Close) from the Meta-world environment [30].
The DM Control tasks require our agent to learn to coordinate multiple torque-controlled actuators.
The Meta-world end-effector is controlled in a 3DOF task space with a 1DOF parallel jaw gripper modeled on a
Sawyer Robot Arm.
This environment setting requires our agent to learn to manipulate external objects with complex physical interactions with the world.
In these manipulation tasks, we now have to model the robot and object-object interactions, such as in the case of Button Press and Drawer
Close where the constrained object joint must be activated along a single axis.
Below we describe the network architecture, details of our training procedures, and the results on these two datasets.
All episode trajectories begin from randomized starting states of the robot and interactive objects (when applicable).
Network architectures Our image encoder function g θ consists of a pair of convolutional neural networks g θ 1 and g θ 2 .
These models are of similar architectures, but the number of input channels is 1 for g θ 1 and 2 for g θ 2 .
We re-use the encoder architecture proposed by [17].
The architecture is a succession of four 5 × 5 stride-2 convolutional layers with 64, 128, 256, and 512 filters.
These convolutional layers are followed by two 1 × 1 convolutional layers of 512 and 128 filters.
All layers before the last are followed by BatchNorm [13] and a Leaky ReLU [9] activation function ( leak = 2 ).
The image decoding function q γ has an inverse architecture to the encoder, except that transposed convolutional layers are employed for the last 4 layers.
Additional network details are presented in
Tables 3 and 4 in Appendix A.
The sequence encoder function, f ω , is a 2-layer LSTM with 128 as input and output sizes.
The output sequence is followed by a fully connected layer with input and output size of 128.
The next-state predictor is a sequence of two 128 × 128 fully connected layers with Leaky ReLU activation between the two layers.
We employ an image encoding function, g θ , to learn a latent representation from high-dimensional observations.
This function is trained with a self-supervised method that combines different surrogate objectives.
However, much modern research [5; 28; 19] has shown the benefit of using pre-trained or foundation models trained on a large corpus for downstream tasks.
We illustrate the opportunity of using this strategy with our setup by employing
EfficientNet [23], a convolutional model trained on ImageNet, as a backbone image encoder.
In this
[Figure]
t
[Figure]
t =t =t =t = t =t =
[Figure]
setting, the function g θ is composed of the backbone model with the weights frozen followed by a 3-layer multilayer perceptron (MLP) of size 1280 × 512 × 128 with trainable parameters.
Though we utilize EfficientNet-B0 as a backbone, there are many reasonable choices for pre-trained encoders.
Training: Alignment Phase For each task, we build a dataset of expert episodic trajectories with colored image observations at each timestep by running a trained RL agent and rendering visual images of the environment as needed.
We also build a second dataset of trajectories with a randomly defined policy using the same process.
In this work, we employ DrQ-v2 agents as our experts for imitation, though any reasonable expert (image-based RL, planning agents, human, or animal experts) could be used since we do not need access to the state of the agent.
In the case, where we train an image encoder from scratch (BootIfOL), the demonstration datasets D e and D a , of size 5000 × 51 × 64 × 64 (trajectory × timestep × height × width), are used as input.
In EffBootIfOL, where we leverage large-scale pre-training, we utilize smaller datasets of demonstrations for finetuning: 1500 × 61 × 224 × 224 (trajectory × timestep × height × width).
Training: Interactive Phase
The parameters of the encoding functions are updated during N train =
375 K training steps of the Interactive Phase.
After these training steps, the parameters of the encoding functions are frozen.
Our RL agent training is performed over N π = 1550 K training steps.
We employ DrQ-v2 [29], an image-based RL agent based off of the DDPG [16] architecture, as our RL policy learner with default hyperparameters.
Additional hyperparameters are available in Table 7.
General results We perform the training on the Reacher Hard, Finger Turn Easy, Hopper Stand, and Walker Run tasks from the Deepmind Control Suite [22].
We compare our algorithm to the
Context Translation (CT) [17] method for IfO by varying the number n of expert trajectory samples used at each training episode to predict the CT agent trajectory.
We also baseline against ViRL [1] and GAIfO [25].
Demonstrated across 4 tasks in DM Control, our method shows strong performance in learning to complete tasks given visual demonstrations as shown in Table 1, where GAIfO showed poor performance on all tasks.
To facilitate the comparison, we have scaled the average returns between 0 and 1 in Table 1.
The expert has the highest average return, one, and the average return of the random agent is zero.
We present the non-scaled absolute values in Appendix B. We also show promising results in complex visual scenes by employing a pre-trained backbone on Meta-world environments as shown in Table 2.
Average episodic returns over RL training in the Interactive Phase
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
to CT and ViRL agents on Reacher Hard and Hopper Stand tasks.
We observe that agent progresses quickly in both cases as compared to other baselines.
Although ViRL is able to do well in Hopper Stand, it is unable to tackle all environments (e.g. Reacher Hard).
are presented in Figure 4.
We show visual examples of our agents acting in Hopper Stand and Drawer
Close tasks in Figure 3.
We present the examples for other tasks in Figure 9 and describe them in
Appendix E. We observe that in tasks such as Walker Run, where a great degree of coordination between joints is necessary to control the robot, IfO agents struggle to reach even 10% of the expert’s reward (see Table 1).
[Table]
[Figure]
Results of Evaluation on Meta-World Considering the Meta-world environment [30], we study precisely the importance of a pre-trained convolutional network as a feature extraction model.
The visual complexity offered by Meta-world tasks allows us to evaluate the relevance of an accurate image encoder during the training of the encoding functions.
We evaluate our new agent, trained with a learned distance metric using a pre-trained image encoding, as described in the Network
Architectures section ( 3), on the tasks Button Press, Drawer Close, and Plate Slide of the Meta-world environment.
In Table 2, we compare the average return of this agent with our initial agent.
The results in Table 2 demonstrate the importance of the image encoding function for the effectiveness of the learned distance metric.
In particular, these results show that the success of the reward function learning depends on the level of accuracy in the interpretation of each frame.
We observe a particularly weak result concerning the Plate Slide task.
We hypothesize that this failure is due to a difficulty related to the sequence encoding function, which has a different role.
Despite the accuracy of the feature extractor, the task remains difficult to imitate by our agent.
Ablating the Alignment Phase In our algorithm, we integrate the Alignment Phase, described in
Section 2, allowing us to bootstrap the encoding functions f and g on two trajectory distributions: (1)
Expert’s trajectories, (2) random trajectories.
This particularity is not present in most methods such as GAN-like approaches.
We hypothesize that providing, from the beginning of the agent’s training, significant and consistent rewards offers an important gain on the search for the optimal imitation policy.
We evaluate the importance of this step by removing the Alignment Phase in two ways: (1) the encoding functions are trained during 375K total training steps (standard case) in the Interactive
Phase; (2) the encoding functions are trained during all the agent’s training in the Interactive Phase (similarly to ViRL and GAIfO).
We present the results of this experiment in Figure 5.
We observe that the use of the alignment phase leads to drastically better performance, highlighting this as a critical phase.
Although training these for the full interactive phase can lead to some progress, it is not nearly as efficient as the bootstrapped approach that includes the Alignment Phase.
Ablating the use of a Learned Image-Encoder for RL To date, in our experiments, the RL agent is learned through a dissociate policy function π and q-value function Q as proposed [29].
In this architecture, the policy function π is composed of a CNN E that internally encodes the observation images, followed by an MLP that feeds the image encodings to return actions.
E is shared between the policy pi and the associated q value function Q , which feeds encoding-action pairs to return q-values.
A natural question is whether we can directly link our learned image encoding function g θ to pi and Q by dropping the CNN E .
The interest of this approach is to transfer the learning of the g to the agent in order to learn directly the extracted features.
We know that in modern RL approaches, using low-dimensional state vectors generally improves agent performance and efficiency, because of higher precision in the description of the state.
We are interested in learning if our learned observation feature descriptions - optimized with a pixel-based reconstruction loss - provide enough information about the state to enable an RL policy to learn efficient and quality control policies without needing access to either the true simulator state or raw pixels.
Specifically, we replace E with an MLP shared between π and Q .
The image encodings returned by g are the input data of the MLP E .
Note in this experiment we freeze the parameters of g θ with respect to the RL update and call this ablation
the
[Figure]
r e t u r n o v e r
[Figure]
t e p
[Figure]
p i s o d
[Figure]
s
[Figure]
encoding-based agent .
We present in Figure 6a the average return of the original image-based agent and the encoding-based agent on the Reacher Hard task over training.
This leads us to believe that the image encodings returned by g present biases that limit the understanding of the environment by the policy, thus preventing this policy from achieving performance similar to agents with access to image-based states directly.
We suppose that this bias is because g and f are trained jointly with learning objectives that are not optimized for the policy function.
Effect of Encoder Training Length in Interactive Phase
In the Interactive Phase , we train the trajectory and image encodings for N train steps before switching to only using a standard RL algorithm.
We now study how the length of this phase affects the final reward reached by the agent.
We evaluated our algorithm with N train set to a period of 0, 100K, 375K, and 1550K total training steps.
Updates to encoder parameters happen every N update = 50 steps.
The results of these experiments on the Reacher Hard task are presented in Figure 6b.
We observe that when the encoding functions are not updated during the Interactive Phase, learning does not proceed, as the rewards provided by the Alignment Phase are not sufficient.
Increasing the number of N train steps initially has a positive effect on average returns per episode, reaching close to the expert return per episode in the case of N train = 375K steps.
However, when the encoding functions are not frozen training during the Interactive Phase, the learned reward function becomes brittle due to changing state encoding.
Thus we observe that N train =
1550K steps eventually led to a collapse in the policy.
Conclusion
We present a new method of Imitation from Observation that relies on the encoding of agent behavior using exemplar demonstrations from visual observations.
This method uses self-supervised learning of states and sequences to progressively train a reward function for use by a reinforcement learning agent.
We demonstrate the strength of this method on a set of 7 simulated robotic tasks which have access to a limited set of expert demonstrations.
We note that our adopted problem framework: imitation from image observations, though important for its potential practical applications, is challenging from both the computational and observational perspectives for tasks that require precise control.
In this paper, we explore this challenge by training our agents with access to information from task-specific and pre-trained image encoders and show that the fidelity of the encoding function is critically important for downstream control tasks.
Our approach to increase the performance of the learned representation is to keep the data generated by the agent during the Interactive Phase.
In future work, we see expanding the sequence optimization functions so that they evaluate not just the two agent trajectory distributions we have now (expert and non-expert), but distributions of many performance levels which are generated as the agent trains.
A Neural network architectures
[Table]
[Table]
Conv + BNorm + LReLU
[Table]
TConv + BNorm + LReLU
[Table]
TConv + BNorm + LReLU
TConv + BNorm +
[Table]
[Table] [Table]
B Estimation of our models and baselines with absolute average episodic return
We detail in Tables 5 and 6 the non-scaled average episodic returns of our models and baselines on each of the 7 tasks of the experiments.
C Encoding function losses
During the alignment phase, we evaluate the intermediary loss terms L Z , L seq , L ae and L triplet , on the expert trajectories and random trajectories.
These different functions are used to bootstrap the encoding functions for the next phase and each have a well-defined learning objective as explained in
Section 2.
We detail the evolution of these loss terms during the alignment phase in Figure 8.
In the interactive phase, during the training of the imitation agent, the encoding functions are progressively
[Table] [Table]
[Figure]
trained on the new trajectories of the agent and on the expert
We present in the
Figure 7, the evolution of L seq loss function.
We can see a slight tendency for L seq to increase.
This effect is due to the sequence encoding function, which is increasingly unable to distinguish expert sequences from agent sequences.
D Hyperparameters
Hyperparameters of our method and the default values are presented in Table 7.
E Demonstrations
We present in Figure 9, examples of the execution of each of the tasks by the expert and the agent, under the same initial conditions.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table]
[Figure]
[Table] [Table]
Total
[Table]
of training steps during which encoders are trained
[Table]
375K
Periodicity of encoder parameter update (in training step) ( N update )
Total number of agent training steps (
[Table]
1.55 M
[Table] [Table]
Adam [15]
[Figure]