Original: Spectrum Random Masking for Generalization in Image-based Reinforcement Learning
Revised: Spectrum Random Masking for Generalization in Image-based Reinforcement Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Paper ID:
Revised: Paper ID:

Original: Abstract
Revised: Abstract

Original: Generalization in image-based reinforcement learning (RL) aims to learn a robustpolicy that could be applied directly on unseen visual environments, which is achallenging task since agents usually tend to overfit to their training environment.
Revised: Generalization in image-based reinforcement learning (RL) aims to learn a robustpolicy that could be applied directly on unseen visual environments, which is achallenging task since agents usually tend to overfit to their training environment.

Original: To handle this problem, a natural approach is to increase the data diversity byimage based augmentations.
Revised: To handle this problem, a natural approach is to increase the data diversity byimage based augmentations.

Original: However, different with most vision tasks such asclassification and detection, RL tasks are not always invariant to spatial basedaugmentations due to the entanglement of environment dynamics and visual appearance.
Revised: However, different with most vision tasks such asclassification and detection, RL tasks are not always invariant to spatial basedaugmentations due to the entanglement of environment dynamics and visual appearance.

Original: In this paper, we argue with two principles for augmentations in RL:
Revised: In this paper, we argue with two principles for augmentations in RL:

Original: First , the augmented observations should facilitate learning a universal policy,which is robust to various distribution shifts.
Revised: First , the augmented observations should facilitate learning a universal policy,which is robust to various distribution shifts.

Original: Second , the augmented data should beinvariant to the learning signals such as action and reward.
Revised: Second , the augmented data should beinvariant to the learning signals such as action and reward.

Original: Following these rules,we revisit image-based RL tasks from the view of frequency domain and propose anovel augmentation method, namely Spectrum Random Masking (SRM),whichis able to help agents to learn the whole frequency spectrum of observation forcoping with various distributions and compatible with the pre-collected action andreward corresponding to original observation.
Revised: Following these rules,we revisit image-based RL tasks from the view of frequency domain and propose anovel augmentation method, namely Spectrum Random Masking (SRM),whichis able to help agents to learn the whole frequency spectrum of observation forcoping with various distributions and compatible with the pre-collected action andreward corresponding to original observation.

Original: Extensive experiments conducted on
Revised: Extensive experiments conducted on

Original: DMControl Generalization Benchmark demonstrate the proposed SRM achievesthe state-of-the-art performance with strong generalization potentials.
Revised: DMControl Generalization Benchmark demonstrate the proposed SRM achievesthe state-of-the-art performance with strong generalization potentials.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement Learning (RL) from image-based observations is a task where representation anddecision-making are jointly learned from visual signals.
Revised: Reinforcement Learning (RL) from image-based observations is a task where representation anddecision-making are jointly learned from visual signals.

Original: Although this type of RL methods has madesignificant progresses due to the development of deep Convolution Neural Networks (CNNs), mostof them suffer from drastic performance decline on previously unseen environments [23, 42, 3]
Revised: Although this type of RL methods has madesignificant progresses due to the development of deep Convolution Neural Networks (CNNs), mostof them suffer from drastic performance decline on previously unseen environments [23, 42, 3]

Original: especially for image-based inputs [42, 31, 3, 46, 26].
Revised: especially for image-based inputs [42, 31, 3, 46, 26].

Original: The reason is that classical RL methods assumethe environments of training and testing are identical [16].
Revised: The reason is that classical RL methods assumethe environments of training and testing are identical [16].

Original: However, there usually exists significantdistributions divergences between them in real-world scenarios.
Revised: However, there usually exists significantdistributions divergences between them in real-world scenarios.

Original: The high dimensionality of visualinputs coupled with the shifted distributions of environment make the generalization of RL tasks evenmore challenging [29].
Revised: The high dimensionality of visualinputs coupled with the shifted distributions of environment make the generalization of RL tasks evenmore challenging [29].

Original: To handle this challenge, one natural way is to decrease the distribution gap of training and testing databy simulating the diversity of test environments in training.
Revised: To handle this challenge, one natural way is to decrease the distribution gap of training and testing databy simulating the diversity of test environments in training.

Original: Recent studies show that the appropriatedata augmentation could facilitate the generalization of the policy network [39].
Revised: Recent studies show that the appropriatedata augmentation could facilitate the generalization of the policy network [39].

Original: Most of these dataaugmentation methods have been carried out directly in the spatial domain [18], such as flipping androtation.
Revised: Most of these dataaugmentation methods have been carried out directly in the spatial domain [18], such as flipping androtation.

Original: However, the dynamic of environment has the disadvantages of lacking sensitivity to suchspatial-based data augmentation [27].
Revised: However, the dynamic of environment has the disadvantages of lacking sensitivity to suchspatial-based data augmentation [27].

Original: An example is illustrated in Fig.
Revised: An example is illustrated in Fig.

Original: 1: when horizontal flipping isapplied on observations, the corresponding left and right actions should be reversed, thus collectedrewards could not reflect the correct response.
Revised: 1: when horizontal flipping isapplied on observations, the corresponding left and right actions should be reversed, thus collectedrewards could not reflect the correct response.

Original: Considering above problem, instead of working in the spatial domain, we aim to study the generalization of image-based RL from a frequency domain perspective.
Revised: Considering above problem, instead of working in the spatial domain, we aim to study the generalization of image-based RL from a frequency domain perspective.

Original: Our motivation stems from
Revised: Our motivation stems from

Original: [Figure]
Revised: [Figure]

Original: √ √
Revised: √ √

Original: [Figure]
Revised: [Figure]

Original: the recent studies Fourier analysis in computer vision [40]: different types of spatial corruptionsinfluence the model robustness toward different frequency ranges.
Revised: the recent studies Fourier analysis in computer vision [40]: different types of spatial corruptionsinfluence the model robustness toward different frequency ranges.

Original: For example, the high frequencydata augmentations such as random Gaussian noise bias the model towards utilizing low frequencyinformation in the input, and improve robustness to corruptions with high frequencies.
Revised: For example, the high frequencydata augmentations such as random Gaussian noise bias the model towards utilizing low frequencyinformation in the input, and improve robustness to corruptions with high frequencies.

Original: Moreover,single spatial-based data augmentation rarely improves model robustness of corruption types withdifferent frequency characteristics.
Revised: Moreover,single spatial-based data augmentation rarely improves model robustness of corruption types withdifferent frequency characteristics.

Original: Inspired by this phenomenon, we conjecture that the observation’s
Revised: Inspired by this phenomenon, we conjecture that the observation’s

Original: Fourier statistics for the same RL task also vary across different environment distributions.
Revised: Fourier statistics for the same RL task also vary across different environment distributions.

Original: In Fig.
Revised: In Fig.

Original: 2,we visualize the observation’s spectrum magnitude of the the original environment as well as theaveraged delta between 4 shifted environment distributions on DeepMind Control suite [32].
Revised: 2,we visualize the observation’s spectrum magnitude of the the original environment as well as theaveraged delta between 4 shifted environment distributions on DeepMind Control suite [32].

Original: Theshifted environments show distinct spectrum difference with the original environment, with each onevarying at different frequency regions.
Revised: Theshifted environments show distinct spectrum difference with the original environment, with each onevarying at different frequency regions.

Original: Knowing that a deep model often tends to bias on certainfrequency band [37], we hypothesis that the diverse spectrum pattern across environment distributionsis the key challenge of image-based RL generalization.
Revised: Knowing that a deep model often tends to bias on certainfrequency band [37], we hypothesis that the diverse spectrum pattern across environment distributionsis the key challenge of image-based RL generalization.

Original: Based on the above analyses, we focus on the Frequency-based augmentation instead of the spatialbased one by introducing a Spectrum Random Masking (SRM) regularization, as shown in Fig.
Revised: Based on the above analyses, we focus on the Frequency-based augmentation instead of the spatialbased one by introducing a Spectrum Random Masking (SRM) regularization, as shown in Fig.

Original: Our proposed SRM can be easily compatible for most existing RL benchmarks.
Revised: Our proposed SRM can be easily compatible for most existing RL benchmarks.

Original: To prevent themodel from focusing on a certain frequency range on input images, SRM randomly discards partialfrequency of observations at training phase, and forces the policy to select the appropriate actionwith remaining information.
Revised: To prevent themodel from focusing on a certain frequency range on input images, SRM randomly discards partialfrequency of observations at training phase, and forces the policy to select the appropriate actionwith remaining information.

Original: In this way, we can increase the robustness to different corruptions(high, middle or low frequency characteristics) by considering the entire frequency distribution.
Revised: In this way, we can increase the robustness to different corruptions(high, middle or low frequency characteristics) by considering the entire frequency distribution.

Original: Specifically, an input observation is operated by three steps: (1) the Fast Fourier transformation (FFT),(2)
Revised: Specifically, an input observation is operated by three steps: (1) the Fast Fourier transformation (FFT),(2)

Original: Spectrum random masking, and (3) the inverse Fourier transformation (IFFT), which enhancesthe diversity of training observations while maintaining the main content.
Revised: Spectrum random masking, and (3) the inverse Fourier transformation (IFFT), which enhancesthe diversity of training observations while maintaining the main content.

Original: To summarize, this paper has the following contributions: (1) We provide a new frequency perspectivefor data augmentation in RL.
Revised: To summarize, this paper has the following contributions: (1) We provide a new frequency perspectivefor data augmentation in RL.

Original: In contrast to previous spatial-based methods, the proposed SRM areperformed consistently for task-agnostic RL environments and could deal with diverse distributionby focusing on the whole frequency information.
Revised: In contrast to previous spatial-based methods, the proposed SRM areperformed consistently for task-agnostic RL environments and could deal with diverse distributionby focusing on the whole frequency information.

Original: The proposed SRM is a plug-and-play methodthat does not require any extra parameters or any additional observations from the environment.
Revised: The proposed SRM is a plug-and-play methodthat does not require any extra parameters or any additional observations from the environment.

Original: Moreover, it is complementary to existing data augmentation and regularization approaches.
Revised: Moreover, it is complementary to existing data augmentation and regularization approaches.

Original: (3) Weconduct comprehensive experiments to demonstrate SRM could bring performance gains on variousimage-based environments with high generalization potentials.
Revised: (3) Weconduct comprehensive experiments to demonstrate SRM could bring performance gains on variousimage-based environments with high generalization potentials.

Original:  Background
Revised:  Background

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: A RL problem is typically described as a Markov Decision Process (MDP)
Revised: A RL problem is typically described as a Markov Decision Process (MDP)

Original: M = < S , A , P , R , γ > , where S and A denotes the state space and action space respectively.
Revised: M = < S , A , P , R , γ > , where S and A denotes the state space and action space respectively.

Original: P ( s t +1 | s t , a t ) is the state transition, R ( s t , a t ) is the reward function, and γ ∈ [0 , 1) is the discountfactor.
Revised: P ( s t +1 | s t , a t ) is the state transition, R ( s t , a t ) is the reward function, and γ ∈ [0 , 1) is the discountfactor.

Original: Here we consider a Partially Observable MDP (POMDP)
Revised: Here we consider a Partially Observable MDP (POMDP)

Original: M
Revised: M

Original: = <
Revised: = <

Original: O , A , P , R , γ > forthe continuous control task because S is often not directly obtained from raw images, where O isthe high-dimensional observation space of images.
Revised: O , A , P , R , γ > forthe continuous control task because S is often not directly obtained from raw images, where O isthe high-dimensional observation space of images.

Original: Following the convention [24], the POMDP isconverted into an MDP by stacking several consecutive environment frames { o t , o t − 1 , o t − 2 } into asingle state s t ∈ O .
Revised: Following the convention [24], the POMDP isconverted into an MDP by stacking several consecutive environment frames { o t , o t − 1 , o t − 2 } into asingle state s t ∈ O .

Original: [Figure]
Revised: [Figure]

Original: The goal of typical RL is to find an optimal policy π ∗ θ which maximizes the expected cumulativereward over the entire distribution of MDPs:
Revised: The goal of typical RL is to find an optimal policy π ∗ θ which maximizes the expected cumulativereward over the entire distribution of MDPs:

Original: [Equation]
Revised: [Equation]

Original: where π θ is parameterized by the learnable parameters θ .
Revised: where π θ is parameterized by the learnable parameters θ .

Original: In this work, rather than learning an optimalpolicy on a single MDP, we consider a generalizable π θ , which is able to obtain high discountedreturn on a set of MDPs M =
Revised: In this work, rather than learning an optimalpolicy on a single MDP, we consider a generalizable π θ , which is able to obtain high discountedreturn on a set of MDPs M =

Original: <
Revised: <

Original: O , A , P , R , γ > , where the observations in O are unseen in training.
Revised: O , A , P , R , γ > , where the observations in O are unseen in training.

Original: Soft Actor-Critic.
Revised: Soft Actor-Critic.

Original: Soft Actor-Critic (SAC) [8, 9] is a popular off-policy method for continuouscontrol tasks.
Revised: Soft Actor-Critic (SAC) [8, 9] is a popular off-policy method for continuouscontrol tasks.

Original: It learns a state-action value function Q ( s, a ) and a stochastic policy π ( a | s ) to find theoptimal policy based on the maximum entropy RL framework:
Revised: It learns a state-action value function Q ( s, a ) and a stochastic policy π ( a | s ) to find theoptimal policy based on the maximum entropy RL framework:

Original: [Equation]
Revised: [Equation]

Original: where α is the temperature parameter which determines the relative importance of the entropy termagainst the reward.
Revised: where α is the temperature parameter which determines the relative importance of the entropy termagainst the reward.

Original: α = 0 degenerates SAC into conventional RL.
Revised: α = 0 degenerates SAC into conventional RL.

Original: H ( · ) provides a substantialimprovement in exploration and robustness by enabling the probability of action output to be diverseas much as possible, rather than being concentrated on one action.
Revised: H ( · ) provides a substantialimprovement in exploration and robustness by enabling the probability of action output to be diverseas much as possible, rather than being concentrated on one action.

Original: Specially, Q ( s, a ) is approximatedby minimizing the soft Bellman residual:
Revised: Specially, Q ( s, a ) is approximatedby minimizing the soft Bellman residual:

Original: [Equation]
Revised: [Equation]

Original: where V s t +1 is the soft target value network approximated as:
Revised: where V s t +1 is the soft target value network approximated as:

Original: [Equation]
Revised: [Equation]

Original: where Q is the target Q function, and the weights can be an exponentially moving average of theweights in Q .
Revised: where Q is the target Q function, and the weights can be an exponentially moving average of theweights in Q .

Original: In the policy improvement step, the policy is updated by minimizing the divergencebetween the policy and the exponential of the soft Q function:
Revised: In the policy improvement step, the policy is updated by minimizing the divergencebetween the policy and the exponential of the soft Q function:

Original: [Equation]
Revised: [Equation]

Original:  Methodology3.1 Spectrum Random Masking
Revised:  Methodology3.1 Spectrum Random Masking

Original: We firstly introduce how to generate new samples by spectrum random masking strategy in thissection.
Revised: We firstly introduce how to generate new samples by spectrum random masking strategy in thissection.

Original: Suppose we have a grayscale image observation 1 o i ∈
Revised: Suppose we have a grayscale image observation 1 o i ∈

Original: R H × W , the Fourier domain spectrum
Revised: R H × W , the Fourier domain spectrum

Original: F ( o i ) can be calculated by fast Fourier transformation [25]:
Revised: F ( o i ) can be calculated by fast Fourier transformation [25]:

Original: [Equation]
Revised: [Equation]

Original: (6) where u ∈ [0 , H − 1] and v ∈ [0 , W − 1] are horizontal and vertical indexes of frequency domaincomponents, and o i ( h, w ) is the pixel value at position ( h, w ) .
Revised: (6) where u ∈ [0 , H − 1] and v ∈ [0 , W − 1] are horizontal and vertical indexes of frequency domaincomponents, and o i ( h, w ) is the pixel value at position ( h, w ) .

Original: Note the subscript i is just the index
Revised: Note the subscript i is just the index

Original: RGB
Revised: RGB

Original: , we the operation each color
Revised: , we the operation each color

Original: [Figure]
Revised: [Figure]

Original: of the sample and irrelevant with the time step.
Revised: of the sample and irrelevant with the time step.

Original: For better processing and visualization, we use theshifted version of F ( o i ) with the lowest frequency component placed in the center of the spectrum.
Revised: For better processing and visualization, we use theshifted version of F ( o i ) with the lowest frequency component placed in the center of the spectrum.

Original: After transforming images from spatial to frequency domain, we construct a binary masking matrix
Revised: After transforming images from spatial to frequency domain, we construct a binary masking matrix

Original: M ∈ { 0 , 1 } H × W .
Revised: M ∈ { 0 , 1 } H × W .

Original: We introduce two radius parameters r 1 , r 2 ∈ [0 , 0 . 5] to control the position andratio of the spectral neighborhood to be masked 2 ( r 1 < r 2 ).
Revised: We introduce two radius parameters r 1 , r 2 ∈ [0 , 0 . 5] to control the position andratio of the spectral neighborhood to be masked 2 ( r 1 < r 2 ).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, M is constructedby setting the region in the annulus between two concentric circles (with radius r 1 and r 2 ) to 0.∆ r
Revised: 3, M is constructedby setting the region in the annulus between two concentric circles (with radius r 1 and r 2 ) to 0.∆ r

Original: = r 2 − r 1 decides the size of mask regions.
Revised: = r 2 − r 1 decides the size of mask regions.

Original: The area ratio of masking region is defined asr p =
Revised: The area ratio of masking region is defined asr p =

Original: r 22 − r 21 .
Revised: r 22 − r 21 .

Original: Hence we propose 3 typical types of masking strategies: erasing, noise and swapping.
Revised: Hence we propose 3 typical types of masking strategies: erasing, noise and swapping.

Original: If the frequency component lies between r 1 and r 2 , it will be masked; otherwise, the frequencysignals will be kept:
Revised: If the frequency component lies between r 1 and r 2 , it will be masked; otherwise, the frequencysignals will be kept:

Original: [Equation]
Revised: [Equation]

Original: (7) where Z is calculated by the specific masking strategy.
Revised: (7) where Z is calculated by the specific masking strategy.

Original: For the erasing strategy, we set Z to 0 H × W ,and only the non-masked frequency components will be fully explored in follow-up training.
Revised: For the erasing strategy, we set Z to 0 H × W ,and only the non-masked frequency components will be fully explored in follow-up training.

Original: Therandom noise strategy sets Z to a random noise image.
Revised: Therandom noise strategy sets Z to a random noise image.

Original: The swapping strategy replaces the removedfrequency components with the same frequency components from another image I by setting Z
Revised: The swapping strategy replaces the removedfrequency components with the same frequency components from another image I by setting Z

Original: = I ,which draws inspiration from Cutmix [41] and Mixup [45].
Revised: = I ,which draws inspiration from Cutmix [41] and Mixup [45].

Original: Considering that the third-party data I isnot always available, during training, we randomly shuffle the training batch and pick I =
Revised: Considering that the third-party data I isnot always available, during training, we randomly shuffle the training batch and pick I =

Original: o j ( j ̸ = i )from the shuffled batch, where o j has the same batch index with o i .
Revised: o j ( j ̸ = i )from the shuffled batch, where o j has the same batch index with o i .

Original: The masking strategy also influences the supervised signals since it is correlated with o i .
Revised: The masking strategy also influences the supervised signals since it is correlated with o i .

Original: Supposethat a i , r
Revised: Supposethat a i , r

Original: i is the supervised signals for observation o i , where a i and r i stands for action and reward,respectively.
Revised: i is the supervised signals for observation o i , where a i and r i stands for action and reward,respectively.

Original: For erasing and random noise strategy, a i and r i are kept unchanged.
Revised: For erasing and random noise strategy, a i and r i are kept unchanged.

Original: For the swappingstrategy, the supervised signals depended on the area ratio of masking region r p .
Revised: For the swappingstrategy, the supervised signals depended on the area ratio of masking region r p .

Original: If r p < 0 . 5 , wekeep a i and r i unchanged, else we replace a i , r i with a j and r j .
Revised: If r p < 0 . 5 , wekeep a i and r i unchanged, else we replace a i , r i with a j and r j .

Original: Once the masked Fourier spectrum ˆ F ( o i )( u, v ) is obtained, we inverse shift and restore it to thespatial domain by the corresponding inverse Fourier transformation F − 1 :
Revised: Once the masked Fourier spectrum ˆ F ( o i )( u, v ) is obtained, we inverse shift and restore it to thespatial domain by the corresponding inverse Fourier transformation F − 1 :

Original: [Equation]
Revised: [Equation]

Original: For observation o i , its next observation o i ′ also goes through the same SRM operation as o i .
Revised: For observation o i , its next observation o i ′ also goes through the same SRM operation as o i .

Original: Algorithm 1 details how to generate new samples by SRM.
Revised: Algorithm 1 details how to generate new samples by SRM.

Original: 3.2 Reinforcement Learning with SRM
Revised: 3.2 Reinforcement Learning with SRM

Original: In off-policy RL, observations are sampled from a replay buffer.
Revised: In off-policy RL, observations are sampled from a replay buffer.

Original: One intuitive way to use dataaugmentation in RL is to perform augmentation before passing them to the agent for training.
Revised: One intuitive way to use dataaugmentation in RL is to perform augmentation before passing them to the agent for training.

Original: Input : Sample transitions a r o i
Revised: Input : Sample transitions a r o i

Original: W .
Revised: W .

Original: Output : Augmented transitions { o i , ˆ a i , ˆ r
Revised: Output : Augmented transitions { o i , ˆ a i , ˆ r

Original: i
Revised: i

Original: ˆ o ′ i | i
Revised: ˆ o ′ i | i

Original: = 1 , · · · , K }
Revised: = 1 , · · · , K }

Original: Init : r 1 ← Rand (0 , 0 . 5) , r 2 ← Rand ( r 1 , 0 . 5) , ∆ r = r 2 − r 1 , r p = r 22 − r 21 .
Revised: Init : r 1 ← Rand (0 , 0 . 5) , r 2 ← Rand ( r 1 , 0 . 5) , ∆ r = r 2 − r 1 , r p = r 22 − r 21 .

Original: 1: for i = 1 , · · · , K do 2:
Revised: 1: for i = 1 , · · · , K do 2:

Original: Fast Fourier transformation according to Eq.
Revised: Fast Fourier transformation according to Eq.

Original: 6: F ( o i )
Revised: 6: F ( o i )

Original: ←
Revised: ←

Original: o i 3: if Masking strategies is ‘Erasing’ then 4: ˆ F ( o i ) =
Revised: o i 3: if Masking strategies is ‘Erasing’ then 4: ˆ F ( o i ) =

Original: M · F ( o i ) + (1 − M ) · F ( 0 ) , ˆ a i ← a i , ˆ r i ← r i 5: else if Masking strategies is ‘Noise’ then 6: ˆ F ( o i ) =
Revised: M · F ( o i ) + (1 − M ) · F ( 0 ) , ˆ a i ← a i , ˆ r i ← r i 5: else if Masking strategies is ‘Noise’ then 6: ˆ F ( o i ) =

Original: M · F ( o i ) + (1 − M ) · F ( N ) , ˆ a i ← a i , ˆ r i ← r i 7: else if Masking strategies is ‘Swapping’ then 8:
Revised: M · F ( o i ) + (1 − M ) · F ( N ) , ˆ a i ← a i , ˆ r i ← r i 7: else if Masking strategies is ‘Swapping’ then 8:

Original: Random Shuffle Batch: { o j , a j , r j , o ′ j | j = 1 , · · · , K } ˆ F ( o i ) =
Revised: Random Shuffle Batch: { o j , a j , r j , o ′ j | j = 1 , · · · , K } ˆ F ( o i ) =

Original: M · F ( o i ) + (1 − M ) · F ( o j ) 9: if r p ≤ 0 .
Revised: M · F ( o i ) + (1 − M ) · F ( o j ) 9: if r p ≤ 0 .

Original: 5 then 10: ˆ a i ← a i , ˆ r
Revised: 5 then 10: ˆ a i ← a i , ˆ r

Original: i ← r i 11: else
Revised: i ← r i 11: else

Original: 12: ˆ a i ← a j , ˆ r i ← r j 13: end if 14: end if 15:
Revised: 12: ˆ a i ← a j , ˆ r i ← r j 13: end if 14: end if 15:

Original: Inverse Fourier transformation according to Eq.
Revised: Inverse Fourier transformation according to Eq.

Original: 8: ˆ o i ← ˆ F ( o i ) 16:
Revised: 8: ˆ o i ← ˆ F ( o i ) 16:

Original: Execute Steps (2) ∼ (15) with o ′ i to get ˆ o ′ i 17: end for
Revised: Execute Steps (2) ∼ (15) with o ′ i to get ˆ o ′ i 17: end for

Original: Meanwhile, the augmentation procedure on stacked consecutive state frames s t = { o t , o t − 1 , o t − 2 }should be consistent to keep temporal information (such as positions and velocities) invariant.
Revised: Meanwhile, the augmentation procedure on stacked consecutive state frames s t = { o t , o t − 1 , o t − 2 }should be consistent to keep temporal information (such as positions and velocities) invariant.

Original: Givenan augmentation function τ ( · , v ) with parameter v ∈ V , the augmented observation ˆ s t ∈
Revised: Givenan augmentation function τ ( · , v ) with parameter v ∈ V , the augmented observation ˆ s t ∈

Original: ˆ O of theoriginal observation s t ∈ O can be obtained by ˆ s t =
Revised: ˆ O of theoriginal observation s t ∈ O can be obtained by ˆ s t =

Original: τ ( s t , v ) .
Revised: τ ( s t , v ) .

Original: Optimally, τ : O × V → ˆ O shouldmaintain Q-values and policy π of the agent:
Revised: Optimally, τ : O × V → ˆ O shouldmaintain Q-values and policy π of the agent:

Original: [Equation]
Revised: [Equation]

Original: where a t ∈ A is the action of s t .
Revised: where a t ∈ A is the action of s t .

Original: However, practically it is unstable to learn value and policyfunctions even from the original observations, not to mention the augmented data that vary wildly inappearances.
Revised: However, practically it is unstable to learn value and policyfunctions even from the original observations, not to mention the augmented data that vary wildly inappearances.

Original: As a result, only several weak data augmentations such as random shift and randomcropping could be employed in a plain way in RL optimization, as shown in DrQ [39] and RAD [18].
Revised: As a result, only several weak data augmentations such as random shift and randomcropping could be employed in a plain way in RL optimization, as shown in DrQ [39] and RAD [18].

Original: In contrast, strong data augmentations such as random convolution and rotation are always beenapplied in a specific way.
Revised: In contrast, strong data augmentations such as random convolution and rotation are always beenapplied in a specific way.

Original: In SVEA [12], both original and augmented images are used to optimizethe Q-function, but only the former are adopted to compute Q-target of the Bellman equation.
Revised: In SVEA [12], both original and augmented images are used to optimizethe Q-function, but only the former are adopted to compute Q-target of the Bellman equation.

Original: In
Revised: In

Original: SECANT [6], expert policy with weak augmentations guides student policy with strong augmentationsfor learning robust representations.
Revised: SECANT [6], expert policy with weak augmentations guides student policy with strong augmentationsfor learning robust representations.

Original: Attractively, as we will show in the experiment section, SRM hasthe merit of both weak and strong data augmentations: it can be plugged into any image-based RLmethods in a plain way, while providing significant performance improvement.
Revised: Attractively, as we will show in the experiment section, SRM hasthe merit of both weak and strong data augmentations: it can be plugged into any image-based RLmethods in a plain way, while providing significant performance improvement.

Original: During training, we apply SRM with a probability of 0.5 on each batch of observations.
Revised: During training, we apply SRM with a probability of 0.5 on each batch of observations.

Original: No dataaugmentation is applied at test time.
Revised: No dataaugmentation is applied at test time.

Original: To evaluate the effectiveness of SRM, we adopt two popularmethods DrQ [39] and SVEA [12] as our base algorithms.
Revised: To evaluate the effectiveness of SRM, we adopt two popularmethods DrQ [39] and SVEA [12] as our base algorithms.

Original: The former is a naive method where dataaugmentations are plugged into RL algorithms directly, and the corresponding SAC objectives arerespectively replaced by:
Revised: The former is a naive method where dataaugmentations are plugged into RL algorithms directly, and the corresponding SAC objectives arerespectively replaced by:

Original: [Equation]
Revised: [Equation]

Original: The latter is delicately designed where Q-target is computed strictly from unaugmented data toalleviate the problem of high-variance, and its goal is:
Revised: The latter is delicately designed where Q-target is computed strictly from unaugmented data toalleviate the problem of high-variance, and its goal is:

Original: [Equation]
Revised: [Equation]

Original: By jointly solving the Q-function with policy and value network in Eqs.
Revised: By jointly solving the Q-function with policy and value network in Eqs.

Original: (4) and (5), the RL systemshas the potential to learn from diverse observations for policy learning.
Revised: (4) and (5), the RL systemshas the potential to learn from diverse observations for policy learning.

Original:  Experiments
Revised:  Experiments

Revised: [Figure]

Original: 4.1 Task Setup and Evaluation Protocol
Revised: 4.1 Task Setup and Evaluation Protocol

Original: Environments We conduct our experiment on 5 tasks from DeepMind Control Suite (DMControl) [32], which offers a set of challenging continuous control tasks and has been widely used as acommon benchmark for visual RL [10, 1].
Revised: Environments We conduct our experiment on 5 tasks from DeepMind Control Suite (DMControl) [32], which offers a set of challenging continuous control tasks and has been widely used as acommon benchmark for visual RL [10, 1].

Original: There are 2 kinds of observations in DMControl: statevector-based and image-based, and the latter is used in our experiment.
Revised: There are 2 kinds of observations in DMControl: statevector-based and image-based, and the latter is used in our experiment.

Original: It has been evidenced that thelearning of state-based observations is easier than high dimensional image-based observations [17].
Revised: It has been evidenced that thelearning of state-based observations is easier than high dimensional image-based observations [17].

Original: To evaluate the generalization, we train the agent from DMControl and report results tested on
Revised: To evaluate the generalization, we train the agent from DMControl and report results tested on

Original: DMControl Generalization Benchmark (DMControl-GB) [13], which includes four distinct testdistributions with slight/strong color/video attack: color-easy, color-hard, video-easy and video-hard..
Revised: DMControl Generalization Benchmark (DMControl-GB) [13], which includes four distinct testdistributions with slight/strong color/video attack: color-easy, color-hard, video-easy and video-hard..

Original: Implementation Details For a fair comparison, we implement all methods following [12], wherethe same hyperparameters and network architecture are adopted.
Revised: Implementation Details For a fair comparison, we implement all methods following [12], wherethe same hyperparameters and network architecture are adopted.

Original: We use a 11-layer feed-forwardconvolution network as the shared encoder, which is followed by independent linear projections forthe actor and critic.
Revised: We use a 11-layer feed-forwardconvolution network as the shared encoder, which is followed by independent linear projections forthe actor and critic.

Original: During training, the masking ratio and position of SRM are randomly chosen,and the ranges of r 1 and ∆ r are set as [0 , 0 . 5] and [0 , 0 .
Revised: During training, the masking ratio and position of SRM are randomly chosen, andthe ranges of r 1 and ∆ r are set as [0 , 0 . 5] and [0 , 0 .

Original: 1] for each batch of observations, respectively.
Revised: for each batch of observations, respectively.

Original: Note SRM is a data augmentation method and needs to combine with other learning baselines.
Revised: Note SRM is a data augmentation method and needs to combine with other learning baselines.

Original: Tovalidate the effectiveness, 3 state-of-the-art baselines are employed: DrQ [39], SVEA-C [12] and
Revised: Tovalidate the effectiveness, 3 state-of-the-art baselines are employed: DrQ [39], SVEA-C [12] and

Original: SVEA-O [12], denoted as “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” respectively.
Revised: SVEA-O [12], denoted as “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” respectively.

Original: Oand C represent random convolution and random overlay respectively, which are set as the defaultaugmentation used in SVEA [12].
Revised: Oand C represent random convolution and random overlay respectively, which are set as the defaultaugmentation used in SVEA [12].

Original: For DrQ, SVEA-C and SVEA-O, the better performances betweenthe reported ones of the original paper and our re-run ones using the source codes are reported in ourexperiment.
Revised: For DrQ, SVEA-C and SVEA-O, the better performances betweenthe reported ones of the original paper and our re-run ones using the source codes are reported in ourexperiment.

Original: We provide 3 candidate masking strategies erasing, noise and swapping in Algorithm 1,all of them can achieve higher performance than baseline as shown in Table 5 .
Revised: We provide 3 candidate masking strategies erasing, noise and swapping in Algorithm 1,all of them can achieve higher performance than baseline as shown in Table 5 .

Original: Since erasing performsbetter than others, it is chosen as the default masking strategy in the following experiments if nototherwise mentioned.
Revised: Since erasing performsbetter than others, it is chosen as the default masking strategy in the following experiments if nototherwise mentioned.

Original: All methods are trained for 500K frames.
Revised: All methods are trained for 500K frames.

Original: We run 5 times with random seedsand report the averaged results.
Revised: We run 5 times with random seedsand report the averaged results.

Original: [Table]
Revised: [Table]

Original: Hard

Original: [Table]

Original: +

Original: [Table]

Original: walker,

Original: [Table]
Revised: [Table]

Original: walk

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: finger,

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 4.2 Comparisons
Revised: 4.2 Comparisons

Original: Comparisons with state-of-the-arts The comparisons are exhibited in Table 1.
Revised: Comparisons with state-of-the-arts The comparisons are exhibited in Table 1.

Original: For the directcomparison, “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” outperform DrQ, SVEA-C and
Revised: For the directcomparison, “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” outperform DrQ, SVEA-C and

Original: SVEA-O with a clear margin, respectively.
Revised: SVEA-O with a clear margin, respectively.

Original: In addition, the usage of SRM reduces the standarddeviation and shows the strong leaning stability.
Revised: In addition, the usage of SRM reduces the standarddeviation and shows the strong leaning stability.

Original: It indicates that performing augmentation in thefrequency domain is effective for image-based RL tasks.
Revised: It indicates that performing augmentation in thefrequency domain is effective for image-based RL tasks.

Original: In addition, the performance of othermethods such as CURL [17], PAD [11], RAD [18] and SODA [13] are also reported.
Revised: In addition, the performance of othermethods such as CURL [17], PAD [11], RAD [18] and SODA [13] are also reported.

Original: Since ourmethod is implemented with different learning baselines with them, the results are just provided forreference.
Revised: Since ourmethod is implemented with different learning baselines with them, the results are just provided forreference.

Original: Overall, it is evident that SRM provides new state-of-the-art performance for the task.
Revised: Overall, it is evident that SRM provides new state-

Original: Comparisons with spatial-based data augmentation Here we compare SRM with 5 spatial-baseddata augmentation methods and 1 removal regularization strategy, including random convolution [19], random rotation [7], random cutout [4], random flip [18] and dropout [30].
Revised: Comparisons with spatial-based data augmentation Here we compare SRM with 5 spatial-baseddata augmentation methods and 1 removal regularization strategy, including random convolution [19],random rotation [7], random cutout [4], random flip [18] and dropout [30].

Original: DrQ is employed as thebaseline learning method.
Revised: DrQ is employed as thebaseline learning method.

Original: When utilizing each data augmentation method, 50% of frames are remainun-augmented.
Revised: When utilizing each data augmentation method, 50% of frames are remainun-augmented.

Original: This is because we find that directly using all kinds of data augmentation is harmfulto policy stability.
Revised: This is because we find that directly using all kinds of data augmentation is harmfulto policy stability.

Original: The dropout layer is added after every nonlinearity layer in the MLP of actor andcritic networks and the dropout probability is set to 0.1.
Revised: The dropout layer is added after every nonlinearity layer in the MLP of actor andcritic networks and the dropout probability is set to 0.1.

Original: Since the video hard task is extremely hard and the learning of DrQ is oscillating, we show learningcurves of the different data augmentations in color easy, color hard and video easy.
Revised: Since the video hard task is extremely hard and the learning of DrQ is oscillating, we show learningcurves of the different data augmentations in color easy, color hard and video easy.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4,the spatial data augmentation methods are less effective and even degrade performance, especiallyfor flip and rotation.
Revised: 4,the spatial data augmentation methods are less effective and even degrade performance, especiallyfor flip and rotation.

Original: The reason is that these augmentations inevitably changes the environmentdynamics, leading to inaccurate state transitions.
Revised: The reason is that these augmentations inevitably changes the environmentdynamics, leading to inaccurate state transitions.

Original: In contrast, SRM outperforms the baseline in mostscenarios and achieve comparable results in the rest scenarios.
Revised: In contrast, SRM outperforms the baseline in mostscenarios and achieve comparable results in the rest scenarios.

Original: It demonstrates that the generalizationability of frequency based SRM is stronger than its spatial-based counterparts.
Revised: It demonstrates that the generalizationability of frequency based SRM is stronger than its spatial-based counterparts.

Original: Note that the “randomcov” performs well on color hard and is less effective than SRM in other situations.
Revised: Note that the “randomcov” performs well on color hard and is less effective than SRM in other situations.

Original: The reason is thatapplying random convolution mainly changes the low frequency components (i.e., colors and textures)of an image, which makes the agent robust to color changes.
Revised: The reason is thatapplying random convolution mainly changes the low frequency components (i.e., colors and textures)of an image, which makes the agent robust to color changes.

Original: However, when facing environmentsthat differs significantly on higher frequency components, random convolution becomes less effective.
Revised: However, when facing environmentsthat differs significantly on higher frequency components, random convolution becomes less effective.

Original: Compared with “random cov”, SRM works more generally to different scenarios.
Revised: Compared with “random cov”, SRM works more generally to different scenarios.

Original: Fig.
Revised: Fig.

Original: 4 also tells that, although the removal based methods dropout and cutout has been proved to beable to improve generalization for discriminative models, they decrease the performance of RL tasks.
Revised: 4 also tells that, although the removal based methods dropout and cutout has been proved to beable to improve generalization for discriminative models, they decrease the performance of RL tasks.

Original: For cutout, randomly cutting a region of an input image is prone to information loss, especially whenthe target region (e.g., the cartpole in cartpole swingup task) is removed.
Revised: For cutout, randomly cutting a region of an input image is prone to information loss, especially whenthe target region (e.g., the cartpole in cartpole swingup task) is removed.

Original: The performance decreaseof dropout can be attributed to the chaotic nature of the RL training when facing with random droppedhidden units.
Revised: The performance decreaseof dropout can be attributed to the chaotic nature of the RL training when facing with random droppedhidden units.

Original: In contrast, each frequency component in the Fourier spectrum has the global vision ofthe whole image, thus SRM does not remove critical image regions.
Revised: In contrast, each frequency component in the Fourier spectrum has the global vision ofthe whole image, thus SRM does not remove critical image regions.

Original: Meanwhile, as an augmentationmethod, SRM also does not influence hidden units of networks.
Revised: Meanwhile, as an augmentationmethod, SRM also does not influence hidden units of networks.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Original: Spectrum Masking Strategies There are four different frequency masking strategies includingerasing, noise, swapping and a mixture of the first three.
Revised: Spectrum Masking Strategies There are four different frequency masking strategies includingerasing, noise, swapping and a mixture of the first three.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, we study the choiceof masking value on the performance of RL and have the following observations: First, all of thesemasking strategies are able to improve the average performance.
Revised: 5, we study the choice [Figure] of masking value on the performance of RL and have the following observations: First, all of thesemasking strategies are able to improve the average performance.

Original: This benefits from the mechanism of [Figure] random masking which attempt to strengthen the whole frequency learning while suppress overfittingto certain frequency range.
Revised: This benefits from the mechanism ofrandom masking which attempt to strengthen the whole frequency learning while suppress overfittingto certain frequency range.

Original: This also demonstrates that the agent may underfit to certain frequencyrange and overfit to other range when directly training with original observations.
Revised: This also demonstrates that the agent may underfit to certain frequencyrange and overfit to other range when directly training with original observations.

Original: Second, with theincreasing complexity of frequency disturbance, the performance of mixture is sometimes muchworse than solely using erasing or swapping.
Revised: Second, with theincreasing complexity of frequency disturbance, the performance of mixture is sometimes muchworse than solely using erasing or swapping.

Original: This phenomenon can be attributed to introduce morevariance by mixture.
Revised: This phenomenon can be attributed to introduce morevariance by mixture.

Original: Spectrum Masking Position
Revised: Spectrum Masking Position

Original: To explore how the low and high frequency affect policy training, weset r 1 to 0.1, 0.2, 0.3, 0.4 respectively, and ∆ r to 0 .
Revised: To explore how the low and high frequency affect policy training, weset r 1 to 0.1, 0.2, 0.3, 0.4 respectively, and ∆ r to 0 .

Original: According to Frequency Principle (F-Principle),
Revised: According to Frequency Principle (F-Principle),

Original: DNN tends to captures the frequency from low to high components in ascending complexity order.
Revised: DNN tends to captures the frequency from low to high components in ascending complexity order.

Original: This is consistent with our results as shown in Fig.
Revised: This is consistent with our results as shown in Fig.

Original: At the early training stage, masking lowfrequency lead to slower improvement, while masking high frequency leads to a similar improvementwith the non-SRM baseline.
Revised: At the early training stage, masking lowfrequency lead to slower improvement, while masking high frequency leads to a similar improvementwith the non-SRM baseline.

Original: In the later period, remaining high frequency is able to achieve higherresults than remaining low frequency.
Revised: In the later period, remaining high frequency is able to achieve higherresults than remaining low frequency.

Original: Moreover, SRM could get the best results in most benchmarksthan fixed masking position.
Revised: Moreover, SRM could get the best results in most benchmarksthan fixed masking position.

Original: It demonstrates that learning frequency from low to high uniformly in arandom way is beneficial.
Revised: It demonstrates that learning frequency from low to high uniformly in arandom way is beneficial.

Original: Spectrum Masking Ratio To better control the masking ratio, we set △ r to 0.05, 0.1,0.2,0.3 and 0.4respectively.
Revised: Spectrum Masking Ratio To better control the masking ratio, we set △ r to 0.05, 0.1,0.2,0.3 and 0.4respectively.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: Notably, SRM with random masking ratio consistentlyoutperforms the DrQ baseline under all parameters setting, especially for harder environment.
Revised: Notably, SRM with random masking ratio consistentlyoutperforms the DrQ baseline under all parameters setting, especially for harder environment.

Original: Inaddition, we can observe that masking ratio has little influence on original environment, which can beattributed to the remaining of a decent percentage of un-augmented frames.
Revised: Inaddition, we can observe that masking ratio has little influence on original environment, which can beattributed to the remaining of a decent percentage of un-augmented frames.

Original:  Related works
Revised:  Related works

Original: Generalization in RL draws increasing attention in recent years.
Revised: Generalization in RL draws increasing attention in recent years.

Original: Existing methods can be roughlycategorized into two groups: (1) increasing distribution similarity between training and testingenvironments by augmentation methods (such as data augmentation and environment generation) [15,19, 39, 17, 18, 13, 27] and (2) handling differences between the training and testing MDPs (suchas regularisation) [14, 21, 43, 44, 5, 1, 20, 34, 22, 28, 35].
Revised: Existing methods can be roughlycategorized into two groups: (1) increasing distribution similarity between training and testingenvironments by augmentation methods (such as data augmentation and environment generation) [15,19, 39, 17, 18, 13, 27] and (2) handling differences between the training and testing MDPs (suchas regularisation) [14, 21, 43, 44, 5, 1, 20, 34, 22, 28, 35].

Original: Our method belongs to the first groupbut works in the frequency domain, which is ignored by most current RL methods.
Revised: Our method belongs to the first groupbut works in the frequency domain, which is ignored by most current RL methods.

Original: To explain thegeneralization behaviors of CNN, recent works in computer vision also provide new insights fromthe frequency domain aspect [37, 33, 40, 38, 36, 2].
Revised: To explain thegeneralization behaviors of CNN, recent works in computer vision also provide new insights fromthe frequency domain aspect [37, 33, 40, 38, 36, 2].

Original: However, they mainly consider the domain shiftbetween fixed datasets on discrimination tasks (e.g., classification and segmentation).
Revised: However, they mainly consider the domain shiftbetween fixed datasets on discrimination tasks (e.g., classification and segmentation).

Original: In contrast, weconsider the dynamic environment in RL as well as the joint representation learning and decisionmaking process.
Revised: In contrast, weconsider the dynamic environment in RL as well as the joint representation learning and decisionmaking process.

Original: Moreover, CNN tends to rely on either low or high frequency components forfixed tasks [37], existing methods rarely consider regularizing the model on all frequency ranges.
Revised: Moreover, CNN tends to rely on either low or high frequency components forfixed tasks [37], existing methods rarely consider regularizing the model on all frequency ranges.

Original: Differently, our method considers the broad frequency difference between different RL environmentdistributions and regularizes the model on all frequency ranges uniformly.
Revised: Differently, our method considers the broad frequency difference between different RL environmentdistributions and regularizes the model on all frequency ranges uniformly.

Original:  Conclusions
Revised:  Conclusions

Original: In this work, we investigated the generalization of image-based RL from a frequency domainperspective.
Revised: In this work, we investigated the generalization of image-based RL from a frequency domainperspective.

Original: Motivated by the distinct spectral impacts of different environment distribution shifts,we develop SRM, a simple yet effective frequency domain augmentation method
Revised: Motivated by the distinct spectral impacts of different environment distribution shifts,we develop SRM, a simple yet effective frequency domain augmentation method.

Original: Experiments show that SRM improves model robustness under various distribution shifts without violating theenvironment dynamics.
Revised: Experimentsshow that SRM improves model robustness under various distribution shifts without violating theenvironment dynamics.

Original: The effectiveness of SRM highlights a promising future research direction to take into account the frequency domain intrinsic mechanism of RL models.
Revised: The effectiveness of SRM highlights a promising future research directionto take into account the frequency domain intrinsic mechanism of RL models.

Original: As RL tasks areessentially different from vision tasks, we believe that different conclusions will be made to providefurther intuition and benefit the model performance.
Revised: As RL tasks areessentially different from vision tasks, we believe that different conclusions will be made to providefurther intuition and benefit the model performance.

Original: Limitations: As an image-based augmentation method, our proposed SRM resort to enhancing thediversity of image-base observations while maintaining the crucial reward-related content, whichfacing challenges when image observations are incomplete or only state information can be observed.
Revised: Limitations: As an image-based augmentation method, our proposed SRM resort to enhancing thediversity of image-base observations while maintaining the crucial reward-related content, whichfacing challenges when image observations are incomplete or only state information can be observed.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See conclusions.
Revised: [Yes] See conclusions.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Seesupplemental material. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] Seesupplemental material.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplemental material.
Revised: [Yes] See supplemen- tal material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.(c)

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Tab. 1.
Revised: [Yes] See Tab. 1.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See supplemental material.4.
Revised: [Yes] See supplemental material.4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.

Original: Weuse public environments with proper citations.
Revised: We use public environments with proper citations.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data used in this paper does not containsoffensive content.
Revised: The data used in this paper does not contains offensive content.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: , if

Original: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]