Original: Spectrum Random Masking for Generalization in Image-based Reinforcement Learning
Revised: Spectrum Random Masking for Generalization in Image-based Reinforcement Learning

Original: Anonymous Author(s)

Original: Paper ID:

Original: Abstract
Revised: Abstract

Original: Generalization in image-based reinforcement learning (RL) aims to learn a robustpolicy that could be applied directly on unseen visual environments, which is achallenging task since agents usually tend to overfit to their training environment.
Revised: Generalization in image-based reinforcement learning (RL) aims to learn a robust policy that could be applied directly on unseen visual environments, which is a challenging task since agents usually tend to overfit to their training environment.

Original: To handle this problem, a natural approach is to increase the data diversity byimage based augmentations.
Revised: To handle this problem, a natural approach is to increase the data diversity by image based augmentations.

Original: However, different with most vision tasks such asclassification and detection, RL tasks are not always invariant to spatial basedaugmentations due to the entanglement of environment dynamics and visual appearance.
Revised: However, different with most vision tasks such as classification and detection, RL tasks are not always invariant to spatial based augmentations due to the entanglement of environment dynamics and visual appearance.

Original: In this paper, we argue with two principles for augmentations in RL:
Revised: In this paper, we argue with two principles for augmentations in RL:

Original: First , the augmented observations should facilitate learning a universal policy,which is robust to various distribution shifts.
Revised: First , the augmented observations should facilitate learning a universal policy, which is robust to various distribution shifts.

Original: Second , the augmented data should beinvariant to the learning signals such as action and reward.
Revised: Second , the augmented data should be invariant to the learning signals such as action and reward.

Original: Following these rules,we revisit image-based RL tasks from the view of frequency domain and propose anovel augmentation method, namely Spectrum Random Masking (SRM),whichis able to help agents to learn the whole frequency spectrum of observation forcoping with various distributions and compatible with the pre-collected action andreward corresponding to original observation.
Revised: Following these rules, we revisit image-based RL tasks from the view of frequency domain and propose a novel augmentation method, namely Spectrum Random Masking (SRM),which is able to help agents to learn the whole frequency spectrum of observation for coping with various distributions and compatible with the pre-collected action and reward corresponding to original observation.

Original: Extensive experiments conducted on
Revised: Extensive experiments conducted on

Original: DMControl Generalization Benchmark demonstrate the proposed SRM achievesthe state-of-the-art performance with strong generalization potentials.
Revised: DMControl Generalization Benchmark demonstrate the proposed SRM achieves the state-of-the-art performance with strong generalization potentials.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement Learning (RL) from image-based observations is a task where representation anddecision-making are jointly learned from visual signals.
Revised: Reinforcement Learning (RL) from image-based observations is a task where representation and decision-making are jointly learned from visual signals.

Original: Although this type of RL methods has madesignificant progresses due to the development of deep Convolution Neural Networks (CNNs), mostof them suffer from drastic performance decline on previously unseen environments [23, 42, 3] especially for image-based inputs [42, 31, 3, 46, 26].
Revised: Although this type of RL methods has made significant progresses due to the development of deep Convolution Neural Networks (CNNs), most of them suffer from drastic performance decline on previously unseen environments [24, 44, 4], especially for image-based inputs [44, 32, 4, 48, 27].

Original: The reason is that classical RL methods assumethe environments of training and testing are identical [16].
Revised: The reason is that classical RL methods assume the environments of training and testing are identical [17].

Original: However, there usually exists significantdistributions divergences between them in real-world scenarios.
Revised: However, there usually exists significant distributions divergences between them in real-world scenarios.

Original: The high dimensionality of visualinputs coupled with the shifted distributions of environment make the generalization of RL tasks evenmore challenging [29].
Revised: The high dimensionality of visual inputs coupled with the shifted distributions of environment make the generalization of RL tasks even more challenging [30].

Original: To handle this challenge, one natural way is to decrease the distribution gap of training and testing databy simulating the diversity of test environments in training.
Revised: To handle this challenge, one natural way is to decrease the distribution gap of training and testing data by simulating the diversity of test environments in training.

Original: Recent studies show that the appropriatedata augmentation could facilitate the generalization of the policy network [39].
Revised: Recent studies show that the appropriate data augmentation could facilitate the generalization of the policy network [41].

Original: Most of these dataaugmentation methods have been carried out directly in the spatial domain [18], such as flipping androtation.
Revised: Most of these data augmentation methods have been carried out directly in the spatial domain [19], such as flipping and rotation.

Original: However, the dynamic of environment has the disadvantages of lacking sensitivity to suchspatial-based data augmentation [27].
Revised: However, the dynamic of environment has the disadvantages of lacking sensitivity to such spatial-based data augmentation [28].

Original: An example is illustrated in Fig.
Revised: An example is illustrated in Fig.

Original: 1: when horizontal flipping isapplied on observations, the corresponding left and right actions should be reversed, thus collectedrewards could not reflect the correct response.
Revised: 1: when horizontal flipping is applied on observations, the corresponding left and right actions should be reversed, thus collected rewards could not reflect the correct response.

Original: Considering above problem, instead of working in the spatial domain, we aim to study the generalization of image-based RL from a frequency domain perspective. Our motivation stems from
Revised: 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: [Figure]
Revised: [Figure]

Original: √ √
Revised: √ √

Original: [Figure]
Revised: [Figure]

Revised: Considering above problem, instead of working in the spatial domain, we aim to study the generalization of image-based RL from a frequency domain perspective.

Original:  the recent studies Fourier analysis in computer vision [40]: different types of spatial corruptionsinfluence the model robustness toward different frequency ranges.
Revised: Our motivation stems from the recent studies Fourier analysis in computer vision [42]: different types of spatial corruptions influence the model robustness toward different frequency ranges.

Original: For example, the high frequencydata augmentations such as random Gaussian noise bias the model towards utilizing low frequencyinformation in the input, and improve robustness to corruptions with high frequencies.
Revised: For example, the high frequency data augmentations such as random Gaussian noise bias the model towards utilizing low frequency information in the input, and improve robustness to corruptions with high frequencies.

Original: Moreover,single spatial-based data augmentation rarely improves model robustness of corruption types withdifferent frequency characteristics.
Revised: Moreover, single spatial-based data augmentation rarely improves model robustness of corruption types with different frequency characteristics.

Original: Inspired by this phenomenon, we conjecture that the observation’s
Revised: Inspired by this phenomenon, we conjecture that the observation’s

Original: Fourier statistics for the same RL task also vary across different environment distributions.
Revised: Fourier statistics for the same RL task also vary across different environment distributions.

Original: In Fig.
Revised: In Fig.

Original: 2,we visualize the observation’s spectrum magnitude of the the original environment as well as theaveraged delta between 4 shifted environment distributions on DeepMind Control suite [32].
Revised: 2, we visualize the observation’s spectrum magnitude of the the original environment as well as the averaged delta between 4 shifted environment distributions on DeepMind Control suite [34].

Original: Theshifted environments show distinct spectrum difference with the original environment, with each onevarying at different frequency regions.
Revised: The shifted environments show distinct spectrum difference with the original environment, with each one varying at different frequency regions.

Original: Knowing that a deep model often tends to bias on certainfrequency band [37], we hypothesis that the diverse spectrum pattern across environment distributionsis the key challenge of image-based RL generalization.
Revised: Knowing that a deep model often tends to bias on certain frequency band [39], we hypothesis that the diverse spectrum pattern across environment distributions is the key challenge of image-based RL generalization.

Original: Based on the above analyses, we focus on the Frequency-based augmentation instead of the spatialbased one by introducing a Spectrum Random Masking (SRM) regularization, as shown in Fig.
Revised: Based on the above analyses, we focus on the Frequency-based augmentation instead of the spatialbased one by introducing a Spectrum Random Masking (SRM) regularization, as shown in Fig.

Original: Our proposed SRM can be easily compatible for most existing RL benchmarks.
Revised: Our proposed SRM can be easily compatible for most existing RL benchmarks.

Original: To prevent themodel from focusing on a certain frequency range on input images, SRM randomly discards partialfrequency of observations at training phase, and forces the policy to select the appropriate actionwith remaining information.
Revised: To prevent the model from focusing on a certain frequency range on input images, SRM randomly discards partial frequency of observations at training phase, and forces the policy to select the appropriate action with remaining information.

Original: In this way, we can increase the robustness to different corruptions(high, middle or low frequency characteristics) by considering the entire frequency distribution.
Revised: In this way, we can increase the robustness to different corruptions (high, middle or low frequency characteristics) by considering the entire frequency distribution.

Original: Specifically, an input observation is operated by three steps: (1) the Fast Fourier transformation (FFT),(2) Spectrum random masking, and (3) the inverse Fourier transformation (IFFT), which enhancesthe diversity of training observations while maintaining the main content.
Revised: Specifically, an input observation is operated by three steps: (1) the Fast Fourier transformation (FFT), (2) Spectrum random masking, and (3) the inverse Fourier transformation (IFFT), which enhances the diversity of training observations while maintaining the main content.

Original: To summarize, this paper has the following contributions: (1) We provide a new frequency perspectivefor data augmentation in RL.
Revised: To summarize, this paper has the following contributions: (1) We provide a new frequency perspective for data augmentation in RL.

Original: In contrast to previous spatial-based methods, the proposed SRM areperformed consistently for task-agnostic RL environments and could deal with diverse distributionby focusing on the whole frequency information.
Revised: In contrast to previous spatial-based methods, the proposed SRM are performed consistently for task-agnostic RL environments and could deal with diverse distribution by focusing on the whole frequency information.

Original: The proposed SRM is a plug-and-play methodthat does not require any extra parameters or any additional observations from the environment.
Revised: The proposed SRM is a plug-and-play method that does not require any extra parameters or any additional observations from the environment.

Original: Moreover, it is complementary to existing data augmentation and regularization approaches.
Revised: Moreover, it is complementary to existing data augmentation and regularization approaches.

Original: (3) Weconduct comprehensive experiments to demonstrate SRM could bring performance gains on variousimage-based environments with high generalization potentials.
Revised: (3) We conduct comprehensive experiments to demonstrate SRM could bring performance gains on various image-based environments with high generalization potentials.

Original:  Background
Revised:  Background

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: A RL problem is typically described as a Markov Decision Process (MDP)
Revised: A RL problem is typically described as a Markov Decision Process (MDP)

Original: M = < S , A , P , R , γ > , where S and A denotes the state space and action space respectively.
Revised: M = < S , A , P , R , γ > , where S and A denotes the state space and action space respectively [33].

Original: P ( s t +1 | s t , a t ) is the state transition, R ( s t , a t ) is the reward function, and γ ∈ [0 , 1) is the discountfactor.
Revised: P ( s t +1 | s t , a t ) is the state transition, R ( s t , a t ) is the reward function, and γ ∈ [0 , 1) is the discount factor.

Original: Here we consider a Partially Observable MDP (POMDP)
Revised: Here we consider a Partially Observable MDP (POMDP)

Original:  M = < O , A , P , R , γ > forthe continuous control task because S is often not directly obtained from raw images, where O isthe high-dimensional observation space of images.
Revised: [2] M = < O , A , P , R , γ > for the continuous control task because S is often not directly obtained from raw images, where O is

Original: Following the convention [24], the POMDP isconverted into an MDP by stacking several consecutive environment frames { o t , o t − 1 , o t − 2 } into asingle state s t ∈ O .

Original: [Figure]
Revised: [Figure]

Original: The goal of typical RL is to find an optimal policy π ∗ θ which maximizes the expected cumulativereward over the entire distribution of MDPs:
Revised: The goal of typical RL is to find an optimal policy π ∗ θ which maximizes the expected cumulative reward over the entire distribution of MDPs [33]:

Original: [Equation]
Revised: [Equation]

Original: where  π θ isparameterized by the learnable parameters θ .
Revised: where ρ π ( s t , a t ) is the state-action marginal of the trajectory distribution.

Original: In this work, rather than learning an optimalpolicy on a single MDP, we consider a generalizable π θ , which is able to obtain high discountedreturn on a set of MDPs M =
Revised: In this work, rather than learning an optimal policy on a single MDP, we consider a generalizable π θ , which is able to obtain high discounted return on a set of MDPs M =

Original: <
Revised: <

Original: O , A , P , R , γ > , where the observations in O are unseen in training.
Revised: O , A , P , R , γ > , where the observations in O are unseen in training.

Original: Soft Actor-Critic.
Revised: Soft Actor-Critic.

Original: Soft Actor-Critic (SAC) [8, 9] is a popular off-policy method for continuouscontrol tasks.
Revised: Soft Actor-Critic (SAC) [9, 10] is a popular off-policy method for continuous control tasks.

Original: It learns a state-action value function Q ( s, a ) and a stochastic policy π ( a | s ) to find theoptimal policy based on the maximum entropy RL framework:
Revised: It learns a state-action value function Q ( s, a ) and a stochastic policy π ( a | s ) to find the optimal policy based on the maximum entropy RL framework:

Original: [Equation]
Revised: [Equation]

Original: where α is the temperature parameter which determines the relative importance of the entropy termagainst the reward.
Revised: where α is the temperature parameter which determines the relative importance of the entropy term against the reward.

Original: α = 0 degenerates SAC into conventional RL.
Revised: α = 0 degenerates SAC into conventional RL.

Original: H ( · ) provides a substantialimprovement in exploration and robustness by enabling the probability of action output to be diverseas much as possible, rather than being concentrated on one action.
Revised: H ( · ) provides a substantial improvement in exploration and robustness by enabling the probability of action output to be diverse as much as possible, rather than being concentrated on one action.

Original: Specially, Q ( s, a ) is approximatedby minimizing the soft Bellman residual:
Revised: Specially, Q ( s, a ) is approximated by minimizing the soft Bellman residual:

Original: [Equation]
Revised: [Equation]

Original: where  V s t +1 is the soft target value network approximated as:
Revised: where D denotes the replay buffer, and V s t +1 is the soft target value network approximated as:

Original: [Equation]
Revised: [Equation]

Original: where Q is the target Q function, and the weights can be an exponentially moving average of theweights in Q .
Revised: where Q is the target Q function, and the weights can be an exponentially moving average of the weights in Q .

Original: In the policy improvement step, the policy is updated by minimizing the divergencebetween the policy and the exponential of the soft Q function:
Revised: In the policy improvement step, the policy is updated by minimizing the divergence between the policy and the exponential of the soft Q function:

Original: [Equation]
Revised: [Equation]

Original:  Methodology3.1 Spectrum Random Masking
Revised:  Methodology 3.1 Spectrum Random Masking

Original: We firstly introduce how to generate new samples by spectrum random masking strategy in thissection.
Revised: We firstly introduce how to generate new samples by spectrum random masking strategy in this section.

Original: Suppose we have a grayscale image observation 1 o i ∈
Revised: Suppose we have a grayscale image observation 1 o i ∈

Original: R H × W , the Fourier domain spectrum
Revised: R H × W , the Fourier domain spectrum

Original: F ( o i ) can be calculated by fast Fourier transformation [25]:
Revised: F ( o i ) can be calculated by fast Fourier transformation [26]:

Original: [Equation]
Revised: [Equation]

Original: (6) where u ∈ [0 , H − 1] and v ∈ [0 , W − 1] are horizontal and vertical indexes of frequency domaincomponents, and o i ( h, w ) is the pixel value at position ( h, w ) .
Revised: [Figure] where u ∈ [0 , H − 1] and v ∈ [0 , W − 1] are horizontal and vertical indexes of frequency domain components, and o i ( h, w ) is the pixel value at position ( h, w ) .

Original: Note the subscript i is just the index RGB , we the operation each color
Revised: Note the subscript i is just the index of the sample and irrelevant with the time step.

Original: [Figure]

Original: of the sample and irrelevant with the time step.

Original: For better processing and visualization, we use theshifted version of F ( o i ) with the lowest frequency component placed in the center of the spectrum.
Revised: For better processing and visualization, we use the shifted version of F ( o i ) with the lowest frequency component placed in the center of the spectrum.

Original: After transforming images from spatial to frequency domain, we construct a binary masking matrix
Revised: After transforming images from spatial to frequency domain, we construct a binary masking matrix

Original: M ∈ { 0 , 1 } H × W .
Revised: M ∈ { 0 , 1 } H × W .

Original: We introduce two radius parameters r 1 , r 2 ∈ [0 , 0 . 5] to control the position andratio of the spectral neighborhood to be masked 2 ( r 1 < r 2 ).
Revised: We introduce two radius parameters r 1 , r 2 ∈ [0 , 0 . 5] to control the position and ratio of the spectral neighborhood to be masked 2 ( r 1 < r 2 ).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, M is constructedby setting the region in the annulus between two concentric circles (with radius r 1 and r 2 ) to 0.∆ r
Revised: 3, M is constructed by setting the region in the annulus between two concentric circles (with radius r 1 and r 2 ) to 0.

Original:  = r 2 − r 1 decides the size of mask regions.
Revised: ∆ r = r 2 − r 1 decides the size of mask regions.

Original: The area ratio of masking region is defined asr p =
Revised: The area ratio of masking region is defined as r p =

Original: r 22 − r 21 .
Revised: r 22 − r 21 .

Original: Hence we propose 3 typical types of masking strategies: erasing, noise and swapping.
Revised: Hence we propose 3 typical types of masking strategies: erasing, noise and swapping.

Original: If the frequency component lies between r 1 and r 2 , it will be masked; otherwise, the frequencysignals will be kept:
Revised: If the frequency component lies between r 1 and r 2 , it will be masked; otherwise, the frequency signals will be kept:

Original: [Equation]
Revised: [Equation]

Original: (7) where Z is calculated by the specific masking strategy.
Revised: where Z is calculated by the specific masking strategy.

Original: For the erasing strategy, we set Z to 0 H × W ,and only the non-masked frequency components will be fully explored in follow-up training.
Revised: For the erasing strategy, we set Z to 0 H × W , and only the non-masked frequency components will be fully explored in follow-up training.

Original: Therandom noise strategy sets Z to a random noise image.
Revised: The random noise strategy sets Z to a random noise image.

Original: The swapping strategy replaces the removedfrequency components with the same frequency components from another image I by setting Z
Revised: The swapping strategy replaces the removed frequency components with the same frequency components from another image I by setting Z

Original: = I ,which draws inspiration from Cutmix [41] and Mixup [45].
Revised: = I , which draws inspiration from Cutmix [43] and Mixup [47].

Original: Considering that the third-party data I isnot always available, during training, we randomly shuffle the training batch and pick I =
Revised: Considering that the third-party data I is not always available, during training, we randomly shuffle the training batch and pick I =

Original: o j ( j ̸ = i )from the shuffled batch, where o j has the same batch index with o i .
Revised: o j ( j ̸ = i ) from the shuffled batch, where o j has the same batch index with o i .

Original: The masking strategy also influences the supervised signals since it is correlated with o i .
Revised: The masking strategy also influences the supervised signals since it is correlated with o i .

Original: Supposethat a i , r i is the supervised signals for observation o i , where a i and r i stands for action and reward,respectively.
Revised: Suppose that a i , r i is the supervised signals for observation o i , where a i and r i stands for action and reward, respectively.

Original: For erasing and random noise strategy, a i and r i are kept unchanged.
Revised: For erasing and random noise strategy, a i and r i are kept unchanged.

Original: For the swappingstrategy, the supervised signals depended on the area ratio of masking region r p .
Revised: For the swapping strategy, the supervised signals depended on the area ratio of masking region r p .

Original: If r p < 0 . 5 , wekeep a i and r i unchanged, else we replace a i , r i with a j and r j .
Revised: If r p < 0 . 5 , we keep a i and r i unchanged, else we replace a i , r i with a j and r j .

Original: Once the masked Fourier spectrum ˆ F ( o i )( u, v ) is obtained, we inverse shift and restore it to thespatial domain by the corresponding inverse Fourier transformation F − 1 :
Revised: Once the masked Fourier spectrum ˆ F ( o i )( u, v ) is obtained, we inverse shift and restore it to the spatial domain by the corresponding inverse Fourier transformation F − 1 :

Original: [Equation]
Revised: [Equation]

Original: For observation o i , its next observation o i ′ also goes through the same SRM operation as o i .
Revised: For observation o i , its next observation o i ′ also goes through the same SRM operation as o i .

Original: Algorithm 1 details how to generate new samples by SRM.
Revised: Algorithm 1 details how to generate new samples by SRM

Original: In off-policy RL, observations are sampled from a replay buffer. One intuitive way to use dataaugmentation in RL is to perform augmentation before passing them to the agent for training. Input :Sample transitions a r o i
Revised: Output :

Original: W .
Revised: ˆ o

Original: Output : Augmented transitions { o i , ˆ a i , ˆ r

Original: i
Revised: i

Original: ˆ o ′ i | i = 1 , · · · , K }
Revised: K Init : r

Original: Init : r 1 ←Rand (0 , 0 . 5) , r 2 ← Rand( r 1 , 0 . 5) , ∆ r= r2 − r1 , r p = r 22 − r 21 .
Revised: ← r 2 ← Rand ∆ r r − r r p = r 22 − r 21 .

Original: 1: for i = 1 , · · · , K do 2:
Revised: 1: for i = 1 , · · · , K do 2:

Original: Fast Fourier transformation according to Eq.
Revised: Fast Fourier transformation according to Eq.

Original: 6: F ( o i )
Revised: 6: F ( o i )

Original: ←
Revised: ←

Original: o i 3: if Masking strategies is ‘Erasing’ then 4: ˆ F ( o i ) =
Revised: o i 3: if Masking strategies is ‘Erasing’ then 4: ˆ F ( o i ) =

Original: M · F ( o i ) + (1 − M ) · F ( 0 ) , ˆ a i ← a i , ˆ r i ← r i 5: else if Masking strategies is ‘Noise’ then 6: ˆ F ( o i ) =
Revised: M · F ( o i ) + (1 − M ) · F ( 0 ) , ˆ a i ← a i , ˆ r i ← r i 5: else if Masking strategies is ‘Noise’ then 6: ˆ F ( o i ) =

Original: M · F ( o i ) + (1 − M ) · F ( N ) , ˆ a i ← a i , ˆ r i ← r i 7: else if Masking strategies is ‘Swapping’ then 8:
Revised: M · F ( o i ) + (1 − M ) · F ( N ) , ˆ a i ← a i , ˆ r i ← r i 7: else if Masking strategies is ‘Swapping’ then 8:

Original: Random Shuffle Batch: { o j , a j , r j , o ′ j | j = 1 , · · · , K } ˆ F ( o i ) =
Revised: Random Shuffle Batch: { o j , a j , r j , o ′ j | j = 1 , · · · , K } ˆ F ( o i ) =

Original: M · F ( o i ) + (1 − M ) · F ( o j ) 9: if r p ≤ 0 .
Revised: M · F ( o i ) + (1 − M ) · F ( o j ) 9: if r p ≤ 0 .

Original: 5 then 10: ˆ a i ← a i , ˆ r
Revised: 5 then 10: ˆ a i ← a i , ˆ r

Original: i ← r i 11: else
Revised: i ← r i 11: else

Original: 12: ˆ a i ← a j , ˆ r i ← r j 13: end if 14: end if 15:
Revised: 12: ˆ a i ← a j , ˆ r i ← r j 13: end if 14: end if 15:

Original: Inverse Fourier transformation according to Eq.
Revised: Inverse Fourier transformation according to Eq.

Original: 8: ˆ o i ← ˆ F ( o i ) 16:
Revised: 8: ˆ o i ← ˆ F ( o i ) 16:

Original: Execute Steps (2) ∼ (15) with o ′ i to get ˆ o ′ i 17: end for
Revised: Execute Steps (2) ∼ (15) with o ′ i to get ˆ o ′ i 17: end for

Revised: 3.2 Reinforcement Learning with SRM

Revised: off

Revised: , observations are sampled from a replay buffer.

Revised: One intuitive way to use data augmentation in RL is to perform augmentation before passing them to the agent for training [41, 19].

Original: Meanwhile, the augmentation procedure on stacked consecutive state frames s t = { o t , o t − 1 , o t − 2 }should be consistent to keep temporal information (such as positions and velocities) invariant.
Revised: Meanwhile, the augmentation procedure on stacked consecutive state frames s t = { o t , o t − 1 , o t − 2 } should be consistent to keep temporal information (such as positions and velocities) invariant.

Original: Givenan augmentation function τ ( · , v ) with parameter v ∈ V , the augmented observation ˆ s t ∈
Revised: Given an augmentation function τ ( · , v ) with parameter v ∈ V , the augmented observation ˆ s t ∈

Original: ˆ O of theoriginal observation s t ∈ O can be obtained by ˆ s t=
Revised: ˆ O of the original observation s t ∈ O can be obtained by ˆ s t

Original:  τ ( s t , v ) .
Revised: = τ ( s t , v ) .

Original: Optimally, τ : O × V → ˆ O shouldmaintain Q-values and policy π of the agent:
Revised: Optimally, τ : O × V → ˆ O should maintain Q-values and policy π of the agent:

Original: [Equation]
Revised: [Equation]

Original: where a t ∈ A is the action of s t .
Revised: where a t ∈ A is the action of s t .

Original: However, practically it is unstable to learn value and policyfunctions even from the original observations, not to mention the augmented data that vary wildly inappearances.
Revised: However, practically it is unstable to learn value and policy functions even from the original observations, not to mention the augmented data that vary wildly in appearances.

Original: As a result, only several weak data augmentations such as random shift and randomcropping could be employed in a plain way in RL optimization, as shown in DrQ [39] and RAD [18].
Revised: As a result, only several weak data augmentations such as random shift and random cropping could be employed in a plain way in RL optimization, as shown in DrQ [41] and RAD [19].

Original: In contrast, strong data augmentations such as random convolution and rotation are always beenapplied in a specific way.
Revised: In contrast, strong data augmentations such as random convolution and rotation are always been applied in a specific way.

Original: In SVEA [12], both original and augmented images are used to optimizethe Q-function, but only the former are adopted to compute Q-target of the Bellman equation.
Revised: In SVEA [13], both original and augmented images are used to optimize the Q-function, but only the former are adopted to compute Q-target of the Bellman equation.

Original: In
Revised: In

Original: SECANT [6], expert policy with weak augmentations guides student policy with strong augmentationsfor learning robust representations.
Revised: SECANT [7], expert policy with weak augmentations guides student policy with strong augmentations for learning robust representations.

Original: Attractively, as we will show in the experiment section, SRM hasthe merit of both weak and strong data augmentations: it can be plugged into any image-based RLmethods in a plain way, while providing significant performance improvement.
Revised: Attractively, as we will show in the experiment section, SRM has the merit of both weak and strong data augmentations: it can be plugged into any image-based RL methods in a plain way, while providing significant performance improvement.

Original: During training, we apply SRM with a probability of 0.5 on each batch of observations.
Revised: During training, we apply SRM with a probability of 0.5 on each batch of observations.

Original: No dataaugmentation is applied at test time.
Revised: No data augmentation is applied at test time.

Original: To evaluate the effectiveness of SRM, we adopt two popularmethods DrQ [39] and SVEA [12] as our base algorithms.
Revised: To evaluate the effectiveness of SRM, we adopt two popular methods DrQ [41] and SVEA [13] as our base algorithms.

Original: The former is a naive method where dataaugmentations are plugged into RL algorithms directly, and the corresponding SAC objectives arerespectively replaced by:
Revised: The former is a naive method where data augmentations are plugged into RL algorithms directly, and the corresponding SAC objectives are respectively replaced by:

Original: [Equation]
Revised: [Equation]

Original: The latter is delicately designed where Q-target is computed strictly from unaugmented data toalleviate the problem of high-variance, and its goal is:
Revised: The latter is delicately designed where Q-target is computed strictly from unaugmented data to alleviate the problem of high-variance, and its goal is:

Original: [Equation]
Revised: [Equation]

Original: By jointly solving the Q-function with policy and value network in Eqs.
Revised: By jointly solving the Q-function with policy and value network in Eqs.

Original: (4) and (5), the RL systemshas the potential to learn from diverse observations for policy learning.
Revised: (4) and (5), the RL systems has the potential to learn from diverse observations for policy learning.

Original:  Experiments 
Revised:  Experiments 4.1 Task Setup and Evaluation Protocol

Original: [Figure] 4.1 Task Setup and Evaluation Protocol

Original: Environments We conduct our experiment on 5 tasks from DeepMind Control Suite (DMControl) [32], which offers a set of challenging continuous control tasks and has been widely used as acommon benchmark for visual RL [10, 1].
Revised: Environments We conduct our experiment on 5 tasks from DeepMind Control Suite (DMControl) [34], which offers a set of challenging continuous control tasks and has been widely used as a common benchmark for visual RL [11, 1].

Original: There are 2 kinds of observations in DMControl: statevector-based and image-based, and the latter is used in our experiment.
Revised: There are 2 kinds of observations in DMControl: state vector-based and image-based, and the latter is used in our experiment.

Original: It has been evidenced that thelearning of state-based observations is easier than high dimensional image-based observations [17].
Revised: It has been evidenced that the learning of state-based observations is easier than high dimensional image-based observations [18].

Original: To evaluate the generalization, we train the agent from DMControl and report results tested on
Revised: To evaluate the generalization, we train the agent from DMControl and report results tested on

Original: DMControl Generalization Benchmark (DMControl-GB) [13], which includes four distinct testdistributions with slight/strong color/video attack: color-easy, color-hard, video-easy and video-hard..
Revised: DMControl Generalization Benchmark (DMControl-GB) [14], which includes four distinct test distributions with slight/strong color/video attack: color-easy, color-hard, video-easy and video-hard..

Original: Implementation Details For a fair comparison, we implement all methods following [12], wherethe same hyperparameters and network architecture are adopted.
Revised: Implementation Details For a fair comparison, we implement all methods following [13], where the same hyperparameters and network architecture are adopted.

Original: We use a 11-layer feed-forwardconvolution network as the shared encoder, which is followed by independent linear projections forthe actor and critic.
Revised: We use a 11-layer feed-forward convolution network as the shared encoder, which is followed by independent linear projections for the actor and critic.

Original: During training, the masking ratio and position of SRM are randomly chosen, andthe ranges of r 1 and ∆ r are set as [0 , 0 . 5] and [0 , 0 .
Revised: During training, the masking ratio and position of SRM are randomly chosen, and the ranges of r 1 and ∆ r are set as [0 , 0 . 5] and [0 , 0 .

Original: for each batch of observations, respectively.
Revised: for each batch of observations, respectively.

Original: Note SRM is a data augmentation method and needs to combine with other learning baselines.
Revised: Note SRM is a data augmentation method and needs to combine with other learning baselines.

Original: Tovalidate the effectiveness, 3 state-of-the-art baselines are employed: DrQ [39], SVEA-C [12] and
Revised: To validate the effectiveness, 3 state-of-the-art baselines are employed: DrQ [41], SVEA-C [13] and

Original: SVEA-O [12], denoted as “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” respectively.
Revised: SVEA-O [13], denoted as “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” respectively.

Original: Oand C represent random convolution and random overlay respectively, which are set as the defaultaugmentation used in SVEA [12].
Revised: O and C represent random convolution and random overlay respectively, which are set as the default augmentation used in SVEA [13].

Original: For DrQ, SVEA-C and SVEA-O, the better performances betweenthe reported ones of the original paper and our re-run ones using the source codes are reported in ourexperiment.
Revised: For DrQ, SVEA-C and SVEA-O, the better performances between the reported ones of the original paper and our re-run ones using the source codes are reported in our [Table] [Table] experiment.

Original: We provide 3 candidate masking strategies erasing, noise and swapping in Algorithm 1,all of them can achieve higher performance than baseline as shown in Table 5 .
Revised: We provide 3 candidate masking strategies erasing, noise and swapping in Algorithm 1, all of them can achieve higher performance than baseline as shown in Table 5 .

Original: Since erasing performsbetter than others, it is chosen as the default masking strategy in the following experiments if nototherwise mentioned.
Revised: Since erasing performs better than others [Table] it is chosen as the default masking strategy in the following experiments if not otherwise mentioned.

Original: All methods are trained for 500K frames.
Revised: All methods are trained for 500K frames.

Original: We run 5 times with random seedsand report the averaged results.[Table] Hard [Table]
Revised: We run 5 times with random seeds and report the averaged results.

Original: +

Original: [Table]

Original: walker,

Original: [Table]

Original: walk

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: finger,

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 4.2 Comparisons
Revised: 4.2 Comparisons

Original: Comparisons with state-of-the-arts The comparisons are exhibited in Table 1.
Revised: Comparisons with state-of-the-arts The comparisons are exhibited in Table 1.

Original: For the directcomparison, “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” outperform DrQ, SVEA-C and
Revised: For the direct comparison, “DrQ+SRM”, “SVEA-C+SRM” and “SVEA-O+SRM” outperform DrQ, SVEA-C and

Original: SVEA-O with a clear margin, respectively.
Revised: SVEA-O with a clear margin, respectively.

Original: In addition, the usage of SRM reduces the standarddeviation and shows the strong leaning stability.
Revised: In addition, the usage of SRM reduces the standard deviation and shows the strong leaning stability.

Original: It indicates that performing augmentation in thefrequency domain is effective for image-based RL tasks.
Revised: It indicates that performing augmentation in the frequency domain is effective for image-based RL tasks.

Original: In addition, the performance of othermethods such as CURL [17], PAD [11], RAD [18] and SODA [13] are also reported.
Revised: In addition, the performance of other methods such as CURL [18], PAD [12], RAD [19] and SODA [14] are also reported.

Original: Since ourmethod is implemented with different learning baselines with them, the results are just provided forreference.
Revised: Since our method is implemented with different learning baselines with them, the results are just provided for reference.

Original: Overall, it is evident that SRM provides new state-of-the-art performance for the task.
Revised: Overall, it is evident that SRM provides new state-of-the-art performance for the task.

Original: Comparisons with spatial-based data augmentation Here we compare SRM with 5 spatial-baseddata augmentation methods and 1 removal regularization strategy, including random convolution [19], random rotation [7], random cutout [4], random flip [18] and dropout [30].
Revised: Comparisons with spatial-based data augmentation Here we compare SRM with 5 spatial-based data augmentation methods and 1 removal regularization strategy, including random convolution [20], random rotation [8], random cutout [5], random flip [19] and dropout [31].

Original: DrQ is employed as thebaseline learning method.
Revised: DrQ is employed as the baseline learning method.

Original: When utilizing each data augmentation method, 50% of frames are remainun-augmented.
Revised: When utilizing each data augmentation method, 50% of frames are remain un-augmented.

Original: This is because we find that directly using all kinds of data augmentation is harmfulto policy stability.
Revised: This is because we find that directly using all kinds of data augmentation is harmful to policy stability.

Original: The dropout layer is added after every nonlinearity layer in the MLP of actor andcritic networks and the dropout probability is set to 0.1.
Revised: The dropout layer is added after every nonlinearity layer in the MLP of actor and critic networks and the dropout probability is set to 0.1.

Original: Since the video hard task is extremely hard and the learning of DrQ is oscillating, we show learningcurves of the different data augmentations in color easy, color hard and video easy.
Revised: Since the video hard task is extremely hard and the learning of DrQ is oscillating, we show learning curves of the different data augmentations in the benchmarks of color-easy, color-hard and video-easy.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4,the spatial data augmentation methods are less effective and even degrade performance, especiallyfor flip and rotation.
Revised: 4, the spatial data augmentation methods are less effective and even degrade performance, especially for flip and rotation.

Original: The reason is that these augmentations inevitably changes the environmentdynamics, leading to inaccurate state transitions.
Revised: The reason is that these augmentations inevitably changes the environment dynamics, leading to inaccurate state transitions.

Original: In contrast, SRM outperforms the baseline in mostscenarios and achieve comparable results in the rest scenarios.
Revised: In contrast, SRM outperforms the baseline in most scenarios and achieve comparable results in the rest scenarios.

Original: It demonstrates that the generalizationability of frequency based SRM is stronger than its spatial-based counterparts.
Revised: It demonstrates that the generalization ability of frequency based SRM is stronger than its spatial-based counterparts.

Original: Note that the “randomcov” performs well on color hard and is less effective than SRM in other situations.
Revised: Note that the “random cov” performs well on color hard and is less effective than SRM in other situations.

Original: The reason is thatapplying random convolution mainly changes the low frequency components (i.e., colors and textures)of an image, which makes the agent robust to color changes.
Revised: The reason is that applying random convolution mainly changes the low frequency components (i.e., colors and textures) of an image, which makes the agent robust to color changes.

Original: However, when facing environmentsthat differs significantly on higher frequency components, random convolution becomes less effective.
Revised: However, when facing environments that differs significantly on higher frequency components, random convolution becomes less effective.

Original: Compared with “random cov”, SRM works more generally to different scenarios.
Revised: Compared with “random cov”, SRM works more generally to different scenarios.

Original: Fig.
Revised: Fig.

Original: 4 also tells that, although the removal based methods dropout and cutout has been proved to beable to improve generalization for discriminative models, they decrease the performance of RL tasks.
Revised: 4 also tells that, although the removal based methods dropout and cutout has been proved to be able to improve generalization for discriminative models, they decrease the performance of RL tasks.

Original: For cutout, randomly cutting a region of an input image is prone to information loss, especially whenthe target region (e.g., the cartpole in cartpole swingup task) is removed.
Revised: For cutout, randomly cutting a region of an input image is prone to information loss, especially when the target region (e.g., the cartpole in cartpole swingup task) is removed.

Original: The performance decreaseof dropout can be attributed to the chaotic nature of the RL training when facing with random droppedhidden units.
Revised: The performance decrease of dropout can be attributed to the chaotic nature of the RL training when facing with random dropped hidden units.

Original: In contrast, each frequency component in the Fourier spectrum has the global vision ofthe whole image, thus SRM does not remove critical image regions.
Revised: In contrast, each frequency component in the Fourier spectrum has the global vision of the whole image, thus SRM does not remove critical image regions.

Original: Meanwhile, as an augmentationmethod, SRM also does not influence hidden units of networks.
Revised: Meanwhile, as an augmentation method, SRM also does not influence hidden units of networks.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Original: Spectrum Masking Strategies There are four different frequency masking strategies includingerasing, noise, swapping and a mixture of the first three.
Revised: Spectrum Masking Strategies There are four different frequency masking strategies including erasing, noise, swapping and a mixture of the first three.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, we study the choiceof masking value on the performance of RL and have the following observations: First, all of thesemasking strategies are able to improve the average performance.
Revised: 5, we study the choice of masking value on the performance of RL and have the following observations: First, all of these masking strategies are able to improve the average performance.

Original: This benefits from the mechanism of
Revised: This benefits from the mechanism of

Original: [Figure]
Revised: [Figure]

Original: random masking which attempt to strengthen the whole frequency learning while suppress overfittingto certain frequency range.
Revised: random masking which attempt to strengthen the whole frequency learning while suppress overfitting to certain frequency range.

Original: This also demonstrates that the agent may underfit to certain frequencyrange and overfit to other range when directly training with original observations.
Revised: This also demonstrates that the agent may underfit to certain frequency range and overfit to other range when directly training with original observations.

Original: Second, with theincreasing complexity of frequency disturbance, the performance of mixture is sometimes muchworse than solely using erasing or swapping.
Revised: Second, with the increasing complexity of frequency disturbance, the performance of mixture is sometimes much worse than solely using erasing or swapping.

Original: This phenomenon can be attributed to introduce morevariance by mixture.
Revised: This phenomenon can be attributed to introduce more variance by mixture.

Original: Spectrum Masking Position
Revised: Spectrum Masking Position

Original: To explore how the low and high frequency affect policy training, weset r 1 to 0.1, 0.2, 0.3, 0.4 respectively, and ∆ r to 0 .
Revised: To explore how the low and high frequency affect policy training, we set r 1 to 0.1, 0.2, 0.3, 0.4 respectively, and ∆ r to 0 .

Original: According to Frequency Principle (F-Principle),
Revised: According to Frequency Principle (F-Principle),

Original: DNN tends to captures the frequency from low to high components in ascending complexity order.
Revised: DNN tends to captures the frequency from low to high components in ascending complexity order.

Original: This is consistent with our results as shown in Fig.
Revised: This is consistent with our results as shown in Fig.

Original: At the early training stage, masking lowfrequency lead to slower improvement, while masking high frequency leads to a similar improvementwith the non-SRM baseline.
Revised: At the early training stage, masking low frequency lead to slower improvement, while masking high frequency leads to a similar improvement with the non-SRM baseline.

Original: In the later period, remaining high frequency is able to achieve higherresults than remaining low frequency.
Revised: In the later period, remaining high frequency is able to achieve higher results than remaining low frequency.

Original: Moreover, SRM could get the best results in most benchmarksthan fixed masking position.
Revised: Moreover, SRM could get the best results in most benchmarks than fixed masking position.

Original: It demonstrates that learning frequency from low to high uniformly in arandom way is beneficial.
Revised: It demonstrates that learning frequency from low to high uniformly in a random way is beneficial.

Original: Spectrum Masking Ratio To better control the masking ratio, we set △ r to 0.05, 0.1,0.2,0.3 and 0.4respectively.
Revised: Spectrum Masking Ratio To better control the masking ratio, we set △ r to 0.05, 0.1,0.2,0.3 and 0. respectively.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: Notably, SRM with random masking ratio consistentlyoutperforms the DrQ baseline under all parameters setting, especially for harder environment.
Revised: Notably, SRM with random masking ratio consistently outperforms the DrQ baseline under all parameters setting, especially for harder environment.

Original: Inaddition, we can observe that masking ratio has little influence on original environment, which can beattributed to the remaining of a decent percentage of un-augmented frames.
Revised: In addition, we can observe that masking ratio has little influence on original environment, which can be attributed to the remaining of a decent percentage of un-augmented frames.

Original:  Related works
Revised:  Related works

Original: Generalization in RL draws increasing attention in recent years.
Revised: Generalization in RL draws increasing attention in recent years.

Original: Existing methods can be roughlycategorized into two groups: (1) increasing distribution similarity between training and testingenvironments by augmentation methods (such as data augmentation and environment generation) [15,19, 39, 17, 18, 13, 27] and (2) handling differences between the training and testing MDPs (suchas regularisation) [14, 21, 43, 44, 5, 1, 20, 34, 22, 28, 35].
Revised: Existing methods can be roughly categorized into two groups: (1) increasing distribution similarity between training and testing environments by augmentation methods (such as data augmentation and environment generation) [16, 20, 41, 18, 19, 14, 28] and (2) handling differences between the training and testing MDPs (such as regularisation) [15, 22, 45, 46, 6, 1, 21, 36, 23, 29, 37].

Original: Our method belongs to the first groupbut works in the frequency domain, which is ignored by most current RL methods.
Revised: Our method belongs to the first group but works in the frequency domain, which is ignored by most current RL methods.

Original: To explain thegeneralization behaviors of CNN, recent works in computer vision also provide new insights fromthe frequency domain aspect [37, 33, 40, 38, 36, 2].
Revised: To explain the generalization behaviors of CNN, recent works in computer vision also provide new insights from the frequency domain aspect [39, 35, 42, 40, 38, 3].

Original: However, they mainly consider the domain shiftbetween fixed datasets on discrimination tasks (e.g., classification and segmentation).
Revised: However, they mainly consider the domain shift between fixed datasets on discrimination tasks (e.g., classification and segmentation).

Original: In contrast, weconsider the dynamic environment in RL as well as the joint representation learning and decisionmaking process.
Revised: In contrast, we consider the dynamic environment in RL as well as the joint representation learning and dedecision-making process.

Original: Moreover, CNN  tends to rely on either low  or high frequency components forfixed tasks [37], existing methods rarely consider regularizing the model on all frequency ranges.
Revised: Moreover, CNN tendssince CNN tends to rely on either low frequency or high frequency components for fixed tasks during training [39], existing methods rarely consider regularizing the model on all frequency ranges.

Original: Differently, our method considers the broad frequency difference between different RL environmentdistributions and regularizes the model on all frequency ranges uniformly.
Revised: Differently, our method considers the broad frequency difference between different RL environment distributions and regularizes the model on all frequency ranges uniformly.

Original:  Conclusions
Revised:  Conclusions

Original: In this work, we investigated the generalization of image-based RL from a frequency domainperspective.
Revised: In this work, we investigated the generalization of image-based RL from a frequency domain perspective.

Original: Motivated by the distinct spectral impacts of different environment distribution shifts,we develop SRM, a simple yet effective frequency domain augmentation method
Revised: Motivated by the distinct spectral impacts of different environment distribution shifts, we develop Spectrum Random Masking (SRM), a simple yet effective frequency domain augmentation methodwhich randomly discards certain frequency bands of the observation.

Original: Experiments show that SRM improves model robustness under various distribution shifts without violating theenvironment dynamics.
Revised: Experimental results show that SRM improves model robustness under various distribution shifts without violating the environment dynamics.

Original: The effectiveness of SRM highlights a promising future research direction  to take into account the frequency domain intrinsic mechanism of RL models.
Revised: The effectiveness of SRM highlights a promising future research direction for researchers to take into account the frequency domain intrinsic mechanism of RL models.

Original: As RL tasks areessentially different from  vision tasks, we believe that different conclusions will be made to providefurther intuition and benefit the model performance.
Revised: As RL tasks are essentially different from conventional computer vision tasks (e.g., image recognition), we believe that different conclusions will be made to provide further intuition and benefit the model performance.

Original: Limitations: As an image-based augmentation method, our proposed SRM resort to enhancing thediversity of image-base observations while maintaining the crucial reward-related content, whichfacing challenges when image observations are incomplete or only state information can be observed.  
Revised: Limitations: As an image-based augmentation method, our proposed SRM resort to enhancing the diversity of image-base observations while maintaining the crucial reward-related content, which facing challenges when image observations are incomplete or only state information can be observed.  Acknowledgments and Disclosure of Funding

Revised: This work is partially supported by grants from the Key-Area Research and Development Program of

Revised: Guangdong Province under contact No. 2020B0101380001 and grants from the National Natural

Revised: Science Foundation of China under contract No. 62027804, No. 61825101 and No. 62088102.

Revised: The computing resources of Pengcheng Cloudbrain are used in this research.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See conclusions.
Revised: [Yes] See conclusions.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Seesupplemental material.
Revised: [Yes] See supplemental material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplemental material.
Revised: [Yes] See supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Tab. 1.
Revised: [Yes] See Tab. 1.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See supplemental material.4.
Revised: [Yes] See supplemental material.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.

Original: Weuse public environments with proper citations.
Revised: We use public environments with proper citations.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data used in this paper does not containsoffensive content.
Revised: The data used in this paper does not contains offensive content.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and 
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: screenshots

Original: , if

Original: applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]