Original: Reinforcement Learning with Automated Auxiliary Loss Search
Revised: Reinforcement Learning with Automated Auxiliary Loss Search

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: A good state representation is crucial to solving complicated reinforcement learning(RL) challenges.
Revised: A good state representation is crucial to solving complicated reinforcement learning(RL) challenges.

Original: Many recent works focus on designing auxiliary losses forlearning informative representations.
Revised: Many recent works focus on designing auxiliary losses forlearning informative representations.

Original: Unfortunately, these handcrafted objectivesrely heavily on expert knowledge and may be sub-optimal.
Revised: Unfortunately, these handcrafted objectivesrely heavily on expert knowledge and may be sub-optimal.

Original: In this paper, wepropose a principled and universal method for learning better representationswith auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS),which automatically searches for top-performing auxiliary loss functions for RL.
Revised: In this paper, wepropose a principled and universal method for learning better representationswith auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS),which automatically searches for top-performing auxiliary loss functions for RL.

Original: Speciﬁcally, based on the collected trajectory data, we deﬁne a general auxiliaryloss space of size 7 . 5 ⇥ 10 20 and explore the space with an efﬁcient evolutionarysearch strategy.
Revised: Speciﬁcally, based on the collected trajectory data, we deﬁne a general auxiliaryloss space of size 7 . 5 ⇥ 10 20 and explore the space with an efﬁcient evolutionarysearch strategy.

Original: Empirical results show that the discovered auxiliary loss (namely,
Revised: Empirical results show that the discovered auxiliary loss (namely,

Original: A2-winner ) signiﬁcantly improves the performance on both high-dimensional(image) and low-dimensional (vector) unseen tasks with much higher efﬁciency,showing promising generalization ability to different settings and even differentbenchmark domains.
Revised: A2-winner ) signiﬁcantly improves the performance on both high-dimensional(image) and low-dimensional (vector) unseen tasks with much higher efﬁciency,showing promising generalization ability to different settings and even differentbenchmark domains.

Original: We conduct a statistical analysis to reveal the relationsbetween patterns of auxiliary losses and RL performance.
Revised: We conduct a statistical analysis to reveal the relationsbetween patterns of auxiliary losses and RL performance.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) has achieved remarkable progress in games [39, 59, 63], ﬁnancialtrading [11] and robotics [16].
Revised: Reinforcement learning (RL) has achieved remarkable progress in games [31, 47, 50], ﬁnancialtrading [8] and robotics [13].

Original: However, in its core part, without designs tailored to speciﬁc tasks,general RL paradigms are still learning implicit representations from critic loss (value predictions)and actor loss (maximizing cumulative reward).
Revised: However, in its core part, without designs tailored to speciﬁc tasks,general RL paradigms are still learning implicit representations from critic loss (value predictions)and actor loss (maximizing cumulative reward).

Original: In many real-world scenarios where observations arecomplicated (e g., images) or incomplete (e g., partial observable), training an agent that is able toextract informative signals from those inputs becomes incredibly sample-inefﬁcient.
Revised: In many real-world scenarios where observations arecomplicated (e g., images) or incomplete (e g., partial observable), training an agent that is able toextract informative signals from those inputs becomes incredibly sample-inefﬁcient.

Original: Therefore, many recent works have been devoted to obtaining a good state representation, whichis believed to be one of the key solutions to improve the efﬁcacy of RL [27, 28].
Revised: Therefore, many recent works have been devoted to obtaining a good state representation, whichis believed to be one of the key solutions to improve the efﬁcacy of RL [23, 24].

Original: One of the mainstreams is adding auxiliary losses to update the state encoder.
Revised: One of the mainstreams is adding auxiliary losses to update the state encoder.

Original: Under the hood, it resorts to informativeand dense learning signals in order to encode various prior knowledge and regularization [49], andobtain better latent representations.
Revised: Under the hood, it resorts to informativeand dense learning signals in order to encode various prior knowledge and regularization [40], andobtain better latent representations.

Original: Over the years, a series of works have attempted to ﬁgure outthe form of the most helpful auxiliary loss for RL.
Revised: Over the years, a series of works have attempted to ﬁgure outthe form of the most helpful auxiliary loss for RL.

Original: Quite a few advances have been made, includingobservation reconstruction [64], reward prediction [24], environment dynamics prediction [49, 8, 43],etc.
Revised: Quite a few advances have been made, includingobservation reconstruction [51], reward prediction [20], environment dynamics prediction [40, 6, 35],etc.

Original: But we note two problems in this evolving process: (i) each of the loss designs listed above areobtained through empirical trial-and-errors based on expert designs, thus heavily relying on humanlabor and expertise; (ii) few works have used the ﬁnal performance of RL as an optimization objectiveto directly search the auxiliary loss, indicating that these designs could be sub-optimal.
Revised: But we note two problems in this evolving process: (i) each of the loss designs listed above areobtained through empirical trial-and-errors based on expert designs, thus heavily relying on humanlabor and expertise; (ii) few works have used the ﬁnal performance of RL as an optimization objectiveto directly search the auxiliary loss, indicating that these designs could be sub-optimal.

Original: To resolve the issues of the existing handcrafted solution mentioned above, we decide to automatethe process of designing the auxiliary loss functions of RL and propose a principled solution named
Revised: To resolve the issues of the existing handcrafted solution mentioned above, we decide to automatethe process of designing the auxiliary loss functions of RL and propose a principled solution named

Original: Automated Auxiliary Loss Search (A2LS).
Revised: Automated Auxiliary Loss Search (A2LS).

Original: A2LS formulates the problem as a bi-level optimization
Revised: A2LS formulates the problem as a bi-level optimization

Original: [Table]
Revised: [Table]

Original: (cid:48)(cid:88)(cid:87)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81) (cid:47)(cid:82)(cid:86)(cid:86)(cid:3)(cid:53)(cid:72)(cid:77)(cid:72)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81) (cid:53)(cid:72)(cid:77)(cid:72)(cid:70)(cid:87) where we try to ﬁnd the best auxiliary loss, which, to the most extent, helps train a good RL agent.
Revised: (cid:48)(cid:88)(cid:87)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81) (cid:47)(cid:82)(cid:86)(cid:86)(cid:3)(cid:53)(cid:72)(cid:77)(cid:72)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81) (cid:53)(cid:72)(cid:77)(cid:72)(cid:70)(cid:87) where we try to ﬁnd the best auxiliary loss, which, to the most extent, helps train a good RL agent.

Original: The outer loop searches for auxiliary losses based on RL performance to ensure the searched lossesalign with the RL objective, while the inner loop performs RL training with the searched auxiliaryloss function.
Revised: The outer loop searches for auxiliary losses based on RL performance to ensure the searched lossesalign with the RL objective, while the inner loop performs RL training with the searched auxiliaryloss function.

Original: Speciﬁcally, A2LS utilizes an evolutionary strategy to search the conﬁguration ofauxiliary losses over a novel search space of size 7 . 5 ⇥ 10 20 that covers many existing solutions.
Revised: Speciﬁcally, A2LS utilizes an evolutionary strategy to search the conﬁguration ofauxiliary losses over a novel search space of size 7 . 5 ⇥ 10 20 that covers many existing solutions.

Original: Bysearching on a small set of simulated training environments of continuous control from Deepmind
Revised: Bysearching on a small set of simulated training environments of continuous control from Deepmind

Original: Control suite (DMC) [55], A2LS ﬁnalizes a loss, namely A2-winner .
Revised: Control suite (DMC) [43], A2LS ﬁnalizes a loss, namely A2-winner .

Original: To evaluate the generalizability of the discovered auxiliary loss A2-winner , we test A2-winner ona wide set of test environments , including both image-based and vector-based (with proprioceptivefeatures like positions, velocities and accelerations as inputs) tasks.
Revised: To evaluate the generalizability of the discovered auxiliary loss A2-winner , we test A2-winner ona wide set of test environments , including both image-based and vector-based (with proprioceptivefeatures like positions, velocities and accelerations as inputs) tasks.

Original: Extensive experiments show thesearched loss function is highly effective and largely outperforms strong baseline methods.
Revised: Extensive experiments show thesearched loss function is highly effective and largely outperforms strong baseline methods.

Original: Moreimportantly, the searched auxiliary loss generalizes well to unseen settings such as (i) differentrobots of control; (ii) different data types of observation; (iii) partially observable settings; (iv)different network architectures; and (v) even to a totally different discrete control domain (Atari 2600games [2]).
Revised: Moreimportantly, the searched auxiliary loss generalizes well to unseen settings such as (i) differentrobots of control; (ii) different data types of observation; (iii) partially observable settings; (iv)different network architectures; and (v) even to a totally different discrete control domain (Atari 2600games [1]).

Original: In the end, we make detailed statistical analyses on the relation between RL performanceand patterns of auxiliary losses based on the data of whole evolutionary search process, providinguseful insights on future studies of auxiliary loss designs and representation learning in RL.
Revised: In the end, we make detailed statistical analyses on the relation between RL performanceand patterns of auxiliary losses based on the data of whole evolutionary search process, providinguseful insights on future studies of auxiliary loss designs and representation learning in RL.

Original:  Problem Formulation and Background
Revised:  Problem Formulation and Background

Original: We consider the standard Markov Decision Process (MDP) E where the state, action and rewardat time step t are denoted as ( s t , a t , r t ) .
Revised: We consider the standard Markov Decision Process (MDP) E where the state, action and rewardat time step t are denoted as ( s t , a t , r t ) .

Original: The sequence of rollout data sampled by the agent in theepisodic environment is ( s 0 , . . .
Revised: The sequence of rollout data sampled by the agent in theepisodic environment is ( s 0 , . . .

Original: , s t , a t , r t , s t +1 , · · · , s T ) , where T represents the episode length.
Revised: , s t , a t , r t , s t +1 , · · · , s T ) , where T represents the episode length.

Original: Suppose the RL agent is parameterized by !
Revised: Suppose the RL agent is parameterized by !

Original: (either the policy ⇡ or the state-action value function Q ),with a state encoder g ✓ parameterized by ✓ ✓ !
Revised: (either the policy ⇡ or the state-action value function Q ),with a state encoder g ✓ parameterized by ✓ ✓ !

Original: which plays a key role for representation learning in
Revised: which plays a key role for representation learning in

Original: RL.
Revised: RL.

Original: The agent is required to maximize its cumulative rewards in environment E by optimizing ! ,noted as R ( ! ; E ) =
Revised: The agent is required to maximize its cumulative rewards in environment E by optimizing ! ,noted as R ( ! ; E ) =

Original: E ⇡ [
Revised: E ⇡ [

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: In this paper, we aim to ﬁnd the optimal auxiliary loss function L Aux such that the agent can reach thebest performance by optimizing !
Revised: In this paper, we aim to ﬁnd the optimal auxiliary loss function L Aux such that the agent can reach thebest performance by optimizing !

Original: under a combination of an arbitrary RL loss function L RL togetherwith an auxiliary loss L Aux .
Revised: under a combination of an arbitrary RL loss function L RL togetherwith an auxiliary loss L Aux .

Original: Formally, our optimization goal is:
Revised: Formally, our optimization goal is:

Original: [Equation]
Revised: [Equation]

Original: (1) where  is a hyper-parameter balancing the relative weight of the auxiliary loss.
Revised: (1) where  is a hyper-parameter balancing the relative weight of the auxiliary loss.

Original: The left part (innerloop) of Figure 1 illustrates how data and gradients ﬂow in RL training when an auxiliary loss isenabled.
Revised: The left part (innerloop) of Figure 1 illustrates how data and gradients ﬂow in RL training when an auxiliary loss isenabled.

Original: Some instances of L RL and L Aux are given in
Revised: Some instances of L RL and L Aux are given in

Original: Appendix B. Unfortunately, existing auxiliarylosses L Aux are handcrafted, which heavily rely on expert knowledge, and may not generalize wellin different scenarios as shown in the experiment part.
Revised: Appendix B. Unfortunately, existing auxiliarylosses L Aux are handcrafted, which heavily rely on expert knowledge, and may not generalize wellin different scenarios as shown in the experiment part.

Original: To ﬁnd better auxiliary loss functions forrepresentation learning in RL, we introduce our principled solution in the following section.
Revised: To ﬁnd better auxiliary loss functions forrepresentation learning in RL, we introduce our principled solution in the following section.

Original: [Table]
Revised: [Table]

Original:  Automated Auxiliary Loss Search
Revised:  Automated Auxiliary Loss Search

Original: To meet our goal of ﬁnding top-performing auxiliary loss functions without expert assignment, weturn to the help of automated loss search, which has shown promising results in the automatedmachine learning (AutoML) community [31, 32, 60].
Revised: To meet our goal of ﬁnding top-performing auxiliary loss functions without expert assignment, weturn to the help of automated loss search, which has shown promising results in the automatedmachine learning (AutoML) community [27, 28, 48].

Original: Correspondingly, we propose Automated
Revised: Correspondingly, we propose Automated

Original: Auxiliary Loss Search (A2LS), a principled solution for resolving the above bi-level optimizationproblem in Equation 1.
Revised: Auxiliary Loss Search (A2LS), a principled solution for resolving the above bi-level optimizationproblem in Equation 1.

Original: A2LS resolves the inner problem as a standard RL training procedure; for theouter one, A2LS deﬁnes a ﬁnite and discrete search space (Section 3.1), and designs a novel evolutionstrategy to efﬁciently explore the space (Section 3.2).
Revised: A2LS resolves the inner problem as a standard RL training procedure; for theouter one, A2LS deﬁnes a ﬁnite and discrete search space (Section 3.1), and designs a novel evolutionstrategy to efﬁciently explore the space (Section 3.2).

Original: 3.1 Search Space Design
Revised: 3.1 Search Space Design

Original: We have argued that almost all existing auxiliary losses require expert knowledge, and we expect tosearch for a better one automatically.
Revised: We have argued that almost all existing auxiliary losses require expert knowledge, and we expect tosearch for a better one automatically.

Original: To this end, it is clear that we should design a search space thatsatisﬁes the following desiderata.
Revised: To this end, it is clear that we should design a search space thatsatisﬁes the following desiderata.

Original: • Generalization : the search space should cover most of the existing handcrafted auxiliary losses toensure the searched results can be no worse than handcrafted losses;
Revised: • Generalization : the search space should cover most of the existing handcrafted auxiliary losses to ensure the searched results can be no worse than handcrafted losses;

Original: • Atomicity : the search space should be composed of several independent dimensions to ﬁt into any
Revised: • Atomicity : the search space should be composed of several independent dimensions to ﬁt into any

Original: general search algorithm  and support an efﬁcient 
Revised: general search algorithm [30] and support an efﬁcient search scheme;

Original: search scheme;

Original: • Sufﬁciency : the search space should be large enough to contain the top-performing solutions
Revised: • Sufﬁciency : the search space should be large enough to contain the top-performing solutions.

Original: Given the criteria, we conclude and list some existing auxiliary losses in Table 1 and ﬁnd theircommonalities, as well as differences.
Revised: Given the criteria, we conclude and list some existing auxiliary losses in Table 1 and ﬁnd theircommonalities, as well as differences.

Original: We realize that these losses share similar components andcomputation ﬂow.
Revised: We realize that these losses share similar components andcomputation ﬂow.

Original: As shown in Figure 2, when training the RL agent, the loss ﬁrstly selects a sequence{ s t , a t , r t } i + k t
Revised: As shown in Figure 2, when training the RL agent, the loss ﬁrstly selects a sequence{ s t , a t , r t } i + k t

Original: = i from the replay buffer, when k is called horizon .
Revised: = i from the replay buffer, when k is called horizon .

Original: The agent then tries to predict someelements in the sequence (called target ) based on another picked set of elements from the sequence(called source ).
Revised: The agent then tries to predict someelements in the sequence (called target ) based on another picked set of elements from the sequence(called source ).

Original: Finally, the loss calculates and minimizes the prediction error (rigorously deﬁnedwith operator ).
Revised: Finally, the loss calculates and minimizes the prediction error (rigorously deﬁnedwith operator ).

Original: To be more speciﬁc, the encoder part g ✓ of the agent, ﬁrst encodes the source intolatent representations, which is further fed into a predictor h to get a prediction y ;
Revised: To be more speciﬁc, the encoder part g ✓ of the agent, ﬁrst encodes the source intolatent representations, which is further fed into a predictor h to get a prediction y ;

Original: the auxiliary loss iscomputed by the prediction y and the target
Revised: the auxiliary loss iscomputed by the prediction y and the target

Original: ˆ y that is translated from the target by a target encoder g ˆ ✓ ,using an operator f .
Revised: ˆ y that is translated from the target by a target encoder g ˆ ✓ ,using an operator f .

Original: The target encoder is updated in an momentum manner as shown in Figure 2(details are given in Appendix C.1.2).

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: where seq source , seq target ✓ { s t , a t , r t } i + k t
Revised: where seq source , seq target ✓ { s t , a t , r t } i + k t

Original: = i are both subsets of the candidate sequence.
Revised: = i are both subsets of the candidate sequence.

Original: And forsimplicity, we will denote g ✓ ( s t , a t , r t , s t +1 , · · · ) as short for [ g ✓ ( s t ) , a t , r t , g ✓ ( s t +1 ) , · · · ] for the 100 rest of this paper (the encoder g only deals with states { s i } ).
Revised: And forsimplicity, we will denote g ✓ ( s t , a t , r t , s t +1 , · · · ) as short for [ g ✓ ( s t ) , a t , r t , g ✓ ( s t +1 ) , · · · ] for the 99 rest of this paper (the encoder g only deals with states { s i } ).

Original: Thereafter, we observe that these 101 existing auxiliary losses differ in two dimensions, i.e., input elements and operator , where inputelements are further combined by horizon , source and target .
Revised: Thereafter, we observe that these 100 existing auxiliary losses differ in two dimensions, i.e., input elements and operator , where inputelements are further combined by horizon , source and target .

Original: These differences compose our searchdimensions of the whole space.
Revised: These differences compose our searchdimensions of the whole space.

Original: We then illustrate the search ranges of these dimensions in detail.
Revised: We then illustrate the search ranges of these dimensions in detail.

Original: Input elements.
Revised: Input elements.

Original: The input elements denote all inputs to the loss functions, which can be furtherdisassembled as horizon , source and target .
Revised: The input elements denote all inputs to the loss functions, which can be furtherdisassembled as horizon , source and target .

Original: Different from previous automated loss search works, thetarget here is not “ground-truth” because auxiliary losses in RL have no labels beforehand.
Revised: Different from previous automated loss search works, thetarget here is not “ground-truth” because auxiliary losses in RL have no labels beforehand.

Original: Instead,both source and target are generated via interacting with the environment in a self-supervised manner.
Revised: Instead,both source and target are generated via interacting with the environment in a self-supervised manner.

Original: Particularly, the input elements ﬁrst determine a candidate sequence { s t , a t , r t } i + k t
Revised: Particularly, the input elements ﬁrst determine a candidate sequence { s t , a t , r t } i + k t

Original: = i with horizon k .
Revised: = i with horizon k .

Revised: Then, it chooses two subsets from the candidate sequence as source and target respectively.

Revised: Forexample, the subsets can be { s t } , { s t , s t +1 } , or { s t , r t +1 , a t +2 } , { s t , s t +1 , a t +1 } , etc.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: I selects a candidate sequence { s t , a t , r t } i + k t
Revised: I selects a candidate sequence { s t , a t , r t } i + k t

Original: = i with horizon k ; then determine a source and a target as arbitrary subsets of the sequence; an encoder g ✓ ﬁrst encodes the source into latent representations, which is fed into a predictor h to get a prediction y ; the auxiliary loss is computed over the prediction y and the ground truth ˆ y that is translated from the target by a target encoder g ˆ ✓ , using a operator f .
Revised: = i with horizon k ; then determine a source and a target as arbitrary subsets of the sequence; an encoder g ✓ ﬁrst encodes the source into latent representations, which is fed into a predictor h to get a prediction y ; the auxiliary loss is computed over the prediction y and the ground truth ˆ y that is translated from the target by a target encoder g ˆ ✓ , using a operator f .

Original: Then, it chooses two subsets from the candidate sequence as source and target respectively.

Original: Forexample, the subsets can be { s t } , { s t , s t +1 } , or { s t , r t +1 , a t +2 } , { s t , s t +1 , a t +1 } , etc.

Original: Operator.
Revised: Operator.

Original: Given a prediction y and its target ˆ y , the auxiliary loss is computed by an operatorf , which is often a similarity measure.
Revised: Given a prediction y and its target ˆ y , the auxiliary loss is computed by an operatorf , which is often a similarity measure.

Original: In our work, we cover all different operators f used bythe previous works, including inner product (Inner) [20, 54], bilinear inner product (Bilinear)
Revised: In our work, we cover all different operators f used bythe previous works, including inner product (Inner) [17, 42], bilinear inner product (Bilinear)

Original: [27],cosine similarity (Cosine) [4], mean squared error (MSE) [43, 8] and normalized mean squared error(N-MSE) [48].
Revised: [23],cosine similarity (Cosine) [3], mean squared error (MSE) [35, 6] and normalized mean squared error(N-MSE) [39].

Original: Additionally, other works also utilize contrastive objectives, e.g., InfoNCE loss [41],incorporating the trick to sample un-paired predictions and targets as negative samples and maximizethe distances between them.
Revised: Additionally, other works also utilize contrastive objectives, e.g., InfoNCE loss [33],incorporating the trick to sample un-paired predictions and targets as negative samples and maximizethe distances between them.

Original: This technique is orthogonal to the ﬁve similarity measures mentionedabove, so we make it optional and create 5 ⇥ 2 = 10 different operators in total.
Revised: This technique is orthogonal to the ﬁve similarity measures mentionedabove, so we make it optional and create 5 ⇥ 2 = 10 different operators in total.

Original: Final design.
Revised: Final design.

Original: In the light of preceding discussion, with the deﬁnition of input elements and operator ,we ﬁnish the design of the search space, which satisfactorily meets the desiderata mentioned above.
Revised: In the light of preceding discussion, with the deﬁnition of input elements and operator ,we ﬁnish the design of the search space, which satisfactorily meets the desiderata mentioned above.

Original: Speciﬁcally, the space is generalizable to cover most of the existing handcrafted auxiliary losses;additionally, the atomicity is embodied by the compositionality that all input elements work withany operator ; most importantly, the search space is sufﬁciently large with a total size of 7 . 5 ⇥ 10 20(detailed calculation can be found in Appendix E) to ﬁnd better solutions.
Revised: Speciﬁcally, the space is generalizable to cover most of the existing handcrafted auxiliary losses;additionally, the atomicity is embodied by the compositionality that all input elements work withany operator ; most importantly, the search space is sufﬁciently large with a total size of 7 . 5 ⇥ 10 20(detailed calculation can be found in Appendix E) to ﬁnd better solutions.

Original: 3.2 Search Strategy
Revised: 3.2 Search Strategy

Original: The success of evolution strategies in exploring large, multi-dimensional search space has been provenin many works [23, 6].
Revised: The success of evolution strategies in exploring large, multi-dimensional search space has been provenin many works [19, 4].

Original: Similarly, A2LS adopts an evolutionary algorithm [45] to search for topperforming auxiliary loss functions over the designed search space.
Revised: Similarly, A2LS adopts an evolutionary algorithm [37] to search for topperforming auxiliary loss functions over the designed search space.

Original: In its essence, the evolutionaryalgorithm (i) keeps a population of loss function candidates; (ii) evaluates their performance; (iii)eliminates the worst and evolves into a new better population.
Revised: In its essence, the evolutionaryalgorithm (i) keeps a population of loss function candidates; (ii) evaluates their performance; (iii)eliminates the worst and evolves into a new better population.

Original: Note that step (ii) of “evaluating” isvery costly because it needs to train the RL agents with dozens of different auxiliary loss functions.
Revised: Note that step (ii) of “evaluating” isvery costly because it needs to train the RL agents with dozens of different auxiliary loss functions.

Original: Therefore, our key technical contribution contains how to further reduce the search cost (Section 3.2.1)and how to make an efﬁcient search procedure (Section 3.2.2).
Revised: Therefore, our key technical contribution contains how to further reduce the search cost (Section 3.2.1)and how to make an efﬁcient search procedure (Section 3.2.2).

Original: 3.2.1 Search Space Pruning
Revised: 3.2.1 Search Space Pruning

Original: In our preliminary experiment, we ﬁnd out the dimension of operator in the search space can besimpliﬁed.
Revised: In our preliminary experiment, we ﬁnd out the dimension of operator in the search space can besimpliﬁed.

Original: In particular, MSE outperforms all the others by signiﬁcant gaps in most cases.
Revised: In particular, MSE outperforms all the others by signiﬁcant gaps in most cases.

Original: So weeffectively prune other choices of operators except MSE.
Revised: So weeffectively prune other choices of operators except MSE.

Original: See Appendix D.1 for complete comparativeresults and an ablation study on the effectiveness of search space pruning.
Revised: See Appendix D.1 for complete comparativeresults and an ablation study on the effectiveness of search space pruning.

Original: 3.2.2 Evolution Procedure
Revised: 3.2.2 Evolution Procedure

Original: Our evolution procedure roughly contains four important components: (i) evaluation and selection :a population of candidate auxiliary losses is evaluated through an inner loop of RL training, then weselect the top candidates for the next evolution stage (i.e., generation); (ii) mutation : the selectedcandidates mutate to form a new population and move to the next stage; (iii) loss rejection : ﬁlter 
Revised: Our evolution procedure roughly contains four important components: (i) evaluation and selection :a population of candidate auxiliary losses is evaluated through an inner loop of RL training, then weselect the top candidates for the next evolution stage; (ii) mutation : the selected candidates mutate toform a new population; (iii) loss rejection : ﬁlter out and skip evaluating invalid auxiliary losses forthe next stage; and (iv) bootstrapping initial population : assign more chance to initial auxiliarylosses that may contain useful patterns by prior knowledge for higher efﬁciency. The step-by-step

Original: [Table]
Revised: [Table]

Original: (cid:54)(cid:72)(cid:79)(cid:72)(cid:70)(cid:87)(cid:72)(cid:71) (cid:49)(cid:82)(cid:87)(cid:3)(cid:54)(cid:72)(cid:79)(cid:72)(cid:70)(cid:87)(cid:72)(cid:71)out and skip evaluating invalid auxiliary losses for the next stage; and (iv) bootstrapping initialpopulation : assign more chance to initial auxiliary losses that may contain useful patterns by priorknowledge for higher efﬁciency. The step-by-step evolution algorithm is provided in Algorithm 1 inthe appendix, and an overview of the A2LS pipeline is illustrated in Figure 1.
Revised: (cid:54)(cid:72)(cid:79)(cid:72)(cid:70)(cid:87)(cid:72)(cid:71) (cid:49)(cid:82)(cid:87)(cid:3)(cid:54)(cid:72)(cid:79)(cid:72)(cid:70)(cid:87)(cid:72)(cid:71) evolution algorithm is provided in Algorithm 1 in the appendix, and an overview of the A2LSpipeline is illustrated in Figure 1.

Original: We next describe themin detail.
Revised: We next describe them in detail.

Original: Evaluation and selection.
Revised: Evaluation and selection.

Original: At each evolution stage, we ﬁrst train a population of candidates witha population size P = 100 by the inner loop of RL training.
Revised: At each evolution stage, we ﬁrst train a population of candidates witha population size P = 100 by the inner loop of RL training.

Original: The candidates are then sorted bycomputing the approximated area under learning curve (AULC)
Revised: The candidates are then sorted bycomputing the approximated area under learning curve (AULC)

Original: [14, 52], which is a single metricreﬂecting both the convergence speed and the ﬁnal performance [58] with low variance of results.
Revised: [11, 41], which is a single metricreﬂecting both the convergence speed and the ﬁnal performance [46] with low variance of results.

Original: After each training stage, the top-25% candidates are selected to generate the population for the nextstage.
Revised: After each training stage, the top-25% candidates are selected to generate the population for the nextstage.

Original: We include an ablation study on the effectiveness of AULC in Appendix D.3.
Revised: We include an ablation study on the effectiveness of AULC in Appendix D.3.

Original: Mutation.
Revised: Mutation.

Original: To obtain a new population of auxiliary loss functions, we propose a novel mutationstrategy.
Revised: To obtain a new population of auxiliary loss functions, we propose a novel mutationstrategy.

Original: First, we represent both the source and the target of the input elements as a pair of binarymasks, where each bit of the mask represents selected by 1 or not selected by 0.
Revised: First, we represent both the source and the target of the input elements as a pair of binarymasks, where each bit of the mask represents selected by 1 or not selected by 0.

Original: For instance,given a candidate sequence { s t , a t , r t , s t +1 , a t +1 , r t +1 } , the binary mask of this subset sequence{ s t , a t , r t +1 } is denoted as 110001 .
Revised: For instance,given a candidate sequence { s t , a t , r t , s t +1 , a t +1 , r t +1 } , the binary mask of this subset sequence{ s t , a t , r t +1 } is denoted as 110001 .

Original: Afterward, we adopt four types of mutations, also shown in
Revised: Afterward, we adopt four types of mutations, also shown in

Original: Figure 3: (i) replacement (50% of the population): ﬂip the given binary mask with probabilityp = 1 2 · (3 k +3) with the horizon length k ; (ii) crossover (20%): generate a new candidate by randomlycombining the mask bits of two candidates with the same horizon length in the population; (iii)horizon decrease and horizon increase (10%): append new binary masks to the tail or delete existingbinary masks at the back.
Revised: Figure 3: (i) replacement (50% of the population): ﬂip the given binary mask with probabilityp = 1 2 · (3 k +3) with the horizon length k ; (ii) crossover (20%): generate a new candidate by randomlycombining the mask bits of two candidates with the same horizon length in the population; (iii)horizon decrease and horizon increase (10%): append new binary masks to the tail or delete existingbinary masks at the back.

Original: (iv) random generation (20%): every bit of the binary mask is generatedfrom a Bernoulli distribution B (0 . 5) .
Revised: (iv) random generation (20%): every bit of the binary mask is generatedfrom a Bernoulli distribution B (0 . 5) .

Original: Loss rejection protocol.
Revised: Loss rejection protocol.

Original: Since the auxiliary loss needs to be differentiable with respect to theparameters of the state encoder, we perform a gradient ﬂow check on randomly generated lossfunctions during evolution and skip evaluating invalid auxiliary losses.
Revised: Since the auxiliary loss needs to be differentiable with respect to theparameters of the state encoder, we perform a gradient ﬂow check on randomly generated lossfunctions during evolution and skip evaluating invalid auxiliary losses.

Original: Concretely, the followingconditions must be satisﬁed to make a valid loss function: (i) having at least one state element inseq source to make sure the gradient of auxiliary loss can propagate back to the state encoder; (ii)seq target is not empty; (iii) the horizon should be within a reasonable range ( 1  k  10 in ourexperiments).
Revised: Concretely, the followingconditions must be satisﬁed to make a valid loss function: (i) having at least one state element inseq source to make sure the gradient of auxiliary loss can propagate back to the state encoder; (ii)seq target is not empty; (iii) the horizon should be within a reasonable range ( 1  k  10 in ourexperiments).

Original: If a loss is rejected, we repeat the mutation to ﬁll the population.
Revised: If a loss is rejected, we repeat the mutation to ﬁll the population.

Original: Bootstrapping initial population.
Revised: Bootstrapping initial population.

Original: To improve the computational efﬁciency so that the algorithm canﬁnd reasonable loss functions quickly, we incorporate prior knowledge into the initialization of thesearch.
Revised: To improve the computational efﬁciency so that the algorithm canﬁnd reasonable loss functions quickly, we incorporate prior knowledge into the initialization of thesearch.

Original: Particularly, before the ﬁrst stage of evolution, we bootstrap the initial population with a priordistribution that assigns high probability to auxiliary loss functions containing useful patterns likedynamics and reward prediction.
Revised: Particularly, before the ﬁrst stage of evolution, we bootstrap the initial population with a priordistribution that assigns high probability to auxiliary loss functions containing useful patterns likedynamics and reward prediction.

Original: More implementation details are provided in Appendix C.3.
Revised: More implementation details are provided in Appendix C.3.

Original:  Evolution and Searched Results
Revised:  Evolution and Searched Results

Original: As mentioned in Section 1, we expect to ﬁnd auxiliary losses that align with the RL objective andgeneralize well to unseen test environments .
Revised: As mentioned in Section 1, we expect to ﬁnd auxiliary losses that align with the RL objective andgeneralize well to unseen test environments .

Original: To do so, we use A2LS to search over a small set oftraining environments , and then test the searched results on a wide range of test environments .
Revised: To do so, we use A2LS to search over a small set oftraining environments , and then test the searched results on a wide range of test environments .

Original: In thissection, we ﬁrst introduce the evolution on training environments and search results.
Revised: In thissection, we ﬁrst introduce the evolution on training environments and search results.

Original: [Figure]
Revised: [Figure]

Original: 4.1 Evolution on Training Environments
Revised: 4.1 Evolution on Training Environments

Original: The training environments are chosen as three image-based (observations for agents are images)continuous control tasks in DMC benchmark [55], Cheetah-Run, Reacher-Easy, and Walker-Walk.
Revised: The training environments are chosen as three image-based (observations for agents are images)continuous control tasks in DMC benchmark [43], Cheetah-Run, Reacher-Easy, and Walker-Walk.

Original: For each environment, we set the total budget to 16k GPU hours (on NVIDIA P100) and terminatethe search when the resource is exhausted.
Revised: For each environment, we set the total budget to 16k GPU hours (on NVIDIA P100) and terminatethe search when the resource is exhausted.

Original: Due to computation complexity, we only run one seedfor each inner loop RL training, but we try to prevent such randomness by cross validation (see
Revised: Due to computation complexity, we only run one seedfor each inner loop RL training, but we try to prevent such randomness by cross validation (see

Original: Section 4.2).
Revised: Section 4.2).

Original: We use the same network architecture and hyperparameters conﬁg as CURL [27] (see
Revised: We use the same network architecture and hyperparameters conﬁg as CURL [23] (see

Original: Appendix C.4.1 for details) to train the RL agents.
Revised: Appendix C.4.1 for details) to train the RL agents.

Original: To evaluate the population during evolution, wemeasure A2LS as compared to SAC, SAC-wo-aug, and CURL, where we randomly crop imagesfrom 100 ⇥ 100 to 84 ⇥ 84 as data augmentation (the same technique used in CURL[27]) for allmethods except SAC-wo-aug.
Revised: To evaluate the population during evolution, wemeasure A2LS as compared to SAC, SAC-wo-aug, and CURL, where we randomly crop imagesfrom 100 ⇥ 100 to 84 ⇥ 84 as data augmentation (the same technique used in CURL[23]) for allmethods except SAC-wo-aug.

Original: The whole evolution process on three environments is demonstrated in
Revised: The whole evolution process on three environments is demonstrated in

Original: Figure 4.
Revised: Figure 4.

Original: Even in the early stages (e.g., stage 1), some of the auxiliary loss candidates already surpassbaselines, indicating the high potential of automated loss search.
Revised: Even in the early stages (e.g., stage 1), some of the auxiliary loss candidates already surpassbaselines, indicating the high potential of automated loss search.

Original: The overall AULC scores of thepopulation continue to improve when more stages come in.
Revised: The overall AULC scores of thepopulation continue to improve when more stages come in.

Original: Judging from the trend, we believe theperformances could improve even more if we had further increased the budget.
Revised: Judging from the trend, we believe theperformances could improve even more if we had further increased the budget.

Original: 4.2 Searched Results: A2-winner
Revised: 4.2 Searched Results: A2-winner

Original: [Figure]
Revised: [Figure]

Original: li z e d AU

Original: L C S c o r e ( w . r . t l o w es t sc o r e ) Although some candidates in the population have achieved remarkable AULC scores in the evolution (Figure 4), they wereonly evaluated with one random seed in one environment,making their robustness under question.
Revised: Although some candidates in the population have achieved remarkable AULC scores in the evolution (Figure 4), they wereonly evaluated with one random seed in one environment,making their robustness under question.

Original: To ensure that weﬁnd a consistently-useful auxiliary loss, we conduct a crossvalidation.
Revised: To ensure that weﬁnd a consistently-useful auxiliary loss, we conduct a crossvalidation.

Original: We ﬁrst choose the top 5 candidates of stage-5 ofthe evolution on Cheetah-Run (detailed top candidates duringthe whole evolution procedure are provided in Appendix F).
Revised: We ﬁrst choose the top 5 candidates of stage-5 ofthe evolution on Cheetah-Run (detailed top candidates duringthe whole evolution procedure are provided in Appendix F).

Original: For each of the ﬁve candidates, we repeat the RL training onall three training environments , shown in Figure 5.
Revised: For each of the ﬁve candidates, we repeat the RL training onall three training environments , shown in Figure 5.

Original: Finally, we mark the best among ﬁve (green barin Figure 5) as our ﬁnal searched result.
Revised: Finally, we mark the best among ﬁve (green barin Figure 5) as our ﬁnal searched result.

Original: We call it A2-winner , which has the following form:
Revised: We call it A2-winner , which has the following form:

Original: [Equation]
Revised: [Equation]

Original:  Generalization Experiments
Revised:  Generalization Experiments

Original: To verify the effectiveness of the searched results, we conduct various generalization experimentson a wide range of test environments in depth.
Revised: To verify the effectiveness of the searched results, we conduct various generalization experimentson a wide range of test environments in depth.

Original: Implementation details and more ablation studies aregiven in Appendix C and Appendix D.
Revised: Implementation details and more ablation studies aregiven in Appendix C and Appendix D.

Original: Generalize to unseen image-based tasks.
Revised: Generalize to unseen image-based tasks.

Original: We ﬁrst investigate the generalizability of A2-winnerto unseen image-based tasks by training agents with A2-winner on common DMC tasks andcompare with model-based and model-free baselines that use different auxiliary loss functions(see Appendix C.5 for details about baseline methods).
Revised: We ﬁrst investigate the generalizability of A2-winnerto unseen image-based tasks by training agents with A2-winner on common DMC tasks andcompare with model-based and model-free baselines that use different auxiliary loss functions(see Appendix C.5 for details about baseline methods).

Original: The results are summarized in Table 2where A2-winner greatly outperforms other baseline methods on most tasks, including unseen testenvironments .
Revised: The results are summarized in Table 2where A2-winner greatly outperforms other baseline methods on most tasks, including unseen testenvironments .

Original: This implies that A2-winner is a robust and effective auxiliary loss for image-basedcontinuous control tasks to improve both the efﬁciency and ﬁnal performance.
Revised: This implies that A2-winner is a robust and effective auxiliary loss for image-basedcontinuous control tasks to improve both the efﬁciency and ﬁnal performance.

Original: The baseline methods are PlaNet [19], Dreamer [18], SAC+AE [64], SLAC [30], image-based
Revised: The baseline methods are PlaNet [16], Dreamer [15], SAC+AE [51], SLAC [26], image-based

Original: SAC [17].
Revised: SAC [14].

Original: Performance values of all baselines are referred to [27], except for Image SAC.
Revised: Performance values of all baselines are referred to [23], except for Image SAC.

Original: Learning curves of all 12 DMC environments are included in Appendix D.2.
Revised: Learning curves of all 12 DMC environments are included in Appendix D.2.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Generalize to totally differentbenchmark domains.
Revised: Generalize to totally differentbenchmark domains.

Original: To further verify the generalizability of A2-winneron totally different benchmarkdomains other than DMC tasks, weconduct experiments on the Atari2600 Games [2], where we take Efﬁcient Rainbow [56] as the base RL algorithm and add A2-winnerto obtain a better state representation.
Revised: To further verify the generalizability of A2-winneron totally different benchmarkdomains other than DMC tasks, weconduct experiments on the Atari2600 Games [1], where we take Efﬁcient Rainbow [44] as the base RL algorithm and add A2-winnerto obtain a better state representation.

Original: Results are shown in Table 3 where A2-winner outperformsall baselines, showing strong evidence of the generalization and potential usages of A2-winner.
Revised: Results are shown in Table 3 where A2-winner outperformsall baselines [Table] showing strong evidence of the generalization and potential usages of A2-winner

Original: Note that the base RL algorithm used in Atari is a value-based method, indicating that A2-winnergeneralizes well to both value-based and policy-based RL algorithms.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Generalize to different observationtypes.
Revised: Generalize to different observationtypes.

Original: To see whether A2-winner(searched in image-based environments) is able to generalize to theenvironments with different observation types, we test A2-winner onvector-based (inputs for RL agentsare proprioceptive features such as positions, velocities and accelerations)tasks of DMC and list the results in
Revised: To see whether A2-winner(searched in image-based environments) is able to generalize to theenvironments with different observation types, we test A2-winner onvector-based (inputs for RL agentsare proprioceptive features such as positions, velocities and accelerations)tasks of DMC and list the results in

Original: Table 4.
Revised: Table 4.

Original: Concretely, we compare
Revised: Concretely, we compare

Original: A2-winner with SAC-Identity, SACand CURL, where SAC-Identity doesnot have state encoder while the others share the same state encoder architecture (See Appendix C.1.1 and
Revised: A2-winner with SAC-Identity, SACand CURL, where SAC-Identity doesnot have state encoder while the others share the same state encoder architecture (See Appendix C.1.1 and

Original: Appendix D.7 for detailed implementations).
Revised: Appendix D.7 for detailed implementations).

Original: To our delight, A2-winner still outperforms allbaselines in 12 out of 18 environments, showing A2-winner can also beneﬁt RL performance invector-based observations.
Revised: To our delight, A2-winner still outperforms allbaselines in 12 out of 18 environments, showing A2-winner can also beneﬁt RL performance invector-based observations.

Original: Moreover, the performance gain is particularly signiﬁcant in more complexenvironments like Humanoid, where SAC barely learns anything at 1000K time steps.
Revised: Moreover, the performance gain is particularly signiﬁcant in more complexenvironments like Humanoid, where SAC barely learns anything at 1000K time steps.

Original: In order to geta deeper understanding of this phenomenon, we additionally visualize the Q loss landscape for bothmethods in Appendix D.8.
Revised: In order to geta deeper understanding of this phenomenon, we additionally visualize the Q loss landscape for bothmethods in Appendix D.8.

Original: More generalization experiments.
Revised: More generalization experiments.

Original: We provided more in-depth experiments for evaluating thegeneralizability of A2-winner to (i) different hypothesis spaces, i.e., different encoder architecturesand (ii) partially observable scenarios.
Revised: We provided more in-depth experiments for evaluating thegeneralizability of A2-winner to (i) different hypothesis spaces, i.e., different encoder architecturesand (ii) partially observable scenarios.

Original: Refer to Appendix D.4 for more details.
Revised: Refer to Appendix D.4 for more details.

Original: In summary, we ﬁnd
Revised: In summary, we ﬁnd

Original: A2-winner with more expressive encoder architecture could achieve better performance (indicatingthat A2-winner does not overﬁt one certain encoder architecture), and A2-winner generalizes wellto partially observable environments with obvious advantages over baselines.
Revised: A2-winner with more expressive encoder architecture could achieve better performance (indicatingthat A2-winner does not overﬁt one certain encoder architecture), and A2-winner generalizes wellto partially observable environments with obvious advantages over baselines.

Original: To search or not?
Revised: To search or not?

Original: As shown above, the searched result A2-winner can generalize well to all kindsof different settings.
Revised: As shown above, the searched result A2-winner can generalize well to all kindsof different settings.

Original: A natural question here is, however, for a new type of domain, why not performa new evolution search, instead of simply using the previously searched result?
Revised: A natural question here is, however, for a new type of domain, why not perform a new evolution search, instead of simply using the previously searched result?

Original: To compare thesetwo solutions, we conduct another evolutionary search similar to Section 4.1 but replaced the threeimage-based tasks with three vector-based ones (marked by † in Table 4) from scratch.
Revised: To compare thesetwo solutions, we conduct another evolutionary search similar to Section 4.1 but replaced the threeimage-based tasks with three vector-based ones (marked by † in Table 4) from scratch.

Original: More detailsare summarized in Appendix D.6.
Revised: More detailsare summarized in Appendix D.6.

Original: We name the searched result as “ A2-winner-v ”.
Revised: We name the searched result as “ A2-winner-v ”.

Original: As shown in
Revised: As shown in

Original: Table 4, A2-winner-v is a very strong-performing loss for vector-based tasks, even stronger than
Revised: Table 4, A2-winner-v is a very strong-performing loss for vector-based tasks, even stronger than

Original: A2-winner .
Revised: A2-winner .

Original: Actually, A2-winner-v is able to outperform baselines in 16 out of 18 environments(with 15 unseen test environments ), while A2-winner only outperforms baselines in 12 out of 18environments.
Revised: Actually, A2-winner-v is able to outperform baselines in 16 out of 18 environments(with 15 unseen test environments ), while A2-winner only outperforms baselines in 12 out of 18environments.

Original: However, please note that it costs another 5k GPU hours (on NVIDIA P100) tosearch for A2-winner-v while there is no additional cost to directly use A2-winner .
Revised: However, please note that it costs another 5k GPU hours (on NVIDIA P100) tosearch for A2-winner-v while there is no additional cost to directly use A2-winner .

Original: It is a trade-offbetween lower computational cost and better performance.
Revised: It is a trade-offbetween lower computational cost and better performance.

Original:  Analysis of Auxiliary Loss Functions
Revised:  Analysis of Auxiliary Loss Functions

Original: In this section, we analyze all the loss functions we have evaluated during the evolution procedure asa whole dataset in order to gain some insights into the role of auxiliary loss in RL performance.
Revised: In this section, we analyze all the loss functions we have evaluated during the evolution procedure asa whole dataset in order to gain some insights into the role of auxiliary loss in RL performance.

Original: Bydoing so, we hope to shed light on future auxiliary loss designs.
Revised: Bydoing so, we hope to shed light on future auxiliary loss designs.

Original: We will also release this “dataset”publicly to facilitate future research.
Revised: We will also release this “dataset”publicly to facilitate future research.

Original: Typical patterns.
Revised: Typical patterns.

Original: We say that an auxiliary loss candidate has a certain pattern if the pattern’s sourceis a subset of the candidate’s source , and the pattern’s target is a subset of the candidate’s target .
Revised: We say that an auxiliary loss candidate has a certain pattern if the pattern’s sourceis a subset of the candidate’s source , and the pattern’s target is a subset of the candidate’s target .

Original: For instance, a loss candidate of { s t , a t } !
Revised: For instance, a loss candidate of { s t , a t } !

Original: { s t +1 , s t +2 } has the pattern { s t , a t } !
Revised: { s t +1 , s t +2 } has the pattern { s t , a t } !

Original: { s t +1 } , and 280 does not have the pattern { a t , s t +1 } ! {
Revised: { s t +1 } , and 276 does not have the pattern { a t , s t +1 } ! {

Original: s t } .
Revised: s t } .

Original: We then try to analyze whether a certain pattern is 281 helpful to representation learning in RL in expectation.
Revised: We then try to analyze whether a certain pattern is 277 helpful to representation learning in RL in expectation.

Original: Speciﬁcally, we analyze the following patterns: (i) forward dynamics { s t , a t } !
Revised: Speciﬁcally, we analyze the following patterns: (i) forward dynamics { s t , a t } !

Original: { s t +1 } ; (ii) 283 inverse dynamics { a t , s t +1 } !
Revised: { s t +1 } ; (ii) 279 inverse dynamics [Equation]

Original: { s t } ; (iii) reward prediction { s t , a t } !
Revised: s t } ; (iii) reward prediction { s t , a t } !

Original: { r t } ; (iv) action inference{ s t , s t +1 } !
Revised: { r t } ; (iv) action inference{ s t , s t +1 } !

Original: { a t } and (v) state reconstruction in the latent space { s t } ! {
Revised: { a t } and (v) state reconstruction in the latent space { s t } ! {

Original: s t } .
Revised: s t } .

Original: For each of these 285 patterns, we categorize all the loss functions we have evaluated into (i) with or (ii) without this pattern.
Revised: For each of these 281 patterns, we categorize all the loss functions we have evaluated into (i) with or (ii) without this pattern.

Original: We then calculate the average RL performances of these two categories, summarized in Table 5.
Revised: We then calculate the average RL performances of these two categories, summarized in Table 5.

Original: Some interesting observations are as follows.
Revised: Some interesting observations are as follows.

Original: (i) Forward dynamics is helpful in most tasks and improves RL performance on Reacher-Easy
Revised: (i) Forward dynamics is helpful in most tasks and improves RL performance on Reacher-Easy

Original: (image) and Cheetah-Run (vector) signiﬁcantly (p-value < 0.05).
Revised: (image) and Cheetah-Run (vector) signiﬁcantly (p-value < 0.05).

Original: (ii) State reconstruction in the latent space improves RL performance in image-based tasks but
Revised: (ii) State reconstruction in the latent space improves RL performance in image-based tasks but

Original: undermines vector-based tasks.
Revised: undermines vector-based tasks.

Original: The improvements in image-based tasks could be attributed
Revised: The improvements in image-based tasks could be attributed

Original: to the combination of augmentation techniques, which, combined with reconstruction loss,
Revised: to the combination of augmentation techniques, which, combined with reconstruction loss,

Original: enforces the extraction of meaningful features.
Revised: enforces the extraction of meaningful features.

Original: In contrast, no augmentation is used in the vector-based setting, and thus the encoder learns no useful representations.
Revised: In contrast, no augmentation is used in thevector-based setting, and thus the encoder learns no useful representations.

Original: This also explains
Revised: This also explains

Original: why CURL performs poorly in vector-based experiments.
Revised: why CURL performs poorly in vector-based experiments.

Original: (iii)
Revised: (iii)

Original: In the vector-based setting, some typical human-designed patterns (e
Revised: In the vector-based setting, some typical human-designed patterns (e

Original: g., reward prediction, inverse dynamics, and action inference) can be very detrimental to RL performance, implying that some renowned techniques in loss designs might not work well under atypical settings.
Revised: g., reward prediction,inverse dynamics, and action inference) can be very detrimental to RL performance, implyingthat some renowned techniques in loss designs might not work well under atypical settings.

Original: Number of Sources and Targets.
Revised: Number of Sources and Targets.

Original: We further investigate whether it is more beneﬁcial to use a smallnumber of sources to predict a large number of targets ( n target > n source , e.g., using s t to predicts t +1 , s t +2 , s t +3 ), or the other way around ( n target < n source , e.g., using s t , s t +1 , s t +2 to predict 302 s t +
Revised: We further investigate whether it is more beneﬁcial to use a smallnumber of sources to predict a large number of targets ( n target > n source , e. g ., using s t to predicts t +1 , s t +2 , s t +3 ), or the other way around ( n target < n source , e.g., using s t , s t +1 , s t +2 to predict 298 s t +3 ).

Original: Statistical results are shown in Table 5, where we ﬁnd that auxiliary losses with more stateson the target side have a signiﬁcant advantage over losses with more states on the source side.
Revised: Statistical results are shown in Table 5, where we ﬁnd that auxiliary losses with more stateson the target side have a signiﬁcant advantage over losses with more states on the source side.

Original: Thisresult echoes recent works [54, 48]: predicting more states leads to strong performance gains.
Revised: Thisresult echoes recent works [42, 39]: predicting more states leads to strong performance gains.

Original:  Related Work
Revised:  Related Work

Original: Reinforcement Learning with Auxiliary Losses.
Revised: Reinforcement Learning with Auxiliary Losses.

Original: Usage of auxiliary tasks for learning better staterepresentations and improving the sample efﬁciency of RL agents, especially on image-based tasks,has been explored in many recent works.
Revised: Usage of auxiliary tasks for learning better staterepresentations and improving the sample efﬁciency of RL agents, especially on image-based tasks,has been explored in many recent works.

Original: A number of manually designed auxiliary objectives areshown to boost RL performance, including observation reconstruction [64], reward prediction [24],dynamics prediction [8] and contrastive learning objectives [27, 48, 54].
Revised: A number of manually designed auxiliary objectives areshown to boost RL performance, including observation reconstruction [51], reward prediction [20],dynamics prediction [6] and contrastive learning objectives [23, 39, 42].

Original: It is worth noting thatmost of these works focus on image-based settings, and only a limited number of works study thevector-based setting [40, 43].
Revised: It is worth noting thatmost of these works focus on image-based settings, and only a limited number of works study thevector-based setting [32, 35].

Original: Although people may think that vector-based settings can beneﬁt less
Revised: Although people may think that vector-based settings can beneﬁt less

Original: [Table]
Revised: [Table]

Original: from auxiliary tasks due to their lower-dimensional state space, we show in our paper that there isstill much potential for improving their performance with better learned representations.
Revised: from auxiliary tasks due to their lower-dimensional state space, we show in our paper that there isstill much potential for improving their performance with better learned representations.

Original: Compared to the previous works, we point out two major advantages of our approach.
Revised: Compared to the previous works, we point out two major advantages of our approach.

Original: (i)
Revised: (i)

Original: Insteadof handcrafting an auxiliary loss with expert knowledge, A2LS automatically searches for the bestauxiliary loss, relieving researchers from such tedious work.
Revised: Insteadof handcrafting an auxiliary loss with expert knowledge, A2LS automatically searches for the bestauxiliary loss, relieving researchers from such tedious work.

Original: (ii) A2LS is a principled approachthat can be used in arbitrary RL settings.
Revised: (ii) A2LS is a principled approachthat can be used in arbitrary RL settings.

Original: We discover great auxiliary losses that bring signiﬁcantperformance improvement in image-based and the rarely studied vector-based settings.
Revised: We discover great auxiliary losses that bring signiﬁcantperformance improvement in image-based and the rarely studied vector-based settings.

Original: Automated Reinforcement Learning.
Revised: Automated Reinforcement Learning.

Original: RL training is notoriously sensitive to hyper-parametersand environment changes [22].
Revised: RL training is notoriously sensitive to hyper-parametersand environment changes [18].

Original: Recently, many works attempted to take techniques in AutoML toalleviate human intervention, for example, hyper-parameter optimization [10, 44, 62, 67], rewardsearch [12, 57] and network architecture search [47, 13].
Revised: Recently, many works attempted to take techniques in AutoML toalleviate human intervention, for example, hyper-parameter optimization [7, 36, 49, 53], rewardsearch [9, 45] and network architecture search [38, 10].

Original: In contrast to these methods which optimizea new conﬁguration for each environment, we search for auxiliary loss functions that generalize acrossdifferent settings such as (i) different robots of control; (ii) different data types of observation; (iii)partially observable settings; (iv) different network architectures; (v) different benchmark domains.
Revised: In contrast to these methods which optimize anew conﬁguration for each environment, we search for auxiliary loss functions that generalize acrossdifferent settings such as (i) different robots of control; (ii) different data types of observation; (iii)partially observable settings; (iv) different network architectures; (v) different benchmark domains.

Original: Automated Loss Design.
Revised: Automated Loss Design.

Original: In the AutoML community, it has become a trend to design good lossfunctions that can outperform traditional and handcrafted ones.
Revised: In the AutoML community, it has become a trend to design good lossfunctions that can outperform traditional and handcrafted ones.

Original: To be speciﬁc, to resolve computervision tasks, AM-LFS [31] deﬁnes the loss function search space as a parameterized probabilitydistribution of the hyper-parameters of softmax loss.
Revised: To be speciﬁc, to resolve computervision tasks, AM-LFS [27] deﬁnes the loss function search space as a parameterized probabilitydistribution of the hyper-parameters of softmax loss.

Original: A recent work, AutoLoss-Zero [32], proposesto search loss functions with primitive mathematical operators.
Revised: A recent work, AutoLoss-Zero [28], proposesto search loss functions with primitive mathematical operators.

Original: For RL, existing works focus on searching for a better RL objective, EPG [23] and MetaGenRL [25]deﬁne the search space of loss functions as parameters of a low complexity neural network.
Revised: For RL, existing works focus on searching for a better RL objective, EPG [19] and MetaGenRL [21]deﬁne the search space of loss functions as parameters of a low complexity neural network.

Original: Recently,[6] deﬁnes the search space of RL loss functions as a directed acyclic graph and discovers two
Revised: Recently,[4] deﬁnes the search space of RL loss functions as a directed acyclic graph and discovers two

Original: DQN-like regularized RL losses.
Revised: DQN-like regularized RL losses.

Original: Note that none of these works investigates auxiliary loss functions,which are crucial to facilitate representation learning in RL and to make RL successful in highlycomplex environments.
Revised: Note that none of these works investigates auxiliary loss functions,which are crucial to facilitate representation learning in RL and to make RL successful in highlycomplex environments.

Original: To the best of our knowledge, our work is the ﬁrst attempt to search forauxiliary loss functions that can signiﬁcantly improve RL performance.
Revised: To the best of our knowledge, our work is the ﬁrst attempt to search forauxiliary loss functions that can signiﬁcantly improve RL performance.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: We present A2LS, a principled and universal framework for automated auxiliary loss design for RL.
Revised: We present A2LS, a principled and universal framework for automated auxiliary loss design for RL.

Original: By searching on training environments with this framework, we discover a top-performing auxiliaryloss function A2-winner that generalizes well to a diverse set of test environments .
Revised: By searching on training environments with this framework, we discover a top-performing auxiliaryloss function A2-winner that generalizes well to a diverse set of test environments .

Original: Furthermore, wepresent an in-depth investigation of the statistical relations between auxiliary loss patterns and RLperformance.
Revised: Furthermore, wepresent an in-depth investigation of the statistical relations between auxiliary loss patterns and RLperformance.

Original: We hope our studies provide insights that will deepen the understanding of auxiliarylosses in RL, and shed light on how to make RL more efﬁcient and practical.
Revised: We hope our studies provide insights that will deepen the understanding of auxiliarylosses in RL, and shed light on how to make RL more efﬁcient and practical.

Original: Limitations of ourcurrent work lie in that searching requires an expensive computational cost.
Revised: Limitations of ourcurrent work lie in that searching requires an expensive computational cost.

Original: In the future, we plan toincorporate more delicate information such as higher-order information such as [15] of the inner-loop
Revised: In the future, we plan toincorporate more delicate information such as higher-order information such as [12] of the inner-loop

Original: RL training procedure to derive more efﬁcient auxiliary loss search methods.
Revised: RL training procedure to derive more efﬁcient auxiliary loss search methods.