Original: Understanding the World Through Action
Revised: Understanding the World Through Action

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract:
Revised: Abstract:

Original: The recent history of machine learning research has taught us that machine learning methods can be most effective when they are provided with verylarge, high-capacity models, and trained on very large and diverse datasets.
Revised: The recent history of machine learning research has taught us that machine learning methods can be most effective when they are provided with very large, high-capacity models, and trained on very large and diverse datasets.

Original: Thishas spurred the community to search for ways to remove any bottlenecks to scale.
Revised: This has spurred the community to search for ways to remove any bottlenecks to scale.

Original: Often the foremost among such bottlenecks is the need for human effort, includingthe effort of curating and labeling datasets.
Revised: Often the foremost among such bottlenecks is the need for human effort, including the effort of curating and labeling datasets.

Original: As a result, considerable attention inrecent years has been devoted to utilizing unlabeled data, which can be collectedin vast quantities.
Revised: As a result, considerable attention in recent years has been devoted to utilizing unlabeled data, which can be collected in vast quantities.

Original: However, some of the most widely used methods for training onsuch unlabeled data themselves require human-designed objective functions thatmust correlate in some meaningful way to downstream tasks.
Revised: However, some of the most widely used methods for training on such unlabeled data themselves require human-designed objective functions that must correlate in some meaningful way to downstream tasks.

Original: I will argue that ageneral, principled, and powerful framework for utilizing unlabeled data can bederived from reinforcement learning, using general purpose unsupervised or selfsupervised reinforcement learning objectives in concert with ofﬂine reinforcementlearning methods that can leverage large datasets.
Revised: I will argue that a general, principled, and powerful framework for utilizing unlabeled data can be derived from reinforcement learning, using general purpose unsupervised or selfsupervised reinforcement learning objectives in concert with ofﬂine reinforcement learning methods that can leverage large datasets.

Original: I will discuss how such a procedure is more closely aligned with potential downstream tasks, and how it couldbuild on existing techniques that have been developed in recent years.
Revised: I will discuss how such a procedure is more closely aligned with potential downstream tasks, and how it could build on existing techniques that have been developed in recent years.

Original:  Introduction
Revised:  Introduction

Original: Machine learning systems have mastered a breadth of challenging problems in domains rangingfrom computer vision [1] to speech recognition [2] and natural language processing [3] and yetthe question of how to design learning-enable systems that match the ﬂexibility and generality ofhuman reasoning remains out of reach.
Revised: Machine learning systems have mastered a breadth of challenging problems in domains ranging from computer vision [1] to speech recognition [2] and natural language processing [3] and yet the question of how to design learning-enable systems that match the ﬂexibility and generality of human reasoning remains out of reach.

Original: This has prompted considerable discussion about whatthe “missing ingredient” in modern machine learning might be, with a number of hypotheses putforward as the big question that the ﬁeld must resolve.
Revised: This has prompted considerable discussion about what the “missing ingredient” in modern machine learning might be, with a number of hypotheses put forward as the big question that the ﬁeld must resolve.

Original: Is the missing ingredient causal reasoning [4],inductive bias [5], better algorithms for self-supervised or unsupervised learning [6] or somethingelse entirely?
Revised: Is the missing ingredient causal reasoning [4], inductive bias [5], better algorithms for self-supervised or unsupervised learning [6] or something else entirely?

Original: This question is difﬁcult one and any answer must necessarily involve a great deal of guesswork butthe lessons we can glean from recent progress in artiﬁcial intelligence can provide us with severalguiding principles.
Revised: This question is difﬁcult one and any answer must necessarily involve a great deal of guesswork but the lessons we can glean from recent progress in artiﬁcial intelligence can provide us with several guiding principles.

Original: The ﬁrst lesson is in the “unreasonable” effectiveness of large, generic models supplied with largeamounts of training data.
Revised: The ﬁrst lesson is in the “unreasonable” effectiveness of large, generic models supplied with large amounts of training data.

Original: As eloquently articulated by Sutton [7] in his essay on the “bitter lesson,”as well as a number of other researchers in machine learning, a persistent theme in the recent historyof ML research has been that methods that effectively leverage large amounts of computation andlarge amounts of data tend to often outperform methods that rely on manually engineered priors andheuristics.
Revised: As eloquently articulated by Sutton [7] in his essay on the “bitter lesson,” as well as a number of other researchers in machine learning, a persistent theme in the recent history of ML research has been that methods that effectively leverage large amounts of computation and large amounts of data tend to often outperform methods that rely on manually engineered priors and heuristics.

Original: While a full discussion of the reasons for this trend are outside the scope of this article,in short they could be summarized (or perhaps caricatured) as follows: when we engineer biases orpriors for our models, we are injecting our own imperfect knowledge of how the world works, whichbiases the model toward some solutions over others.
Revised: While a full discussion of the reasons for this trend are outside the scope of this article, in short they could be summarized (or perhaps caricatured) as follows: when we engineer biases or priors for our models, we are injecting our own imperfect knowledge of how the world works, which biases the model toward some solutions over others.

Original: When the model instead gleans this knowledgefrom data, it will arrive at conclusions that are more accurate than those that we engineered ourselves, and therefore will work better.
Revised: When the model instead gleans this knowledge from data, it will arrive at conclusions that are more accurate than those that we engineered ourselves, and therefore will work better.

Original: Indeed, a similar pattern has been observed in how peopleacquire proﬁciency.
Revised: Indeed, a similar pattern has been observed in how people acquire proﬁciency.

Original: As discussed by Dreyfus and Dreyfus [8], “rule-based” reasoning that followsrules we can articulate clearly tends to only provide people with “novice-level” performance at var- ious skills, while “expert-level” performance is associated with a mess of special cases, exceptions,and patterns that people struggle to articulate clearly, and yet can leverage seamlessly in the momentwhen the situation demands it.
Revised: As discussed by Dreyfus and Dreyfus [8], “rule-based” reasoning that follows rules we can articulate clearly tends to only provide people with “novice-level” performance at var- ious skills, while “expert-level” performance is associated with a mess of special cases, exceptions, and patterns that people struggle to articulate clearly, and yet can leverage seamlessly in the moment when the situation demands it.

Original: As Dreyfus points out, real human experts are rarely able to articulatethe rules they actually follow when exhibiting their expertise, and so it should come as no surprisethat in the same way that we must acquire expertise from experience, so must our machines.
Revised: As Dreyfus points out, real human experts are rarely able to articulate the rules they actually follow when exhibiting their expertise, and so it should come as no surprise that in the same way that we must acquire expertise from experience, so must our machines.

Original: Andto do that, they will need powerful, high-capacity models that impose comparatively few biases andcan handle the large amounts of experience that will be needed.
Revised: And to do that, they will need powerful, high-capacity models that impose comparatively few biases and can handle the large amounts of experience that will be needed.

Original: The second, more recent lesson is that manual labeling and supervision do not scale nearly as wellas unsupervised or self-supervised learning.
Revised: The second, more recent lesson is that manual labeling and supervision do not scale nearly as well as unsupervised or self-supervised learning.

Original: We’ve already seen that unsupervised pre-training hasbecome standard in natural language processing [9], and perhaps will soon become standard acrossother ﬁelds.
Revised: We’ve already seen that unsupervised pre-training has become standard in natural language processing [9], and perhaps will soon become standard across other ﬁelds.

Original: In a sense, this lesson is a corollary to the ﬁrst one: if large models and large datasetsare the most effective, then anything that limits how large the models and datasets can be willeventually end up as the bottleneck.
Revised: In a sense, this lesson is a corollary to the ﬁrst one: if large models and large datasets are the most effective, then anything that limits how large the models and datasets can be will eventually end up as the bottleneck.

Original: Human supervision can be one such bottleneck: if all datamust be labeled manually by a person, then less data will be available to the system to learn from.
Revised: Human supervision can be one such bottleneck: if all data must be labeled manually by a person, then less data will be available to the system to learn from.

Original: However, here we reach a conundrum: current methods for learning without human labels often violate the principles outlined in the ﬁrst lesson, requiring considerable human insight (which is oftendomain-speciﬁc!)
Revised: However, here we reach a conundrum: current methods for learning without human labels often violate the principles outlined in the ﬁrst lesson, requiring considerable human insight (which is often domain-speciﬁc!)

Original: to engineer self-supervised learning objectives that allow large models to acquiremeaningful knowledge from unlabeled datasets.
Revised: to engineer self-supervised learning objectives that allow large models to acquire meaningful knowledge from unlabeled datasets.

Original: These include relatively straightforward tasks suchas language modeling [10], as well as comparatively more esoteric tasks, such as predicting whethertwo transformed images were produced by the same original image, or two different ones [11].
Revised: These include relatively straightforward tasks such as language modeling [10], as well as comparatively more esoteric tasks, such as predicting whether two transformed images were produced by the same original image, or two different ones [11].

Original: Thelatter is a widely used and successful approach in modern self-supervised learning in computer vision.
Revised: The latter is a widely used and successful approach in modern self-supervised learning in computer vision.

Original: While such approaches can be effective up to a point, it may well be that the next bottleneckwe will face is in deciding how to train large models without requiring manual labeling or manual design of self-supervised objectives, so as to acquire models that distill a deep and meaningfulunderstanding of the world and can perform downstream tasks with robustness generalization, andeven a degree of common sense.
Revised: While such approaches can be effective up to a point, it may well be that the next bottleneck we will face is in deciding how to train large models without requiring manual labeling or manual design of self-supervised objectives, so as to acquire models that distill a deep and meaningful understanding of the world and can perform downstream tasks with robustness generalization, and even a degree of common sense.

Original: I will argue that such methodology could be developed out of current algorithms for learning-basedcontrol (reinforcement learning), though it will require a number of substantial algorithmic innovations that will allow such methods to go signiﬁcantly beyond the kinds of problems they’ve beenable to tackle so far.
Revised: I will argue that such methodology could be developed out of current algorithms for learning-based control (reinforcement learning), though it will require a number of substantial algorithmic innovations that will allow such methods to go signiﬁcantly beyond the kinds of problems they’ve been able to tackle so far.

Original: Central to this idea is the notion that, in order to control the environment in diverse and goal-directed ways, autonomous agents will necessarily need to develop an understandingof their environment that is causal and generalizable, and hence will address many of the shortcomings of current supervised models.
Revised: Central to this idea is the notion that, in order to control the environment in diverse and goal-directed ways, autonomous agents will necessarily need to develop an understanding of their environment that is causal and generalizable, and hence will address many of the shortcomings of current supervised models.

Original: At the same time, this will require going beyond the currentparadigm in reinforcement learning in two important ways.
Revised: At the same time, this will require going beyond the current paradigm in reinforcement learning in two important ways.

Original: First, reinforcement learning algorithmsrequire a task goal (i.e., a reward function) to be speciﬁed by hand by the user, and then learnthe behaviors necessary to accomplish that task goal.
Revised: First, reinforcement learning algorithms require a task goal (i.e., a reward function) to be speciﬁed by hand by the user, and then learn the behaviors necessary to accomplish that task goal.

Original: This of course greatly limits their ability tolearn without human supervision.
Revised: This of course greatly limits their ability to learn without human supervision.

Original: Second, reinforcement learning algorithms in common use todayare not inherently data-driven , but rather learn from online experience, and although such methodscan be deployed directly in real-world environments [12], online active data collection limits theirgeneralization in such settings [13], and many uses of reinforcement learning instead take place insimulation, where there are few opportunities to learn about how the real world works.
Revised: Second, reinforcement learning algorithms in common use today are not inherently data-driven , but rather learn from online experience, and although such methods can be deployed directly in real-world environments [12], online active data collection limits their generalization in such settings [13], and many uses of reinforcement learning instead take place in simulation, where there are few opportunities to learn about how the real world works.

Original:  Learning Through Action
Revised:  Learning Through Action

Original: Insofar as artiﬁcial intelligence systems are useful, it is because they provide inferences that can beused to make decisions, which in turn affect something in the world.
Revised: Insofar as artiﬁcial intelligence systems are useful, it is because they provide inferences that can be used to make decisions, which in turn affect something in the world.

Original: Therefore, it is reasonable toconclude that a general learning objective should be one that provides impetus to learn those thingsthat are most useful for affecting the world in meaningful ways.
Revised: Therefore, it is reasonable to conclude that a general learning objective should be one that provides impetus to learn those things that are most useful for affecting the world in meaningful ways.

Original: Making decisions that create desiredoutcomes is the purview of reinforcement learning and control.
Revised: Making decisions that create desired outcomes is the purview of reinforcement learning and control.

Original: Therefore, we should consider howreinforcement learning can provide the sort of automated and principled objectives for training highcapacity models that can endow them with the ability to understand, reason, and generalize.
Revised: Therefore, we should consider how reinforcement learning can provide the sort of automated and principled objectives for training highcapacity models that can endow them with the ability to understand, reason, and generalize.

Original: However, this will require addressing the two limitations: reinforcement learning requires manuallydeﬁned reward functions, and it requires an active learning paradigm that is difﬁcult to reconcilewith the need to train on large and diverse datasets.
Revised: However, this will require addressing the two limitations: reinforcement learning requires manuallydeﬁned reward functions, and it requires an active learning paradigm that is difﬁcult to reconcile with the need to train on large and diverse datasets.

Original: To address the issue with objectives, we candevelop algorithms that, instead of aiming to perform a single user-speciﬁed task, rather aim toaccomplish whatever outcomes they infer are possible in the world.
Revised: To address the issue with objectives, we can develop algorithms that, instead of aiming to perform a single user-speciﬁed task, rather aim to accomplish whatever outcomes they infer are possible in the world.

Original: Potential objectives for suchmethods could include learning to reach any feasible state, learning to maximize mutual information between latent goals and outcomes, or learning through principled intrinsic motivation objectivesthat lead to broad coverage of possible outcomes.
Revised: Potential objectives for such methods could include learning to reach any feasible state, learning to maximize mutual information between latent goals and outcomes, or learning through principled intrinsic motivation objectives that lead to broad coverage of possible outcomes.

Original: To address the issue with data, we must developreinforcement learning algorithms that can effectively utilize previously collected datasets.
Revised: To address the issue with data, we must develop reinforcement learning algorithms that can effectively utilize previously collected datasets.

Original: Theseare ofﬂine reinforcement learning algorithms [13], and they can provide a path toward training RLsystems on broad and diverse datasets in much the same manner as in supervised learning, followedby some amount of active online ﬁnetuning to attain the best performance [14].
Revised: These are ofﬂine reinforcement learning algorithms [13], and they can provide a path toward training RL systems on broad and diverse datasets in much the same manner as in supervised learning, followed by some amount of active online ﬁnetuning to attain the best performance [14].

Original: To provide a hypothetical example of a system that instantiates these ideas, imagine a robot that performs a variety of manipulation tasks When given a user-speciﬁed goal, the robot performs that goal.
Revised: To provide a hypothetical example of a system that instantiates these ideas, imagine a robot that performs a variety of manipulation tasks When given a user-speciﬁed goal, the robot performs that goal.

Original: However, in its “spare time,” the robot imagines potential outcomes that it can produce, and then“practices” taking actions to produce them.
Revised: However, in its “spare time,” the robot imagines potential outcomes that it can produce, and then “practices” taking actions to produce them.

Original: Each such practice session deepens its understanding ofthe causal structure in the world.
Revised: Each such practice session deepens its understanding of the causal structure in the world.

Original: Of course, the notion of a real-world commercially deployed robotic system that “plays” with itsenvironment in this way might seem far-fetched (it is also of course not a new idea [15]).
Revised: Of course, the notion of a real-world commercially deployed robotic system that “plays” with its environment in this way might seem far-fetched (it is also of course not a new idea [15]).

Original: Thisis precisely why ofﬂine RL is important: since an ofﬂine algorithm would be comparatively indifferent to the source of the experience, the fraction of time that the robot spends accomplishinguser-speciﬁed objectives versus “playing” could be adjusted to either extreme, and even a systemthat spends all of its time performing user-speciﬁed tasks can still use all of its collected experienceas ofﬂine training data for learning to achieve any outcome.
Revised: This is precisely why ofﬂine RL is important: since an ofﬂine algorithm would be comparatively indifferent to the source of the experience, the fraction of time that the robot spends accomplishing user-speciﬁed objectives versus “playing” could be adjusted to either extreme, and even a system that spends all of its time performing user-speciﬁed tasks can still use all of its collected experience as ofﬂine training data for learning to achieve any outcome.

Original: Such a system would still “play” withits environment, but only virtually, in its “memories.”
Revised: Such a system would still “play” with its environment, but only virtually, in its “memories.”

Original: While robotic systems might be the most obvious domain in which to instantiate this design, it isnot restricted to robotics, nor to systems that are embodied in the world in an analogous way to people.
Revised: While robotic systems might be the most obvious domain in which to instantiate this design, it is not restricted to robotics, nor to systems that are embodied in the world in an analogous way to people.

Original: Any system that has a well-deﬁned notion of actions can be trained in this way: recommendersystems, autonomous vehicles, systems for inventory management and logistics, dialogue systems,and so forth.
Revised: Any system that has a well-deﬁned notion of actions can be trained in this way: recommender systems, autonomous vehicles, systems for inventory management and logistics, dialogue systems, and so forth.

Original: Online exploration may not be feasible in many of these settings, but learning with unsupervised outcome-driven objectives via ofﬂine RL is still possible.
Revised: Online exploration may not be feasible in many of these settings, but learning with unsupervised outcome-driven objectives via ofﬂine RL is still possible.

Original: As mentioned previously, MLsystems are useful insofar as they enable making intelligent decisions.
Revised: As mentioned previously, ML systems are useful insofar as they enable making intelligent decisions.

Original: It therefore stands to reasonthat any useful ML system is situated in a sequential process where decision-making is possible, andtherefore such a self-supervised learning procedure should be applicable.
Revised: It therefore stands to reason that any useful ML system is situated in a sequential process where decision-making is possible, and therefore such a self-supervised learning procedure should be applicable.

Original:  Unsupervised and Self-Supervised Reinforcement Learning
Revised:  Unsupervised and Self-Supervised Reinforcement Learning

Original: An unsupervised or self-supervised reinforcement learning method should fulﬁll two criteria: itshould learn behaviors that control the world in meaningful ways, and it should provide some mechanism to learn to control it in as many ways as possible.
Revised: An unsupervised or self-supervised reinforcement learning method should fulﬁll two criteria: it should learn behaviors that control the world in meaningful ways, and it should provide some mechanism to learn to control it in as many ways as possible.

Original: This leaves considerable room for interpretation, for example concerning the term “meaningful,” and a wide range of potential deﬁnitions havebeen offered in the literature [16, 17, 18].
Revised: This leaves considerable room for interpretation, for example concerning the term “meaningful,” and a wide range of potential deﬁnitions have been offered in the literature [16, 17, 18].

Original: This problem should not be confused with the closely related problem of exploration, which has also often been formulated as a problem of attaining broadcoverage [19], but which is not generally concerned with learning to control the world in meaningful ways in the absence of a task objective.
Revised: This problem should not be confused with the closely related problem of exploration, which has also often been formulated as a problem of attaining broad coverage [19], but which is not generally concerned with learning to control the world in meaningful ways in the absence of a task objective.

Original: That is, exploration methods provide an objective forcollecting data, rather than utilizing it.
Revised: That is, exploration methods provide an objective for collecting data, rather than utilizing it.

Original: Perhaps the most direct way to formulate a self-supervised RL objective is to frame it as the problemof reaching a goal state [20].
Revised: Perhaps the most direct way to formulate a self-supervised RL objective is to frame it as the problem of reaching a goal state [20].

Original: The problem then corresponds to training a goal-conditioned policyπ ( a | s , g ) with some choice of reward function r ( s , g ) .
Revised: The problem then corresponds to training a goal-conditioned policy π ( a | s , g ) with some choice of reward function r ( s , g ) .

Original: Though this reward itself might constitutes amanually designed objective, it’s possible to derive frameworks where the reward function is a consequence of solving a well-deﬁned inference problem, such as the problem of predicting the actionthat is most likely to lead a particular outcome [21, 22].
Revised: Though this reward itself might constitutes a manually designed objective, it’s possible to derive frameworks where the reward function is a consequence of solving a well-deﬁned inference problem, such as the problem of predicting the action that is most likely to lead a particular outcome [21, 22].

Original: This problem formulation provides considerable depth, with connections to density estimation [21], variational inference [23, 22], model-basedreinforcement learning [24, 25, 26], and exploration [27].
Revised: This problem formulation provides considerable depth, with connections to density estimation [21], variational inference [23, 22], model-based reinforcement learning [24, 25, 26], and exploration [27].

Original: What does a policy trained to reach all possible goals learn about the world?
Revised: What does a policy trained to reach all possible goals learn about the world?

Original: As argued both inrecent work [25, 26] and classic literature in RL [24], solving such goal-conditioned RL problemscorresponds to learning a kind of dynamics model.
Revised: As argued both in recent work [25, 26] and classic literature in RL [24], solving such goal-conditioned RL problems corresponds to learning a kind of dynamics model.

Original: Intuitively, being able to bring about any potential desired outcome requires a deep understanding of how actions affect the environment overa long horizon.
Revised: Intuitively, being able to bring about any potential desired outcome requires a deep understanding of how actions affect the environment over a long horizon.

Original: Of course, one might then wonder – why not just learn a dynamics model directly,of the form more commonly used in model-based RL?
Revised: Of course, one might then wonder – why not just learn a dynamics model directly, of the form more commonly used in model-based RL?

Original: Model learning is likely also an effectiveway to utilize diverse datasets without a speciﬁc user-provided objective.
Revised: Model learning is likely also an effective way to utilize diverse datasets without a speciﬁc user-provided objective.

Original: However, there is goodreason to believe that goal-conditioned RL either solves a very similar problem [24, 25, 26] or, potentially, actually solves a problem that is more closely connected to downstream tasks – unlikemodel-based RL, where the model objective is largely disconnected from actually bringing aboutdesired outcomes, the goal-conditioned RL objective is connected to long-horizon outcomes verydirectly.
Revised: However, there is good reason to believe that goal-conditioned RL either solves a very similar problem [24, 25, 26] or, potentially, actually solves a problem that is more closely connected to downstream tasks – unlike model-based RL, where the model objective is largely disconnected from actually bringing about desired outcomes, the goal-conditioned RL objective is connected to long-horizon outcomes very directly.

Original: Therefore, insofar as the end goal of an ML system is to bring about desired outcomes, wewould expect that the objective of goal-conditioned RL would be well-aligned.
Revised: Therefore, insofar as the end goal of an ML system is to bring about desired outcomes, we would expect that the objective of goal-conditioned RL would be well-aligned.

Original: However, current approaches suffer from a number of major limitations.
Revised: However, current approaches suffer from a number of major limitations.

Original: Even standard goalconditioned RL methods can be difﬁcult to use and unstable.
Revised: Even standard goalconditioned RL methods can be difﬁcult to use and unstable.

Original: But even more importantly, goalreaching does not span the full set of possible tasks that could be speciﬁed in RL.
Revised: But even more importantly, goalreaching does not span the full set of possible tasks that could be speciﬁed in RL.

Original: Even if anagent learns to successfully accomplish every outcome that is possible in a given environment,there may not exist a single desired outcome that would maximize an arbitrary user-speciﬁed reward function.
Revised: Even if an agent learns to successfully accomplish every outcome that is possible in a given environment, there may not exist a single desired outcome that would maximize an arbitrary user-speciﬁed reward function.

Original: It may still be that such a goal-conditioned policy would have learned powerful andbroadly applicable features, and could be readily ﬁnetuned to the downstream task, but an interestingproblem for future work is to better understand whether more universal self-supervised objectivescould lift this limitation.
Revised: It may still be that such a goal-conditioned policy would have learned powerful and broadly applicable features, and could be readily ﬁnetuned to the downstream task, but an interesting problem for future work is to better understand whether more universal self-supervised objectives could lift this limitation.

Original: A number of methods have been proposed for unsupervised acquisition ofskills [28, 29, 30], so we might reasonably ask whether more general and principled self-supervisedreinforcement learning objectives could be derived on this basis.
Revised: A number of methods have been proposed for unsupervised acquisition of skills [28, 29, 30], so we might reasonably ask whether more general and principled self-supervised reinforcement learning objectives could be derived on this basis.

Original:  Ofﬂine Reinforcement Learning
Revised:  Ofﬂine Reinforcement Learning

Original: As discussed previously, ofﬂine RL can make it possible to apply self-supervised or unsupervised
Revised: As discussed previously, ofﬂine RL can make it possible to apply self-supervised or unsupervised

Original: RL methods even in settings where online collection is infeasible, and such methods can serve asone of the most powerful tools for incorporating large and diverse datasets into self-supervised RL.
Revised: RL methods even in settings where online collection is infeasible, and such methods can serve as one of the most powerful tools for incorporating large and diverse datasets into self-supervised RL.

Original: This is likely to be essential to make this a truly viable and general tool for large-scale representationlearning.
Revised: This is likely to be essential to make this a truly viable and general tool for large-scale representation learning.

Original: However, ofﬂine RL presents a number of challenges [13].
Revised: However, ofﬂine RL presents a number of challenges [13].

Original: Foremost among these is thatofﬂine RL requires answering counterfactual questions: given data that shows one outcome, canwe predict what would have happened if we had taken a different action?
Revised: Foremost among these is that ofﬂine RL requires answering counterfactual questions: given data that shows one outcome, can we predict what would have happened if we had taken a different action?

Original: This is of course verychallenging in general.
Revised: This is of course very challenging in general.

Original: Nonetheless, our understanding of ofﬂine RL has progressed signiﬁcantlyover the past few years.
Revised: Nonetheless, our understanding of ofﬂine RL has progressed signiﬁcantly over the past few years.

Original: Besides understanding how distributional shift affects ofﬂine RL [31], theperformance of ofﬂine RL algorithms has advanced considerably, and new algorithms have beendeveloped that provide robustness guarantees [32], ﬁnetune online after ofﬂine pretraining [14], andtackle a range of other problems in the ofﬂine RL setting.
Revised: Besides understanding how distributional shift affects ofﬂine RL [31], the performance of ofﬂine RL algorithms has advanced considerably, and new algorithms have been developed that provide robustness guarantees [32], ﬁnetune online after ofﬂine pretraining [14], and tackle a range of other problems in the ofﬂine RL setting.

Original: Advances in ofﬂine RL have the potential to signiﬁcantly increase the applicability of self-supervised
Revised: Advances in ofﬂine RL have the potential to signiﬁcantly increase the applicability of self-supervised

Original: RL methods.
Revised: RL methods.

Original: Using the tools of ofﬂine RL, it is possible to construct self-supervised RL methodsthat do not require any exploration on their own.
Revised: Using the tools of ofﬂine RL, it is possible to construct self-supervised RL methods that do not require any exploration on their own.

Original: Much like the “virtual play” mentioned in Section 2, we can utilize ofﬂine RL in combination with goal-conditioned policies to learn entirely frompreviously collected data [33, 34, 35].
Revised: Much like the “virtual play” mentioned in Section 2, we can utilize ofﬂine RL in combination with goal-conditioned policies to learn entirely from previously collected data [33, 34, 35].

Original: However, major challenges remain.
Revised: However, major challenges remain.

Original: Ofﬂine RL algorithmsinherit many of the difﬁculties of standard (deep) RL methods, including sensitivity to hyperparameters, but these difﬁculties are further exacerbated by the fact that we cannot perform multiple onlinetrials to determine the best hyperparameters.
Revised: Ofﬂine RL algorithms inherit many of the difﬁculties of standard (deep) RL methods, including sensitivity to hyperparameters, but these difﬁculties are further exacerbated by the fact that we cannot perform multiple online trials to determine the best hyperparameters.

Original: In supervised learning we can deal with such issues byusing a validation set, but a corresponding equivalent in ofﬂine RL is lacking.
Revised: In supervised learning we can deal with such issues by using a validation set, but a corresponding equivalent in ofﬂine RL is lacking.

Original: We need algorithmsthat are more stable and reliable, as well as effective methods for evaluation, in order to make suchapproaches truly broadly applicable.
Revised: We need algorithms that are more stable and reliable, as well as effective methods for evaluation, in order to make such approaches truly broadly applicable.

Original:  Concluding Remarks
Revised:  Concluding Remarks

Original: In this article, I discussed how self-supervised reinforcement learning combined with ofﬂine RLcould enable scalable representation learning.
Revised: In this article, I discussed how self-supervised reinforcement learning combined with ofﬂine RL could enable scalable representation learning.

Original: The motivation behind this approach is that, insofar that learned models are useful, it is because they allow us to make decisions that bring aboutthe desired outcome in the world.
Revised: The motivation behind this approach is that, insofar that learned models are useful, it is because they allow us to make decisions that bring about the desired outcome in the world.

Original: Therefore, self-supervised training with the goal of bringingabout any possible outcome should provide such models with the requisite understanding of howthe world works.
Revised: Therefore, self-supervised training with the goal of bringing about any possible outcome should provide such models with the requisite understanding of how the world works.

Original: Self-supervised RL objectives, such as those in goal-conditioned RL, have a closerelationship with model learning, and fulﬁlling such objectives is likely to require policies to gaina functional and causal understanding of the environment in which they are situated.
Revised: Self-supervised RL objectives, such as those in goal-conditioned RL, have a close relationship with model learning, and fulﬁlling such objectives is likely to require policies to gain a functional and causal understanding of the environment in which they are situated.

Original: However, forsuch techniques to be useful, it must be possible to apply them at scale to real-world datasets.
Revised: However, for such techniques to be useful, it must be possible to apply them at scale to real-world datasets.

Original: Ofﬂine
Revised: Ofﬂine

Original: RL can play this role, because it enables using large, diverse previously collected datasets.
Revised: RL can play this role, because it enables using large, diverse previously collected datasets.

Original: Puttingthese pieces together may lead to a new class of algorithms that can understand the world throughaction, leading to methods that are truly scalable and automated.
Revised: Putting these pieces together may lead to a new class of algorithms that can understand the world through action, leading to methods that are truly scalable and automated.