Original: Intra-agent speech permits zero-shot task acquisition
Revised: Intra-agent speech permits zero-shot task acquisition

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Human language learners are exposed to a trickle of informative, context-sensitivelanguage, but a flood of raw sensory data.
Revised: Human language learners are exposed to a trickle of informative, context-sensitivelanguage, but a flood of raw sensory data.

Original: Through both social language use andinternal processes of rehearsal and practice, language learners are able to buildhigh-level, semantic representations that explain their perceptions.
Revised: Through both social language use andinternal processes of rehearsal and practice, language learners are able to buildhigh-level, semantic representations that explain their perceptions.

Original: Here, we takeinspiration from such processes of “inner speech” in humans (Vygotsky, 1934) tobetter understand the role of intra-agent speech in embodied behaviour.
Revised: Here, we takeinspiration from such processes of “inner speech” in humans (Vygotsky, 1934) tobetter understand the role of intra-agent speech in embodied behaviour.

Original: First, weformally pose intra-agent speech as a semi-supervised problem and develop twoalgorithms that enable visually grounded captioning with little labeled languagedata.
Revised: First, weformally pose intra-agent speech as a semi-supervised problem and develop twoalgorithms that enable visually grounded captioning with little labeled languagedata.

Original: We then experimentally compute scaling curves over different amountsof labeled data and compare the data efficiency against a supervised learningbaseline.
Revised: We then experimentally compute scaling curves over different amountsof labeled data and compare the data efficiency against a supervised learningbaseline.

Original: Finally, we incorporate intra-agent speech into an embodied, mobilemanipulator agent operating in a 3D virtual world, and show that with as fewas 150 additional image captions, intra-agent speech endows the agent with theability to manipulate and answer questions about a new object without any relatedtask-directed experience (zero-shot).
Revised: Finally, we incorporate intra-agent speech into an embodied, mobilemanipulator agent operating in a 3D virtual world, and show that with as fewas 150 additional image captions, intra-agent speech endows the agent with theability to manipulate and answer questions about a new object without any relatedtask-directed experience (zero-shot).

Original: Taken together, our experiments suggest thatmodelling intra-agent speech is effective in enabling embodied agents to learn newtasks efficiently and without direct interaction experience.
Revised: Taken together, our experiments suggest thatmodelling intra-agent speech is effective in enabling embodied agents to learn newtasks efficiently and without direct interaction experience.

Original:  Introduction
Revised:  Introduction

Original: Contemporary language models learn from troves of text comprising billions, and sometimes trillionsof tokens.
Revised: Contemporary language models learn from troves of text comprising billions, and sometimes trillionsof tokens.

Original: This is not the same situation faced by embodied language learners, such as human children,whose raw sensory data vastly exceeds their social-linguistic experiences.
Revised: This is not the same situation faced by embodied language learners, such as human children,whose raw sensory data vastly exceeds their social-linguistic experiences.

Original: Psychologists have longnoted the existence of non-social, intra-personal dialogue—such as inner speech (silent) or privatespeech (aloud)—which provides additional linguistic experiences beyond those encountered in socialsettings.
Revised: Psychologists have longnoted the existence of non-social, intra-personal dialogue—such as inner speech (silent) or privatespeech (aloud)—which provides additional linguistic experiences beyond those encountered in socialsettings.

Original: Indeed, one possible reason for intra-personal speech may be to support the development ofsocial speech [1–5], though the inverse causal relation may also be possible [1, 6].
Revised: Indeed, one possible reason for intra-personal speech may be to support the development ofsocial speech [1–5], though the inverse causal relation may also be possible [1, 6].

Original: Intra-personal speech may also affect broader aspects of our cognition: the process wherein we privately speak about our perceptions subsequently impacts how we attend to, remember, or manipulatethe objects of our perception [7–12].
Revised: Intra-personal speech may also affect broader aspects of our cognition: the process wherein we privately speak about our perceptions subsequently impacts how we attend to, remember, or manipulatethe objects of our perception [7–12].

Original: For example, our short-term memories tend to commit errorsfor items that are phonetically but not visually or semantically similar [8, 9], and “self-talk” affectstask performance on executive tasks both in children [11, 12] and in adults [7], and has been widelyused to improve athletic performance [10].
Revised: For example, our short-term memories tend to commit errorsfor items that are phonetically but not visually or semantically similar [8, 9], and “self-talk” affectstask performance on executive tasks both in children [11, 12] and in adults [7], and has been widelyused to improve athletic performance [10].

Original: Inspired by these ideas, here we ask if we can leverage intra-agent speech as a tool for artificialintelligence research.
Revised: Inspired by these ideas, here we ask if we can leverage intra-agent speech as a tool for artificialintelligence research.

Original: Specifically, we ask the following questions: First, is it possible to modelintra-agent speech as a machine learning problem, enabling us to make up for a paucity of languagedata when given access to a large amount of unlabeled data?
Revised: Specifically, we ask the following questions: First, is it possible to modelintra-agent speech as a machine learning problem, enabling us to make up for a paucity of languagedata when given access to a large amount of unlabeled data?

Original: Second, can mechanisms for learningintra-agent speech impact the subsequent behavior of an embodied agent? 
Revised: Second, can mechanisms for learningintra-agent speech impact the subsequent behavior of an embodied agent? Training

Original: Regarding the first, we formally structure intra-agent speech as a semi-supervised problem.
Revised: Hand me the drum. Inference

Original: Inparticular, we model intra-agent speech samples as variational latent variables representing images,
Revised: There is a drum on the floor. Put a pear on the table. I can see a drum in front of me. Caption about new object (rare)

Original: [Figure]
Revised: Self supervision about new object Tasks about known objects Regarding the first, we formally structure intra-agent speech as a semi-supervised problem.

Original: and use this perspective to devise two algorithms enabling visual captioning given a paucity ofdirectly labeled data.
Revised: Inparticular, we model intra-agent speech samples as variational latent variables representing images,and use this perspective to devise two algorithms enabling visual captioning given a paucity ofdirectly labeled data.

Original: Regarding the second, we go on to show how embedding such algorithms in anembodied agent not only permits the capacity to speak about any given perception (that is, to caption ,which is precisely what is being optimized for) but rather also permits zero-shot behaviors involvingnew objects that were never directly implicated in any task-directed experience (see Figure 1).
Revised: Regarding the second, we go on to show how embedding such algorithms in anembodied agent not only permits the capacity to speak about any given perception (that is, to caption ,which is precisely what is being optimized for) but rather also permits zero-shot behaviors involvingnew objects that were never directly implicated in any task-directed experience (see Figure 1).

Original:  Approach
Revised:  Approach

Original: At a high level, we structured intra-agent speech as the production of language generated from inputs(which in our case manifests as captions of images), with two additional concerns: First, there willbe a preponderance of unlabeled data (i.e., pure images) relative to labeled data (i.e., images withassociated captions) in our captioning dataset, much like the situations faced by human childrenin nature and semi-supervised learning in machine learning.
Revised: At a high level, we structured intra-agent speech as the production of language generated from inputs(which in our case manifests as captions of images), with two additional concerns: First, there willbe a preponderance of unlabeled data (i.e., pure images) relative to labeled data (i.e., images withassociated captions) in our captioning dataset, much like the situations faced by human childrenin nature and semi-supervised learning in machine learning.

Original: Second, the linguistic representationseventually produced will provide auxiliary supervision for an embodied, acting agent so that theymay influence interactive behavior.
Revised: Second, the linguistic representationseventually produced will provide auxiliary supervision for an embodied, acting agent so that theymay influence interactive behavior.

Original: In humans, the learning of external and internal speech, the rehearsal of inner speech, and thepractice of tasks can develop concurrently.
Revised: While in humans both inner speech and behavior develop simultaneously, here the system is trained intwo stages for simplicity.

Original: In the first stage, we pretrain a captioning model bysemi-supervised learning, using a small amount of labeled languagedata and the large amount ofagent’s experience as unlabeled data.This model is able to commentate about visual observations – acritical function of “inner speech”.
Revised: First, we develop a semi-supervised language learning method to efficientlypre-train a capacity for intra-agent speech when given access to large amounts of unlabeled data andsmall amounts of labeled language data.

Original: This model is then used to commentate (via what we have named“intra-agent speech”) on further experiences to shape representations as the agent goes on to learnspecific tasks, and we show comparisons of the agent’s performance with and without this intra-agentspeech process.
Revised: We then explore how this capacity for intra-agent speechcan be leveraged to shape subsequent interactive behavior in an embodied agent.

Original: A diagram of the semi-supervised learning approach to intra-agent speech and its role in acquiringbehaviors is shown in Figure 2.
Revised: A diagram of thesemi-supervised learning approach to intra-agent speech and its role in acquiring behaviors is shownin Figure 2.

Original: 2.1 Methods for Semi-supervised Language Learning
Revised: 2.1 Methods for Semi-supervised Language Learning

Original: Our data D are composed of a dataset D p of paired data { x , y } and a dataset D u of unpaired data{ x } .
Revised: Our data D are composed of a dataset D p of paired data { x , y } and a dataset D u of unpaired data{ x } .

Original: In our interpretation, y corresponds to language (a caption) describing the image x .
Revised: In our interpretation, y corresponds to language (a caption) describing the image x .

Original: Whenobserving unpaired data, we treat y as a latent variable that must be inferred to explain the observeddata [13].
Revised: Whenobserving unpaired data, we treat y as a latent variable that must be inferred to explain the observeddata [13].

Original: We develop two variants, one generative and one contrastive, of the semi-supervisedlanguage learning algorithm in what follows.[Figure]
Revised: We develop two variants, one generative and one contrastive, of the semi-supervisedlanguage learning algorithm in what follows.

Original: 2.1.1 Generative variant
Revised: 2.1.1 Generative variant

Original: The preponderance of our data is unlabeled.
Revised: The preponderance of our data is unlabeled.

Original: To model these data we can maximise a variational lowerbound on the log marginal distribution, log p θ ( x ) , corresponding to posterior inference.
Revised: To model these data we can maximise a variational lowerbound on the log marginal distribution, log p θ ( x ) , corresponding to posterior inference.

Original: We willuse a learned approximation to the true posterior distribution, q ω ( y | x ) .
Revised: We willuse a learned approximation to the true posterior distribution, q ω ( y | x ) .

Original: We call this approximatedistribution an “image-conditional language encoder” (or sometimes, simply, caption model ) sinceits functional role is to convert images into a discrete latent code that is, ideally, a semanticallymeaningful language caption:(cid:88)
Revised: We call this approximatedistribution an “image-conditional language encoder” (or sometimes, simply, caption model ) sinceits functional role is to convert images into a discrete latent code that is, ideally, a semanticallymeaningful language caption:

Original: [Equation]
Revised: [Equation]

Original: In Equation 1, the space of latent codes (captions) is discrete and vast, complicating optimization.
Revised: In Equation 1, the space of latent codes (captions) is discrete and vast, complicating optimization.

Original: However, we can form a stochastic approximation by sampling from q ω ( y | x ) to estimate a gradient.
Revised: However, we can form a stochastic approximation by sampling from q ω ( y | x ) to estimate a gradient.

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we pretrain the prior p ϕ ( y ) as a language model that learns on all of our caption data,and fix it during this optimisation phase.
Revised: For simplicity, we pretrain the prior p ϕ ( y ) as a language model that learns on all of our caption data,and fix it during this optimisation phase.

Original: Thus, optimising J u corresponds to an entropy-regularised
Revised: Thus, optimising J u corresponds to an entropy-regularised

Original: RL problem [14–16].
Revised: RL problem [14–16].

Original: The encoder q ω ( y | x ) serves as a policy that achieves reward for matching thelanguage model prior while enabling the language-conditional decoder to reconstruct the image.
Revised: The encoder q ω ( y | x ) serves as a policy that achieves reward for matching thelanguage model prior while enabling the language-conditional decoder to reconstruct the image.

Original: While this reward pressures the image-conditional language encoder to produce linguistically meaningful latent codes, we can also exert additional pressure by using a small number of imageswith associated captions (“labeled data”), and leverage supervised learning to learn log p θ ( x |y ) , log p ϕ ( y ) , log q ω ( y | x ) :
Revised: While this reward pressures the image-conditional language encoder to produce linguistically meaningful latent codes, we can also exert additional pressure by using a small number of imageswith associated captions (“labeled data”), and leverage supervised learning to learn log p θ ( x |y ) , log p ϕ ( y ) , log q ω ( y | x ) :

Original: [Equation]
Revised: [Equation]

Original: Over the combined paired and unpaired datasets, our complete objective is to maximise J =
Revised: Over the combined paired and unpaired datasets, our complete objective is to maximise J =

Original: J p + J u .
Revised: J p + J u .

Original: 2.1.2 Contrastive variant
Revised: 2.1.2 Contrastive variant

Original: The language-conditional image decoder p θ ( x | y ) is a potentially complicated term to model.
Revised: The language-conditional image decoder p θ ( x | y ) is a potentially complicated term to model.

Original: To understand if this term represents a liability for the approach and to generate an algorithmwith potentially complementary strengths, we also developed a contrastive, energy-based approach, amounting to multi-class classification, to approximate it [17–19].
Revised: To understand if this term represents a liability for the approach and to generate an algorithmwith potentially complementary strengths, we also developed a contrastive, energy-based approach,amounting to multi-class classification, to approximate it [17–19].

Original: If we have a multi-class classifierthat discriminates if a batch element y j is paired to a batch element x j , and we express its crossentropy loss for the correct index c
Revised: If we have a multi-class classifierthat discriminates if a batch element y j is paired to a batch element x j , and we express its crossentropy loss for the correct index c

Original: = j as L ( { x b } Bb =1 , y j , c = j ) , then we can show that thegenerative reconstruction loss log p θ ( x j | y j ) is proportional to L ( { x b } Bb =1 , y j , c = j ) up to constantfactors with respect to y (Appendix A.1).
Revised: = j as L ( { x b } Bb =1 , y j , c = j ) , then we can show that thegenerative reconstruction loss log p θ ( x j | y j ) is proportional to L ( { x b } Bb =1 , y j , c = j ) up to constantfactors with respect to y (Appendix A.1).

Original: Thus, we can substitute L ( { x b } Bb =1 , y j , c = j ) for batchelement j , log p θ ( x j | y j ) , in Equation 2, and the gradients with respect to ω remain the same inexpectation.
Revised: Thus, we can substitute L ( { x b } Bb =1 , y j , c = j ) for batchelement j , log p θ ( x j | y j ) , in Equation 2, and the gradients with respect to ω remain the same inexpectation.

Original: We represented the softmax in the classifier as
Revised: We represented the softmax in the classifier as

Original: [Equation]
Revised: [Equation]

Original: , where f and g are networks thatcompress the image into a vector and the caption into a vector, respectively.
Revised: , where f and g are networks thatcompress the image into a vector and the caption into a vector, respectively.

Original: To train this classifier,we adopted a similar approach to recent models for visual language representation learning [20, 21]by minimizing the sum of two losses, one for matching each image to its paired caption in the batch,and one for matching each caption to its paired image:
Revised: To train this classifier,we adopted a similar approach to recent models for visual language representation learning [20, 21]by minimizing the sum of two losses, one for matching each image to its paired caption in the batch,and one for matching each caption to its paired image:

Original: [Equation]
Revised: [Equation]

Original: The networks f and g are trained at the same time as the rest of the model, and samples fromq ω
Revised: The networks f and g are trained at the same time as the rest of the model, and samples fromq ω

Original: ( y j | x j ) are included as positive examples for the classifier along with the original paired data.
Revised: ( y j | x j ) are included as positive examples for the classifier along with the original paired data.

Original: 2.1.3 Intra-agent speech architecture and optimization
Revised: 2.1.3 Intra-agent speech architecture and optimization

Original: Image-conditional language encoder.
Revised: Image-conditional language encoder.

Original: The image-conditional language encoder received inputimages with resolution 96 × 72 .
Revised: The image-conditional language encoder received inputimages with resolution 96 × 72 .

Original: We used the ResNet architecture described in [22], with strides(1 , 1 , 2 , 2) , 3 × 3 kernel size, and channel sizes (64 , 64 , 128 , 256) for a total of 20 layers.
Revised: We used the ResNet architecture described in [22], with strides(1 , 1 , 2 , 2) , 3 × 3 kernel size, and channel sizes (64 , 64 , 128 , 256) for a total of 20 layers.

Original: Languagewas produced by a 4 -layer causal transformer with 256 embedding size and 4 attention heads [23],which attended to the 432 hyper-pixel vectors generated from the ResNet, and produced a sequenceof logits corresponding to a 4 , 000 token vocabulary.
Revised: Languagewas produced by a 4 -layer causal transformer with 256 embedding size and 4 attention heads [23],which attended to the 432 hyper-pixel vectors generated from the ResNet, and produced a sequenceof logits corresponding to a 4 , 000 token vocabulary.

Original: Language targets were encoded with a SentencePiece byte-pair encoder [24] trained using language data from [25].
Revised: Language targets were encoded with a SentencePiece byte-pair encoder [24] trained using language data from [25].

Original: Generative semi-supervised model.
Revised: Generative semi-supervised model.

Original: The language prior comprised a separate transformer pretrained on all the caption labels in the training data (and hence, did not have image inputs).
Revised: The language prior comprised a separate transformer pretrained on all the caption labels in the training data (and hence, did not have image inputs).

Original: For theimage decoder, we first pre-trained a VQ-VAE [26] on all unlabeled images to define the VQ-VAE’scodebook, compressing each image into 432 tokens with a vocabulary of 512 .
Revised: For theimage decoder, we first pre-trained a VQ-VAE [26] on all unlabeled images to define the VQ-VAE’scodebook, compressing each image into 432 tokens with a vocabulary of 512 .

Original: We then used an8-layer transformer with 512 embedding size with causal masking to model the VQ-VAE tokensautoregressively.
Revised: We then used an8-layer transformer with 512 embedding size with causal masking to model the VQ-VAE tokensautoregressively.

Original: The conditioning captions were tokenized and embedded as in the image-conditionallanguage encoder, and then processed by a distinct transformer with 4 layers, 256 embedding size,and 4 heads, whose output was then cross-attended by the image decoder to produce a reconstruction.
Revised: The conditioning captions were tokenized and embedded as in the image-conditionallanguage encoder, and then processed by a distinct transformer with 4 layers, 256 embedding size,and 4 heads, whose output was then cross-attended by the image decoder to produce a reconstruction.

Original: Contrastive semi-supervised model.
Revised: Contrastive semi-supervised model.

Original: For the contrastive model, images were first encoded using a
Revised: For the contrastive model, images were first encoded using a

Original: ResNet with the same architecture as the image-conditional language encoder, and globally meanpooled into one vector with dimension 1024 .
Revised: ResNet with the same architecture as the image-conditional language encoder, and globally meanpooled into one vector with dimension 1024 .

Original: Language was first encoded using a transformer withthe same 4-layer architecture described above, and then flattened and compressed with a 2-layer MLPwith dimensions (2048 , 1024) .
Revised: Language was first encoded using a transformer withthe same 4-layer architecture described above, and then flattened and compressed with a 2-layer MLPwith dimensions (2048 , 1024) .

Original: Optimization.
Revised: Optimization.

Original: We used V-MPO [16] to optimize q ω ( y | x ) .
Revised: We used V-MPO [16] to optimize q ω ( y | x ) .

Original: The loss was optimized by the Adamoptimizer [27] with β 1 = 0 .
Revised: The loss was optimized by the Adamoptimizer [27] with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 and a learning rate of 2 × 10 − 4 with early stopping atthe lowest log-likelihood over captions in validation data.
Revised: 999 and a learning rate of 2 × 10 − 4 with early stopping atthe lowest log-likelihood over captions in validation data.

Original: We trained all models with a batch sizeof 128 except for the contrastive classifier model, which also received 2 , 048 unlabeled images perbatch, giving a total batch size of 2 , 176 .
Revised: We trained all models with a batch sizeof 128 except for the contrastive classifier model, which also received 2 , 048 unlabeled images perbatch, giving a total batch size of 2 , 176 .

Original: We trained our models using Tensor Processing Units(TPUv3)
Revised: We trained our models using Tensor Processing Units(TPUv3)

Original: In all experiments the early stopping criteria was reached within 150 K optimizationsteps.
Revised: In all experiments the early stopping criteria was reached within 150 K optimizationsteps.

Original: 2.2 Agent training
Revised: 2.2 Agent training

Original: The agent was trained by behavioral cloning on a large corpus of human interactions in the Playhouseenvironment [22].
Revised: The agent was trained by behavioral cloning on a large corpus of human interactions in the Playhouseenvironment [22].

Original: Briefly, it comprises a 3D simulated environment wherein two embodied avatarsinteract with natural language to cooperatively accomplish tasks involving object manipulation,navigation, and question answering.
Revised: Briefly, it comprises a 3D simulated environment wherein two embodied avatarsinteract with natural language to cooperatively accomplish tasks involving object manipulation,navigation, and question answering.

Original: In addition to receiving visual observations and producing motoractions, agents in this environment also receive language inputs and produce language outputs.
Revised: In addition to receiving visual observations and producing motoractions, agents in this environment also receive language inputs and produce language outputs.

Original: Thisallows them to answer questions, participate in dialogue, ask for clarifications, and so on.
Revised: Thisallows them to answer questions, participate in dialogue, ask for clarifications, and so on.

Original: Our approach to training the linguistic representations of the agent therefore has two steps.
Revised: Our approach to training the linguistic representations of the agent therefore has two steps.

Original: First, byusing semi-supervised language learning, we can amplify the impact of sparse language annotationsprovided for a small number of images in learning the intra-agent speech module.
Revised: First, byusing semi-supervised language learning, we can amplify the impact of sparse language annotationsprovided for a small number of images in learning the intra-agent speech module.

Original: In turn, using thistrained module, we are able to sample captions at high frequency as the agent acts, providing theagent with dense language supervision.
Revised: In turn, using thistrained module, we are able to sample captions at high frequency as the agent acts, providing theagent with dense language supervision.

Original: Therefore, while the agent trains using behavioral cloning,we can sample captions from the pretrained intra-agent speech module, and provide these as eitherauxiliary inputs or targets to the agent so that they may influence the agent parameters optimized toproduce embodied behaviors.
Revised: Therefore, while the agent trains using behavioral cloning,we can sample captions from the pretrained intra-agent speech module, and provide these as eitherauxiliary inputs or targets to the agent so that they may influence the agent parameters optimized toproduce embodied behaviors.

Original: For our first loss – the caption loss – captions arising as intra-agent speech samples were provided astargets for the agent’s language output policy:
Revised: For our first loss – the caption loss – captions arising as intra-agent speech samples were provided astargets for the agent’s language output policy:

Original: [Equation]
Revised: [Equation]

Original: where π l is the language action policy of the agent, y c is the caption sample, o represents all visualand text observations, and K represents the maximum unroll length.
Revised: where π l is the language action policy of the agent, y c is the caption sample, o represents all visualand text observations, and K represents the maximum unroll length.

Original: To allow the agent to distinguishwhether it was being tasked with emitting a caption of its current observation, or whether it was beingtasked with emitting language for the purposes of interactive behavior (see [22]), the input of thelanguage policy is summed with a learnable embedding of the indicator variable representing whetherthe target is captioning or language output from demonstration.
Revised: To allow the agent to distinguishwhether it was being tasked with emitting a caption of its current observation, or whether it wasbeing tasked with emitting language for the purposes of interactive behavior (see [22]), the agent alsoreceived a corresponding binary indicator embedding as input.

Original: While the caption loss trains the agent to speak about what it sees, the agent’s language encoding (vialinguistic inputs) remains untrained.
Revised: While the caption loss trains the agent to speak about what it sees, the agent’s language encoding (vialinguistic inputs) remains untrained.

Original: To this end we introduced a second loss: the caption-matchingloss.
Revised: To this end we introduced a second loss: the caption-matchingloss.

Original: In the caption-matching loss, the agent is required to predict whether a given caption, provided asinput, matches the observed image.
Revised: In the caption-matching loss, the agent is required to predict whether a given caption, provided asinput, matches the observed image.

Original: Negative samples (i.e., captions that do not match the observation)are captions associated with images from elsewhere in the batch [29].
Revised: Negative samples (i.e., captions that do not match the observation)are captions associated with images from elsewhere in the batch [29].

Original: To perform this prediction,an auxiliary classifier D is attached to the network of the agent that encodes both visual and textobservations in a separate training pass from the behavioral cloning optimization. 
Revised: To perform this prediction,an auxiliary classifier D is attached to the network of the agent that encodes both visual and textobservations in a separate training pass from the behavioral cloning optimization. [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here o X represents the visual observation, and the caption sample is fed as a text observation.
Revised: Here o X represents the visual observation, and the caption sample is fed as a text observation.

Original: Theroll () function rolls the index over batches, so that y c roll ( b ) represents a caption from another batchelement as a negative example.
Revised: Theroll () function rolls the index over batches, so that y c roll ( b ) represents a caption from another batchelement as a negative example.

Original: The roll function was implemented as roll ( b ) =
Revised: The roll function was implemented as roll ( b ) =

Original: ( b + 1) mod B .
Revised: ( b + 1) mod B .

Original: And finally, the behavioral cloning loss over human language and movement action sequences is asdescribed in Interactive Agents Team [22]:
Revised: And finally, the behavioral cloning loss over human language and movement action sequences is asdescribed in Interactive Agents Team [22]:

Original: [Equation]
Revised: [Equation]

Original:  where the π m and a m represents the movement policy and action, respectively.
Revised: (7) where the π m and a m represents the movement policy and action, respectively.

Original: The total loss is the sum of these losses L =
Revised: The total loss is the sum of these losses L =

Original: L C + L CM + L BC .
Revised: L C + L CM + L BC .

Original: Other than the addition of the captionloss and caption-matching loss, we follow Interactive Agents Team [22] for the agent architectureand training hyper-parameters.
Revised: Other than the addition of the captionloss and caption-matching loss, we follow Interactive Agents Team [22] for the agent architectureand training hyper-parameters.

Original:  Experiments
Revised:  Experiments

Original: 3.1 Semi-supervised captioning
Revised: 3.1 Semi-supervised captioning

Original: Hypothesis.
Revised: Hypothesis.

Original: We first sought to validate our method of semi-supervised learning.
Revised: We first sought to validate our method of semi-supervised learning.

Original: We hypothesizedthat with the additional unlabeled images, the semi-supervised model would be more data efficient(with respect to labelled data) than a supervised baseline.
Revised: We hypothesizedthat with the additional unlabeled images, the semi-supervised model would be more data efficient(with respect to labelled data) than a supervised baseline.

Original: Data.
Revised: Data.

Original: As mentioned, the domain in which we tested our methods is called the Playhouse, whichoriginated in Interactive Agents Team [22].
Revised: As mentioned, the domain in which we tested our methods is called the Playhouse, whichoriginated in Interactive Agents Team [22].

Original: The authors compiled approximately three years worth ofinteraction data, comprising about 2 billion image frames.
Revised: The authors compiled approximately three years worth ofinteraction data, comprising about 2 billion image frames.

Original: The authors have confirmed no personallyidentifiable information or offensive content is contained in the dataset and gave consent for us toaccess it
Revised: The authors have confirmed no personallyidentifiable information or offensive content is contained in the dataset and gave consent for us toaccess it.

Original: As there are no associated captions with any frames, we treat each frame in this dataset as asingle image and used all the data as our “unpaired” dataset
Revised: As there are no associated captions with any frames, we treat each frame in this dataset as asingle image, and used all the data as our “unpaired” dataset.

Original: For the “paired” dataset, we engaged with crowd raters to provide corresponding captions for 78Kuniformly sampled images from the unpaired dataset.
Revised: For the “paired” dataset, we engaged with crowd raters to provide corresponding captions for 78Kuniformly sampled images from the unpaired dataset.

Original: Raters were instructed to describe what theysaw in the image, with particular reference to a randomly selected object indicated in a bounding box.
Revised: Raters were instructed to describe what theysaw in the image, with particular reference to a randomly selected object indicated in a bounding box.

Original: Detailed instructions are included in Appendix A.2.
Revised: Detailed instructions are included in Appendix A.2.

Original: On average, each image contain 12 .

Original: 9 objects,and its corresponding caption describe 1 .

Original: 2 of them.

Original: Evaluation.
Revised: Evaluation.

Original: We measured the log probability of captions in the validation set, CIDEr score [30],object precision and “color-object pair precision”.
Revised: We measured the log probability of captions in the validation set, CIDEr score [30]and “color-object accuracy”.

Original: As the images were generated from the Playhouseenvironment, we had access to the ground truth object identities and colors present in each image,allowing us to check if any object and its color mentioned in a caption were indeed present in theimage.
Revised: As the images were generated from the Playhouse environment, wehad access to the ground truth object identities and colors present in each image, allowing us tocheck if color-object pairs mentioned in a caption were indeed present in the image.

Original: We calculated object precision to be the proportion of correct objects among all mentionedin sampled captions, normalized by the same calculation from human captions.The color-objectpair precision is calculated in the same manner, except for any mentions of an object and its color.
Revised: We calculatedcolor-object accuracy to be the proportion of correct color-object pairs among all mentioned insampled captions, normalized by the same calculation from human captions.

Original: As there are usually many objects in an image and captions tend to mention just a few, the recallcounterpart of this metric (number of correct color-object pairs divided by those present in the image)is not very informative.
Revised: As there are usuallymany objects in an image and captions tend to mention just a few, the recall counterpart of this metric(number of correct color-object pairs divided by those present in the image) is not very informative.

Original: Captions were greedily sampled for the calculation of CIDEr scoreobjectprecision and color-object pair precision.
Revised: Captions were greedily sampled for the calculation of CIDEr score and color-object accuracy.

Original: Results.
Revised: Results.

Original: Figure 3 shows sample data used to train the our models, as well as samples generatedfrom the generative semi-supervised model.
Revised: Figure 3 shows sample data used to train the our models, as well as samples generatedfrom the generative semi-supervised model.

Original: Models were able to both produce coherent and relevantcaptions given an image (e.g., "I can see a white bed where a green ball and a green duck are onit") and also generated realistic visual depictions of the Playhouse when conditioned on a languagedescription.
Revised: Models were able to both produce coherent and relevantcaptions given an image (e.g., "I can see a white bed where a green ball and a green duck are onit") and also generated realistic visual depictions of the Playhouse when conditioned on a languagedescription.

Original: Both semi-supervised methods performed better than the supervised baseline across log prob, CIDErand color-object pair precision, including settings where we artificially constrained the size of thelabeled dataset (Figure 4).
Revised: Both semi-supervised methods performed better than the supervised baseline across allmetrics, including settings where we artificially constrained the size of the labeled dataset (Figure 4).

Original: Notably, when using the full amount of data our semi-supervised methodsexceeded human color-object pair precision.
Revised: Notably, when using the full amount of data our semi-supervised methods exceeded human colorobject accuracy.

Original: To further put the performance gain in context, byleveraging the unpaired dataset of images the semi-supervised methods performed at a level equivalentto training a supervised model on 3 × more labeled data [31].
Revised: To further put the performance gain in context, by leveraging the unpaired dataset ofimages the semi-supervised methods performed at a level equivalent to training a supervised modelon 3 × more labeled data [31].

Original: Interestingly, the semi-supervised methods perform similarly in the object precision metric, despitean advantage in color-object pair precision.

Original: This result suggest that the semi-supervised methods mayhave learned to bind color to objects from the unlabelled images, and this results in a performancegain compared to pure supervised methods.

Original: [Figure]
Revised: [Figure]

Original: 3.2 Learning captions of new objects
Revised: 3.2 Learning captions of new objects

Original: Hypothesis.
Revised: Hypothesis.

Original: Upon validating our methodology and amassing the relevant data, we were well-poisedto ask the following questions: do semi-supervised methods of intra-agent speech allow a model toquickly learn to speak about a new object with little supervision?
Revised: Upon validating our methodology and amassing the relevant data, we were well-poisedto ask the following questions: do semi-supervised methods of intra-agent speech allow a model toquickly learn to speak about a new object with little supervision?

Original: Data.
Revised: Data.

Original: To address the hypothesis, we first chose an object – a drum – and filtered our labeled captiondata to remove all instances where the drum is either depicted visually or spoken about verbally.
Revised: To address the hypothesis, we first chose an object – a drum – and filtered our labeled captiondata to remove all instances where the drum is either depicted visually or spoken about verbally.

Original: While not strictly necessary, from the unlabeled dataset we also removed all instances where theunderlying (and unobserved) instruction included mention of a drum, which ensured that all instances
Revised: While not strictly necessary, from the unlabeled dataset we also removed all instances where theunderlying (and unobserved) instruction included mention of a drum, which ensured that all instances

Original: [Figure]
Revised: [Figure]

Original: of drum appearances were truly “passive”.
Revised: of drum appearances were truly “passive”.

Original: For example, the unlabeled data may have includedtrajectories of experience wherein a players was tasked with removing a ball from a shelf, but theremight have also happened to be a drum in view in the room.
Revised: For example, the unlabeled data may have includedtrajectories of experience wherein a players was tasked with removing a ball from a shelf, but theremight have also happened to be a drum in view in the room.

Original: Evaluation.
Revised: Evaluation.

Original: We evaluated the model using three different metrics.
Revised: We evaluated the model using three different metrics.

Original: We first created a validationdataset of all captions mentioning drum with their associated image, and evaluated the model’saverage log likelihood on this drum dataset (drum caption log likelihood).
Revised: We first created a validationdataset of all captions mentioning drum with their associated image, and evaluated the model’saverage log likelihood on this drum dataset (drum caption log likelihood).

Original: The drum recall, whichis the probability that them model mentions a drum given that a drum is present, measures how themodel recognizes drums in the images.
Revised: The drum true positiverate, which is the probability that it mentions a drum given that a drum is present, measures how themodel recognizes drums in the images.

Original: We also calculate its counterpart, the probability that a drumis present given a drum  is mentioned, as the drum precision, to measure the accuracy of any drummentions from the model.
Revised: We also calculate its counterpart, the probability of the modelerroneously mentioning a drum when it is not present as the drum false positive rate.

Original: The perfect scores for drum recall and drum precision are 1 as a resultof titering the drum data: we have removed all the data where drum is in the image or caption, andadded back a small amount where drum is mentioned in the caption.

Original: This has a consequence that forany image with drum (although very few), its caption will always mention drum.

Original: Results.
Revised: Results.

Original: We then performed the same experiment as in Section 3.1 at varying levels of re-introducedlabeled drum data, as seen in Figure 4.
Revised: We then performed the same experiment as in Section 3.1 at varying levels of re-introducedlabeled drum data, as seen in Figure 4.

Original: As expected, the model’s ability to speak about drums scaleswith the amount of labeled drum data with which it is trained.
Revised: As expected, the model’s ability to speak about drums scaleswith the amount of labeled drum data with which it is trained.

Original: Notably, the model’s recall reachesapproximately 70% given only 585 labeled samples of drums.
Revised: Notably, the model’s true positive ratereaches approximately 70% given only 585 labeled samples of drums.

Original: In contrast, the supervised method,where the new drum data is trained in concatenation with the non-drum data, reaches just over 20%.
Revised: In contrast, the supervisedmethod reaches just over 20%.

Original: The increase of drum recall in the semi-supervised method is also accompanied with an increase ofprecision, where the semi-supervised method reaches 80%, versus 40% of the supervised method.

Original: These results indicate that the semi-supervised methods are able to learn about new objects, implicitly ,by virtue of their language-conditioned reconstruction objective.
Revised: This indicates that the semi-supervised methods are able to learnabout new objects, implicitly , by virtue of their language-conditioned reconstruction objective.

Original: That is, the mere presence of drumsin the visual observations was sufficient, in conjunction with a small amount of labeled drum captions,to elicit linguistic understanding of drums as relevant objects in the Playhouse.
Revised: Thatis, the mere presence of drums in the visual observations was sufficient, in conjunction with a smallamount of labeled drum captions, to elicit linguistic understanding of drums as relevant objects in the Playhouse.

Original: [Figure]
Revised: [Figure]

Original:  to manipulate a new object
Revised: 3.3 Learning to manipulate a new object

Original: Hypothesis.
Revised: Hypothesis.

Original: As a final test we tackled the final question: Can the ability to quickly learn tospeak about an object confer advantages for learning behaviors involving the new object (such asmanipulating the object, or answering questions about it)?
Revised: As a final test we tackled the final question: Can the ability to quickly learn tospeak about an object confer advantages for learning behaviors involving the new object (such asmanipulating the object, or answering questions about it)?

Original: Methods and data.
Revised: Methods and data.

Original: To answer this question we leveraged the semi-supervised models built in
Revised: To answer this question we leveraged the semi-supervised models built in

Original: Section 3.2 to provide auxiliary intra-agent speech objectives.
Revised: Section 3.2 to provide auxiliary intra-agent speech objectives.

Original: While we employed a two-stage process(learning to caption, and subsequently learning from those captions to shape behavior) for expedienceand ease of experimental setup, in principle the two learning procedures could occur  in parallel ina unified agent.
Revised: While we employed a two-stageprocess (learning to caption, and subsequently learning from those captions to shape behavior) forexpedience and ease of experimental setup, in principle the two learning procedures could occur inparallel in a unified agent.

Original: The agent then proceeded with behavioral cloning-based training as inInteractive
Revised: The agent then proceeded with behavioral cloning-based training as in

Original:  Agents Team [22] on data that was filtered to not include any drum-based instructions (resulting in9 , 344 removed interactions).
Revised: Interactive Agents Team [22] on data that was filtered to not include any drum-based instructions(resulting in 9 , 344 removed interactions).

Original: While after filter the dataset do not contain any demonstration of druminteraction, in some episodes the drum object is visible (larger than a pixel) in the background in4.4% of the image frames.

Original: Evaluation.
Revised: Evaluation.

Original: We evaluated agent performance on two tasks: in the “lift drum” task, the agent receivesthe instruction “lift a drum”, and is required to find the drum in a randomized Playhouse environmentand lift it.
Revised: We evaluated agent performance on two tasks: in the "lift drum" task, the agent receivesthe instruction "lift a drum", and is required to find the drum in a randomized Playhouse environmentand lift it.

Original: In the “ask color drum” task, a single drum is spawned in the environment.
Revised: In the "ask color drum" task, a single drum is spawned in the environment.

Original: The agentreceives the instruction “what is the color of the drum?”, and is then required to output language thatanswers the question (See Appendix A.4 for details).
Revised: The agentreceives the instruction "what is the color of the drum?", and is then required to output language thatanswers the question (See Appendix A.4 for details).

Original: At the end of eachepisode, the task environmentgives out a score of 0 or 1, indicating whether the task is completed successfully (note that thesescores are used for evaluation purposes only, and not for training).We collected human performanceon each of the tasks. Our participants receive an average of 0 . 98 for the “lift a drum” task, and 0 .
Revised: We collected human performance on each of thetasks and normalized reward by the average human score (note that rewards are used for evaluationpurposes only, and not for training).

Original: 51for the “ask color drum” task.

Original: The low score of “ask color drum” is primary due to the stringentreward function that require exact match of the canonical color name, and the fact that many colorsare similar to the canonical name (for example, the canonical “cyan” color is can often referred as“light blue” by human participants).

Original: We calculate the final score by dividing the agent score by theaverage human score, so that a final score of 0 represents no success, and a 1 represents human-levelperformance.

Original: Results.
Revised: Results.

Original: As seen in Figure 6, a baseline agent trained without any drum-based instructions performspoorly on drum-based evaluation tasks, indicating that there is minimal transfer to be had whenlearning from the base interaction dataset.
Revised: As seen in Figure 6, a baseline agent trained without any drum-based instructions performspoorly on drum-based evaluation tasks, indicating that there is minimal transfer to be had whenlearning from the base interaction dataset.

Original: On the other hand, a model trained with drum-basedinstructions (that is, one that trains on the 9 , 344 episodes that we had filtered) displays an expectedupper bound for this agent of just over 75% on "lifting drums", and just under 80% for answeringquestions about a drum’s color, similar to previously reported results [22].
Revised: On the other hand, a model trained with drum-basedinstructions (that is, one that trains on the 9 , 344 episodes that we had filtered) displays an expectedupper bound for this agent of just over 75% on "lifting drums", and just under 80% for answeringquestions about a drum’s color, similar to previously reported results [22].

Original: Agents trained with both the caption loss and caption-matching loss, but without any drum-basedinteraction data achieve approximately 70% of the performance attained by models trained directlyon drum-based interaction data.
Revised: Agents trained with both the caption loss and caption-matching loss, but without any drum-basedinteraction data achieve approximately 70% of the performance attained by models trained directlyon drum-based interaction data.

Original: This is a substantial increase over the baseline model, and directlyindicates that learning to speak about drums, which is afforded by training a model on as few as585 labeled drum examples using semi-supervised methods, allows an agent to subsequently exhibitbehaviors that implicate the manipulation or mention of drums, without affecting performance on aknown object (Appendix A.5).
Revised: This is a substantial increase over the baseline model, and directlyindicates that learning to speak about drums, which is afforded by training a model on as few as585 labeled drum examples using semi-supervised methods, allows an agent to subsequently exhibitbehaviors that implicate the manipulation or mention of drums, without affecting performance on aknown object (Appendix A.5).

Original: We then tested whether we could reduce the number of labeled captions even further, and observedthat with as few as 150 labeled captions we observed similar performance on drum-color questionanswering tasks, and only a slight decrease in drum lifting task performance.
Revised: We then tested whether we could reduce the number of labeled captions even further, and observedthat with as few as 150 labeled captions we observed similar performance on drum-color questionanswering tasks, and only a slight decrease in drum lifting task performance.

Original:  Related Work
Revised:  Related Work

Original: Components.
Revised: Components.

Original: The modeling in this study builds on components developed for generative modeling(VQVAE-2: [32]) and contrastive learning (CLIP: [20]).
Revised: The modeling in this study builds on components developed for generative modeling(VQVAE-2: [32]) and contrastive learning (CLIP: [20]).

Original: Task generalization in embodied agents.
Revised: Task generalization in embodied agents.

Original: Our work is in the tradition of grounded languageresearch in virtual environments [33–35].
Revised: Our work is in the tradition of grounded languageresearch in virtual environments [33–35].

Original: Recently, there have been efforts to demonstrate zeroshot task acquisition using a variety of means including with the use of pretrained models forlanguage encoders [36] or visual-language models for zero-shot navigation [37].
Revised: Recently, there have been efforts to demonstrate zeroshot task acquisition using a variety of means including with the use of pretrained models forlanguage encoders [36] or visual-language models for zero-shot navigation [37].

Original: The majority ofthese works have focused on generalization to novel phrasings of task instructions [38, 39] with somerecent studies exploring generalization to new tasks given background knowledge from previoustasks [22, 40].
Revised: The majority ofthese works have focused on generalization to novel phrasings of task instructions [38, 39] with somerecent studies exploring generalization to new tasks given background knowledge from previoustasks [22, 40].

Original: Similar to phenomena of zero-shot generalization in supervised classification andobject detection [20, 21, 41], these studies have often relied on a large, diverse dataset providing [Figure] supervision – where held-out tasks represent a novel recombination of parts of the training dataset (e.g.,“compositional” generalization or systematicity [42]).
Revised: Similar to phenomena of zero-shot generalization in supervised classification andobject detection [20, 21, 41], these studies have often relied on a large, diverse dataset providingsupervision – where held-out tasks represent a novel recombination of parts of the training dataset (e.g.,“compositional” generalization or systematicity [42]).

Original: Another line of research has focused on fewshot generalization [43–46] through meta-learning over a distribution of tasks – with generalizationto other tasks within the distribution’s support.
Revised: Another line of research has focused on fewshot generalization [43–46] through meta-learning over a distribution of tasks – with generalizationto other tasks within the distribution’s support.

Original: Our work is complementary to both lines of work –and presents a means by which a different learning objective can embody “mental practice” enablinggeneralisation.
Revised: Our work is complementary to both lines of work –and presents a means by which a different learning objective can embody “mental practice” enablinggeneralisation.

Original: Semi-supervised learning with natural language.
Revised: Semi-supervised learning with natural language.

Original: The semi-supervised methods we used to train theintra-agent speech module are related to many previous methods that use language as a latent variable. 
Revised: The semi-supervised methods we used to train theintra-agent speech module are related to many previous methods that use language as a latent variable. [Figure]

Original: For example, Kingma et al.
Revised: For example, Kingma et al.

Original: [13] developed a semi-supervised, variational autoencoder (VAE) withdiscrete latent variables, and this work has also been extended to use language as the latent variable.
Revised: [13] developed a semi-supervised, variational autoencoder (VAE) withdiscrete latent variables, and this work has also been extended to use language as the latent variable.

Original: VAEs with language latent variables have also been studied in language summarization [47], and inthe context of multi-agent communication with natural language [48, 49].
Revised: VAEs with language latent variables have also been studied in language summarization [47].

Original: Our work may present avaluable contribution to this area, showcasing the utility of modern generative and contrastive modelsin supporting data-efficient learning.
Revised: Ourwork may present a valuable contribution to this area, showcasing the utility of modern generative andcontrastive models in supporting data-efficient learning.

Original: Our work’s analysis of the scaling curve [31] properties ofsemi-supervised captioning [50–52] may provide additional analytical tools for the field.
Revised: Our work’s analysis of the scaling curve [31]properties of semi-supervised captioning [48–50] may provide additional analytical tools for the field.

Original: Language-conditioned image generation.
Revised: Language-conditioned image generation.

Original: The use of language to create sophisticated, controllablegenerative models has seen rapid development in recent years.
Revised: The use of language to create sophisticated, controllablegenerative models has seen rapid development in recent years.

Original: Recent works based on diffusionmodels [53–55] and autoregressive models have produced some of the most impressive results.
Revised: Recent works based on diffusionmodels [51–53] and autoregressive models have produced some of the most impressive results.

Original: Notable work includes the DALL · E family of models [56, 57] and GLIDE [58].
Revised: Notable work includes the DALL · E family of models [54, 55] and GLIDE [56].

Original: Other work in thearea has also employed VAEs [59] and GANs [60–64].
Revised: Other work in thearea has also employed VAEs [57] and GANs [58–62].

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: In this work we built a model for intra-agent speech by leveraging semi-supervised learning techniques. We have first shown that the semi-supervised methods outperforms the supervised model in captioningby leveraging unlabeled images.
Revised: In this work we built a model for intra-agent speech by leveraging semi-supervised learning techniquesthat maximize the impact of a small number of labeled samples in a sea of unlabeled samples.

Original: We further demonstrated that with 150 captions of a novel object,such models can help embodied agents execute interactive tasks with the novel object without havingto learn from direct interactive experience (zero-shot), reaching 70% performance of a baseline agenttrained with 9 , 000 interaction episodes.
Revised: Wefurther demonstrated how such models can help embodied agents execute interactive tasks withouthaving to learn from direct interactive experience (zero-shot).

Original: Although we developed the method in a virtual environment, it can be easily generalized to real worldapplications, applied with either good or bad intent.
Revised: Although we developed the method in a virtual environment, it can be easily generalized to real worldapplications, applied with either good or bad intent.

Original: Embodied language agents acting as robots couldbe instructed to perform a wide manner of tasks, including harmful ones.
Revised: Embodied language agents acting as robots couldbe instructed to perform a wide manner of tasks, including harmful ones.

Original: Thus, if this work is to beapplied in real settings, we advise extra caution to minimize unintended or harmful consequences.
Revised: Thus, if this work is to beapplied in real settings, we advise extra caution to minimize unintended or harmful consequences.

Original: This work is a first step into leveraging ideas from the psychology of inner and private speech inhumans.
Revised: This work is a first step into leveraging ideas from the psychology of inner and private speech inhumans.

Original: While our source of speech was captioning, humans have a richer repertoire of techniques for intra-personal dialogue that affect modalities beyond vision.
Revised: While our source of speech was captioning, humans have a richer repertoire of techniquesfor intra-personal dialogue that affect modalities beyond vision.

Original: Direct next steps are, therefore, toexplore a possible role of intra-agent speech in abstracting over time, and observing its effects onplanning and reasoning.
Revised: Direct next steps are, therefore, toexplore a possible role of intra-agent speech in abstracting over time, and observing its effects onplanning and reasoning.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?.•
Revised: [Yes] See Section ?.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.(c)
Revised: [Yes] See Section 5.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The code andthe data are proprietary.(b) Did you specify all the training details (e
Revised: [No] The code and the data are proprietary. (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] See Figure 4, 5 and 6.
Revised: [Yes] See Figure 4, 5 and 6.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 3.1.
Revised: [Yes] See Section 3.1.

Original: We trained ourmodels using Tensor Processing Units (TPUv3), in all experiments the early stopping criteria is reached within 150 K optimization steps.
Revised: We trained ourmodels using Tensor Processing Units (TPUv3), in all experiments the early stoppingcriteria is reached within 150 K optimization steps.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We have cited
Revised: [Yes] We have cited

Original: Interactive Agents Team [22].
Revised: Interactive Agents Team [22].

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: The assets are proprietary.
Revised: The assets are proprietary.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data
Revised: (d) Did you discuss whether and how consent was obtained from people whose data

Original: you’re using/curating?
Revised: you’re using/curating?

Original: [Yes] Consent for using the data was obtained from Interactive
Revised: [Yes] Consent for using the data was obtained from Interactive

Original: Agents Team [22] under proprietary contract.
Revised: Agents Team [22] under proprietary contract.

Original: See Section 3.1.
Revised: See Section 3.1.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See Section 3.1.
Revised: [Yes] See Section 3.1.

Original: Interactive Agents Team [22] has confirmed no personal identifiable information or harmful content is contained in
Revised: Interactive Agents Team [22]has confirmed no personal identifiable information or harmful content is contained in

Original: the dataset.
Revised: the dataset.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [Yes] See Appendix A.2.
Revised: [Yes] See Appendix A.2.

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [Yes] The full details of our study design were
Revised: [Yes] The full details of our study design were

Original: [Table]
Revised: [Table]

Original: reviewed by our institution’s independent ethical review committee.
Revised: reviewed by our institution’s independent ethical review committee.

Original: See Appendix A.2.
Revised: See Appendix A.2.

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [No] The compensation rates were reviewed by
Revised: [No] The compensation rates were reviewed by

Original: [Table]
Revised: [Table]

Original: our institution’s independent ethical review committee as part of the study design.
Revised: our institution’s independent ethical review committee as part of the study design.

Original: SeeAppendix A.2.
Revised: SeeAppendix A.2.

Original: [Table]
Revised: [Table]

Original: A AppendixA.1 Contrastive cross-entropy approximates generative log-likelihood
Revised: A AppendixA.1 Derivation for using contrastive loss to approximate generative log likelihood

Original: Consider a batch of images { x b } Bb =1 and a caption y corresponding to one of the images x j .
Revised: Consider a batch of images { x b } Bb =1 and a caption y corresponding to one of the images x j .

Original: Thiscaption will come either from the paired dataset or as a sample from the image-conditional languageencoder q ω ( y | x j ) .
Revised: Thiscaption will come either from the paired dataset or as a sample from the image-conditional languageencoder q ω ( y | x j ) .

Original: The posterior distribution over the classification of the image index c is
Revised: The posterior distribution over the classification of the image index c is

Original: [Equation]
Revised: [Equation]

Original: Therefore, the multi-class cross-entropy over the correct index is
Revised: Therefore, the multi-class cross-entropy over the correct index is

Original: [Equation]
Revised: [Equation]

Original: We can manipulate the third term into the form of an expectation
Revised: We can manipulate the third term into the form of an expectation

Original: [Equation]
Revised: [Equation]

Original: and for large B, we can approximate 1 B (cid:80)
Revised: and for large B, we can approximate 1 B (cid:80)

Original: b p ( x b | y ) p ( x b ) ≈ E p( x ) (cid:2) p ( x | y
Revised: b p ( x b | y ) p ( x b ) ≈ E p

Original: [Equation]
Revised: [Equation]

Original:  p ( x ) (cid:3) =
Revised: x ) (cid:2) p ( x | y ) p ( x ) (cid:3) =

Original: (cid:82) x p ( x | y )
Revised: (cid:82) x p ( x | y )

Original: = 1 .
Revised: = 1 .

Original: In this largebatch limit, the multi-class cross-entropy is proportional to the language-conditional image decoderlog-likelihood up to constant factors in y :
Revised: Therefore,in this large batch limit, the multi-class cross-entropy is proportional to the language-conditionalimage decoder log-likelihood up to constant factors in y :

Original: [Equation]
Revised: [Equation]

Revised: | y )

Original: ) + constant [Equation]
Revised: + constant ( y ) .

Original: Therefore, if we have a multi-class classifier that discriminates if a batch element y is paired to abatch element x , we can substitute thegenerative log likelihood with the classifier’s loss:{+loss in equation 
+}
Revised: Therefore, if we have a multi-class classifier that discriminates if a batch element y is paired to abatch element x , we can substitute the classifier’s {+loss in equation 
+}

Original: [Equation]
Revised: [Equation]

Original: and gradients with respect to q ω ( y | x j ) are the same in expectation.
Revised: and gradients with respect to q ω ( y | x j ) are the same in expectation.

Revised: A.2 Interface and instruction for caption data collection

Original: [Figure]
Revised: [Figure]

Original: You will be shown an image of a room, where one of the objects is highlighted with a dotted red rectangle .

Original: Follow the script, think of a description of the object and its surroundings and type it in the text field.

Original: The description should also have enough details such that another player in the room can easily find and understand which object you are referring to.

Original: During data collection, the caption interface displayed a single frame randomly selected from human-human interaction data as described by [22] at a resolution of 320 × 240 . An object from the image was randomly selected and highlighted by a bounding box. The participants were prompted to “[d]escribe the object in the red dotted rectangle and its location relative to other objects or landmarks”. After the participants input the caption in the text-box and clicked “submit”, the is refreshed and text-box is cleared for the of next
Revised: A.3 Additional samples from generative semi-supervised model

Original: Numbered in interface

Original: The prompt for for the participants to

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: A.4 Details about lift/ ask color tasks
Revised: A .4 Details about lift

Original: We modified the Playhouse environment [22] to create the lift drum and askcolor drum tasks. Thesetasks were used for evaluation only, and the agent was never trained in these tasks. For these tasks, We first initialize a randomized playhouse environment as described in [22], which represents arandomized multi-room environment.
Revised: / ask

Original: We then spawn a drum object in the room where the agentavatar is spawned. The color of the drum object is randomly selected from the following list of10 colors: “red”, “yellow”, “blue”, “white”, “green”, “pink”, “purple”, “orange”, “aquamarine”,“magenta”. Lift task. In the lift task, the instruction “ Lift the drum. ” appears after a random delay of upto 10 seconds.
Revised: color tasks

Original: A reward of 1 is given if the agent lifts any drum object, and the episode terminatesafter reward is emitted. If the agent lifts any other object, or times out after 2 minutes, the episodeterminates with a reward of 0 . Ask about color task. In the ask about color task, the instruction “ What is the color of thedrum? ” appears after a random delay of up to 10 seconds.
Revised: [Figure]

Original: If the agent emits language that matchesthe color of the drum, a reward of 1 is given, and the episode terminates. Otherwise if the agentoutputs any other text, or times out after 2 minutes with no language output, the episode terminateswith a reward of 0. For each agent, we averaged rewards collected from 1 , 000 episodes for each task. We also collectedhuman scores on these tasks, and used it to normalize the agent reward to a range of [0 , 1] .
Revised: A.

Original: Thehuman normalized score is reported in the manuscript.

Original: A.5 Resultson control objects
Revised: 5 Results

Revised: on control objects

Original: [Table]
Revised: [Figure]

Original: Lift Drum

Original: Teddy bear

Original: S585G +CM / G +D

Original: +M +CB

Original: S585G +CM / G +D

Original: +M +CB