Original: ℓ Gym: Natural Language Visual Reasoning with Reinforcement Learning
Revised: lil Gym: Natural Language Visual Reasoning with Reinforcement Learning

Original: Anonymous Author(s)

Original: Affiliation

Original: Address email

Original:  Introduction
Revised:  Introduction

Original: The inclusion of natural language context in reinforcement learning (RL) provides both opportunitiesand challenges.
Revised: Reinforcement learning (RL) with natural language context poses important opportunities and challenges.

Original: Language provides an expressive and accessible conduit for task specification, sothat RL agents can address a broad set of tasks in a given environment, rather than learn a singlebehavior.
Revised: Language provides an expressive and accessible conduit for task specification, so that RL agents can address a broad set of tasks, rather than learn a single behavior.

Original: For  language learning, RL is a promising avenue for language use and acquisition throughworld interaction.
Revised: For natural language processing (NLP), RL is a promising avenue for language use and acquisition with world interaction.

Original: Fulfilling this potential requires addressing core reasoning challenges: the RLagent must reason about both high-level language concepts, low-level actions, and the relationsbetween them.
Revised: Fulfilling this potential requires addressing core reasoning challenges: the RL agent must reason about both high-level language concepts, low-level actions, and the relations between them.

Original: Despite significant interest and promising approaches, it has been challenging to create expressive
Revised: Despite significant interest and promising approaches, it has been challenging to create expressive

Original: RL benchmarks with natural language.
Revised: RL benchmarks with natural language.

Original: Existing benchmarks often make various simplifications,such as using synthetic language that does not reflect the complexity of natural language [Shinagawaet al., 2018, El-Nouby et al.,2018, 2019] and using reward functions that do not accurately capturelanguage semantics [Misra et al., 2017]
Revised: Existing approaches often make various simplifications, such as using synthetic language [Côté et al., 2018, Co-Reyes et al., 2019] or heuristic approximation, for example via demonstration data [Misra et al., 2017]. While these approaches open new avenues for research, they either do not explore the full complexity of natural language or introduce unexpected artifacts into learning through meaning approximations.

Original: We present ℓ Gym,  a reinforcement learning benchmark for natural language visual reasoning. ℓ Gymis designed to address the above issues.
Revised: We present lil Gym, 1 a reinforcement learning benchmark for natural language visual reasoning that addresses the above issues.

Original: It includes the semantically diverse natural language of the 
Revised: It includes the semantically diverse natural language of the Natural

Original: NLVR corpus [Suhr et al., 2017], which has highly compositional human-written language andrequires complex grounded reasoning.
Revised: Language for Visual Reasoning (NLVR) corpus [Suhr et al., 2017], which has highly compositional human-written language and requires complex grounded reasoning.

Original: ℓ Gym provides an executable evaluation function for everystatement in the NLVR corpus, and these annotations align the reward function with the languagesemantics of the agent’s underlying reasoning task.
Revised: lil Gym provides an executable evaluation function for every statement in the NLVR corpus, and these annotations align the reward function with the language semantics of the agent’s underlying reasoning task.

Original: We experiment with standard on-policy RLalgorithms.
Revised: We experiment with standard on-policy RL algorithms.

Original: Our experimental results show that while existing methods are able to achieve non-trivialperformance, the complex visual reasoning required by ℓ Gym forms a challenging open problem.
Revised: Our experimental results show that while existing methods are able to achieve non-trivial performance, the complex visual reasoning required by lil Gym forms a challenging open problem.

Original:  The ℓ Gym Benchmark
Revised:  The lil Gym Benchmark

Original: ℓ Gym consists of a collection of environments that share a common backbone.
Revised: lil Gym consists of a collection of environments that share a common backbone.

Original: The backbone is a2D plane that is manipulated by placing and removing objects of different types.
Revised: The backbone is a 2D plane that is manipulated by placing and removing objects of different types.

Original: Each environmentinstance is a Markov Decision Process (MDP) created by pairing a natural language statement and atarget boolean value with a configuration of the shared backbone.
Revised: Each environment instance is a Markov Decision Process (MDP) created by pairing a natural language statement and a target boolean value with a configuration of the shared backbone.

Original: The goal of the agent in each environment is to manipulate it by adding and removing objects so that the truth-value of the statementwith regard to the environment is the target boolean.
Revised: The goal of the agent in each environment is to manipulate it by adding and removing objects so that the truth-value of the statement with regard to the environment is the target boolean.

Original: The learning problem ℓ Gym presents is to induce a policy that generalizes across MDPs.
Revised: The learning problem lil Gym presents is to induce a policy that generalizes across MDPs.

Original: We splitthe MDPs to training, development, and held-out testing sets.
Revised: We split the MDPs to training, development, and held-out testing sets.

Original: The training environments are to beused for parameter estimation, while the two other sets are for testing during development and forfinal held-out testing to report approach performance.
Revised: The training environments are to be used for parameter estimation, while the two other sets are for testing during development and for final held-out testing to report approach performance.

Revised: [Figure]

Original: There are two dimensions of configuration: appearance and starting condition.
Revised: There are two dimensions of configuration: appearance and starting condition.

Original: The appearancedetermines the state space, transition function, and action space.
Revised: The appearance determines the state space, transition function, and action space.

Original: The starting condition determines

Original: [Figure] the agent’s goal. The appearance of the environment can be (a) TOWER : the objects include squaresonly, and they can be stacked into towers  only; or SCATTER : objects of different types can be freelydistributed.
Revised: The appearance of the environment can be (a) TOWER : the objects include squares only, and they can be stacked into towers in specific positions only; or SCATTER : objects of different types can be freely distributed.

Revised: The starting condition determines the agent’s goal.

Original: The starting condition and agent objective can be: (a) SCRATCH :
Revised: The starting condition and agent objective can be: (a) SCRATCH :

Original: the environment startswithout any objects and the goal is to modify it so that the statement’s truth-value is True ; or (b) 
Revised: the environment starts without any objects and the goal is to modify it so that the statement’s truthvalue is True ; or (b) FLIPIT :

Original: FLIPIT : the environment starts with a set of objects and the agent’s goal is to flip the truth-valueof the statement.
Revised: the environment starts with a set of objects and the agent’s goal is to flip the truth-value of the statement.

Original: There are four configurations: TOWER-SCRATCH , TOWER-FLIPIT , SCATTER-SCRATCH , and
Revised: There are four configurations: TOWER-SCRATCH , TOWER-FLIPIT , SCATTER-SCRATCH , and

Original: SCATTER-FLIPIT .
Revised: SCATTER-FLIPIT .

Original: Each configuration forms a Contextual Markov Decision Process [CMDP;
Revised: Each configuration forms a Contextual Markov Decision Process [CMDP;

Original: Hallak et al., 2015].
Revised: Hallak et al., 2015].

Original: CMDP is an abstraction over a set of Markov Decision Processes (MDPs)to account for a context that remains constant throughout the interaction with an MDP.
Revised: CMDP is an abstraction over a set of MDPs to account for a context that remains constant throughout the interaction with an MDP.

Original: We set thecontext to include the statement and the target boolean the interaction is conditioned on.
Revised: We set the context to include the statement and the target boolean the interaction is conditioned on.

Original: A CMDP isa tuple ( C , S , A , M ( c )) , where C is the context space, S the state space,
Revised: A CMDP is a tuple ( C , S , A , M ( c )) , where C is the context space, S the state space,

Original: A the action space, and Ma function mapping a context c ∈ C to an MDP M ( c )
Revised: A the action space, and M a function mapping a context c ∈ C to an MDP M ( c )

Original: = ( S , A , T, R c , β c ) .
Revised: = ( S , A , T, R c , β c ) .

Original: Here, T : S × A → Sis a transition function, R c : S × A → R the reward function, and β c an initial state distribution.
Revised: Here, T : S ×A → S is a transition function, R c : S ×A → R the reward function, and β c an initial state distribution.

Original: This means that a CMDP is a collection of MDPs that share the same state space, action space andtransition function.
Revised: This means that a CMDP is a set of MDPs that share the same states and actions.

Original: Table 2 in Appendix A shows the number of MDPs under each configuration.
Revised: Table 2 in Appendix A shows the number of MDPs under each configuration.

Original: The policy takes as input both the current state and the context that created the MDP.
Revised: The policy takes as input both the current state and the context that created the MDP.

Original: The learningproblem is to estimate parameters θ for a policy π θ : S × C
Revised: The learning problem is to estimate parameters θ for a policy π θ : S × C

Original: → A .
Revised: → A .

Original: Contexts A context c ∈ C is a pair c = (¯ x, b ) , where ¯
Revised: Contexts A context c ∈ C is a pair c = (¯ x, b ) , where ¯

Original: x is a natural language statement andb ∈ { True , False } is a target boolean value for the statement ¯ x with respect to the state s .
Revised: x is a natural language statement and b ∈ { True , False } is a target boolean value for the statement ¯ x with respect to the state s .

Original: Thetarget boolean value in SCRATCH is always True .
Revised: The target boolean value in SCRATCH is always True .

Original: In FLIPIT , the target boolean value can eitherbe True or False .
Revised: In FLIPIT , the target boolean value can either be True or False .

Original: States A state s ∈ S is an RGB image.
Revised: States A state s ∈ S is an RGB image.

Original: Images in ℓ Gym are divided into three box regions ofidentical dimensions by two gray separators (Figure 1).
Revised: Images in lil Gym are divided into three box regions of identical dimensions by two gray separators (Figure 1).

Original: The objects in ℓ Gym have three properties,each can take multiple values: shape ( CIRCLE , SQUARE or TRIANGLE ), color ( BLACK , BLUE , or
Revised: The objects in lil Gym have three properties, each can take multiple values: shape ( CIRCLE , SQUARE or TRIANGLE ), color ( BLACK , BLUE , or

Original: YELLOW ), and size ( SMALL , MEDIUM or LARGE ).
Revised: YELLOW ), and size ( SMALL , MEDIUM or LARGE ).

Original: In TOWER , states are constrained to have stacks ofup to four SQUARE s of MEDIUM size and any color at the center of each box.
Revised: In TOWER , states are constrained to have stacks of up to four SQUARE s of MEDIUM size and any color at the center of each box.

Original: SCATTER states supportall object shapes, sizes, and colors, and they may be positioned freely.
Revised: SCATTER states support all object shapes, sizes, and colors, and they may be positioned freely.

Original: In both conditions, objectscannot cross image boundaries or into the separators.
Revised: In both conditions, objects cannot cross image boundaries or into the separators.

Original: The choice between SCRATCH or FLIPITdoes not influence the state space.
Revised: The choice between SCRATCH or FLIPIT does not influence the state space.

Original: Actions and Transitions There are three action types STOP , ADD , and REMOVE .
Revised: Actions and Transitions There are three action types STOP , ADD , and REMOVE .

Original: STOP terminatesthe episode and does not require any parameters.
Revised: STOP terminates the episode and does not require any parameters.

Original: The truth-value of the statement is only evaluatedand compared to the target boolean after the STOP action is taken.
Revised: The truth-value of the statement is only evaluated and compared to the target boolean after the STOP action is taken.

Original: ADD adds objects to the environment, and REMOVE removes objects.
Revised: ADD adds objects to the environment, and REMOVE removes objects.

Original: They take different arguments for TOWER and SCATTER :
Revised: They take different arguments for TOWER and SCATTER :

Original: TOWER :
Revised: TOWER :

Original: Similar to the state space of TOWER , the actions are also more constrained.
Revised: Similar to the state space of TOWER , the actions are also more constrained.

Original: Both ADDand REMOVE take a position argument, which has three possible values correspondingto the three box regions.
Revised: Both ADD and REMOVE take a position argument, which has three possible values corresponding to the three box regions.

Original: Objects are always added or removed at the top of the stack.
Revised: Objects are always added or removed at the top of the stack.

Original: Adding an object on top of a stack of four objects or removing an object from an emptybox are both invalid actions.
Revised: Adding an object on top of a stack of four objects or removing an object from an empty box are both invalid actions.

Original: ADD also takes a color argument.
Revised: ADD also takes a color argument.

Original: Including STOP , there are1 + (3 + 1) × 3 = 13 actions.
Revised: Including STOP , there are 1 + (3 + 1) × 3 = 13 actions.

Original: SCATTER :
Revised: SCATTER :

Original: Unlike TOWER , objects of any type can be placed freely in the box regions.
Revised: Unlike TOWER , objects of any type can be placed freely in the box regions.

Original: Both ADD and
Revised: Both ADD and

Original: REMOVE take 2D coordinates that specify the pixel location.
Revised: REMOVE take 2D coordinates that specify the pixel location.

Original: Adding an object places it so that its top-left coordinates is the given coordinates.
Revised: Adding an object places it so that its top-left coordinates are the given coordinates.

Original: Removing an object will removethe object at the given coordinates.
Revised: Removing an object will remove the object at the given coordinates.

Original: Adding also requires specifying the shape, color,and size.
Revised: Adding also requires specifying the shape, color, and size.

Original: If adding results in objects overlap or boundary crossing with the separators orimage boundaries, the action is invalid.
Revised: If adding results in objects overlap or boundary crossing with the separators or image boundaries, the action is invalid.

Original: Removing from a position that does not includean object is also an invalid action.
Revised: Removing from a position that does not include an object is also an invalid action.

Original: The native resolution of images is 380 × 100 pixels.
Revised: The native resolution of images is 380 × 100 pixels.

Original: Including STOP , there are 1+(380 × 100) ×
Revised: Including STOP , there are 1+(380 × 100) ×

Original: ((3 × 3 × 3)+1)
Revised: ((3 × 3 × 3)+1)

Original: = 1 , 064 , 001 actions.
Revised: = 1 , 064 , 001 actions.

Original: In ourexperiments (Section 4), we use a grid of 19 × 5, giving a total number of 2,661 actions.
Revised: In our experiments (Section 4), we use a grid of 19 × 5, giving a total number of 2,661 actions.

Original: The environment transitions are controlled by the transition function T : S ×
Revised: The environment transitions are controlled by the transition function T : S ×

Original: A → S .
Revised: A → S .

Original: Because thechoice between TOWER and SCATTER configurations determines the action space, it also determinesthe transition function.
Revised: T depends on the choice between TOWER and SCATTER , because this choice determines the action space.

Original: The transition function does not modify the context, which is fixed for a given MDP.
Revised: The transition function does not modify the context, which is fixed for a given MDP.

Original: Reward Function
Revised: Reward Function

Original: The reward function R c is computed with respect to the context pair c = (¯ x, b ) ,where ¯ x is a natural language statementand b is the target boolean value.
Revised: The reward function R c is computed with respect to the context pair c = (¯ x, b ) , and is based on evaluating the truth-value of the natural language statement

Original: The reward is basedon evaluating the truth-value of the natural language statement ¯ x with respect to a state s , andcomparing it to the target boolean b . ℓ Gym includes an executable evaluation function E ¯ x : S×A →{ True , False } for all ¯ x .
Revised: ¯ x with respect to a state s , and comparing it to the target boolean b . lil Gym includes an executable evaluation function E ¯ x : S × A → { True , False } for every statement ¯ x .

Original: We describe how we create these evaluation functions in Appendix A.
Revised: We describe how we create these evaluation functions in Section A.3.

Original: The agent receives a positive reward for terminating the episode using the STOP action with thestatement evaluation E ¯ x ( s ) equal to the target boolean value b .
Revised: The agent receives a positive reward for terminating the episode using the STOP action with the statement evaluation E ¯ x ( s ) equal to the target boolean value b .

Original: If the statement boolean value E ¯ x ( s )does not equal the target boolean b value when taking the STOP action, the agent receives a negativereward.
Revised: If the statement boolean value E ¯ x ( s ) does not equal the target boolean b value when taking the STOP action, the agent receives a negative reward.

Original: If the episode terminated because the current time step t reached the action horizon H orbecause of an invalid action, the agent also receives a negative reward.
Revised: If the episode terminates because the current time step t reached the action horizon H or because of an invalid action, the agent also receives a negative reward.

Original: Action validity depends onthe current state s and on the configuration, because TOWER and SCATTER have different actionspaces.
Revised: Action validity depends on the current state s and on the configuration, because TOWER and SCATTER have different action spaces.

Original: There is also a verbosity penalty of δ for every other action.
Revised: There is also a verbosity penalty of δ for every other action.

Original: Formally, the reward is:
Revised: Formally, the reward is:

Original: [Equation]
Revised: [Equation]

Original: ) =   1 .a =
Revised:   1 .a =

Original: STOP ∧
Revised: STOP ∧

Original: E ¯ x ( s ) =
Revised: E ¯ x ( s ) =

Original: b − 1 . 

Revised: b − 1 .

Original: a = STOP ∧
Revised: a = STOP ∧

Original: E ¯ x ( s ) ̸
Revised: E ¯ x ( s ) ̸

Original: =
Revised: =

Original: b − 1 . 

Revised: b − 1 .

Original: ( a is invalid in s ) ∨ ( t = H ) − δ otherwise .
Revised: ( a is invalid in s ) ∨ ( t = H ) − δ otherwise .

Original: Initial State Distribution
Revised: Initial State Distribution

Original: The initial state distribution β c is parameterized by the context c ∈ C ,which is different between SCRATCH and FLIPIT .
Revised: The initial state distribution β c is parameterized by the context c ∈ C , which is different between SCRATCH and FLIPIT .

Original: In SCRATCH , the agent modifies an emptyenvironment to satisfy the truth-condition of the statement ¯ x in the context c , so the initial states 0 is always an empty image.
Revised: In SCRATCH , the agent modifies an empty environment to satisfy the truth-condition of the statement ¯ x in the context c , so the initial state s 0 is always an empty image.

Original: The set of initial states β c for every context c ∈ C is the set ofimages associated with the statement ¯ x in the NLVR data.
Revised: The set of initial states β c for every context c ∈ C is the set of images associated with the statement ¯ x in the NLVR data.

Original: In practice, for FLIPIT , this set includesbetween 1 to 43 images.
Revised: In practice, for FLIPIT , this set includes between 1 to 43 images.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: We consider two RL algorithms paired with two models for a total of four algorithm-model pairs.
Revised: We consider two RL algorithms paired with two models for a total of four algorithm-model pairs.

Original: 3.1 Models
Revised: 3.1 Models

Original: In our experiments, we consider two models: CNN+BERT and ViLT.
Revised: In our experiments, we consider two models: C+BERT and ViLT.

Original: These two models learn a jointvisuo-linguistic representation, which is necessary to solve the proposed CMDP configurations.
Revised: These two models learn a joint visuo-linguistic representation, which is necessary to solve the proposed CMDP configurations.

Original: CNN+BERT. We combine a CNN network [Fukushima and Miyake, 1982, LeCun et al., 1989]with BERTtext features [Devlin et al., 2019] to learn a joint visuo-linguistic representation.
Revised: C+BERT We process the statement ¯ x using BERT [Devlin et al., 2019], and do mean pooling across all layers and tokens to get the statement representation.

Original: The
Revised: We use a three-layer

Original: BERT features are fixed and used without fine-tuning. To account for thecontext target boolean, wetrain two embeddings for FLIPIT , and combined it with the other two modalities representations. For SCRATCH , we ignore the context target boolean because it is always True (see Section 2).
Revised: CNN [Fukushima and Miyake, 1982] to embed the image of the current state s . We concatenate the statement representation, image representation, and an embedding for the target boolean b , and process the vector through a multi-layer perceptron (MLP) to compute the action distribution.

Original: ViLT. We also evaluate amodern pretrained  Transformer [Vaswani et al., 2017] model, Visionand-Language Transformer (ViLT) [Kim et al., 2021].
Revised: ViLT ViLT is a pretrained multi-modal Transformer that jointly processes text and image inputs [Kim et al., 2021].

Original: Unlike CNN+BERT, ViLT handles bothmodalities, vision and language, inside a single Transformer instead of separating both components. In preprocessing, we concatenate the sentence with thetext-format target boolean separated by aspecial token <TARGET> . ViLT’s pretrained
Revised: We create a sequence of tokens by concatenating the statement, a token for the target boolean, and image patches, separated by special tokens. The image patches are the same size as the 19 × 5 grid cells, including in TOWER , where the action space does not use a grid.

Original: [Table]
Revised: [Table]

Original: 3.2 Algorithms
Revised: 3.2 Algorithms

Original: We couple each model with two algorithms: PPO and PPO+SF. We choose the on-policy algorithm PPO [Schulman et al., 2017]  tooptimize each of the CMDP proposed in this benchmark because ofits empirical success with large models [Stiennon et al., 2020].
Revised: We use PPO [Schulman et al., 2017] for parameter estimation, 3 with a separate network as a critic. The critic network is identical to the policy, except that we add a tanh activation for the value output.

Original: In our CMDP configurations pairs,the reward is very sparse because the agent has to select the correct sequence ofactions and choosethe STOP action as the last action if statement evaluation E ¯ x ( s ) is equal to the target boolean valueb .
Revised: Because of the large action space, especially for SCATTER , the agent rarely observes positive reward, which requires taking a STOP action at an appropriate state. We design a simple variant of

Original: To combat this sparse reward issue, we maskout all actions exceptfor the STOP action whenreaching a goal state, and the agent needs to select the STOP action.
Revised: PPO called PPO+SF (PPO with stop forcing) to study this issue. PPO+SF is identical to PPO, except that during training, we mask all actions except STOP when the agent reaches a state where selecting

Original: This makes the reward functionless sparse and allows the PPO agent to gain some positive reward sooner.
Revised: STOP will give a positive reward. This modification is present only during training.

Original: We denote the policywith masked actions as PPO+Stop Forcing (PPO+SF).
Revised: All testing is done under the same conditions, without stop forcing.

Original:  Experiments
Revised:  Experiments

Original: For the experiments we focus on two questions.
Revised: For the experiments we focus on two questions.

Original: How do the algorithm-model pairs perform?
Revised: How do the algorithm-model pairs perform?

Original: For every CMDP, we report accuracy on the development and test sets for both PPO and PPO+SF and both CNN+BERT and ViLT.
Revised: For every CMDP, we report accuracy on the development and test sets for both PPO and PPO+SF and both C+BERT and ViLT.

Original: All modelsare randomly initialized.

Original: For every configuration, we randomly sample 10% of the training data asa held-out validation set kept unchanged throughout the experiments.
Revised: For every configuration, we randomly sample 10% of the training data as a held-out validation set kept unchanged throughout the experiments.

Original: We stop training using thisvalidation set with early stopping and select the model with the best validation accuracy.
Revised: We stop training using this validation set with early stopping and select the model with the best validation accuracy.

Original:  Table 1 shows the accuracy after training all algorithm-model pairs.
Revised: In Table 1, we observe that the additional guidance of PPO+SF compared to PPO helps with exploration, especially on SCATTER CMDPs. On SCATTER-FLIPIT , PPO+SF improves performance by 23.86% compared to PPO.

Original: We see that  ViLT is performingbetter than CNN+BERT on all configurations except for SCATTER-SCRATCH . ViLT is especiallyoutperforming CNN+BERT on the configurations using FLIPIT , for both TOWER and SCATTER ,showing that having models with more representation capacity is important.
Revised: This illustrates the hard exploration problem that SCATTER CMDPs pose. ViLT generally outperforms C+BERT, except on SCATTER-SCRATCH . This is relatively expected given the joint reasoning architecture and multi-modal pre-training of ViLT.

Original: What types of mistakes do PPO models make?
Revised: What types of mistakes do PPO models make?

Original: We sample 50 erroneous development examplesfrom configurations in the SCATTER-FLIPIT and SCATTER-SCRATCH CMDPs, and analyzetheir mistakes.
Revised: We sample 50 erroneous development examples from configurations in the SCATTER-FLIPIT and SCATTER-SCRATCH CMDPs, and analyze their mistakes.

Original: In SCATTER - SCRATCH trained with PPO, we found that for CNN+BERT, 76%of the mistakes are due to invalid actions, and 24% due to early termination.
Revised: In SCATTER - SCRATCH trained with PPO, we found that for C+BERT, 76% of the mistakes are due to invalid actions, and 24% due to early termination.

Original: Among the invalidactions, 58% are due to trying to put an item that cannot fit in the box, 24% are due to trying toperform an action on a separator, and 18% due to trying to remove an object from a position thatdoes not include an object.
Revised: Among the invalid actions, 58% are due to trying to put an item that cannot fit in the box, 24% are due to trying to perform an action on a separator, and 18% due to trying to remove an object from a position that does not include an object.

Original: For other configurations, the error causes are similar.
Revised: For other configurations, the error causes are similar.

Original:  Conclusion
Revised:  Conclusion

Original: We introduce the ℓ Gym reinforcement learning benchmark for natural language visual reasoning,based on a set of Python program annotations for thousands ofhuman-written natural language statements grounded in visual environments.
Revised: We introduce lil Gym, a reinforcement learning benchmark focused on natural language visual reasoning. lil Gym is designed to be accessible for RL researchers, while still displaying the reasoning richness of natural language.

Original: The statements present a high degree of compositionalityand semantic diversity, and require complex relational reasoning about displayed sets of objects.
Revised: Our data annotation approach allows including expressive and diverse natural language, while still providing accurate and automatic reward computation.

Original: The collection of environments in ℓ Gym are modular and present increasing levels of difficulty. Wewill release our annotated data and code.
Revised: Our strong baselines show that existing methods can achieve non-trivial performance on lil Gym, but there remain significant challenges to be solved and progress to be made.

Original: A The ℓ Gym Data
Revised: A The lil Gym Data

Original: The data used for ℓ Gym is based on the Natural Language  Visual Reasoning  corpus [NLVR; Suhret al., 2017].
Revised: The data used for lil Gym is based on the Natural Language for Visual Reasoning (NLVR) corpus [Suhr et al., 2017].

Original: The NLVR data was initially collected as a supervised learning benchmark.
Revised: The NLVR data was initially collected as a supervised learning benchmark.

Original: Weformalize an interactive task on top of the NLVR data and collect additional annotations for rewardcomputation.
Revised: We formalize an interactive task on top of the NLVR data and collect additional annotations for reward computation.

Original: A.1 The NLVR Corpus
Revised: A.1 Background: the NLVR Corpus

Original: NLVR includes human-written natural language statements paired with synthetic images.
Revised: NLVR includes human-written natural language statements paired with synthetic images.

Original: Each pairis annotated with the boolean truth-value of the statement with regard to the image (i.e., True if thestatement is true with regard to the image, or False otherwise).
Revised: Each pair is annotated with the boolean truth-value of the statement with regard to the image (i.e., True if the statement is true with regard to the image, or False otherwise).

Original: The images are designed to support complex reasoning, including about spatial and set relations.
Revised: The images are designed to support complex reasoning, including about spatial and set relations.

Original: The original learning task posedby NLVR is to classify statement-image pairs as True to indicate the statement is true with regardto the image, or False otherwise.
Revised: The original learning task posed by NLVR is to classify statement-image pairs as True to indicate the statement is true with regard to the image, or False otherwise.

Original: Various approaches were developed to address the NLVR challenge [Suhr et al., 2017, Tan and Bansal, 2018, Goldman et al., 2018, Pavez et al., 2018, Yao et al. ,2018, Hudson and Manning, 2018, Perez et al., 2018, Dasigi et al., 2019, Gupta et al., 2021], and aseparate version using photos was also released [Suhr et al., 2019].
Revised: Various approaches were developed to address the NLVR challenge [Suhr et al., 2017, Tan and Bansal, 2018, Goldman et al., 2018, Pavez et al., 2018, Yao et al., 2018, Hudson and Manning, 2018, Perez et al., 2018, Dasigi et al., Zheng et al., 2020, Gupta et al., 2021], and a separate version using photos was also released [Suhr et al., 2019].

Original: 1 NLVR is roughly balanced, andthe current state-of-the-art using the raw images is 69.7% accuracy [Tan and Bansal, 2018], leavingsignificant room for improvement and illustrating the reasoning challenges the NLVR data presents.
Revised: 5 NLVR is roughly balanced, and the current state-of-the-art using the raw images is 80.6% accuracy [Zheng et al., 2020], leaving significant room for improvement and illustrating the reasoning challenges the NLVR data presents.

Original: Qualitative analysis released with the data  shows a more diverse representation of semantic andcompositional phenomena compared to similar corpora (Table 2 in [Suhr et al., 2017]), includingrequiring joint visual-linguistic reasoning about spatial relations, quantities, and sets of objects.
Revised: Qualitative analysis of the data [Table 2 in Suhr et al., 2017] shows a more diverse representation of semantic and compositional phenomena compared to related corpora [Antol et al., 2015], including requiring joint visual-linguistic reasoning about spatial relations, quantities, and sets of objects.

Original: NLVR also provides an underlying structure representation for every image, which supports easymanipulation of images.
Revised: NLVR also provides an underlying structured representation for every image, which supports easy image manipulation.

Original: The combination of  simple interface for image manipulation with complexreasoning via natural language makes NLVR a strong candidate to support an interactive benchmarkenvironment.
Revised: The combination of a simple interface for image manipulation with complex reasoning via natural language makes NLVR ideal to support an interactive benchmark environment. A.2 Constructing lil Gym from NLVR

Revised: We use the NLVR data to create each of the CMDPs (Table 2).

Revised: SCRATCH CMDPs include contexts for all natural language statements from NLVR, each paired with the empty initial state containing no shapes.

Revised: FLIPIT CMDPs include the natural language statements with their corresponding images, both from NLVR (an example from SCATTER-FLIPIT in Figure 1).

Original: The original dataset is divided into four splits for training, development, public testing, and hiddentesting.
Revised: The images are used as initial states. The target boolean is set so that the initial state does not fulfil it. The split between TOWER and SCATTER also follows from NLVR. Statements corresponding to

Revised: TOWER images in NLVR are included in our TOWER CMDPs, and the same for SCATTER sentences.

Revised: There are two towers height not color.

Revised: [Figure]

Revised: There is a

Revised: 3 different colors and a touching with its top.

Revised: [Figure]

Revised: NLVR has four splits for training, development, public testing, and hidden testing.

Original: We follow the original splits for the training and development sets.
Revised: We follow the original splits for the training and development sets.

Original: Following the rest publicrelease of the hidden testing set, we merge the public and hidden testing sets into a single public testsplit.
Revised: Following the recent public release of the hidden testing set, we merge the public and hidden testing sets into a single public test split.

Original: A.2 Annotations for Automatic Evaluation
Revised: A.3 Annotations for Reward Computation

Original: The NLVR annotations include the truth-value of each statement with regard to the images pairedwith it in the data.
Revised: The NLVR annotations include the truth-value of each statement with regard to the images paired with it in the data.

Original: Once we manipulate the image (i.e., change the state in our interactive environment), the truth-value annotation does  hold.
Revised: Once we manipulate an image (i.e., change the state in our interactive environment), the truth-value annotation does not hold.

Original: A key challenge for creating an interactive environmentusing the NLVR data is the need for  accurate evaluation of the natural language statement for everypossible state (i.e., image), as required for reward computation (Section 2).
Revised: A key challenge for creating an interactive environment using the NLVR data is the need for an accurate evaluation of the natural language statement for every possible state (i.e., image), as required for reward computation (Section 2).

Original: We address this challenge by annotating all statements with  executable  Python programs representing their meaning,denoted as E ¯ x in Section 2.
Revised: We address this challenge by annotating each statement ¯ x with an executable boolean Python program representing its meaning, E ¯ x in Section 2.

Original: The Python programs operate on the underlying structured representation.
Revised: The Python programs operate on the underlying structured representation.

Original: Each program returns True for every image that satisfies the constraintsspecified in the corresponding statement, and False otherwise.
Revised: Each program returns True for every image that satisfies the constraints specified in the corresponding statement, and False otherwise.

Original: In general, there are many statesthat satisfy any given statement, many more than provided with the original NLVR images.
Revised: In general, there are many states that satisfy any given statement, many more than provided with the original NLVR images.

Original: The programs are written using an API defined over the structured representations.
Revised: The programs are written using an API defined over the structured representations.

Original: We base the APIdesign on the logical ontology designed for NLVR’s structured representations by [Goldman et al.,2017].
Revised: We base the API design on the logical ontology designed for NLVR’s structured representations by Goldman et al.

Revised: [2018], which we extend to include a total of 66 functions.

Original: Figure 2 shows two examples of logical forms paired with the corresponding text statementand an image. [Figure]
Revised: Figure 2 shows two examples of logical forms paired with their corresponding statements.

Original: We use the freelancing platform Upwork 2 for annotation.
Revised: We use the freelancing platform Upwork 6 for annotation.

Original: We recruit three annotators based on apreliminary screening of their fluency in English and competency in Python programming.
Revised: We recruit three annotators based on preliminary screening of their fluency in English and competency in Python.

Original: We deduplicate the naturally occurring sentences in the data, collect 2,666 annotations at a total cost of$3,756, and keep 2,661 valid annotations.
Revised: We de-duplicate the naturally occurring sentences in the data, collect 2,666 annotations at a total cost of $3,756, and keep 2,661 valid annotations.

Original: All the sentences in the dataset are randomly distributed to annotators with an example pair of imageand boolean.
Revised: All the sentences in the dataset are randomly distributed to the annotators, each with an example image.

Original: Every sentence is annotated with a logical form by one annotator.
Revised: Every sentence is annotated with a logical form by one annotator.

Original: Each logical formis evaluated against a corresponding hidden validation set, and it needs to pass all the tests.
Revised: Each logical form is evaluated against a corresponding hidden validation set, and must pass all the tests.

Original: B Related work
Revised: B Related Work

Original: Interactive grounded language learning Grounding language learning in an multimodal and interactive framework has been a challenging and long-standing goal [Anderson et al., 2018].
Revised: There is significant and increasing interest in RL conditioned on natural language. Various strategies are deployed to resolve language semantics for reward computation, mostly by strict control of the language or through approximations. Maybe the most common approach is to control the language by using synthetic language backed by a formal representation [Narasimhan et al., 2015, Johnson et al., 2017a,b, Côté et al., 2018,

Original: Thecompetition Kiseleva et al.
Revised: Chevalier-Boisvert et al., 2019, Co-Reyes et al., 2019, Jiang et al., 2020]. Although synthetic language allows studying the problem of learning high-level concepts, many of the complexities of natural language are stripped away, and such approaches run the risk of reducing the language learning challenge to reverse engineering the hand-crafted generation process.

Original: [2022] focused on building collaborative agents provided with naturallanguage instructions, and Shridhar et al. [2020] studied instruction following in an embodied environment. Nguyen et al. [2021] studies interactive learning for agents mapping language request toexecutions.
Revised: An alternative that allows for natural language while retaining the control of its semantics is to generate the target sequence of decisions (i.e., task demonstration), and solicit post-hoc instructional language [Shridhar et al., 2020, 2021, Hanjie et al., 2021].

Original: Our work focuses on both vision and language modalities, while focusing on learningto reason interactively based on visual-linguistic environments displaying complex relations. Reinforcement Learning and Imitation Learning with Language
Revised: While this process uses human-written language, it potentially implicitly retains the regularities of the demonstration generation procedure. Others have carefully designed the underlying environment to simplify termination state evaluation given demonstrations, for example, with a sparse graph-based structure [Anderson et al., 2018, Chen et al., 2019, Ku et al., 2020].

Original: Recent work  in Reinforcement Learning (RL) and Imitation Learning investigated a range of natural language-related tasks[Chevalier-Boisvert et al., 2018, Jiang et al., 2020], such as visual question answering (VQA) [Hudson and Manning, 2018], image captioning [Ren et al., 2017], semantic parsing [Liang et al.
Revised: However, recent work shows the potential for evaluation fidelity issues even in these settings [Jain et al., 2019]. In contrast to previous approaches, we emphasize using human-written natural language and avoid constraining the task to simplify reward evaluation.

Original: , 2016],vision-language navigation [Nguyen et al., 2019b, Blukis et al., 2018], text games [Narasimhanet al., 2015], instruction following [Blukis et al., 2021, Nguyen et al., 2019a]. Several work focusedon using language to represent instructions or to condition the goal [Branavan et al., 2009, Bahdanauet al., 2018, Colas et al., 2020]. There has also been a trend of reward modeling in language task[Bahdanau et al., 2018, Leike et al., 2018]. Language is also used to improve generalization [Hanjieet al., 2021].
Revised: We also opt to not use underlying hand-crafted procedures as stimuli for the writing.

Original: Ramamurthy et al. [2020] is an unimodal framework for sequence tagging, multi-labelclassification and question answering. In Jiang et al.
Revised: lil Gym prioritizes exact reward computation rather than automated approximations to allow for relatively clean benchmarking of learning methods.

Original: [2019], language  is used to study compositionaltask learning. By nature, the compositional nature of language enables the study at different levelsof abstraction and combinatorial generalization.
Revised: Our annotation of natural language statements with programs is inspired by the annotation of data for supervised learning of semantic parsers [Zelle and Mooney, 1996, Zettlemoyer and Collins, 2005,

Original: Visual-linguistic reasoning Grounded reasoning has been studied in a visual-linguistic context ina range of recent work, using synthetic images [Suhr et al., 2017, Johnson et al., 2017, Yang et al.
Revised: Suhr et al., 2018]. The Python API of our environments is based on the semantic parsing work of

Original: ,2018] or real images [Suhr et al., 2018, Liu et al., 2021a] with human annotations, or real imageswith synthetic language [Hudson and Manning, 2019]. More recently, there have been work focusingon investigating the compositionality of visually grounded natural language, either by generating andediting synthetic images [Liu et al., 2021b], or instead generating question-answer pairs about realimages [Bogin et al., 2021]. There has been work investigating linguistic issues about current VQAsystems [Bernardi and Pezzelle, 2021]. Current visual-linguistic systems still present a range ofchallenges, for instance reasoning at different abstraction levels, credit assignment [Minsky, 1961],dealing with language ambiguity, negation, etc.
Revised: Goldman et al.

Original: Our work incorporate these ideas and requiresreasoning about the alignments of inputs parts across modalities.
Revised: Robust semantic parsers can assist in automating our annotation process.