lil G ym: Natural Language Visual Reasoning with Reinforcement Learning (2023)

Abstract: We present lilGym, a new benchmark for language-conditioned reinforcement learning in visual environments. lilGym is based on 2,661 highly-compositional human-written natural language statements grounded in an interactive visual environment. We introduce a new approach for exact reward computation in every possible world state by annotating all statements with executable Python programs. Each statement is paired with multiple start states and reward functions to form thousands of distinct Markov Decision Processes of varying difficulty. We experiment with lilGym with different models and learning regimes. Our results and analysis show that while existing methods are able to achieve non-trivial performance, lilGym forms a challenging open problem. lilGym is available at https://lil.nlp.cornell.edu/lilgym/ .

## introduction the ability to reason about natural language has the potential to transform how reinforcement learning (rl) is used to train grounded agents. language provides an expressive and accessible conduit for task specification, enabling agents to address a broad set of tasks, rather than to learn singlebehavior policies. at the same time, rl is a promising framework to study core grounded language learning problems within interactive environments. prerequisite to realizing this potential are expressive benchmark environments, as has been instrumental for progress in rl more broadly. however, natural language poses unique challenges to building such benchmarks. beyond the design of the environment itself, which must support rich linguistic reasoning, accurate reward computation requires resolving language semantics. existing approaches adopt different strategies to address this issue, most often by using synthetic language (e.g., @xcite , or by removing the there are two boxes which has the same number of objects and three kinds of colors. figure 1: an illustrative example of lilgym. given a natural language statement with a target truth value, true in this example (omitted from the figure), and an initial image, the agent needs to manipulate objects in the environment so that the truth value of the statement with respect to the image is the target truth value. a key challenge in constructing lilgym is accurate reward computation. because of the flexibility of the environments and language, there are many possible equally correct termination states (i.e., goals) for each mdp. correct reward computation at every possible state requires taking into account the semantic nuances of the highly-compositional language. we address this by annotating all statements with executable python programs representing their meaning, effectively creating a supervised semantic parsing corpus (e.g., @xcite . the executable programs allow for exact and reliable reward computation at every possible state. our experiments with lilgym show that existing models demonstrate non-trivial performance given sufficient training time, with multi-modal pre-training providing further benefit. however, there remains significant room for improvement. for example, on the simplest configuration, our agent can solve 76.23% of the test environments, but performance drops significantly to only 16.81% on the most complex configuration. our experiments also confirm the importance of modeling language meaning for reward computation in learning. the lilgym benchmark and trained models are available under the mit license at https: //lil.nlp.cornell.edu/lilgym/ . ## related work rl research has benefited greatly from benchmarks such as atari @xcite and mu-joco @xcite . although both are synthetic with limited potential to train policies directly applicable to realistic environments, their accessibility and focus on core rl problems have made them impactful drivers of algorithm development. for example, the first demonstration of an effective neural network policy @xcite and the development of proximal policy optimization (ppo; @xcite both used these benchmarks, and both were later shown to generalize to more complex scenarios. lilgym is inspired by these benchmarks. rather than aiming for training models that transfer to realistic domains, it aims to enable fundamental algorithmic development by emphasizing semantic diversity and compositional-ity within an accessible benchmark. there is significant interest in rl for training policies conditioned on natural language. this includes multiple efforts at developing rl environments with language-informed tasks, mainly via grounded language learning in 2d or 3d environments or using text games. oftentimes, using synthetic language @xcite @xcite @xcite @xcite @xcite . although synthetic language allows studying the problem of learning high-level concepts, many of the complexities of natural language may be stripped away, and such approaches run the risk of reducing the language learning challenge to reverse engineering the hand-crafted generation process. in contrast, lilgym is based on semantically-diverse human-written language grounded in a visual environment, and requires both the ability of reasoning over highly-compositional language including sets and spatial relations, and precise alignment between statements and states. another approach is to simplify the task so only a few annotated termination states or trajectories are correct @xcite @xcite . this forgoes much of the abstractive potential of natural language, where it can succinctly define extremely large set of states. thereby reducing the utility of language, and not exposing learning algorithms to some of the core challenges it introduces to the rl problem. lilgym does not adopt such simplifications. our experiments show the importance of considering language meaning for reward computation when there is a large set of valid goal states. alternatively, other benchmarks are created by first generating the target sequence of decisions (i.e., task demonstration), and then soliciting posthoc instructional language @xcite @xcite . this process uses human-written language, but retains the regularities of the demonstration generation procedure. lilgym uses language from nlvr, which was crowdsourced via a contrastive task that was shown to elicit high semantic diversity. pervised learning benchmark. we formalize an interactive task using the nlvr data and collect additional annotations for reward computation. nlvr includes human-written natural language statements paired with synthetic images. each pair is annotated with the boolean truth-value of the statement with regard to the image (i.e., true if the statement is true with regard to the image, or false otherwise). the images are designed to support complex reasoning, including about spatial and set relations. the original learning task posed by nlvr is to classify statement-image pairs as true to indicate the statement is true with regard to the image, or false otherwise. nlvr has been studied extensively @xcite @xcite @xcite @xcite @xcite , and a separate version using photos was also released @xcite . 1 qualitative analysis of the nlvr data ( the learning problem lilgym presents is to induce a policy that generalizes across mdps. we split the mdps to training, development, and heldout testing sets. the training environments are for parameter estimation, while the two other sets are 1 we do not use the photographic nlvr2 in this work. for testing during development and for final heldout testing to report approach performance. 2 there are two dimensions of configuration: appearance and starting condition. the appearance determines the state space, transition function, and action space. the appearance of the environment can be (a) tower: the objects include squares only, and they can be stacked into towers in specific positions only; or (b) scatter: objects of different types can be freely distributed. the two leftmost examples in figure 2 are from tower, and the two rightmost are from scatter. tower gives a more constrained problem with much smaller state and action spaces compared to scatter. there are two starting conditions, which also determine the agent's goal: (a) scratch: the environment starts without any objects and the goal is to modify it so that the statement's truth-value is true; or (b) flipit: the environment starts with a set of objects and the agent's goal is to flip the truth-value of the statement, by modifying the environment. the first row of images in figure 2 shows start states in both conditions. scratch generally only requires adding objects, except in cases of correcting for agent's errors, while flipit requires both adding and removing, because there are already objects present. the four configurations are tower-scratch, tower-flipit, scatter-scratch, and scatter-flipit. in our experiments (section 6), we observe the different configurations provide different levels of difficulty. for example, scatter configurations are generally harder than tower, due to the larger state and action spaces. formally, each configuration is a contextual markov decision process (cmdp; @xcite . cmdp is an abstraction over a set of mdps to account for a context that remains constant throughout an interaction with an mdp. the context includes the statement and the target boolean the interaction is conditioned on. a cmdp is a tuple (c, s, a, m(c)), where c is the context space, s the state space, a the action space, and m a function mapping a context @xmath0, a, t, r c , β c ). here, t : s × @xmath1, r c : s × @xmath2, and β c an initial state distribution. this means that a cmdp is a set of mdps that share the same states and actions. the policy learning one tower has exactly 1 black block and 1 yellow block starting label: false target label: true s0 there is no black block as the top of a tower with at most three blocks. starting label: false target label: true there is a box with 2 triangles of same color nearly touching each other. starting label: false target label: true none of the yellow triangles are touching the edge starting label: true target label: false c tower-scratch tower-flipit scatter-scratch scatter-flipit s1 a0 ∼ π(s0, c) r0 ∼ r c (s0, a0) @xmath3, yellow) @xmath4, y0, triangle, yellow, large) @xmath5, y0, triangle, yellow, medium) s2 a1 ∼ π(s1, c) r1 ∼ r c (s1, a1) @xmath6, black) @xmath7, y1, triangle, yellow, small) @xmath8, c) r2 ∼ r c (s2, a2) @xmath9: s × @xmath10, which maps the current state and the context underlying the mdp to an action. the policy must generalize across different contexts from c. figure 2 shows example action trajectories in mdps for each of the four cmdps. states s a state @xmath13. images in lilgym are divided into three box regions of identical dimensions by two dark gray separators (figure 2 ). the objects in lilgym have three properties, each can take multiple values: shape (circle, square, or triangle), color (black, blue, or yellow), and size (small, medium, or large). in tower, states are constrained to have stacks of up to four squares of medium size and any color at the center of each box. scatter states support all object shapes, sizes, and colors, and they may be positioned freely. in both conditions, objects cannot cross image boundaries or into the separators. the choice of starting condition between scratch or flipit does not influence the state space. actions a and transitions t there are three action types stop, add, and remove. stop terminates the episode. the truth-value of the statement is only evaluated and compared to the target boolean after the stop action is taken. add adds objects to the environment, and remove removes objects. add and remove take arguments that differ between tower and scatter: tower: both add and remove take a position argument, which has three possible values corresponding to the three box regions. objects are added or removed at the top of the stack. adding an object on top of a stack of four objects or removing an object from an empty box are both invalid actions. add also takes a color argument. for example, the first action on the left trajectory in figure 2 is adding a yellow square in an empty box. including stop, there are 1 + (3 + 1) × 3 = 13 actions. scatter: unlike tower, objects of any type can be placed freely in the box regions. both add and remove take 2d coordinates that specify pixel location. adding an object places it so that its top-left coordinates are the given coordinates. removing an object will remove the object at the given coordinates. adding also requires specifying the shape, color, and size. the action is invalid if adding results in objects' overlap or boundary crossing with the separators or image boundaries. removing from a position that does not include an object is also an invalid action. the native resolution of images in lilgym is 380×100 pixels. including stop, there are 1 + (380 × 100) × ((3 × 3 × 3) + 1) = 1,064,001 actions. because of the extremely large action space, lilgym also supports a simplification through a coarser grid system for scatter that is automatically mapped to the original resolution (appendix a). the grid simplification includes heuristics that assist in identifying locations to place objects in the original pixel space or objects to remove once a grid cell is selected. in our experiments (section 6), we use a grid simplification of 19×5, giving a total of 2,661 actions. the difficulty of scatter can be adjusted by modifying the grid size, or acting at the original resolution. the transition function t : s × @xmath14, because this choice determines the action space. similar to the action space, the transitions in tower are more constrained compared to scatter. the transition function does not modify the context, which is fixed for a given mdp. ## the lilgym data and annotation we use the nlvr data to create each of the cmdps ( flipit cmdps include the natural language statements with their corresponding images, both from nlvr. the images are used as initial states. the target boolean is set so that the initial state does not fulfil it. the split between tower and scatter also follows from nlvr. statements corresponding to tower images in nlvr are included in our tower cmdps, and the same for scatter sentences. nlvr has four splits for training, development, public testing, and hidden testing. we adopt the original training and development sets splits. following the recent public release of the hidden testing set, we merge the public and hidden testing sets into a single public test split. the nlvr annotations include the truth-value of each statement with regard to the images paired with it in the data. once we manipulate an image (i.e., change the state in our environment), the truthvalue annotation does not necessarily hold. a key challenge for creating an interactive environment using this data is an accurate evaluation of the natural language statement for every possible state (i.e., image) for reward computation (section 4). we address this by annotating each statement x with an executable boolean python program representing its meaning, e x in section 4. this process is inspired by data annotation for supervised semantic parsing @xcite @xcite , where sentences are annotated with formal meaning representations. the python programs operate on the underlying structured representation. each program returns true for every image that satisfies the constraints specified in the corresponding statement, and false otherwise. in general, there are many states that satisfy any given statement, many more than provided with the original nlvr images. the programs are written using an api defined over the structured representations. we base the api design on the ontology designed for nlvr's structured representations by @xcite , which we extend to include 66 functions. figure 7 in appendix b shows two example programs with their corresponding statements. we use the freelancing platform upwork foot_3 for annotation. we recruit three programmers based on preliminary screening of their fluency in english and competency in python. we de-duplicate the naturally occurring sentences in the data, and distribute sentences to annotators randomly, each with a single example nlvr image. each program is evaluated against a corresponding hidden validation set made of all remaining nlvr images paired with the sentence, and must pass all the tests. appendix b provides a screenshot of the interface and more details. we collect 2,666 annotations at a total cost of $3,756, and keep 2,661 valid annotations. ## experiments 6.1 methods we experiment with each of the four cmdps separately, training on the training split and testing on the development and test splits. we sample a validation set from the training split for model selection. for scatter we use a simplified grid action space of 19×5 (section 4). each grid cell is 20×20 pixels. we set the action horizon @xmath15. appendix c provides implementation details. we use ppo @xcite for parameter estimation, 5 with a separate network as a critic. the critic network is identical to the policy, except that we add a tanh activation for the value output. because of the large action space, especially for scatter, the agent rarely observes positive reward, which requires taking a stop action at an appropriate state. we design a simple variant of ppo called ppo+sf (ppo with stop forcing) to study this issue. ppo+sf is identical to ppo, except that during training, we mask all actions except stop when the agent reaches a state where selecting stop will give a positive reward. this modification is present only during training. all testing is done under the same conditions, without stop forcing. we also study the importance of our annotation for reward computation using a reward function for scratch that does not require any annotation beyond what is already in nlvr. the reward uses nlvr images that are associated with each statement (between 1-43) and are labeled with the target boolean. 6 instead of testing the state using a program, it compares the state to the available nlvr images, and only if it equals one of them, the learner receives a positive task completion reward. we experiment with three models: c3+bert, c10+bert and vilt. 7 in c3+bert and c10+bert, we process the statement x using bert @xcite , and do mean pooling across all layers and tokens to get the statement representation. we use a three-layer cnn @xcite in c3+bert to embed the image of the current state s, and a ten-layer cnn in c10+bert. we concatenate the statement and image representations with an embedding for the target boolean b, and use a multi-layer perceptron to compute the action distribution. vilt is a pretrained multi-modal transformer that jointly processes text and image inputs @xcite . we create a sequence of tokens by concatenating the statement, a token for the target boolean, and image patches, separated by special tokens. the image patches are the same size as the 19×5 grid cells, including in tower, where the action space does not use a grid. ## conclusion we introduce lilgym, an rl benchmark that focuses on natural language visual reasoning. lilgym is designed to be accessible for researchers, while 0 50 100 non-stopped rollouts (%) 0 50 100