Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning (2021)

Abstract: Mapping natural language instructions to programs that computers can process is a fundamental challenge. Existing approaches focus on likelihood-based training or using reinforcement learning to fine-tune models based on a single reward. In this paper, we pose program generation from language as Inverse Reinforcement Learning. We introduce several interpretable reward components and jointly learn (1) a reward function that linearly combines them, and (2) a policy for program generation. Fine-tuning with our approach achieves significantly better performance than competitive methods using Reinforcement Learning (RL). On the VirtualHome framework, we get improvements of up to 9.0% on the Longest Common Subsequence metric and 14.7% on recall-based metrics over previous work on this framework (Puig et al., 2018). The approach is data-efficient, showing larger gains in performance in the low-data regime. Generated programs are also preferred by human evaluators over an RL-based approach, and rated higher on relevance, completeness, and human-likeness.

## introduction mapping natural language descriptions to programs is vital for developing agents that can mimic human behavior in the real world. for example, imagine a robot that is instructed to watch television (figure 1 ). the robot needs to parse the language instruction into a sequence of commands for navigating to the living room, then going to the sofa, turning on television, etc. for parsing, the robot needs to map objects mentioned in the instruction to objects in the surrounding environment, and then perform the actions required to complete the task. parsing language into actions has been widely explored in diverse settings @xcite @xcite . most recent approaches leverage supervised learning with maximum likelihood estimation, followed environment walk to living room, switch on television, sit on sofa and watch. nl task description: [walk] <@xmath0: [walk] <@xmath1: reward components: train irrelevance recall repetition longest common subsequence executability model: figure 1 : we frame conversion of a natural language task description into a program that can be executed in an environment as inverse reinforcement learning. we design multiple interpretable reward components encoding preferred characteristics in generated programs. a reward function (optimally combining these components) and a policy for program generation are jointly learnt from expert demonstrations/programs. by a fine-tuning phase of reinforcement learning with a single user-specified reward, encoding signals for things like task completion and executability @xcite . to alleviate the problem of sparse rewards, reward shaping terms are often used to encode aspects like distance from goal state or deviation from labeled trajectories @xcite . reward signals and shaping terms are combined using hyperparameters which need to be manually tuned, and this process becomes tedious. thus, these approaches can't efficiently leverage multiple possible sources of supervision that might be available. this paper explores an approach to alleviate this problem. figure 1 illustrates the core idea of this paper. we use multiple reward components encoding various desired characteristics of a good program to drive program generation, given a natural language task description and an environment. each compo-nent consists of a manually defined reward (such as irrelevance, recall, and repetition) with clear semantics. the example in the figure illustrates the need for multiple reward components. we note that the predicted program mentions all the commands in the same sequence as in the expert program. however, it also generates irrelevant commands and repeats steps. thus even though it scores optimally in the longest common subsequence metric which has been used as a reward for this task @xcite , we need other signals to improve program generation. we investigate subsuming such reward components in an irl framework to jointly learn: (1) a composite reward function combining the reward components with optimal weights, and (2) a policy that closely mimics the expert. the only supervision involved consists of task descriptions paired with labeled (expert) programs. automatically learning the weights of reward components becomes increasingly vital as their number increases. in terms of testbeds, much of previous work on instruction following has focused on block worlds @xcite and navigation tasks @xcite @xcite . we focus on the virtu-alhome environment @xcite . in contrast with previous datasets, it contains stepwise instructions and programs for a large number of realistic household activities, such as making coffee or folding laundry. steps in the programs often involving interacting with objects, or changing the state of the environment. thus, it offers a rich set of realistic challenges including object interactions and commonsense reasoning for program generation. our evaluation shows that using irl to fine-tune a model leads to significant gains over reinforcement learning. in fact, irl leads to improved performance even on some metrics that a reinforcement learning policy directly optimizes. additionally, our approach is data efficient. experimenting with different dataset sizes reveals that the method generalizes better than baseline methods in lowdata scenarios. more significantly, the approach can extend to other domains and provides a general framework for incorporating multiple sources of supervision or inductive biases about a task. our contributions are: ## method each example in our data consists of a natural language description of a task to be performed in an environment, along with its corresponding program (see figure 1 ). in virtualhome, a program is a sequence of commands that can be executed to complete the task being described. a command is an action-object-object triplet, where one or both of the object arguments can be empty based on the action. for example, in the command "[walk] living_room " the action "walk" is followed by an object argument "living_room". similarly the command "[putback] plate cupboard " has two object arguments and "sleep" has none. let @xmath2; 1 ≤ @xmath3, where d i denotes the i th token. let @xmath4; 1 ≤ @xmath5, where c j denotes j th command. our objective is to predict program c given the text description d and an environment e. previous approaches performed likelihood-based pre-training followed by fine-tuning using reinforcement learning @xcite . instead, we use irl to learn the reward function from the expert programs and fine-tune the program generation policy simultaneously. we design a set of rewards with clear semantics that encode the desired characteristics of a good program in our context. in the rest of this section, we describe the model architecture, provide definitions of reward components, and finally describe model training using irl. ## experiments and results in this section, we describe the dataset, metrics used for quantitative evaluation followed by a comparative evaluation of the proposed method with baselines and results of human evaluation foot_1 . ## conclusion we explored an approach for incorporating diverse reward components in instruction following tasks. such components can often be defined by a domain expert, and encode inductive biases about a problem. since reward weights are learned, these models can be robust to spurious reward components. however, the issue of possibly adversarial reward components remains to be explored. while the approach requires access to expert examples, since we focus on scenarios involving rl-based fine-tuning, these are presumed to be already available. the approach can potentially generalize to other domains and applications, and can be fertile ground for directions of future research. hyper-parameter tuning we use five different random seeds for five trials of each experiment: 42, 101, 123, 2020 and 2021. batch size and learning rate are manually tuned in the range {64,128, 256} and {0.001, 0.0005, 0.0001} respectively. based on the results on validation splits we chose batch size as 256 and learning rate as 0.001 to report the results. we use a weight of 0.05 for entropy regularization during policy gradient after trying out weights in the range {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 10 }. we chose 0.05 as weight for entropy regularization on the basis of model performance on validation set which peaks for 0.05 when the model is trained on 70 samples. for adam optimizer the value of coefficients used for computing running averages of gradient and its square are 0.9, 0.999 (default values as per pytorch) respectively. we do not use weight decay during optimization. all the models are trained for 400 epochs. for max-ent irl @xcite , we sample a subset of the training set for reward approximation stage (learning reward component weights). when we have a training set size of 70, we sample 5 expert and generated programs for this stage. similarly we sample 50 for training set size of 221 and 150 for training set of 697 examples. software and hardware specifications all the models are coded using pytorch 1.4.0 foot_2 @xcite and related libraries like numpy @xcite , scipy @xcite etc. the graphical simulator for virtualhome @xcite used in the paper is publicly available foot_3 . we run all experiments on geforce rtx 2080 gpu of size 12 gb. the system has 256 gb ram and 40 cpu cores. the inference process is run in parallel on all the cores. for irl fine-tuning on training set of size 70 it takes around 80 minutes for 400 epochs, which increases to 10 hours when fine-tuning on the training set of 697 samples. doing rl fine-tuning also takes similar amount of time since the time required for just reward weight approximation is quite less.