LETI : Learning to Generate from Textual Interactions (2024)

Abstract: Fine-tuning pre-trained language models (LMs) is essential for enhancing their capabilities.Existing techniques commonly fine-tune on input-output pairs (e.g., instruction tuning) or with numerical rewards that gauge the output quality (e.g., RLHF). We explore LMs’ potential to **le**arn from **t**extual **i**nteractions (**LETI**) that not only check their correctness with *binary labels* but also pinpoint and explain errors in their outputs through *textual feedback*.Our focus is the code generation task, where the model produces code based on natural language instructions. This setting invites a natural and scalable way to acquire textual feedback: the error messages and stack traces from code execution using a Python interpreter. LETI iteratively fine-tunes the model, using the LM objective, on a concatenation of natural language instructions, LM-generated programs, and textual feedback. Prepended to this fine-tuning text, a binary reward token is used to differentiate correct and buggy solutions.LETI requires *no* ground-truth outputs for training and even outperforms a fine-tuned baseline that does. LETI not only improves the performance of LMs on a code generation dataset MBPP, but also generalizes to other datasets. Trained on MBPP, it achieves comparable or better performance than the base LMs on unseen problems in HumanEval. Furthermore, compared to binary feedback, we observe that textual feedback leads to improved generation quality and sample efficiency, achieving the same performance with fewer than half of the gradient steps.LETI is equally applicable in natural language tasks when they can be formulated as code generation, which we empirically verified on event argument extraction.

## introduction large-scale language models have fundamentally shifted the paradigms of natural language process-ing (nlp). based on lms pre-trained on raw text, subsequent fine-tuning stages have proven crucial to enhance their capabilities in solving benchmark nlp tasks and generating texts that align with human preferences. success has been achieved by fine-tuning with direct training signals that measure whether the model, e.g., classifies the input into the right category @xcite , answers a question correctly @xcite , summarizes documents well @xcite , and generates outputs that align with human preferences @xcite . we hypothesize that lms can harness the much richer training signals from textual interactions with the environment (e.g., a human or a python interpreter) that not only check the correctness of lm's outputs but also pinpoint the errors and explain why. we propose leti, a new lm fine-tuning paradigm that aims to explore lms' potential to learn from nuanced textual interactions. we evaluate leti on code generation tasks, where the lm is supposed to generate code pieces to solve tasks described in natural language. this setting invites a natural and scalable way to acquire automatic interactive textual feedback: the stack traces and error message outputs by established programming language (pl) tools such as a python interpreter. leti's improvement process naturally mirrors a typical software development cycle: a human developer writes an initial program, executes it, and improves the program based on feedback obtained from the programming environment until a satisfying solution is found (e.g., successfully executed with no error); furthermore, the human developer learns from mistakes in this process and becomes a (slightly) better developer who can avoid similar mistakes in the future. similarly to the human development process, we provide empirical evidence that leti can learn from past mistakes and avoid similar errors in §3.2. figure 1 : qualitative example of leti improving an lm on code generation by leveraging feedback from a solution evaluator (e.g., a python interpreter). at each leti iteration, the lm is first asked to generate candidate solutions. as a case study, we obtain binary and textual feedback by executing the solution against test cases using a python interpreter. feedback and the generated solutions are used to improve the lm generator for the next leti iteration through feedback-conditioned fine-tuning ( §2.3). this is a code generation (mbpp; @xcite test set example generated by a 2b model optimized with leti. we omit a few iterations and repetitive code for clarity. in leti, a base lm pre-trained on both natural language and code 2 is asked to generate a piece of program conditioning on the natural language instruction, which is then tested on a suite of test cases. leti fine-tunes the model on a concatenation of natural language instruction, lm-generated program, and the textual feedback (e.g., stack traces and error messages) that pinpoints the bug, which is only provided when the generated program fails to solve the task. in addition to textual feedback, we prepend the fine-tuning sequences with a reward token (i.e., binary feedback), which differs for correct (
) and buggy solutions (
), to encourage the lm to generate correct solutions when conditioning @xmath0. leti repeats this procedure for multiple rounds. during this iterative process, leti assumes no instruction-code paired data. we find that leti improves lm's performance 2 almost all modern large language models train on both natural language and code @xcite @xcite . we observe that textual feedback is advantageous in terms of improving the lm compared to baselines that only use binary feedback, as it offers enhanced performance and greater sample efficiency that only requires about half of the gradient steps to reach the same performance for the 2b-scale model ( §3.3). furthermore, we find leti is equally applicable to nlp tasks (e.g., event argument extraction wang et al. 2023a) when they can be formulated as code generation problems ( §3.5). ## leti: learning from textual interactions each iteration, leti prompts the lm ( §2.1) with the natural language problem description to generate a set of n solutions. the solutions are then evaluated on a suite of test cases by a solution evaluator ( §2.2) to generate textual feedback (i.e., stack traces and error messages). this work uses a python interpreter as the solution evaluator to assess lm-generated solutions. the textual feedback is used to fine-tune the lm with feedbackconditioned fine-tuning (fcft, §2.3). we assume no ground-truth solutions while finetuning the lm, as leti directly learns from solution evaluator's feedback. intuitively, fcft leverages textual feedback to associate various types of errors (e.g., syntaxerror) and solutions that commit them. furthermore, with binary feedback, fcft aligns correct or wrong solutions with corresponding pre-pended reward @xmath1, so that better solutions can be sampled from a trained lm by conditioning it @xmath2. the workflow (one iteration) is described in algorithm 1 and fig. a.6. ## related work using feedback to improve code generation. leveraging non-textual feedback from an interpreter, prior work can generate solutions following natural language instructions by sampling and filtering large amounts of programs @xcite , training a model to rank generated solutions @xcite , fine-tuning a code-lm on generated solutions verified by test cases @xcite , or training a reward model and using reinforcement learning (rl) to improve code-lms @xcite . recent work has explored textual feedback (e.g., error messages, human language feedback) to improve lms @xcite . @xcite improves code generation by fine-tuning the original lm on code refinement generated by conditioning on human language feedback; different from our work, their fine-tuned lm uses more expensive human feedback and is not trained directly on the provided textual feedback. @xcite improve code generation by allowing lm to look at self-generated (and/or interpreter) feedback; however, the generator lm was frozen and couldn't generate better code on the original problem without these methods, while leti improves the underlying lm directly. improving lms with reinforcement learning. using ppo, @xcite align lms with human preferences. coderl @xcite follows reinforce @xcite and policy gradient @xcite to improve code-lms with a scalar reward from the interpreter. different from leti that directly leverages textual feedback, these algorithms require either manually crafting @xcite or training @xcite reward/value functions, which could be less scalable for various tasks. another strand of work leverages transformer architecture @xcite to perform rl with sequence modeling @xcite . @xcite @xcite improve lm by performing condition training, similar to conditioning lm on binary feedback f binary in leti. leti goes beyond the aforementioned work conditioning on the coarse-grained label: we are asking the lm to comprehend and improve directly based on textual feedback (e.g., error messages) that generally contains richer information compared to binary feedback. ## conclusion we proposed leti, a new lm fine-tuning paradigm that explores lm's potential to learn from textual interactions. we focused on code generation tasks and showed that one can effectively leverage automatic textual feedback from a python interpreter to improve lms. textual feedback outperforms baselines that only use binary feedback in both generation quality and sample efficiency. furthermore, leti is equally applicable in nlp tasks that can be formulated as code generation, which we empirically verified on event argument extraction.