Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code (2024)

Abstract: This paper presents Coffee-Gym, a comprehensive RL environment for training models that provide feedback on code editing. Coffee-Gym includes two major components: (1) Coffee, a dataset containing humans’ code edit traces for coding questions and human-written feedback for editing erroneous code; (2) CoffeeEval, a reward function that faithfully reflects the helpfulness of feedback by assessing the performance of the revised code in unit tests. With them, Coffee-Gym addresses the unavailability of high-quality datasets for training feedback models with RL, and provides more accurate rewards than the SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback models that outperform baselines in enhancing open-source code LLMs’ code editing, making them comparable with closed-source LLMs. We make the dataset and the model checkpoint publicly available in https://huggingface.co/spaces/Coffee-Gym/Project-Coffee-Gym.

## introduction large language models (llms) have made great progress in code generation @xcite , e.g., achieving human-level performances in code generation benchmarks @xcite . such success makes them powerful tools for assisting human programmers @xcite ; however, they still produce errors @xcite . therefore, code editing, i.e., resolving errors in code, remains an important task for code llms @xcite . studies have utilized natural language (nl) feedback from llms as descriptive guidance in editing wrong codes for code llms. for instance, self-refine @xcite largely improves their code editing using gpt-4's feedback. yet, abilities to generate helpful feedback, as they report, are limited to powerful closed-source llms (e.g., gpt-4). this can lead to a heavy reliance on closed-source llms that may cause not only high computational (e.g., api) cost but also security risks @xcite , limiting their applicability for confidential codes. this work aims to foster building open-source feedback models that produce effective feedback for code editing. an intuitive approach is to apply supervised fine-tuning (sft) on open-source code llms using feedback from gpt-4 (generated based on machines' code editing) @xcite . however, this simplified approach poorly aligns editing performance with the helpfulness of feedback (bottom of figure 1 ) @xcite . inspired by the success of rlhf @xcite , we reformulate feedback modeling with reinforcement learning (rl), where we align feedback models with the helpfulness of feedback during training. since the success of rl highly depends on the initial sft model and a reliable reward function @xcite , we hereby identify 3 main challenges in applying rl to feedback generation for code editing: (1) limited scenarios of errors in modelgenerated code editing datasets for initializing sft model, (2) the lack of pairwise (correct and wrong) feedback to train/test reward functions, (3) absence of validated implementation of reward models. we present coffee-gym, a comprehensive rl environment addressing the above challenges in training feedback models for code editing. first, to tackle data scarcity in sft initialization and reward modeling, we curate coffee, a dataset for code fixing with feedback, which consists of code editing traces of human programmers and human annotated feedback. unlike model-generated data (figure 2 ), coffee includes (1) problems across various difficulties, including those current llms (e.g., gpt-4) cannot solve; (2) pairs of correct and wrong feedback for reward modeling; through experiments, we validate coffee-gym's efficacy in training feedback models. we find that coffeeeval provides more accurate rewards, compared to the current sota reward model, i.e., g-eval @xcite with gpt-4. also, we show that the feedback models trained with coffee-gym generate more helpful feedback, achieving comparable performance to closedsource feedback models in code editing. ## constructing coffee-gym we introduce coffee-gym, a comprehensive rl environment for training nl feedback model for code editing. coffee-gym consists of two major components: (1) coffee, a dataset of humanwritten edit traces with annotated nl feedback, and (2) coffeeeval, an accurate reward model that @xmath0: abc[ord(c)-ord('a')] = s.index(c) print(abc) @xmath1: abc[ord(c)-ord('a')] = s.index(c) print(*abc) given a word s consisting only of lowercase letters, write a program that prints the first occurrence of each letter in the word, or -1 if the letter is not included in the word. your code correctly initializes the list with -1 for each letter, but you need to print the values individually using the operator to unpack the list. input (i.e., word s) ... ... correct output # of instance 44,782 4.19 2.7 35.5 742 674.1 649.4 avg. # of error lines per code avg. # of submissions per user avg. # of test cases per prob. # of total prob. sets avg. solution len. 674.1 avg. wrong code len. avg. feedback len. 269.0 avg. description len. zebra [4, 2 , -1, ..., 0] problem description: dataset statistics wrong code: q y* * c correct code: correct feedback: synthetic test cases: the issue is that you need to use a dictionary to store the ... c incorrect feedback: measures feedback's impact on code editing. ## conclusion in this paper, we present a comprehensive study on building open-source feedback models for code editing. we introduce coffee-gym, an environment for training and evaluating feedback models, and share valuable insights from our experiments. we hope our work will encourage researchers to further explore feedback model development using coffee-gym and our findings, advancing the field of code editing with nl feedback.