Paper: LangProp: A code optimization framework using Language Models applied to driving (ICLR-2024)

Reviewer comment (Evaluation):
After complex training pipelines (online IL + RL), the generated code's performance is still far from hand-crafted rules which has only hundreds of lines (55 vs 76 in longest6). It casts doubts on the effectiveness of the proposed method.

Author response:
While this is a valid criticism, we would like to point out that the results are still remarkable, considering that researchers (including ourselves) have invested a lot of time in optimizing towards the CARLA driving benchmark with the given scenarios, with manual tuning of hyperparameters of the driving policy, whereas the LangProp agent was able to learn a sensible driving policy from imitation learning and infraction penalties, which outperformed many of the handcrafted experts implemented in previous work. In fact, the only expert that we do not outperform is TF++ and our own expert agent. While we acknowledge there is still room for improvement, we do not think this is the shortcoming of the LangProp method itself, but because of its novel approach which still has room for exploration and refinement in future work. The other benefit of the LangProp approach is that, because of its data-driven nature, it could quickly adapt to changes in the assumptions made in the training scenarios. With manually implemented agents, changes in scenarios or assumptions may break the implementation completely, whereas with LangProp one could improve and adapt the driving policy to new sets of assumptions, as long as there is training data for imitation learning and infraction signals provided.

Reviewer comment (Experiments):
The motivation is unclear to me. The authors assume that the perception problem is solved since they feed ground-truth information about the scene to LLM. The experiments is conducted in a further simplified environment only including ego states, other agents' states, traffic lights, and stop signs. However, in the actual driving scenario, there could be lots of complex obstacles or information that should be noticed in the driving scene, which might not be easy to describe precisely in language. (It is noteworthy that 3D occupancy and open-set scene understanding are hot topics discussed in the community). Thus, I am not sure this framework is particularly useful for the actual usage of the planning process.

Author response:
This is a great observation. This is exactly why we suggest the use of LLMs to generate driving policies as “code”, as opposed to using LLMs for scene description in language space, or doing language-to-actions. As you correctly point out, there are many details in the scene that cannot be captured by language, because language does not have the same granularity and spatial resolution as images or other geometric measurements. What we are proposing here is not about describing the scene in language (which, funnily enough, was what we initially considered before we came to the same conclusion as what you have pointed out just now), but using LLMs to refine driving code. While we demonstrated the capabilities of generating driving policies from scratch in our experiments, it is also possible to initialize the driving policy with human-written code, and refine them using the LangProp framework. It is true that the current LLMs are still not as robust as we would like them to be. LangProp is one way of addressing the shortcomings of its generation capabilities, and ensuring that the code quality improves with more training in a data-driven way. As to why we focus more on privileged information problems rather than directly working with RGB-D inputs, this is because “code” has an advantage over neural networks in terms of high-level reasoning and planning, which is where we want to apply a data-driven approach. Neural networks are more capable of handling continuous state-action spaces and low-level control, so rather than replacing these with LLMs, we consider a hybrid scenario where we use these neural network modules to process and extract information about the environment, and then apply the LangProp method to optimize the high-level planner.

Reviewer comment (Novelty):
Novelty. The authors claim that *we are the first to completely translate and apply the training paradigm used in machine learning for iterative code generation.* However, as mentioned by the authors, works like Auto-GPT and MineDojo VOYAGER also have similar paradigms. My research focus is on autonomous driving and I am not an expert in LLM. Thus, I am not sure whether it is an overclaim. I hope the author could explain how your paradigm differs significantly from existing LLM Agent works instead of an implementation in the autonomous driving field.

Author response:
Thank you for this opportunity to clarify our contribution. Both Auto-GPT and Voyager are different in the approaches that they take from ours, and so far, we have not come across work that is similar to LangProp, which extends the paradigm of machine learning to treating “code” as “models that can be optimized”. With this paradigm shift, we are now able to apply classical machine learning techniques (e.g. supervised learning, reinforcement learning) for data-driven code improvement. This broadens the definition of what we consider “learnable” and “trainable”, and can open up many new avenues of research that integrate learning methods into high-level reasoning, planning, and rule-based systems. We won’t repeat the arguments we have made in the paper in favor of the novelty of our method, but we will highlight the differences between our method and existing methods such as Auto-GPT and Voyager.

Auto-GPT is relevant as an embodied agent system, but is a very different paradigm from LangProp. Firstly, it is not a learning paradigm, since what Auto-GPT does is to create a plan to achieve its goal, and execute the plan in finer and finer detail. While it does interact with the environment and use the responses and feedback for executing its next steps, fundamentally its planning capabilities are upper-bounded by the capability of the LLM to create plans in the first place, because there isn’t any further learning signal that refines the plan. Therefore, Auto-GPT is more akin to using the LLM as a pre-trained policy and executing it in the real world (which works surprisingly well). Secondly, Auto-GPT does not have the notion of generating code and executing it. Rather, it sequentially calls APIs and takes other pre-assigned actions. 

Voyager, while more similar to LangProp and has provided the inspiration for our research, is also very different in their approach. Firstly, how they solve Minecraft is more similar to how Auto-GPT solves real-world problems in that they assume that the LLM knows enough about Minecraft such that it can create a step-by-step decomposition of the plan into “skills”, and write code for each skill separately. The implementation and learning method of Voyager is tightly coupled with Minecraft (more specifically, the Mineflayer bot interface), and assumes details specific to Minecraft (e.g. inventory, health, food, equipment, crafting table). Further, as discussed in the literature review, there is no mechanism to optimize or remove a sub-optimal skill once it has been added to the library. 

LangProp is a method that jointly optimizes a code policy towards an objective defined by the user, provided a training dataset. This is in contrast to previous work which does not adopt the machine learning paradigm, i.e. a policy should be learnable from training data given a training objective. With our paradigm, we can extend the machine learning paradigm not only to parameterized neural networks, but also to “code”, which is a better medium to express complex logic, reasoning and planning in an interpretable way. As far as we know, we are the first to propose this way of thinking about code optimization, and treating code as checkpoints just like neural network parameters, simply because the ability of LLMs to optimize code only improved to a practically usable level in the last two years.

Reviewer comment (Evaluation):
The framework's capacity to deliver driving strategies that are both interpretable and transparent raises concerns.

Author response:
Since some of the other reviewers missed this, we would like to highlight that we provide checkpoints (the policy code generated by LLM using the LangProp framework) at https://github.com/langprop-iclr24/LangProp/tree/main/checkpoints. As mentioned in the paper, “online_il_rl_(best)” provides the best checkpoint which was trained using both imitation learning and reinforcement learning losses, and record_0.txt contains the best-performing policy. By interpretability, we mean that the policies that are learnt are directly inspectable and diagnosable since they are in code form. Please note that we do not claim that LangProp can replace neural network models - rather, we see our method to be complementary to more classical machine learning approaches, since by leveraging the capability of LLMs whose strengths are in high-level reasoning and planning, we can use them in combination with deep learning models which excel in working with continuous state-action spaces and low-level control to solve problems which were previously difficult to solve using a learning-based approach (e.g. planning and reasoning).

Reviewer comment (Experiments):
The framework depends on the quality and coverage of LLM, which limits the effect and performance of LangProp. If LLM fails to generate sensible initial code, or struggles with some syntactic or logical issues, then LangProp is also hard to optimize. Existing LLMs may face some generation challenges or high uncertainty.

Author response:
While having a bad initialization of policies is not a problem for our method, since such policies will be discarded during training as better policies emerge, the diversity of responses (tunable by the temperature parameter for the LLM), as well as the number of responses per query, are beneficial to exploring the solution space, thus reaching a feasible solution quickly. It is true that the LLM method does not solve all problems and there may be some problems that are inherently challenging for LLMs to solve. We do not claim that our method can solve all problems - in fact, there are certain classes of problems (e.g. low-level control) for which neural networks are better candidate solutions. LangProp’s contribution is in that we improve the capabilities of LLMs by a large amount by introducing this notion of optimizing the output solution towards the target objective. Without this (zero-shot prediction from an LLM), the generated code may not be aligned with the intention of the user or the task specification, as we found when we generated driving policies either zero-shot or trained on offline imitation learning. A fair assessment of the effectiveness of our method would be to compare zero-shot performance from LLMs and performance after training using LangProp.

Reviewer comment (Evaluation):
Further analysis and in-depth exploration of the suitability of LLM in the context of autonomous driving is essential.

Author response:
We believe that what LangProp proposes (applying LLMs to optimize code in a data-driven way) opens up new possibilities for data-driven code development. While zero-shot applications of LLMs have enabled tools such as GitHub Co-pilot, some suggestions are inaccurate or misaligned with the user’s intentions, whereas if we have data or unit tests that we need to satisfy, the code suggestions can be made much more accurate by first running evaluations on these test cases and choosing the best possible suggestion that satisfies the requirement. We do not imply that autonomous driving is fully solvable with our method. In fact, as we have mentioned above, certain aspects of autonomous driving, such as perception and low-level control are best handled with neural network models, whereas our LangProp method has an advantage in iteratively optimizing code for high-level planning, which can be tested in simulation. Planning is one aspect of autonomous driving that has not yet successfully adopted a data-driven approach, for good reasons, since neural networks often struggle to produce generalizable high-level planning rules and are less interpretable. Therefore, most methods currently in deployment have human-engineered planning algorithms. Our LangProp framework is far from sufficient to replace such systems since it lacks the robustness that human-designed systems have to offer, and more research needs to be done in this direction, since we are the first to propose this approach. We hope that our work will provide inspiration for future research to make the framework more robust and safely deployable in the real world.

Reviewer comment (Experiments):
Lack of variety in experiments. To claim that LangProp is a viable meta-learning framework, we need some more examples than driving. The driving problem in the paper is more like estimating a driving function ("our expert") by a piece of python code. It's practical and useful, but also special. The Reviewer loves to see a few more different applications, especially if the optimization phase (code refinement) will always converge.

Author response:
With regards to your point on a lack of variety in experiments, this is a valid point which we would like to consider. We initially had a larger focus on the autonomous vehicle aspect of the research before we realized that the framework could be made more generally available as a learning framework. We agree that having more example use cases would be a great addition. While we consider setting up experiments for e.g. Minecraft to be on the scale of a separate research project (please also see our response to Reviewer ZuhC), we think it would be very interesting to apply our method to problems such as solving Sudoku or other puzzles and observe what solutions are discovered by LangProp over training. We will get back to you by the end of the rebuttal period regarding these additional examples.

We have also added additional examples featuring Sudoku and CartPole to the repository. Please refer to Appendix A.9, as well as https://github.com/langprop-iclr24/LangProp/tree/main/src/langprop#more-examples for the details on the experiments to show the generality of the framework. We would like to highlight that both experiments required no modifications to the pre-existing code-base, and that the training code for the new experiments is kept minimal and can be applied to other tasks.

Reviewer comment (Experiments):
The paper has some overclaims, and the experimental part is not sufficient enough. It states that this is a universal framework, but why are experiments only conducted in the CARLA simulator for autonomous driving scenarios? If experimental results could be obtained in other more complex and diverse environments, such as Minecraft or tasks related to robotics, it would greatly enhance the credibility of the paper.

Author response:
Thank you very much for taking the time to review the paper. We are delighted that the main message of our paper came across and you see the future potential in this branch of research as much as we do. As you suggest, there are many areas and use cases that could benefit from subsequent research. 

With regards to your point on having more complex and diverse environments being desirable, this is a valid point which we shall consider seriously (more action points in paragraph 4). There is a subtle difference in claiming that our framework is a general learning framework similar to PyTorch lightning (which we claim), and that it can solve all machine learning tasks (which we do not claim) - for instance, LLMs are better at handling high-level planning or reasoning tasks, rather than low-level control tasks, and our intention for this paper is to propose an alternative learning paradigm that allows LLMs to be used to learn high-level planning which has hitherto been a difficult problem for other machine learning approaches (e.g. neural networks). On the other hand, neural networks excel in working with continuous state-action spaces and low-level control. We could add this disclaimer to the paper to prevent any unwanted overclaims.

As for testing this in Minecraft, this would be indeed an exciting application for our LangProp method, and since Voyager also harnesses LLMs, we believe this to be a feasible application. However, there are many specific assumptions made in Voyager that use the domain knowledge of Minecraft, so disentangling these assumptions from the learning paradigm is a non-trivial task. For our autonomous driving experiment, setting up the experiment (including the evaluation benchmark, the baselines, determining what inputs and outputs should be given, and designing the rollout buffer and reinforcement learning pipeline) was a substantial engineering effort, and we expect setting up experiments for Minecraft would be a similar endeavor, hence we considered this to be out of scope. This is not necessarily because LangProp isn’t applicable to these tasks, but because more work is required in designing RL-specific training strategies, and these RL environments do not implement LLM-friendly interfaces ready for plug and play as of now. This is similar to the era in RL research before OpenAI gym interfaces with standardized state and action spaces were defined, and one had to implement their own custom environments or adapt their RL algorithms to work with a particular environment. LangProp is a framework more akin to PyTorch lightning rather than an RL framework (e.g. Stable Baselines / RLLib). 

Having said this, we are keen to add more experiments showcasing the potential diverse use of LangProp (which does not require starting a new project). For example, we are considering adding examples in Sudoku and/or other similar puzzles. We hypothesize there may be some interesting emergent behavior in terms of what solutions are found if we change the learning objective (for example, we could penalize computational time if we want a more computationally efficient solution). We will update you closer to the end of the rebuttal period in terms of this.

Reviewer comment (Experiments):
You have only showcased the LLM's capability for code optimization, which is constrained by the LLM's window size. This means that inference is limited to simple-logic functions, which are far from sufficient for the more complex interactive tasks and long-tail problems in autonomous driving. Though achieving high score in CARLA, it is way far from reality. How can we extend the code generation task to be more complex and diverse? For example, similar to VISPROG[1], to complete more complex tasks through the combination of models. I would like to hear your thoughts on this matter.

Author response:
Your point on the code optimization constrained to the window size is also a very good point. We have also thought about this, and we have had some trial experiments with multiple modular components (similar to the VISPROG example you’ve raised), but we found that this degraded the performance due to each of the modular components is also learnable and may not accurately match the docstring description. Rather than having a complex system which treats this problem separately, we found that including all the source code in one query and jointly optimizing the code tend to have more stable performances. However, we also believe that this multi-modular approach still has potential and is solvable with more research in this direction and ways of propagating learning signals to each of these modular components. (Please see our discussion on this topic in Appendix A.9). 

We do not consider the limitation due to context window size to be a significant drawback, since our framework can be used with any LLM, and there have already been rapid advances in the capabilities of LLMs. For instance, when we conducted our research we used a GPT 3.5 model with a context of 16k tokens, but now there are models that have 128k tokens with cheaper pricing. There is also a lot of research in vector embedding databases to make large repositories that far exceed the context size of LLMs searchable and usable. While there are still drawbacks that you have pointed out, we believe these concerns will be alleviated as the infrastructure around LLMs becomes more mature.

Reviewer comment (Evaluation):
I suggest adding another graph showing the close-loop score versus the number of steps. This would allow for a comparison of scores across all methods and clearly show how the final close-loop score (most important) changes as the number of steps increases. Though it could be time-consuming, a very sparse sampled point would be helpful, just like validation per epoch.

Author response:
As for the close-loop score versus the number of training steps, this would be ideal, however, the close-loop evaluation for each saved checkpoint in CARLA takes about 2-3 days for the training routes, 1-2 days for the test routes, and 6-7 days for the Longest6 routes, so plotting a curve is not very feasible. This is why we have a table showing the close-loop evaluation of the final checkpoint instead.

Reviewer comment (Experiments):
it requires a substantial amount of training data to train the language model and optimize the generated code.

Author response:
LangProp is an approach that actually requires much less domain-specific data because we use a pre-trained LLM and use its code generation capability and its open-domain generalization of language-to-code translation to solve novel tasks. Rather than finetuning the LLM, we show that iterative prompting allows for more successful outcomes compared to one-shot generation of code in a new application domain. The number of samples required for this optimization is small since we do not finetune the LLM itself. Instead, we use the LLM to optimize the code solution.

Reviewer comment (Experiments):
since the framework uses methods of supervised and reinforcement learning, there may be issues with overfitting or undertraining during the training process.

Author response:
This is true for almost all modern machine learning algorithms, and common training strategies to avoid such issues (e.g. train / val / test split, early stopping, data augmentation, training on data representative of the application domain) can be applied to training with LangProp as well.

Reviewer comment (Reproducibility):
the framework requires proper normalization and standardization of inputs and outputs; otherwise, it may result in degraded performance of the generated code.

Author response:
It is true that the training inputs and outputs need to be documented accurately (including the units of measurement of each quantity) in the docstring that defines whatever function you are trying to model. However, we do not see this as a deficiency of our method. In fact, models are bound to fail to perform under inputs that use a wrong normalization or unit (it is required that the user normalizes inputs to models trained on ImageNet, for example), and having such requirements written down explicitly in the docstring is better documentation, and may also be useful for automated assertions of the ranges of inputs and outputs.

Reviewer comment (Experiments):
How does the LangProp framework's evolutionary algorithm-based updating process compare in efficiency and effectiveness with the gradient-based optimization used in traditional neural network training?

Author response:
The short answer is that it is not possible to apply a gradient-based update method directly in our problem setting. The longer answer is that, because we are looking at open domain adaptation of code generation (i.e. the ground truth of the optimal code is not available for a given novel task, and we optimize the code based on its accuracy), we cannot simply fine-tune and regress the language model to generate the ground truth code solution with supervised learning. Alternative approaches to our method which use gradient-based methods would be using reinforcement learning [1] or other forms of planning [2] (e.g. Monte Carlo Tree Search), but these require much more compute due to having sparse rewards (if one considers generating a token from the LLM as an action, and code as states). Furthermore, an incomplete code would fail with an exception (thus a large penalty) so it is difficult to learn an accurate value function and it is unstable to retrain an LLM using such rewards / values.

Another argument against a gradient descent method is that, for our state space (i.e. code), feasible solutions may be far apart in the token representation space and is non-trivial to find one from the other by gradient descent. In contrast, an evolutionary algorithm approach works well in our use case because a mapping from one feasible solution to another (e.g. error correction) can be obtained by using an LLM, and can be used to perform updates to each seed. Once a feasible solution has been discovered, descendants from this seed are also likely to lie within the solution space.

Reviewer comment (Experiments):
How do different prompt template strategies affect the LangProp framework’s ability to generate and refine code?

Author response:
Our research focus is orthogonal to research in prompt engineering strategies (e.g. Chain-of-Thought), since most prompt engineering strategies can still be applied in LangProp and the LangProp framework is agnostic to which prompt engineering strategy one chooses. It is true that a detailed specification of the function’s inputs and outputs as a docstring (including the units and data types of each input and output) is important. While the ablation of prompt engineering strategies is out of scope for this paper, we welcome any follow-up research on this topic.

Reviewer comment (Theory):
Is there going to be a limit one generating correct code since the model parameters are not trained during the process?

Author response:
We experimented with Python code generation for tasks that can be solved by using standard Python libraries, so it is possible that the LLM either needs to be fine-tuned on an unknown programming language or be provided with documentation of a new library’s API if the task requires knowledge of languages or packages unknown to the LLM. We observe, however, that the LLM has the capability of generating code that performs better than what was in its training data. For example, we used GPT 3.5 Turbo model which was only trained on data collected up to September 2021, yet the LangProp agent outperformed human-written expert agents that were developed and published in 2022 and 2023, indicating that our method can generate code that is more performant than what was in the training data. (Note that this is not possible if we do not use LangProp, i.e. if we just use GPT 3.5 zero-shot)

Reviewer comment (Evaluation):
How does the framework address the issue of interpretability of the generated code?

Author response:
The generated code is human-readable, which greatly enhances the interpretability of the system compared to a neural network model. Interpretability can also be enhanced through prompting schemes such as Chain-of-Thought during code generation, or requiring dense commenting of the code.

Reviewer comment (Experiments):
What are the potential applications of the LangProp framework beyond autonomous driving, and how does it generalize to other domains and programming languages?

Author response:
LangProp is highly compatible with applications that have well-defined inputs, outputs, and performance metric. Applications may include writing code for operations on spreadsheets and data structures, generating SQL queries, automated test-driven programming for backend applications, and any other task that typically benefits from automated code generation and correction. As mentioned in the response to Q5, generalization to unknown programming languages may either require an LLM trained or fine-tuned on that programming language, while generalization to a new library may only require passing in the documentation of the library API in the prompt.

Reviewer comment (Evaluation):
How can we quantify the trustworthiness in the generated code? Are there any ways to quantify subjectivity in driving behaviors while getting / obtaining a measure of trust in the outcome or end result of LangProp. I think the strategy in Figure 2 can gain inspiration from a trustworthiness framework developed for deep reinforcement learning in autonomous driving and CNNs: "A general trust framework for multi-agent systems." In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 332-340. 2021 "Trustworthiness Evaluation and Trust-Aware Design of CNN Architectures." In Conference on Lifelong Learning Agents, pp. 1086-1102. PMLR, 2022 A major issue with the proposed framework is that it does not have any way of verifying and validating the safety or trustworthiness which are critical for autonomous driving.

Author response:
Thank you for the suggestion with regards to introducing a trust framework to the autonomous driving solution. It is true that not all the actors in the scene behave in ways typically expected from a rational agent, and hence that amount of uncertainty has to be incorporated in the model. For our LangProp framework, the way this should be incorporated would be in the success metric which dictates which policies should be prioritized. One could, for instance, have auxiliary losses for accurately predicting the actors’ future moves so that the model has some understanding of the actors’ behavior, implicitly modeling their trustworthiness.

Please note that our paper’s focus is more on the learning paradigm, rather than the autonomous driving solution itself. We use CARLA as a useful standardized benchmark to demonstrate the capabilities of the LangProp learning paradigm, where we benchmark against other human-implemented experts). While we consider autonomous driving to be one useful application demonstrating the capabilities of LangProp, we wish to keep the implementation as simple as possible so that we can fairly assess the performance of the learning paradigm while keeping other things mostly standard. It would be interesting to see future work that integrates a trust framework into the system (e.g. by changing the training objective, or encouraging modeling of the actors’ trustworthiness in the prompt).