Evaluating In-Context Learning of Libraries for Code Generation (2024)

Abstract: Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly promising area is their ability to interpret code modules from unfamiliar libraries for solving user-instructed tasks. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations. These results raise several open questions: whether demonstrations of library usage is required, whether smaller (and more open) models also possess such capabilities, etc. In this work, we take a broader approach by systematically evaluating a diverse array of LLMs across three scenarios reflecting varying levels of domain specialization to understand their abilities and limitations in generating code based on libraries defined in-context. Our results show that even smaller open-source LLMs like Llama-2 and StarCoder demonstrate an adept understanding of novel code libraries based on specification presented in-context. Our findings further reveal that LLMs exhibit a surprisingly high proficiency in learning novel library modules even when provided with just natural language descriptions or raw code implementations of the functions, which are often cheaper to obtain than demonstrations. Overall, our results pave the way for harnessing LLMs in more adaptable and dynamic coding environments.

## introduction large language models (llms) pretrained on massive amounts of text and code data @xcite have shown promising results across various code generation tasks @xcite . these models excel at both, writing programs based on natural language instructions, and at generating code † work done partly during an internship at the allen institute for ai. as llms become more widely used, an important area of application is the generation of code in specialized domains. since llms are expensive to finetune, in-context learning (i.e., learning from instructions and examples in the prompt) has emerged as the preferred approach for adapting llms to tasks and domains not seen during training. previous works @xcite @xcite use demonstrations provided in-context to prompt llms to generate code that makes calls to external taskspecific library foot_0 functions. however, there are several open questions surrounding this phenomenon: can we use other easier-to-obtain supervision methods instead of demonstrations for the model to learn a new library? is it possible to adapt smaller, openly accessible llms to generate code that uses novel libraries? can llms in-context learn to use relatively uncommon programming languages? in this work, we attempt to answer these questions. we describe our evaluation framework and summarize our findings below. evaluation framework. we design three distinct scenarios bearing real-world significance to evaluate the code generation abilities of llms: the underlying evaluation approach is similar across all three scenarios. we consider tasks requiring models to generate code given some natural language instruction while making calls to external library functions. accordingly, we define novel programming libraries or even a target programming language that the model must use to generate code. for each example in the dataset for the task, we provide a specification of the library (or language) along with the task instruction in-context to the model. we then execute the generated code and evaluate its correctness. within this framework, we experiment with providing different types of incontext supervision for the library such as natural language descriptions or code implementations. findings. below, we summarize our results. foot_1 (1) we find that the ability to learn and use novel programming libraries is not limited to the largest proprietary models (such as gpt-4 (ope-nai, 2023)), but is also seen in openly-accessible models such as llama-2 @xcite and @xcite . ## learning novel libraries many practical applications of code generation require learning new libraries or frameworks. moreover, programmers need to constantly adapt to changes in existing frameworks. motivated by these practical needs, we study the ability of llms to adapt to novel programming libraries in-context. @xcite showed that llms can be used to solve computer vision tasks by learning how to call the functions of a novel library based on demonstrations provided in-context. we study this phenomenon in more detail with a wider range of models and different types of supervision. ## learning a new programming language in this section, we study the ability of llms to learn a new and unfamiliar programming language in-context. there are many diverse applications and tasks that use niche domain-specific-languages (dsls) @xcite @xcite . ideally, we would want to adapt a general llm to solve these tasks without the added overhead of re-training or finetuning the model. hence, it is important to contextualize how good current llms are at learning unfamiliar languages using just in-context learning. ## discussion in this work, we investigated the abilities of llms to in-context learn novel programming libraries and languages. below we discuss the main takeaways. we observed that the strongest llms can learn novel libraries from just their natural language descriptions as well as the underlying programming language implementations. this holds promise for adapting llms rapidly for different applications without requiring any effort for obtaining paired demonstrations data. moreover, we found that that the ability to acquire novel libraries in-context is not limited to proprietary llms. smaller, openlyaccessible models such as llama-2 and starcoder also exhibit a significantly high ability at learning code modules without requiring any finetuning. we noticed that the choice of supervision provided in-context (between natural language descriptions and programming language implementations) is crucial in determining performance, especially for smaller, openly-accessible models. this finding is beneficial for resource-constrained scenarios, where we can only use particular llms and need to determine the type of supervision to provide, or we only have a particular type of supervision available and need to select the best model suited for it. we found that llms degrade in performance when constrained to use particular libraries in scenarios where they can generate code by themselves. in such cases, llms have a very strong bias to generate code based on their priors and find it difficult to use the library functions provided in-context. finally, we observed that llms show a preliminary but promising ability to learn new programming languages from scratch based on in-context demonstrations or description of the language.