RAR : Retrieval-augmented retrieval for code generation in low resource languages (2024)

Abstract: Language models struggle in generating code for low-resource programming languages, since these are underrepresented in training data. Either examples or documentation are commonly used for improved code generation. We propose to use both types of information together and present retrieval augmented retrieval (RAR) as a two-step method for selecting relevant examples and documentation. Experiments on three low-resource languages (Power Query M, OfficeScript and Excel formulas) show that RAR outperforms independently example and grammar retrieval (+2.81–26.14%). Interestingly, we show that two-step retrieval selects better examples and documentation when used independently as well.

## introduction large language models (llms) struggle to generate low-resource programming languages from natural language due to limited pre-training knowledge @xcite @xcite . previous work improves low-resource code generation with llms using retrieval augmented generation @xcite with relevant examples @xcite or documentation @xcite . there are several challenges in using documentation as context for code generation. first, documentation often does not include how the components are actually pieced together in the form of real code, which makes it difficult for models to understand the syntax and usage for new languages. second, documentation is weakly correlated to the natural language utterance and specific parts of documentation might not be related at all to the utterance but crucial for the code generation. for example, the power query m code to "highlight top 5 projects based on sales" requires a flag orderbydescending to be set, but this is not related to the utterance. third, documentation is often dense and flat, and selecting the right subset of documentation is both challenging and crucial to the success of these systems. similarly, using examples poses some additional challenges. first, obtaining an example bank of code and natural language pairs requires a lot of manual effort. second, models tend to overfit to examples provided in the prompt and do not properly adapt them to the current problem. to address these challenges, we propose retrieval augmented retrieval (rar) for code generation from both examples and documentation. the examples provide specific information and the documentation allows the model to generalize that specific information to the current problem. our approach enhances the retrieval process by leveraging the outputs of an initial retriever, the driver retriever, to guide a secondary retriever, the influenced retriever. this sequential retrieval mechanism ensures relatedness of the retrieved examples and grammar entities, which is required for the model to specialize and generalize accordingly. we evaluate rar on three low-resource languages: (power query) m, excel formulas (excel) and officescript (os). we compare dependent example or documentation selection of rar to multiple existing documentation and example retrieval techniques, showing improvements of +25% on os and +3% on m for documentation; and +1.28% for os, +3.5% for m and +2% on excel for examples. when combining examples and documentation, rar shows improvements of +4% on os, +2% on m and +4% on excel over independent retrieval. we also analyze the impact of using two-step retrieval, including only relevant and irrelevant context items in the prompt, and the token length. we make the following contributions: ## related work multiple techniques have been developed to improve code generation from natural language with llms, including (1) retrieval augmented generation for adding contextually relevant examples to the prompt @xcite @xcite ; (2) execution-guided refinement @xcite ; and ## documentation and examples documentation is the most comprehensive and structured resource @xcite publicly available @xcite for most programming languages. the documentation consists of a grammar (d) that describes how code is built over entities @xcite and examples (e) that depicts how to use and combine elements from d. an example of grammar and examples from a page of the officescript documentation is shown in figure 1 . the grammar (d) serves as a bank for grammar elements over which retrieval is performed. we consider each grammar element g i to be one standalone function or class method. we can use the path of each node in an abstract syntax tree (ast) to extract @xmath0. for example, even if multiple classes have a method getformat, then following a path up the ast allows us to disambiguate which class this method is from. a full example of code and the associated grammar entities is shown in figure 2 . the example corpus (e) is composed of description (utterance) and code pairs (u i , c i ). we only consider examples present in the documentation, which consists of sample code illustrating the usage of grammar elements. if a textual description of an example is not available, we use an llm (gpt-4) to generate it. ## retrieval augmented retrieval rar uses a two-step retrieval where the driver retriever (r d ) influences the influenced retriever (r i ). there are two possible scenarios: ## experimental setup we describe the experimental setup and the conditions set for a fair comparison between our approach and the baseline. ## evaluation we aim to answer the following research questions: rq1 how does dependent retrieval for documentation or examples compare against existing (independent) documentation and example retrieval methods? rq2 does dependent retrieval extract better context than standalone retrieval of examples and documentation? rq3 are both positively and negatively dependent retrieval necessary? rq4 how does the performance vary as a function of increasing context length? rq5 does rar rely on the driver retriever for its performance gain over independent retrievers? ## conclusion we introduce rar, a two-step retrieval technique used to extract relevant context for code generation over low-resource programming languages. our approach claims that off-the-shelf documentation for a language is enough to help an llm generate syntactically and semantically correct programs. we also show how grammar and example work better together. our approach establishes a working relationship between the two, capable of generating sound and reliable programs. the results we outline opens gates for future research, where grammar and example complement each other to formulate unseen programming languages.