Selective Shot Learning for Code Explanation (2025)

Abstract: Code explanation plays a crucial role in the software engineering domain, aiding developers in grasping code functionality efficiently. Recent work shows that the performance of LLMs for code explanation improves in a few-shot setting, especially when the few-shot examples are selected intelligently. State-of-the-art approaches for such Selective Shot Learning (SSL) include token-based and embedding-based methods. However, these SSL approaches have been evaluated on proprietary LLMs, without much exploration on open-source Code-LLMs. Additionally, these methods lack consideration for programming language syntax. To bridge these gaps, we present a comparative study and propose a novel SSL method (SSL_ner) that utilizes entity information for few-shot example selection. We present several insights and show the effectiveness of SSL_ner approach over state-of-the-art methods across two datasets. To the best of our knowledge, this is the first systematic benchmarking of various few-shot examples selection approaches using open-source Code-LLMs for the code explanation task.

## introduction code understanding and explanation @xcite , also known as code summarization @xcite and code comment generation @xcite , is an important problem in the domain of software engineering. it involves generating concise and informative explanations for pieces of source code. this provides the developers with a quick understanding of its functionality aiding in code maintenance, search and retrieval @xcite . for programmers new to a particular programming language, code summaries serve as valuable documentation to familiarize them with the new environment efficiently @xcite . automating the task of code documentation through comments and explanations can therefore prove beneficial in many ways. large language models (llms) have proven their efficiency in a variety of nlp tasks. llms have shown promising results in several software engineering tasks like code generation @xcite , translation @xcite , test case generation @xcite and code explanation @xcite @xcite @xcite . while using llms for the code explanation task, it has been shown that few-shot prompting achieves better results than zero-shot prompting @xcite . hence, selecting examples for few-shot learning is an important design criteria. we use the term selective shot learning (ssl) when few-shot examples are chosen intelligently, instead of being random. ssl approaches for code explanation include token-based and embedding-based methods @xcite without taking into account the language syntax. recent work in the area of code explanation have only considered proprietary llms like codex @xcite , code-davinci-002 @xcite ), text-davinci-003 @xcite , @xcite and gpt-3.5turbo @xcite . however there is a huge gap in proper benchmarking and performance evaluation of several competing, open-source code-llms like @xcite , star-coder @xcite for the code explanation task. to this end, the contributions of the paper are: we make the dataset and code publicly available at https://github.boschdevcloud.com/ hxt2kor/code-explanation . ## related work the code explanation @xcite task is a well studied problem in the domain of software engineering @xcite @xcite . with the advent of deep learning, methods combining neural architectures @xcite @xcite along with software engineering approaches like ast trees @xcite have been proposed. large language models have shown exceptional performance in a plethora of nlg tasks @xcite . the zero-shot and few-shot capabilities of these model make them highly adaptable to many nlp tasks. generic, open-source llms like llama-2 @xcite , alpaca @xcite are trained on open internet datasets. codellms such as star-coder @xcite , codeup @xcite , @xcite and llama-2-coder (manuel romero, 2023) have been either trained or fine-tuned on code-specific datasets containing source codes covering around 80+ programming languages. the large language models, when used for the code explanation task, has shown some encouraging results. the recent approaches @xcite @xcite demonstrate that the llms performs better in the few-shot setup when good examples of the task are provided. hence, deciding the relevant examples is an important design criteria while using llms for the code explanation task. existing approaches involve token-based, embedding-based @xcite and bm-25 along with repository information, data flow graph, ast tree etc. @xcite . however, these methods do not explore the efficacy of codellms. there has been systematic evaluations of transformer models (codet5 and codebert) @xcite and open source code-llms @xcite for code summarization, llms on code search @xcite and non-codellms like gpt, bard for code documentation generation @xcite . this work addresses the lack of systematic benchmarking of selective shot learning (ssl) strategies for code explanation. it analyzes four open-source codellms across two datasets and three ssl methods, without using auxiliary tools like ast or data-flow graphs @xcite . ## dataset in order to perform an extensive evaluation of the performance of the different open source code-llms on the code explanation task, we consider two types of datasets which have different levels of codes and explanations -inline level and function level. we describe each of them in detail: ## selective-shot learning approaches in this section we elaborate the different approaches for selecting relevant demonstrations for the code explanation task. the general pipeline is shown in figure 1 . it is assumed that there is a database containing (code snippet, code explanation) pairs (referred to as training data) from which relevant examples will be selected. similarity is computed between the input code snippet (q) and all code snippets (d i ) in the database, using the approaches selection token , selection semantic and ssl ner described next. from each approach, we find the most relevant k code snippets, along with their explanations, and curate a prompt which is then passed on to an llm to generate the explanation for q. ## experimental setup in this section we describe the experimental design choices used in this paper. evaluation: we use the bleu, meteor and rouge-l scores for evaluating the model generated explanations with respect to the ground truth explanations. these are the most widely used metrics for the task @xcite @xcite . large language models: we evaluate the performance of the different approaches by providing prompts to the following llms -llama-2-coder-7b, codeup-13b-chat, starcoder (15.5b) and codellama-34b-instruct. we use @xmath0. for the universalner llm, we set @xmath1, @xmath2, @xmath3.1. for all codellms, we set @xmath4, @xmath5.7. for the tlc dataset, there are five intents as described in section 3. @xcite uses these intents in the prompt construction. for instance, for a test query code from the intent "how-to-use" they use the prompt: "describe the usage or the expected set-up of using the method". however, we find that including such intent-specific keywords in the prompt does not affect the performance of the open source code llms. we therefore do not include the description of the intents in the prompt. the zero-shot prompt templates used in our experiments are as follows: codellama: [inst] <>you are an expert in programming. below is a line of python code that describes a task. return only one line of summary that appropriately describes the task that the code is performing. you must write only summary without any prefix or suffix explanations. note: the summary should have minimum 1 words and can have on an average 10 words. ## results the empirical results of the code explanation task on the conala dataset are presented in ## conclusion and future work in this paper, we perform a comparative study of several open-source code llms, ssl methods and experiment with two datasets having varying levels of explanations for the code explanation task. we perform a thorough analysis of the methods and the performances of the different codellms that lead to different interesting insights. additionally, we introduce a new selective-shot learning method ssl ner based on code-based ner . empirical results suggest ssl ner to be the best token-based demonstration selection strategy while being inherently interpretable and customizable through the code entities. there are several avenues to extend this work. possibilities of combining ssl ner with embeddings may be studied. we also plan to experiment with repository level code explanations. finetuning the llms by using the relevant examples selected by ssl ner is likely to improve performance. we leave its consideration to future research.