HGA dapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detection (2025)

Abstract: Pre-trained language models (PLMs) are increasingly being applied to code-related tasks. Although PLMs have achieved good results, they do not take into account potential high-order data correlations within the code. We propose three types of high-order correlations in code tokens, i.e. abstract syntax tree family correlation, lexical correlation, and line correlation. We design a tokens and hyperedges generator to capture these high-order data correlations. We improve the architecture of hypergraph neural networks and combine it with adapter tuning to propose a novel hypergraph-based adapter (HGAdapter) to fine-tune PLMs. HGAdapter can encode high-order data correlations and is allowed to be inserted into various PLMs to enhance performance. Experiments were conducted on several public datasets, including six languages of code summarization and code clone detection tasks. Our methods improved the performance of PLMs in datasets to varying degrees. Experimental results validate the introduction of high-order data correlations that contribute to improved effectiveness.

## introduction in recent years, the emergence of pre-trained language models (plms), large language models (llms), and small language models (slms) has driven the application of associated techniques to code-related tasks, including code clone detection, code classification, code summarization, and more @xcite . unlike natural language, code inherently contains richer structural information. consequently, in code-related tasks, numerous language models incorporate syntactic and semantic code structures to enhance task performance @xcite @xcite . however, the high-order data correlation inherent in code remains underexplored in current language models. high-order data correlation refers to the relationship among multiple entities as a unified unit, usually involving more than two entities, distinguishing it from a pairwise relationship @xcite . current language models mainly employ transformerbased encoder or decoder architectures, where the self-attention mechanism considers pairwise relationships between all tokens. however, this framework lacks the capability to capture cohesive grouplevel features where multiple tokens can be treated as an integrated feature unit. our research reveals that in the source code, there exist high-order data correlations that enable the extraction of such integrated feature units. the first category of these correlations originates from the cohesive relationships among syntax tree nodes that share a common parent node in the abstract syntax tree (ast). the code can be parsed into an ast which represents the structure of the code. the text of the code corresponds to the leaf nodes of the ast. tokens that belong to the same ast parent node can have their features computed as a unified whole to capture structural semantics, indicating that they together form a specific code structure. we call it ast family correlation. an example of ast family correlation is shown in figure 1 . figure 1 : ast family correlation. the tokens 'a', '+', and 'b' belong to the same ast parent node, forming an addition operation, and can thus be treated as a whole unit. the second category are correlations within lexical units. in code, there are numerous long lexical constructs, such as function names, variable names, and class names, which frequently comprise multiple semantic components concatenated via camel case or underscore delimiters. when undergoing tokenization, these unified lexical units may be fragmented into discrete tokens. the resulting split tokens can have their features computed as a whole to preserve the characteristics of their parent lexical unit. we call this lexical correlation. an example of lexical correlation is shown in figure 2 . simplecalculator is a class name that was split into three tokens. these tokens can be viewed as a whole to extract features from their original class name. the third category involves tokens within the same line of code. programmers typically write code with lines as the basic unit. tokens on the same line of original code can be processed as a whole, providing additional granularity to represent code organization patterns. we call it line correlation. an example of line correlation is shown in figure 3 . to extract the three types of high-order data correlation from the code, we design a tokens and hyperedges generator. existing language models only consider tokens as pairwise relationships, not as collective units for the above high-order correlation feature encoding. to address this gap, we propose a novel hypergraph-based adapter (hgadapter). hypergraphs are the standard framework for representing high-order correlations. unlike the edges in a standard graph, which only connect two entities, the edges in a hypergraph can connect any number of entities called the hyperedge @xcite . hypergraph neural networks (hgnns) have emerged as the most widely used neural networks to capture high-order data correlations @xcite @xcite . adapters are lightweight modules inserted into pre-trained models for fine-tuning @xcite . we improved the two-stage architecture of hgnns and combined it with adapters to propose hgadapter, we implement it as an adapter module inserted into language models, enabling the extraction and computation of the high-order features to enhance language models performance. we conduct experiments on code summarization, a generation task, and code clone detection, an understanding task. we used public datasets of the two tasks to evaluate our method. experimental results validate that our method improves the performance of language models, with the introduction of high-order data correlations contributing to an improvement in effectiveness. the main contributions of this paper are: ## methods our methodology has two core components: the tokens and hyperedges generator module and the hgadapter module. before code text is input into the language model, it needs to be tokenized. to capture high-order data correlation in the code, we design a tokens and hyperedges generator instead of directly tokenizing the code. to encode highorder data correlation, we design the hgadapter, which is integrated as an adapter module between the plm layers. ## conclusion in this paper, we propose to introduce high-order data correlations within code tokens into language models. we propose ast family correlation, lex-ical correlation, and line correlation. we design a tokens and hyperedges generator to capture the three types of high-order data correlation in the code. we improve the architecture of hgnns and combine it with adapters to propose hgadapter, it can encode high-order data correlations, and it is allowed to be inserted into various plms. we perform experiments on public datasets of code summarization and code clone detection tasks. experimental results show that our method increases the performance of plms, with the introduction of high-order data correlations contributing to an improvement in results. further ablation studies and parameter comparisons further validate the effectiveness of our method. in the future, we will explore the introduction of more high-order data correlations, the integration of more effective parameter fine-tuning methods with hypergraphs, and the application of our approach to a wider range of code-related tasks.