Language Agnostic Code Embeddings (2024)

Abstract: Recently, code language models have achieved notable advancements in addressing a diverse array of essential code comprehension and generation tasks. Yet, the field lacks a comprehensive deep dive and understanding of the code embeddings of multilingual code models. In this paper, we present a comprehensive study on multilingual code embeddings, focusing on the cross-lingual capabilities of these embeddings across different programming languages. Through probing experiments, we demonstrate that code embeddings comprise two distinct components: one deeply tied to the nuances and syntax of a specific language, and the other remaining agnostic to these details, primarily focusing on semantics. Further, we show that when we isolate and eliminate this language-specific component, we witness significant improvements in downstream code retrieval tasks, leading to an absolute increase of up to +17 in the Mean Reciprocal Rank (MRR).

## introduction large language models (llms) have made remarkable progress in code-related tasks, exemplified by models such as codex, which powers github copilot and offers automated code suggestions within integrated development environments (ides) @xcite . these models achieve their proficiency through extensive training on vast code datasets, providing them with versatile contextual understanding for a range of coding tasks @xcite @xcite . however, it's worth noting that decoder-only models may not always be the optimal choice for retrieval tasks when compared to encoder models @xcite @xcite . while previous studies indicate that language models trained on a variety of natural languages exhibit strong cross-lingual traits @xcite , their multilingual representations can be dissected into a language-specific syntax component and a language-agnostic semantic component @xcite . moreover, eliminating the languagespecific elements can enhance retrieval tasks and counteract "language bias", a tendency for representations to cluster by language instead of meaning @xcite @xcite . we aim to determine if similar patterns are evident in multilingual code models pretrained on programming languages (e.g., c, c++, python) as opposed to natural languages (e.g., english, french, spanish). we specifically address: our comprehensive evaluations confirm these patterns in code language models. our contributions are summarized as follows: ## language agnostic code embeddings let m represent a multilingual code language model trained on a set of programming languages {1, . . . , ℓ}. given a code snippet c in a specific programming language l, this model produces an embedding @xmath0, denoted as m(c) = @xmath1. we hypothesize that the embedding @xmath2: a syntax component, e @xmath3, which depends on the programming language l, and semantic component, e a which is language-agnostic. this relationship can be expressed as: next, we introduce the estimation set e, which is used to estimate the language-specific components e s . ## ablation study in this section, we conduct various ablation studies focusing on the estimation set's size and the effects of different kinds of embeddings. ## discussion in our study of multilingual code models, we find that these embeddings can be decomposed into two main components: language-specific and languageagnostic. through extensive experimentation, we conclude that when representations are not aligned during pre-training, the removal of the languagespecific component, utilizing only the languageagnostic component, significantly enhances performance in retrieval tasks. ## limitations in this study, we have focused on exploring representations of encoder-only or encoder-decoder models. however, future work should also investi-gate decoder-only models.