How Programming Concepts and Neurons Are Shared in Code Language Models (2025)

Abstract: Several studies have explored the mechanisms of large language models (LLMs) in coding tasks, but most have focused on programming languages (PLs) in a monolingual setting. In this paper, we investigate the relationship between multiple PLs and English in the concept space of LLMs. We perform a few-shot translation task on 21 PL pairs using two Llama-based models. By decoding the embeddings of intermediate layers during this task, we observe that the concept space is closer to English (including PL keywords) and assigns high probabilities to English tokens in the second half of the intermediate layers. We analyze neuron activations for 11 PLs and English, finding that while language-specific neurons are primarily concentrated in the bottom layers, those exclusive to each PL tend to appear in the top layers. For PLs that are highly aligned with multiple other PLs, identifying language-specific neurons is not feasible. These PLs also tend to have a larger keyword set than other PLs and are closer to the model’s concept space regardless of the input/output PL in the translation task. Our findings provide insights into how LLMs internally represent PLs, revealing structural patterns in the model’s concept space. Code is available at https://github.com/cisnlp/code-specific-neurons.

## introduction most state-of-the-art autoregressive large language models (llms) perform well on coding tasks @xcite @xcite @xcite . including code in the pre-training data has become a common practice in llm pre-training, even for models not specifically designed for coding @xcite . most of these llms involved in coding tasks are pre-trained on multiple programming languages (pls) @xcite @xcite ## materials and methods we use three established methods to uncover the concept space of llms, using datasets from pls at parallel, keyword, and raw levels. ## discussion and implications our findings suggest several strategies for building more efficient multilingual code models. neuron-level interpretability. initially, language-specific components were studied in neural machine translation using small language models @xcite @xcite . later, the role of ffns within llms was explored in several studies, highlighting their function as key-value memories for storing factual and linguistic knowledge @xcite @xcite . however, these analyses typically investigate neuron behavior, focusing on monolingual settings in natural languages and pls. building on methods explored in investigations on the role of ffns within llms and considering clear evidence that llms exhibit significant overlap in their embeddings across languages-particularly among those from the same linguistic family @xcite -several recent studies @xcite @xcite @xcite @xcite @xcite @xcite have investigated the existence of language-specific neurons and internal mechanisms for natural languages, especially within the ffn layers of llms. just as there are many natural languages, there are also many pls. however, no research has explored the existence of language-specific neurons for pls, even though llms are typically pre-trained on a mixture of these languages. building on this, our work adopts the method proposed by @xcite to identify pl-specific neurons. this approach enables a scalable and targeted analysis of neurons for many pls using only raw pl data. interpretability for code. interpretability in language models for code-related tasks remains underexplored, with most research focusing on attention layers @xcite @xcite . our work is closest to @xcite , who analyze ffn layers. they analyze two gpt-based models @xcite for three pls, showing that lower layers capture syntax while higher layers encode abstract concepts and semantics. they demonstrate that concepts are stored in the ffn layers and can be edited without compromising code language model performance. however, their analysis is performed in monolingual settings, while our work investigates the relationship between pls to determine if they share concepts and neurons in coding tasks. ## conclusion in this study, we investigate how llms represent programming languages (pls) in their concept space using the logit lens method. we observe that english and pl keywords appear in intermediate layers, with notable probabilities in the latter half. initially, these keywords surpass the expected output tokens, but as the probabilities of expected tokens increase and overtake those of english and pl keywords, the probabilities of the latter decline. we further investigate the existence of language-specific neurons using the language activation probability entropy (lape) method. our analysis reveals that language-specific neurons can be identified for most languages in the llama 3.1 model, but not for pls such as java and c#, which align closely with other pls. we find that languagespecific neurons are concentrated in the bottom layers, while neurons exclusive to each pl are located in the top layers. these findings deepen our understanding of llms' inner workings in the context of pls and provide valuable insights for interpretability in code-related tasks.