Does Large Language Model Contain Task-Specific Neurons? (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in comprehensively handling various types of natural language processing (NLP) tasks. However, there are significant differences in the knowledge and abilities required for different tasks. Therefore, it is important to understand whether the same LLM processes different tasks in the same way. Are there specific neurons in a LLM for different tasks? Inspired by neuroscience, this paper pioneers the exploration of whether distinct neurons are activated when a LLM handles different tasks. Compared with current research exploring the neurons of language and knowledge, task-specific neurons present a greater challenge due to their abstractness, diversity, and complexity. To address these challenges, this paper proposes a method for task-specific neuron localization based on Causal Gradient Variation with Special Tokens (CGVST). CGVST identifies task-specific neurons by concentrating on the most significant tokens during task processing, thereby eliminating redundant tokens and minimizing interference from non-essential neurons. Compared to traditional neuron localization methods, our approach can more effectively identify task-specific neurons. We conduct experiments across eight different public tasks. Experiments involving the inhibition and amplification of identified neurons demonstrate that our method can accurately locate task-specific neurons.

## introduction large language models (llms) have gained widespread attention due to their powerful capabilities @xcite . based on the unsupervised pre-training followed by instruction finetuning (ift) paradigm @xcite , llms have developed the ability to comprehensively handle various types of natural language processing (nlp) tasks. the advantage of this paradigm is that a single model deployment can perform multiple tasks, showcasing the versatility and efficiency of llms in nlp applications @xcite . however, there are significant differences in the knowledge and abilities required for different tasks by llms. for example, sentiment analysis focuses on adjectives and adverbs @xcite ), text classification emphasizes domain-specific terminology @xcite , while natural language inference prioritizes the relationship between premise and conclusion sentences @xcite . this suggests that the processing patterns of the same llm may vary across different tasks, highlighting the necessity to investigate the underlying mechanisms of task-specific processing in llm. task-specific processing can be traced by examining the neurons activated during the inference of specific tasks. therefore, the two critical questions emerge: are there neurons within the same llm that handle specific tasks?, and if so, how does the model manage and differentiate between various types of tasks? in fact, neuroscience has discovered that different brain areas control distinct behavioral abilities @xcite . for instance, while higher cognitive functions such as learning, reasoning, decision-making and creativity are primarily controlled by the frontal lobe, the neurons involved in these processes remain distinct @xcite . inspired by these findings, we hypothesize that although llms utilize a unified structure and parameters, the neurons engaged in different tasks may vary significantly. motivated by this hypothesis, this paper investigates the existence of task-specific neurons in llms, specifically examining whether the neurons activated by different tasks exhibit distinct patterns. as shown in figure 1 upper, different task-specific neurons in the same llm are marked with different colors. current research has explored specific neurons in llms that primarily handle knowledge @xcite @xcite and languages @xcite . on the one hand, knowledge-specific neurons in llms are key units that store factual information. by adjusting the activity of these neurons, we can control how much certain knowledge is represented in the model @xcite . this ability to manipulate knowledge-specific neurons enables knowledge editing, allowing for the modification of the factual information stored in the model @xcite . on the other hand, language-specific neurons in llms are specialized units that focus on language-related tasks, such as language modeling and machine translation. these neurons are responsible for controlling the quality of language generation in the model @xcite . by manipulating the activity of language-specific neurons, it is possible to influence the way llms generate language, such as translations or responses in different languages. for example, researchers have shown that adjusting these neurons can enable llms to switch between languages more effectively @xcite . utilizing language-specific neurons can help llms better handle linguistic tasks. moreover, current methods cannot directly ex-plore task-specific neurons because they have the following main different characteristics from language-and knowledge-specific neurons: 1) abstractness: unlike specific knowledge and language, the competencies required for tasks are more intricate and challenging to represent within symbolic systems. 2) diversity: while knowledge and language can be examined through a finite number of enumerated examples, task examples exhibit diversity and are difficult to comprehensively enumerate. 3) complexity: the abilities necessary for tasks may exhibit interdependencies or operate independently, often lacking clear demarcations. specifically, we found that task-specific neurons only need to solve the target task; for example, sentiment analysis neurons do not need to contain general english neurons. thus, we believe that not all tokens are crucial for identifying task-specific neurons. a similar approach has been used in tasks such as continuous learning @xcite and computer vision @xcite . different parts of inputs, such as task definitions, and contextual examples, play distinct roles in the inference process over llms @xcite . in biology, studies often use fluorescent markers to track neuronal activity by injecting fluorescent agents and observing their effects to monitor neurons @xcite . inspired by this, we propose a novel task-specific neuron detection method using causal gradient variation with special tokens (cgvst). we introduced fluorescent markers into the llm reasoning process to uncover the crucial role of special tokens. by observing neuron activation patterns triggered by these tokens, we can obtain abstract semantic representations beyond individual examples. this approach also helps to control diversity and complexity by focusing only on the special tokens. finally, we perform causal masked language model prediction for the task, recording the gradient on special tokens, and identifying the most active neurons as the task-specific neurons. we also conduct detailed experiments on 8 different tasks @xcite , including quantitative experiments on neuron inhibition and amplification. quantitative experiments demonstrate that our method can accurately identify task-specific neurons. as shown in figure 1 bottom, inhibition of these neurons significantly reduces performance on the target task, while the effect on other tasks is minimal. conversely, amplifying these neurons improves performance in the related tasks with minimal effect on other tasks. furthermore, we conducted comprehensive analyses of the neurons, including cross-validation and neuron visualization. the codes for this paper are available at github 1 . our contributions can be summarized concisely in the following three aspects: • we propose the existence of task-specific neurons, defining them as neurons that have a significant impact on a specific task while having minimal impact on other tasks. ## task-specific neurons identification in this section, we propose a simple and innovative strategy to identify task-specific neurons. by tracking gradient changes of specific tokens, we can locate neurons essential for task processing. ## experiments in this section, we conduct a series of detailed experiments to investigate task-specific neurons. we introduce 8 distinct tasks, each serving as an objective for identifying specific neurons. the experiments aim to answer the following research questions: rq1: can the proposed method locate neurons? for the located neurons, does inhibiting ( § 4.4) and amplifying them ( § 4.5) have corresponding effects? rq2: do task-specific neurons impact the model's language ability? ( § 4.6). rq3: what is the relationship between neurons corresponding to different tasks? ( § 4.7) and how are they distributed in the model? ( § 4.8). ## related work llms have garnered widespread attention due to their superior performance @xcite . after instruction fine-tuning and alignment, llms demonstrate strong performance across multiple tasks @xcite . especially, llms have achieved further breakthroughs in task performance with in-context learning @xcite . however, llms often produce hallucinations, which impacts their applicability in real-world @xcite . therefore, studies have to explore the mechanisms of llms to understand their operating principles @xcite . among these studies, the internal attribution of llms has received extensive attention, with various components of llms being investigated, including embeddings @xcite , attention @xcite , transformer layer (xu task ground truth base prediction amplification inhibition sa positive negative positive great news! here are the bigge st stars in pop culture ... tc business world business institution known for low fuel c osts 161 below link 60% of the charts predicted... ec surprise anonymous surprise wait solid good despite commun ic while we continu... qa seven 7 seven the beautiful mountain ranges o f the andes de files over... tm similar similar[inst: sente nce1: i've been expe riencing similar man, that was fast! here are the answers to the questions ... ## conclusion in this study, we demonstrated that different nlp tasks activate distinct neurons in llms. using our method, causal gradient variation with special tokens (cgvst), we identified task-specific neurons by focusing on significant tokens during task processing. our experiments across various tasks, including 8 different nlp tasks, confirmed that manipulating these neurons affects task performance.