An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLM s (2024)

Abstract: Large language models (LLMs) have shown strong arithmetic reasoning capabilities when prompted with Chain-of-Thought (CoT) prompts. However, we have only a limited understanding of how they are processed by LLMs. To demystify it, prior work has primarily focused on ablating different components in the CoT prompt and empirically observing their resulting LLM performance change. Yet, the reason why these components are important to LLM reasoning is not explored. To fill this gap, in this work, we investigate “neuron activation” as a lens to provide a unified explanation to observations made by prior work. Specifically, we look into neurons within the feed-forward layers of LLMs that may have activated their arithmetic reasoning capabilities, using Llama2 as an example. To facilitate this investigation, we also propose an approach based on GPT-4 to automatically identify neurons that imply arithmetic reasoning. Our analyses revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding.

## introduction arithmetic reasoning is one of the emergent properties in large language models (llms), which is necessary for them to tackle tasks that require multiple steps to arrive at the correct answer. in recent years, chain-of-thought (cot) has become a popular prompting strategy to elicit reasoning foot_1 in llms @xcite . despite its successes, there is little understanding of what makes it effective and how llms utilize it to facilitate reasoning. to address this concern, a line of research has focused on decomposing the cot prompt into various components and performing ablation studies on them to ascertain the significance of each component on the llm reasoning performance @xcite @xcite . although these studies have yielded several insightful observations on the effect of input on llm's reasoning performance, they do not shed light on how these inputs are being processed internally by llms to perform reasoning. on the other hand, there is a growing body of research in the field of mechanistic interpretability @xcite ) that specifically examines the internals of llms to understand their mechanism. in this vein, @xcite studied the internal mechanism of llms to perform arithmetic calculation, suggesting that attention heads facilitate information traversal, while the feed-forward layer (ff) handles information processing to produce accurate answers for a given computation. however, @xcite only studied the mechanism for a single mathematical computation and doesn't study arithmetic reasoning in full scope. in parallel, some other research demonstrated that llms consist of neurons that can be associated with human-interpretable concepts, which play a crucial role in various capabilities of llms @xcite @xcite . @xcite showed that neurons in the ff layer of a transformer model @xcite form key-value pairs that facilitate next-token prediction by promoting concepts in the vocabulary space. however, none of the prior work has applied the intuition to understand llm reasoning. motivating by the need to form a deeper understanding of how cot prompts elicit reasoning in llms and observing the pivotal role of neurons within the ff layers of llms @xcite , in this work, we propose to investigate the activation of ff neurons in llms as a lens to interpret their arithmetic reasoning capabilities. particularly, we aim to use neuron activation to provide a unified explanation of observations that were only empirically made by prior work @xcite @xcite , as listed in to this end, we first propose an approach that leverages llms (e.g., @xcite ) to automatically search for neurons that are related to arithmetic reasoning (e.g., arithmetic addition, logical connections, etc.). prior work trying to search for concept-relevant neurons relies on human analysis @xcite . for example, @xcite proposed to manually examine a neuron's top promoted tokens and determine if the neuron promotes the given pre-defined concept or not. however, this manual approach becomes impractical for llms with a large number of layers and numerous neurons in each layer. our approach instead decides whether a given neuron expresses a certain concept automatically by prompting the gpt-4 with its top promoted tokens and make a judgment on its represented concept. our experimental results demonstrate the high effectiveness of utilizing gpt-4 for this purpose. subsequently, we apply our proposed approach to identify ff neurons in llama2-7b that promote several concepts relevant to arithmetic reasoning, listed in leveraging the identified reasoning neurons, we performed a series of analyses on observations made by prior work @xcite @xcite , including the importance of textual explanation, equations, arithmetic diversity, and the negligible impact of incorrect labels in cot prompts for elicitating reasoning in llms. specifically, we analyzed the activation patterns of the identified reasoning neurons, such as their activation frequency and strength, to gain insights into these observations. our results reveal that examining the activation of ff neurons in response to different cot prompts can provide valuable insights into why certain cot prompts are more effective in eliciting arithmetic reasoning capabilities in llms. we then conclude the paper with a discussion of future work that can complement the proposed neuron activation analysis with other approaches to form a more complete understanding of llm reasoning. ## neuron discovery using gpt-4 to facilitate the neuron analysis, we first propose an approach for discovering neurons that express concepts related to arithmetic reasoning. to achieve the same goal, @xcite manually examined the top-scoring vocabulary tokens projected by each neuron v l j and annotated its concept. however, this manual search approach can become impractical for llms with deep layers and numerous subupdates per layer. to overcome this inefficiency, we propose a method that leverages gpt-4 to automate the search process. our proposed approach involves two steps. first, for a given llm, we store the t neurons v l j 's with the largest coefficient m l ij from each layer l and at each generation time step i, using a set of examples e that showcase the llm's capability (i.e., arithmetic reasoning in our case) to provide the prompt. we only considered the top-t neurons to narrow our search to the most activated neurons. this returns a set of candidate neurons n . we present this step in algorithm 1. algorithm 1 candidate neuron collection 1: input: a set of examples e implying the capability, a filtering threshold t , the target llm 2: output: a set of candidate neurons n . 3: initialize n ← {} 4: for each example in e : 5: algorithm 2 neuron annotation via gpt-4 1: input: concept cname, a set of seed tokens sname, filtering thresholds p and f , embedding e of llm, and candidate neuron set n . 2: output: a subset of neurons r ⊂ n representing concept cname. 3: initialize r ← {} 4: for each neuron @xmath0: 5: ## experimental setup dataset and model setup we conduct our experiment on the gsm8k dataset @xcite , which is widely used for evaluating the arithmetic reasoning capabilities of llms. it consists of diverse grade school math word problems and only requires basic arithmetic operations to solve, often involving problem-solving steps ranging from two to eight. we use llama2-7b @xcite as our model to investigate the reasoning capabilities in llms. however, we believe that our findings apply to other transformer-based decoderonly llms as well. our experiments are based on the cot prompts obtained from @xcite , with a slight modification to ensure a consistent format in multistep reasoning which makes further analysis easier. each cot prompt consists of eight exemplars. for reproducibility purposes, we provide a complete list of our prompts in the appendix i. before investigating the mechanism of llm reasoning, we have conducted experiments to replicate and validate observations made by prior work ( to answer this question, we apply the proposed approach in section 3 to automatically identify neurons implying a set of 7 concepts, including logical connectors, which plays a crucial role in deciding the reasoning direction, a set of four arithmetic operations (i.e., add, subtract, multiply, and division), and others (equals to and calculation), which are also important to arithmetic reasoning. though they may not fully encompass arithmetic reasoning, these concepts are sufficient for an initial investigation of neuron activation. the seed token set s name for each concept, the identified neuron examples, and the expanded concept tokens found in the identified neurons, are presented in concept seed tokens expanded tokens #of neurons exemplar neurons logical connectors (c logic ) {first, so, meaning, therefore, then, next, hence } {logic, implies, thus, however, accordingly, subsequently, later, corresponding, etc. } 65 l10n9818{then, then, then, then, ..}, l11n3000{therefore, therefore, accordingly, donc,..}, l11n7742, l12n1030 addition (c add ) {add, addition, +, sum, plus } {added, u+002b, adding, ++, increment, total, etc.} 18 l12n4814{added ,addition ,add,..}, l21n7027{+ ,add ,u+4e0e, u+306,..}, l27n10751{+, plus, -, minus,..} subtraction (c sub ) {subtract, -, minus, sub } { -=, negative, u+2212, etc. } 2 l19n7900{-= ,-,minus, 2212, ..}, l25n5227 multiplication (c mul ) {multiply, product, times, mult, *, x } { multip, multi, u+00d7, double, twice, triple, fold, larger, etc. } 5 l16n10193{multip, double, multip, multiply, ..}, l18n4462, l20n6554, l22n1345, l22n1236 division (c div ) {divide, division, div, /, % } { div, divided, divisions, u+00f7, partition, partitions, etc. } 2 l20n10457{div ,divided ,division ,u+00f7,.. }, l26n1378{div, div, div, div, division,.. } equals to (c eq ) { =, total, equals, equal, equivalent } { equality, identical, same, exactly, contain, exact, etc. } 6 l14n7597{identical, difference, differences, equal,..}, l18n7531, l18n1850, l20n3177, l20n5535, l24n154 calculation (c cal ) activation pattern of reasoning neurons our further investigation found out intriguing activation pattern of reasoning neurons throughout an llm's reasoning process. for example, in figure 2 of appendix d, we showed that the logical connector neurons are often activated at the beginning of a generated sentence, whereas arithmetic neurons are mostly activated in response to arithmetic symbols and numbers. once a neuron is activated, it remains activated for a few subsequent time steps. this persistence implies a lasting impact of activated neurons on text generation in its proximity. ## understanding prior observations via the lens of neuron activation in this section, we revisit the major findings from prior work and use the activation of ff neurons in an llm to explain them. for each research question (rq), our analysis will be based on how each cot prompt variant triggers different neuron activation patterns. these observations are summarized in ## conclusions our work is among the first in applying neuron activation analysis to understanding llms in arithmetic reasoning. our results offer valuable insights into the role of neurons and their utility in understanding the internal mechanism of llms. we thus expect this work to pave the way for future research on llm interpretability.