Under the Shadow of B abel: How Language Shapes Reasoning in LLM s (2025)

Abstract: Language is not only a tool for communication but also a medium for human cognition and reasoning. If, as linguistic relativity suggests, the structure of language shapes cognitive patterns, then large language models (LLMs) trained on human language may also internalize the habitual logical structures embedded in different languages. To examine this hypothesis, we introduce BICAUSE, a structured bilingual dataset for causal reasoning, which includes semantically aligned Chinese and English samples in both forward and reversed causal forms. Our study reveals three key findings: (1) LLMs exhibit typologically aligned attention patterns, focusing more on causes and sentence-initial connectives in Chinese, while showing a more balanced distribution in English. (2) Models internalize language-specific preferences for causal components order and often rigidly apply them to atypical inputs, leading to degraded performance, especially in Chinese. (3) When causal reasoning succeeds, model representations converge toward semantically aligned abstractions across languages, indicating a shared understanding beyond surface form. Overall, these results suggest that LLMs not only mimic surface linguistic forms but also internalize the reasoning biases shaped by language. Rooted in cognitive linguistic theory, this phenomenon is for the first time empirically verified through structural analysis of model internals.

## introduction this idea lies at the heart of one of the most debated theories in linguistic history-linguistic relativity. from herder and humboldt in the 18th century to the well-known sapir-whorf hypothesis in the 20th, scholars have long proposed that linguistic structures shape the speaker's perception of reality @xcite . although the hypothesis has been contested, recent empirical work in psycholinguistics-on color terms, spatial reasoning, and temporal concepts-has provided mounting evidence that linguistic variation can influence how humans perceive, infer, and categorize the world. given that human reasoning is expressed through language, and that linguistic structures-especially syntactic patterns-constrain how thoughts are formulated, the cognitive models embedded in different languages may also be encoded within language models trained on them. modern large language models (llms) are primarily trained on internetscale corpora, which inherently reflect the logical structures and reasoning styles of diverse linguistic communities. this leads to a critical question: do different languages lead llms to think differently? and do llms internalize the cognitive biases embedded in different languages? this work addresses a critical gap by systematically examining how llms represent causal reasoning across languages. we introduce bicause, a structured bilingual dataset of semantically and syntactically aligned causal chains in english and chinese-prototypical representatives of fusional and analytic language families. the dataset spans multiple domains, tightly controls for lexical and grammatical variation, and includes both sequential and reversed causal chains, enabling interpretable tests of language-specific biases, which refer to the llms' tendency to apply structural expectations 面包机一旦加热，面包就烤熟，然后香气就扩散 (once toaster heats, bread toasts, then aroma spreads) 香气扩散，是由于面包烤熟，而这源自面包机加热 (aroma spreads, due to bread toasts, which originates from toaster heats) once toaster heats, bread toasts, then aroma spreads aroma spreads, due to bread toasts, which originates from toaster heats figure 1: causal expressions in forward order are common in native chinese, while reversed forms are rare; both are frequent in english. llms internalize such language-specific patterns, leading to divergent reasoning behaviors when processing semantically aligned but structurally different inputs. for chinese, the model over-applies the prior that the initial phrase signals the cause, misallocating attention to [final effect] and failing to infer correctly. in contrast, english models maintain stable attention and accurate predictions under both orders, revealing the deep influence of syntactic structure on model behavior. based on bicause, we develop a fine-grained analytical framework that decomposes each causal chain into 13 interpretable syntactic components and 3 semantic-level causal components. our analysis proceeds along three axes: 1) syntactic attention patterns. we analyze how llms distribute attention to individual syntactic components across languages under the same syntactic structure. we observe that llms exhibit a greater attentional focus on connectives and subjects when processing chinese inputs, whereas verbs receive more attention in the english counterparts. 2) causal structure preferences. we examine the model's attention distribution over causal components across different language-structure combinations. the results show that llms tend to focus more on causal antecedents (causes) in chinese, while paying greater attention to consequences (effects) in english. moreover, we find that chinese-specific causal ordering preferences are internalized and rigidly applied, leading to degraded reasoning accuracy when the input structure deviates from canonical chinese expressions. 3) representation alignment. despite the attention divergence, we find that llms produce highly similar hidden representations for correctly reasoned samples, regardless of language or syntactic structure. this suggests that divergent attention patterns may converge toward a shared, language-agnostic abstraction space-a finding consistent with recent work on multilingual model alignment @xcite @xcite . our main contributions are as follows: 1) we construct a bilingual chinese-english causal reasoning dataset bicause designed for finegrained quantitative analysis. the dataset spans multiple domains and is carefully aligned at both the syntactic and causal component levels, ensuring cross-linguistic comparability and interpretability. 2) to the best of our knowledge, this is the first work that systematically analyzes the internal mechanisms of llms in cross-lingual causal reasoning from both syntactic and semantic component perspectives. our approach provides a new lens for understanding how llms handle structured reasoning across languages. 3) we provide novel empirical evidence that, echoing the claims of linguistic relativity, different linguistic representations not only affect the external reasoning outcomes of llms but also lead to the internalization of language-specific causal expression patterns as stable attention allocation strategies within the model. cross-linguistic performance of llms. multilingual llms show substantial performance gaps across languages, largely due to the dominance of high-resource languages like english in training data and model design @xcite @xcite . @xcite find that llms perform better when prompted in english, and @xcite further show that the syntactic knowledge encoded in these models is also primarily aligned with english. in addition, @xcite further show that language structure affects cross-lingual transferability, with syntactic composition playing a more central role than word order or co-occurrence. to address these issues, researchers have proposed multilingual benchmarks (e.g., @xcite and explored approaches such as structural alignment @xcite , and cross-lingual prompt design @xcite to improve performance in low-resource languages. building on this, we further explore whether language-specific internal differences in llms underlie their cross-lingual performance gaps. ## dataset to analyze the causal reasoning behavior of multilingual llms, we construct a bilingual dataset named bicause (bilingual causal chains). it covers eight domains, including household routine, natural events, school life, healthcare, shopping and retail, workplace activities, public transportation, leisure and recreation. each domain contains 50 samples, resulting in 400 samples in total. details of the data generation process can be found in appendix a. forward causal chains. each sample in the dataset is a 3-step causal chain, following the structure [cause]→[intermediate effect]→[final effect], where cause leads to intermediate effect, and intermediate effect leads to final effect. the english and chinese versions adopt parallel syntactic patterns, where all chinese phrases are semantically equivalent to their english counterparts: ## cross-lingual attention patterns over syntactic components to investigate whether language-specific syntactic preferences are internalized by llms, we focus on analyzing attention patterns. since attention directly governs how a token "sees" others during inference, it provides a window into how the model structurally organizes information and encodes grammatical relationships. in this section, we decompose each chinese and english causal chain into 13 interpretable syntactic components and quantitatively compare attention allocation patterns across languages. our analysis reveals that while the layerwise attention trajectories over corresponding components are highly consistent between chinese and english, the absolute attention weights exhibit clear divergences, suggesting that llms internalize language-specific structural biases and expression preferences. ## cross-lingual attention patterns over causal components in this section, we further examine how multilingual llms allocate attention to different causal components across languages and syntactic structures. building on the component segmentation introduced in section 3, we compute the rcar for each of the three causal components by layer. we additionally introduce the singular vector canonical correlation analysis (svcca) @xcite , a tool for quickly comparing two representations, to measure the overall similarity of layerwise attention patterns across different sentence structures and lanuages. our findings reveal striking differences in attentional focus between chinese and english, even when the underlying causal chains are semantically and syntactically aligned. moreover, within chinese, we observe a rigid transfer of the model's internalized causal ordering preference when confronted with reversed syntactic structures-leading to a drop in reasoning accuracy. in contrast, the english attention distribution appears more adaptive to structural changes, reflecting greater flexibility in its learned causal reasoning schema. ## representation similarity across languages and structures we further analyze the differences in representations at the convergence point of causal information. specifically, for each input sample, we extract the hidden vector at the final token of the causal chain-this position represents the model's state after integrating the causal information @xcite @xcite . for each pair of input forms (e.g., english vs. chinese), we compute the layerwise cosine similarity between the hidden vectors at this position across all layers, considering only samples where both inputs produce correct predictions. as shown in figure 5 , the english-chinese pair exhibits the highest similarity across most layers, indicating that when semantics and structure are aligned, the model's internal reasoning paths are highly synchronized. in contrast, the chinese-reversed chinese pair shows the lowest similarity in the middle layers, suggesting that word order changes disrupt the internal representation more strongly in chinese. this aligns with our earlier findings. notably, all input forms converge to relatively high cosine similarity in the final layers, implying that once the model succeeds in completing the reasoning task, it forms functionally equivalent causal representations, regardless of surface structure. this phenomenon also echoes prior works @xcite . ## conclusion this paper introduces bicause, a structured bilingual dataset designed to support fine-grained analysis of llms' causal reasoning. our findings include: 1. even when semantics and structure are fully aligned, llms exhibit typologically aligned attention patterns; 2. llms internalize languagespecific habitual causal constructions; 3. when reasoning succeeds, models form a shared understanding that goes beyond surface form. these results reveal the language-specific biases of llms in causal reasoning and their impact on performance. future work includes extending bi-cause and advancing research on multilingual interpretability and alignment mechanisms.