Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance? (2024)

Abstract: Recent large language models (LLMs) have demonstrated remarkable generalization abilities in mathematics and logical reasoning tasks.Prior research indicates that LLMs pre-trained with programming language data exhibit high mathematical and reasoning abilities; however, this causal relationship has not been rigorously tested. Our research aims to verify which programming languages and features during pre-training affect logical inference performance. Specifically, we pre-trained decoder-based language models from scratch using datasets from ten programming languages (e.g., Python, C, Java) and three natural language datasets (Wikipedia, Fineweb, C4) under identical conditions. Thereafter, we evaluated the trained models in a few-shot in-context learning setting on logical reasoning tasks: FLD and bAbi, which do not require commonsense or world knowledge. The results demonstrate that nearly all models trained with programming languages consistently outperform those trained with natural languages, indicating that programming languages contain factors that elicit logic inference performance. In addition, we found that models trained with programming languages exhibit a better ability to follow instructions compared to those trained with natural languages. Further analysis reveals that the depth of Abstract Syntax Trees representing parsed results of programs also affects logical reasoning performance. These findings will offer insights into the essential elements of pre-training for acquiring the foundational abilities of LLMs.

## introduction recently, large language models (llms) have demonstrated remarkable generalization abilities in downstream tasks. these tasks include not only fundamental natural language processing tasks, such as machine translation and text classification @xcite , as well as advanced tasks, such as mathematics and logical reasoning @xcite . the generalization ability of llms originates from pre-training on large text corpora, such as redpajama @xcite and @xcite . the corpora often contain content from various domains, such as common crawl, github, arxiv, wikipedia, and stackexchange. however, the relationship between each domain of training data and the abilities of llms is not fully understood. prior research has shown that llms pre-trained with programming language data acquire high mathematical and reasoning abilities @xcite @xcite ; however, this causal relationship has not been rigorously tested. specifically, fair comparisons are often not conducted between models trained on programming language data and those trained on natural language data due to differences in the number of training tokens and model sizes, or because the information is unknown as for closed models. in addition, some prior works have finetuned models using a mixture of programming languages, but they have not conducted detailed analyses regarding the effect of each programming language on the performance of downstream tasks @xcite . we conducted experiments to analyze whether models trained solely on a single programming language generalize better to pure logical reasoning tasks compared to models trained on natural language. specifically, we trained gpt2-124m, gpt2-774m, gpt2-1.5b, and llama-774m @xcite from scratch using datasets from ten programming languages (e.g., python, c, java) and three natural language datasets (wikipedia, fineweb, c4) under the same conditions. we then evaluated each trained model in a few-shot in-context learning (icl) setting on two logical reasoning tasks: formal logic deduction (fld) @xcite and babi @xcite , which do not require commonsense or world knowledge. experimental results demonstrate that nearly all models trained on programming languages consistently outperform those trained on natural language in both fld and babi. these results indicate that it is not a particular programming language that affects logical inference performance; rather programming languages as a whole contain factors that elicit logical inference capabilities. we qualitatively analyzed the output of each trained model and found that models trained with programming data exhibit a better ability to follow instructions compared to those trained with natural languages. in other words, the ability to respond in the correct format, along with logical reasoning ability, is necessary, and training with programming data provides models with both abilities. additional experiments have confirmed that these abilities were preserved to some degree even when the commented-out parts were removed from the code. further analysis reveals that the complexity of code syntax, specifically the number of hierarchies, such as loops and conditional statements (e.g., "if"), also affects logical reasoning performance. moreover, we evaluated the trained models on the glue benchmark @xcite , and found significant differences in performance (accuracy / f1) across languages in semantic equivalence judgment tasks. ## conclusion our study rigorously verified that nearly all models trained on individual programming languages consistently achieve the better logical inference performance than those trained solely on natural language datasets in few-shot icl settings. further analysis reveals that an appropriate level of syntax complexity influences logical reasoning performance. additionally, models trained on programming languages exhibit a greater ability to follow instructions compared to those trained on natural language datasets. moreover, dependencies expressed in code significantly contribute to logical reasoning in few-shot icl settings. we hope these findings will offer insights into the essential elements of pre-training for acquiring the foundational abilities of llms.