Empirical Investigation of Neural Symbolic Reasoning Strategies (2023)

Abstract: Neural reasoning accuracy improves when generating intermediate reasoning steps. However, the source of this improvement is yet unclear. Here, we investigate and factorize the benefit of generating intermediate steps for symbolic reasoning. Specifically, we decompose the reasoning strategy w.r.t. step granularity and chaining strategy. With a purely symbolic numerical reasoning dataset (e.g., A=1, B=3, C=A+3, C?), we found that the choice of reasoning strategies significantly affects the performance, with the gap becoming even larger as the extrapolation length becomes longer. Surprisingly, we also found that certain configurations lead to nearly perfect performance, even in the case of length extrapolation. Our results indicate the importance of further exploring effective strategies for neural reasoning models.

## introduction artificial intelligence researchers have been attempting neural-symbolic integration for a long time (d'avila @xcite . neural models tend to perform better when generating intermediate reasoning steps in addition to the answer. this phenomenon was seen across various reasoning tasks, such as math word problems @xcite @xcite @xcite , commonsense reasoning @xcite , and symbolic reasoning @xcite . however, it is yet unclear which factors in the intermediate step generation bring the benefit. previous studies often used different strategies for step generation in an adhoc manner. to investigate this, we break down the neural reasoning process into two strategies: output strategy and chaining strategy (figure 1 ). the output strategy ( §2.1) determines the granularity of 1 code available at: https://github.com/ao1neko/ reasoning-strategy intermediate reasoning step generation (all at once vs. step-by-step vs. token-by-token). some studies trained the models to generate reasoning steps and a conclusion derived from them at once @xcite @xcite @xcite , some generated a single reasoning step given the input and iterated this process until achieving a conclusion @xcite @xcite , and others iteratively generated sub-goals as well as reasoning steps @xcite . in turn, the chaining strategy ( §2.2) defines the reasoning path direction (shortest path vs. exhaustive path vs. backward path). for example, some studies used a backward chaining process @xcite @xcite , while others adopted exhaustive searches @xcite @xcite . to compare the strategies, we prepared a test bed of numerical reasoning problems in a simplified language (figure 1 ). this format allows for more controlled testing while serving as a necessary condition-should a model fail to solve it, it cannot be expected to adequately generalize to more complex math word problems. we found that both strategies substantially affect the symbolic reasoning performance of neural seq2seq learners. overall, iterative generation outperformed all-at-once outputting, and roughly granular reasoning steps (i.e., shortest-path chaining) lagged behind finely granular steps (i.e., exhaustive and backward chaining). surprisingly, some settings had near-perfect performance even in generalization tests which extrapolate over greater reasoning depths and unseen numbers during training. ## experimental settings problem definition. we evaluated the models' ability to iteratively perform arithmetic operations over given symbols. given a series of equations, the task is to answer the value of a target variable (figure 1 ). each question also has a certain reasoning depth-the number of necessary equations to reach the answer. for example, the depth of the question @xmath0, @xmath1, @xmath2, @xmath3, c? is 3 (@xmath4, @xmath5, @xmath6. each equation defines either an assignment (e.g., @xmath7.g., @xmath8. the addition is mod 100. the question contexts also contain distractors that are not necessary to calculate the answer (e.g., @xmath9. a value assigned to a particular variable is typically referred to in different equations (e.g., @xmath10, @xmath11. numbers, variables, and the ordering of equations are randomly assigned. motivation for using artificial data there are mainly three advantages to this dataset. first, the symbolic format allows easier control of reasoning depth for generalization tests. specifically, we trained a model using instances with shallow (1-5) depths and evaluated them with instances with shallow/deep (1-12) depths. on the other hand, math word problems are harder to control for reasoning depth (e.g., it is not easy to come up with various instances which have a reasoning depth of 10). second, we wanted to avoid the "spurious bias" that natural (math word) texts implicitly bring into the model @xcite @xcite @xcite . third, we assume that our setting is the necessary condition for solving math word problems. it is unreasonable to expect that a model that can't solve this pure numerical reasoning task can solve more complex tasks. in total, we prepared 5k instances for training and 2.4k for testing. ## results models: we used the pre-trained t5-base, t5-large foot_0 @xcite , and bartbase foot_1 @xcite . results of bart-base are in appendix c. note that their pre-defined tokenizers have all the numbers from 0 to 9, and the numerical values in our dataset are divided into digits (e.g., "12" should be "@@1 @@2") in advance, following @xcite . training: the models were first pre-trained using a 10k simple dataset for 30 epochs, then trained with the 5k training set (1k training instances for each reasoning depth.) for 2000 epochs. the experiment setting details are in appendix a. in addition, we prepared 0.2k test instances for each reasoning depth. this pre-training is intended to teach the models primitive operations (i.e., assignment, reference, and addition). the pretraining dataset contains two types of single-depth instances: assign-refer type (e.g., @xmath12,a?) and operate-assign-refer type (e.g., @xmath13, a?). all the results in the paper are averages of the results on three different seeds. ## conclusions we investigated and factorized the reasoning strategy in symbolic numerical reasoning with neural seq2seq models. we found that the combination of step-by-step output and finely granular reasoning leads to successfully performing symbolic reasoning. our results support the potential of neural models for symbolic reasoning.