  Automated mathematical reasoning is a challenging problem that requires an
agent to learn algebraic patterns that contain long-range dependencies. Two
particular tasks that test this type of reasoning are (1) mathematical equation
verification, which requires determining whether trigonometric and linear
algebraic statements are valid identities or not, and (2) equation completion,
which entails filling in a blank within an expression to make it true. Solving
these tasks with deep learning requires that the neural model learn how to
manipulate and compose various algebraic symbols, carrying this ability over to
previously unseen expressions. Artificial neural networks, including recurrent
networks and transformers, struggle to generalize on these kinds of difficult
compositional problems, often exhibiting poor extrapolation performance. In
contrast, recursive neural networks (recursive-NNs) are, theoretically, capable
of achieving better extrapolation due to their tree-like design but are
difficult to optimize as the depth of their underlying tree structure
increases. To overcome this issue, we extend recursive-NNs to utilize
multiplicative, higher-order synaptic connections and, furthermore, to learn to
dynamically control and manipulate an external memory. We argue that this key
modification gives the neural system the ability to capture powerful transition
functions for each possible input. We demonstrate the effectiveness of our
proposed higher-order, memory-augmented recursive-NN models on two challenging
mathematical equation tasks, showing improved extrapolation, stable
performance, and faster convergence. Our models achieve a 1.53% average
improvement over current state-of-the-art methods in equation verification and
achieve a 2.22% Top-1 average accuracy and 2.96% Top-5 average accuracy for
equation completion.
