Learning advanced mathematical computations from examples
Anonymous authors
Paper under double-blind review
Abstract
Using transformers over large generated datasets, we train models to learn mathematical properties of diﬀerential systems, such as local stability, behavior at inﬁnity and controllability.
We achieve near perfect prediction of qualitative characteristics, and good approximations of numerical features of the system.
This demonstrates that neural networks can learn to perform complex computations, grounded in advanced theory, from examples, without built-in mathematical knowledge.
Introduction
Scientists solve problems of mathematics by applying rules and computational methods to the data at hand.
These rules are derived from theory, they are taught in schools or implemented in software libraries, and guarantee that a correct solution will be found.
Over time, mathematicians have developed a rich set of computational tools that can be applied to many problems, and have been said to be “unreasonably eﬀective” (Wigner, 1960).
Deep learning, on the other hand, learns from examples and solves problems by improving a random initial solution, without relying on domain-related theory and computational rules.
Deep networks have proven to be extremely eﬃcient for a large number of tasks, but struggle on relatively simple, rule-driven arithmetic problems (Saxton et al., 2019;
Trask et al., 2018;
Zaremba and Sutskever, 2014).
Yet, recent studies show that deep learning models can learn complex rules from examples.
In natural language processing, models learn to output grammatically correct sentences without prior knowledge of grammar and syntax (Radford et al., 2019), or to automatically map one language into another (Bahdanau et al., 2014; Sutskever et al., 2014).
In mathematics, deep learning models have been trained to perform logical inference (Evans et al., 2018), SAT solving (Selsam et al., 2018) or basic arithmetic (Kaiser and Sutskever, 2015).
Lample and
Charton (2020) showed that transformers can be trained from generated data to perform symbol manipulation tasks, such as function integration and ﬁnding formal solutions of ordinary diﬀerential equations.
In this paper, we investigate the use of deep learning models for complex mathematical tasks involving both symbolic and numerical computations.
We show that models can predict the qualitative and quantitative properties of mathematical objects, without built-in mathematical knowledge.
We consider three advanced problems of mathematics: the local stability and controllability of diﬀerential systems, and the existence and behavior at inﬁnity of solutions of partial diﬀerential equations.
All three problems have been widely researched and have many applications outside of pure mathematics.
They have known solutions that rely on advanced symbolic and computational techniques, from formal diﬀerentiation, Fourier transform, algebraic full-rank conditions, to function evaluation, matrix inversion, and computation of complex eigenvalues.
We ﬁnd that neural networks can solve these problems with a very high accuracy, by simply looking at instances of problems and their solutions, while being totally unaware of the underlying theory.
In one of the quantitative problems where several solutions are possible (predicting control feedback matrix), neural networks are even able to predict diﬀerent solutions that those generated with the mathematical algorithms we used for training.
After reviewing prior applications of deep learning to related areas we introduce the three problems we consider, describe how we generate datasets, and detail how we train our models.
Finally, we present our experiments and discuss their results.
Related work
Applications of neural networks to diﬀerential equations have mainly focused on two themes: numerical approximation and formal resolution.
Whereas most diﬀerential systems and partial diﬀerential equations cannot be solved explicitly, their solutions can be approximated numerically, and neural networks have been used for this purpose (Lagaris et al., 1998; 2000;
Lee and Kang, 1990; Rudd, 2013; Sirignano and Spiliopoulos, 2018).
This approach relies on the universal approximation theorem, that states that any continuous function can be approximated by a neural network with one hidden layer over a wide range of activation functions (Cybenko, 1989;
Hornik et al., 1990; Hornik, 1991; Petersen and Voigtlaender, 2018;
Pinkus, 1999).
This has proven to be especially eﬃcient for high dimensional problems.
For formal resolution, Lample and Charton (2020) proposed several approaches to generate arbitrarily large datasets of functions with their integrals, and ordinary diﬀerential equations with their solutions.
They found that a transformer model (Vaswani et al., 2017) trained on millions of examples could outperform state-of-the-art symbolic frameworks such as
Mathematica or MATLAB (Wolfram-Research, 2019; MathWorks, 2019) on a particular subset of equations.
Their model was used to guess solutions, while veriﬁcation (arguably a simpler task) was left to a symbolic framework (Meurer et al., 2017).
Arabshahi et al.
(2018a;b) proposed to use neural networks to verify the solutions of diﬀerential equations, and found that Tree-LSTMs (Tai et al., 2015) were better than sequential LSTMs (Hochreiter and Schmidhuber, 1997) at generalizing beyond the training distribution.
Other approaches investigated the capacity of neural networks to perform arithmetic operations (Kaiser and Sutskever, 2015; Saxton et al., 2019;
Trask et al., 2018) or to run short computer programs (Zaremba and Sutskever, 2014).
More recently, Saxton et al.
(2019) found that neural networks were good at solving arithmetic problems or at performing operations such as diﬀerentiation or polynomial expansion, but struggled on tasks like prime number decomposition or on primality tests that require a signiﬁcant number of steps to compute.
Unlike the questions considered here, most of those problems can be solved by simple algorithmic computations.
Differential systems and their stability
A diﬀerential system of degree n is a system of n equations of n variables x 1 ( t ) , ..., x n ( t ) , [Equation] i (
[Equation] i = 1 ...n or, in vector form, with x ∈ R n and f : R n → R n ,
[Equation]
Many problems can be set as diﬀerential systems.
Special cases include n-th order ordinary diﬀerential equations (letting x 1 =
y , x 2 = y (cid:48) , ...
x n = y ( n − 1) ), systems of coupled diﬀerential equations, and some particular partial diﬀerential equations (separable equations or equations with characteristics).
Diﬀerential systems are one of the most studied areas of mathematical sciences.
They are found in physics, mechanics, chemistry, biology, and economics as well as in pure mathematics.
Most diﬀerential systems have no explicit solution.
Therefore, mathematicians have studied the properties of their solutions, and ﬁrst and foremost their stability, a notion of paramount importance in many engineering applications.
3. Local stability
Let x e ∈ R n be an equilibrium point, that is, f ( x e ) = 0 .
If all solutions x ( t ) converge to x e when their initial positions x (0) at t = 0 are close enough, the equilibrium is said to be locally stable (see Appendix B for a proper mathematical deﬁnition).
This problem is well known, if f is diﬀerentiable in x e , an answer is provided by the Spectral Mapping Theorem (SMT) (Coron, 2007, Theorem 10.10):
Theorem 3.1.
Let J ( f )( x e ) be the Jacobian matrix of f in x e (the matrix of its partial derivatives relative to its variables).
Let λ be the largest real part of its eigenvalues.
If λ is positive, x e is an unstable equilibrium.
If λ is negative, then x e is a locally stable equilibrium.
Predicting the stability of a given system at a point x e is our ﬁrst problem.
We will also predict λ , which represents the speed of convergence when negative, in a second experiment.
Therefore, to apply the SMT, we need to: diﬀerentiate each function with respect to each variable, obtain the formal Jacobian
[Equation]
2. evaluate J ( x e ) , the Jacobian in x e (a real or complex matrix)
[Equation]
calculate the eigenvalues λ i , i = 1 ...n of J ( [Equation]
e
[Equation]
compute λ = − max(Real
( λ i )) and return the stability (resp.
λ the speed of convergence
[Equation]
441 >
0 → locally stable with decay rate
3. Control theory
One of the lessons of the spectral mapping theorem is that instability is very common.
In fact, unstable systems are plenty in nature (Lagrange points, epidemics, satellite orbits, etc.), and the idea of trying to control them through external variables comes naturally.
This is the controllability problem.
It has a lot of practical applications, including space launch and the landing on the moon, the US Navy automated pilot, or recently autonomous cars (Bernhard et al., 2017; Minorsky, 1930; Funke et al., 2016).
Formally, we are given a system
[Equation]
where x ∈ R n is the state of the system.
We want to ﬁnd a function u ( t ) ∈ R p , the control action, such that, beginning from a position x 0 at t = 0 , we can reach a position x 1 at t =
T (see Appendix B).
The ﬁrst rigorous mathematical analysis of this problem was given by Maxwell (1868), but a turning point was reached in 1963, when Kalman gave a precise condition for a linear system (Kalman et al., 1963), later adapted to nonlinear system:
Theorem 3.2 (Kalman condition) .
Let A = ∂ x f ( x e , u e ) and B = ∂
u f ( x e , u e ) , if
Span { A i Bu : [Equation] m , i
[Equation]
[Equation] n [Equation] (2) then the system is locally controllable around x = x e , u = u e .
When this condition holds, a solution to the control problem that makes the system locally stable in x e is u ( t ) =
u
e + K
( x ( t ) − x e ) (c.f. Coron (2007); Kleinman (1970); Lukes (1968) and appendix D for key steps of the proof), where K is the m × n control feedback matrix: [Equation]
[Equation]
In the non-autonomous case, where f
= f
( x, u, t ) (and A and B ) depends on t , (2) can be replaced by:
[Equation]
where D 0 ( t ) = B ( t ) and D i +1 ( t ) =
D [-
-](cid:48)
i ( t ) − A ( t ) D i ( t ) .
All these theorems make use of advanced mathematical results, such as the Cayley-Hamilton theorem, or LaSalle invariance principle.
Learning them by predicting controllability and computing the control feedback matrix K is our second problem.
To measure whether the system is controllable at a point x e , we need to:
1. diﬀerentiate the system with respect to its internal variables, obtain A ( x, u ) 2.
diﬀerentiate the system with respect to its control variables, obtain B ( x, u ) 3.
evaluate A and B in ( x e , u e ) 4.
calculate the controllability matrix C with (2) (resp. (4) if non-autonomous) 5. calculate the rank d of C , if d = n , the system is controllable 6. (optionally) if d = n , compute the control feedback matrix K with (3) In: f ( x, u )
= (cid:18) sin( x 21 ) + log(1 + x 2) + atan ( ux 1 ) 1+ x 2 − e x 1 x u
, Out:
  n − d =
System is controllable
[Equation]
A step by step derivation of this example is given in Section A of the appendix.
3. Stability of partial differential equations using Fourier Transform
Partial Diﬀerential Equations (PDEs) naturally appear when studying continuous phenomena (e.g. sound, electromagnetism, gravitation).
Over such problems, ordinary diﬀerential systems are not suﬃcient.
Like diﬀerential systems, PDEs seldom have explicit solutions, and studying their stability has many practical applications.
It is also a much more diﬃcult subject, where few general theorems exist.
We consider linear PDEs of the form
[Equation]
where t , x ∈ R n , and u ( t, x ) are time, position, and state.
α =
( α 1 , ..., α n ) ∈ R n is a multi-index and a α are constants.
Famous examples of such problems include the heat equation, transport equations or Schrodinger equation (Evans, 2010).
We want to determine whether a solution u ( t, x ) of (5) exists for a given an initial condition u (0 , x ) = u 0 , and if it tends to zero as t → + ∞ .
This is mathematically answered (see appendix D and Evans (2010); Bahouri et al.
(2011) for similar arguments) by:
Proposition 3.1.
Given u 0 ∈ S (cid:48) ( R n ) , the space of tempered distribution, there exists a solution u ∈ S [-
-](cid:48)
( R n ) if there exists a constant C such that
[Equation]
where (cid:101) u 0 is the Fourier transform of u 0 and f ( ξ ) is the Fourier polynomial associated with the diﬀerential operator D x = (cid:80) |
α |≤ k a α ∂ αx .
In addition, if C > 0 , this solution u ( t, x ) goes to zero when t → + ∞ .
Learning this proposition and predicting, given an input D x and u 0 , whether a solution u exists, if so, whether it vanishes at inﬁnite time, will be our third and last problem.
To predict whether our PDE has a solution under given initial conditions, and determine its behavior at inﬁnity, we need to: ﬁnd the Fourier polynomial f ( ξ ) associated to D x ; ﬁnd the
Fourier transform ˜ u 0 ( ξ ) of u 0 ; minimize f ( ξ ) on F ; output (0,0) if this minimum is inﬁnite, (1,0) is ﬁnite and negative, (1,1) if ﬁnite and positive.
Optionally, output F .
A step by step example is given in Appendix A.
In: D x = 2 ∂ 2 x 0 + 0 . 5 ∂ 2 x 1 + ∂ 4 x 2 − 7 ∂ 2 x 0 ,x 1 − 1 . 5 ∂ x 1 ∂ 2 x 2 ,
Out: (1 , 0) → there exists a solution u ; it does not vanish at t →
+ ∞
Datasets and models
To generate datasets, we randomly sample problems and compute their solutions with mathematical software (Virtanen et al., 2020; Meurer et al., 2017) using the techniques described in Section 3.
For stability and controllability, we generate diﬀerential systems with n equations and n + q variables (i.e. n random functions, q > 0 for controllability).
Following Lample and Charton (2020), we generate random functions by sampling unarybinary trees, and randomly selecting operators, variables and integers for their internal nodes and leaves.
We use + , − , × , /, exp , log , sqrt , sin , cos , tan , sin -1 , cos -1 , tan -1 as operators, and integers between − 10 and 10 as leaves.
When generating functions with n + q variables, we build trees with up to 2( n + q + 1) operators.
Generated trees are enumerated in preﬁx order (normal Polish notation) and converted into sequences of tokens compatible with our models.
Integers and ﬂoating point reals are also represented as sequences: 142 as [INT+, 1 , 4 , 2 ], and 0 .
314 as [FLOAT+, 3 , DOT, 1 , 4 , e,
INT-, 1 ].
A derivation of the size of the problem space is provided in appendix E.
Local stability Datasets for local stability include systems with 2 to 6 equations (in equal proportion).
Functions that are not diﬀerentiable at the equilibrium x e and degenerate systems are discarded.
Since many of the operators we use are undeﬁned at zero, setting x e = 0 would result in biasing the dataset by reducing the frequency of operators like division, square root, or logarithms.
Instead, we select x e with all coordinates equal to 0 . 01 (denoted as x e = [0 . 01] ).
This is, of course, strictly equivalent mathematically to sampling systems with equilibrium at the origin or at any other point.
When predicting overall stability, since stable systems become exponentially rare as dimension increases, we use rejection sampling to build a balanced dataset with 50% stable systems.
When predicting convergence speed, we work from a uniform (i.e. unbalanced) sample.
The value of λ at x e is expressed as a ﬂoating point decimal rounded to 4 signiﬁcant digits.
For this problem, we generate two datasets with over 50 million systems each.
Control theory Datasets for automonous control include systems with 3 to 6 equations, and 4 to 9 variables ( 1 to 3 control variables).
In the non-autonomous case, we generate systems with 2 or 3 equations.
As above, we discard undeﬁned or degenerate systems.
We also skip functions with complex Jacobians in x e (since the Jacobian represents local acceleration, one expects its coordinates to be real).
We have x e = [0 . 5] or [0 . 9] .
In the autonomous case, more than 95% of the systems are controllable.
When predicting controllability, we use rejection sampling to create a balanced dataset.
In the non-autonomous case, we use a uniform sample with 83% controllable cases.
Finally, to predict feedback matrices, we restrict generation to controllable systems and express the matrix as a sequence of ﬂoating point decimals.
All 3 datasets have more than 50 million examples each.
Stability of partial diﬀerential equations using Fourier Transform We generate a diﬀerential operator (a polynomial in ∂ x i ) and an initial condition u 0 .
u 0 is the product of n functions f ( a j x j ) with known Fourier transforms, and d operators exp( ib k x k ) , with 0 ≤ d ≤ 2 n and a j , b k ∈ {− 100 , . .
We calculate the existence of solutions, their behavior when t → + ∞ , and the set of frequencies, and express these three values as a sequence of 2 Booleans and ﬂoating point decimals.
Our dataset is over 50 million examples.
Models and evaluation In all experiments, we use a transformer architecture withattention heads.
We vary the dimension from 64 to 1024 , and the number of layers from 1 to 8 .
We train our models with the Adam optimizer (Kingma and Ba, 2014), a learning rate of 10 − 4 and the learning rate scheduler in Vaswani et al.
(2017), over mini-batches of 1024 examples.
Additional information can be found in appendix F. Training is performed on 8 V100 GPUs with ﬂoat16 operations.
Our qualitative models (predicting stability, controllability and existence of solutions) were trained for about 12 hours, but accuracies close to the optimal values were reached after about 6 hours.
Learning curves for this problem can be found in appendix H. On quantitative models, more training time and examples were needed: 76 hours for convergence speed, 73 hours for control matrices.
Evaluation is performed on held-out validation and test sets of 10000 examples.
We ensure that validation and test examples are never seen during training (given the size of the problem space, this never happens in practice).
Model output is evaluated either by comparing it with the reference solution or using a problem-speciﬁc metric.
Experiments 5. Predicting qualitative properties of differential systems
In these experiments, the model is given n functions f : R n + p → R ( n ∈ { 2 , . . .
, 6 } , p =for stability, p > 0 for controllability)
and is trained to predict whether the corresponding system is stable, resp.
controllable, at a given point x e .
This is a classiﬁcation problem.
To provide a baseline for our results, we use fastText (Joulin et al., 2016), a state-of-the-art text classiﬁcation tool, which estimates, using a bag of words model, the probability of a qualitative feature (stability) conditional to the distribution of tokens and of small ﬁxed sequences (N-grams of up to ﬁve tokens).
Such a model can detect simple correlations between inputs and outputs, such as the impact on stability of the presence of a given operator, or the number of equations in the system. It would also ﬁnd out obvious solutions, due to the speciﬁcs of one problem or glitches in the data generator. FastText was trained over 2 million examples from our dataset (training on a larger set do not improve accuracy).
A 6 -layer transformer with 512 dimensions correctly predicts the system stability in 96 .
4% of the cases.
Since the dataset is balanced, random guessing would achieve 50% .
FastText achieves 60 .
6% , demonstrating that whereas some easy cases can be learnt by simple text classiﬁers, no trivial general solution exists for this dataset.
Prediction accuracy decreases with the degree, but remains high even for large systems (Table 1).
[Table]
For autonomous controllability over a balanced dataset, a 6 -layer transformer withdimensions correctly predicts 97 .
4% of the cases.
The FastText baseline is 70 .
5% , above the 50% chance level.
Whereas accuracy increases with model size (dimension and number of layers), even very small models (dimension 64 and only 1 or 2 layers) achieve performance over 80% , above the FastText baseline (Table 2).
[Table]
For non-autonomous systems, our dataset features systems of degree 2 and 3 , 83% controllable.
FastText achieves 85 .
3% , barely above the chance level of 83% .
This shows that text classiﬁers have diﬃculty handling diﬃcult problems like this one, even in low dimensions.
Our model achieves 99 .
7% accuracy.
Again, small models, that would be unsuitable for natural language processing, achieve near perfect accuracy (Table 3).
[Table]
5. Predicting numerical properties for differential systems
Speed of convergence In these experiments, the model is trained to predict λ , the convergence speed to the equilibrium, up to a certain precision.
Here, we consider predictions to be correct when they fall within 10% of the ground truth.
Further experiments with diﬀerent levels of precision ( 2 , 3 or 4 decimal digits) are provided in Appendix C.
A model with 8 layers and a dimension of 1024 predicts convergence speed with an accuracy of 86 .
6% overall.
While reasonably good results can be achieved with smaller models, the accuracy decrease quickly when model size falls under a certain value [Table] unlike when qualitative properties were predicted.
Table 4 summarizes the results.
[Table]
Control feedback matrices In these experiments, we train the model ( 6 layers,dimensions) to predict a feedback matrix ensuring stability of an autonomous system.
We use two metrics to evaluate accuracy:
1) prediction within 10% of all coeﬃcients in the target matrix K given by (3) and provided in the training set, 2) verifying that the model outputs a correct feedback matrix K 1 , i.e. that all eigenvalues in A + BK 1 have negative real parts.
This makes more mathematical sense, as it veriﬁes that the model provides an actual solution to the control problem (like a diﬀerential equation, a feedback control problem can have many diﬀerent solutions).
Using the ﬁrst metric, 15 . 8% of target matrices K are predicted with less than 10% error.
Accuracy is 50 .
0% for systems with 3 equations, but drops fast as systems becomes larger.
These results are very low, although well above chance level (< 0 .
With the second metric (i.e. the one that actually matters mathematically), we achieve 66 .
5% accuracy, a much better result.
Accuracy decreases with system size, but even degree 6 systems, with 1 × 6 to 3 × 6 feedback matrices, are correctly predicted 41 .
5% of the time.
Therefore, while the model fails to approximate K to a satisfactory level, it does learn to predict correct solutions to the control problem in 66 .
5% of the cases.
This result is very surprising, as it suggests that a mathematical property characterizing feedback matrices might have been learned.
[Table]
5. Predicting qualitative properties of PDEs
In this setting, the model is given a diﬀerential operator D x and an initial condition u 0 .
It is trained to predict if a solution to ∂ t u + D x u = 0 exists and, if so, whether it converges towhen t → + ∞ .
The space dimension (i.e. dimension of x ) is between 2 and 6 .
In a ﬁrst series of experiments the model is only trained to predict the existence and convergence of solutions.
Overall accuracy is 98 .
In a second series, we introduce an auxiliary task by adding to the output the frequency bounds F of u 0 .
We observe it signiﬁcantly contributes to the stability of the model with respect to hyper-parameters.
In particular, without the auxiliary task, the model is very sensitive to the learning rate scheduling and often fails to converge to something better than random guessing.
However, in case of convergence, the model reaches the same overall accuracy, with and without auxiliary task.
Table 6 details the results.
[Table]
Discussion
We studied ﬁve problems of advanced mathematics from widely researched areas of mathematical analysis.
In three of them, we predict qualitative and theoretical features of diﬀerential systems.
In two, we perform numerical computations.
According to mathematical theory, solving these problems requires a combination of advanced techniques, symbolic and numerical, that seem unlikely to be learnable from examples.
Yet, our model achieves more than 95% accuracy on all qualitative tasks, and between 65 and 85% on numerical computations.
When working from synthetic data, a question naturally arises about the impact of data generation on the results of experiments. In particular, one might wonder whether the model is exploiting a defect in the generator or a trivial property of the problems that allows for easier solutions.
We believe this is very unlikely.
First, because our results are consistent over diﬀerent problems, using datasets generated with diﬀerent techniques.
Second, because a trivial solution would be found by the bag of words model we use as a baseline.
And ﬁnally, because we build our datasets by direcly sampling problems from a distribution that includes all possible functions (up to the basis operators and the random number generator). This eliminates the biases that can result from sampling special instances or solutions (Yehuda et al., 2020).
It also means that the training set is an extremely tiny sample of the whole problem space (over the 50 million examples generated, we did not get a single duplicate).
Learning from very large samples often raises questions about overﬁtting and generalisation out of the training distribution.
Due to the size of the problem space, it is very unlikely that the model could memorize a large number of cases and interpolate between them.
Note that because the space of functions from R n to R n has inﬁnite dimension, the universal approximation theorem does not apply here. Note also that for some of our problems (e.g. local stability), mathematical theory states that solutions cannot be obtained by simple interpolation.
To investigate out-of-distribution generalization, we modiﬁed our data generator to produce 10 new test sets for stability prediction.
We changed the distribution of operators and variables, and experimented with systems with longer expressions and more equations than in the training set (see Appendix I). Changes in the distribution of operators and variables have very little impact on accuracy, demonstrating that the model generalizes out of the training distribution. Extending the problem space to systems with longer expressions also results in good performance, which is interesting because we are then generalizing to a larger problem space than the one used for training.
Finally, the model manages a relatively high accuracy ( 78% ) on systems of 6 equations, whereas it was only trained on systems with 2 to 5 equations. This is very surprising, and tends to conﬁrm that some underlying mathematical properties of diﬀerential systems are learned. However, it seems unlikely that the model follows the same mathematical procedure as human solvers.
For instance, problems involving more computational steps, such as non-autonomous controllability, do not result in lower accuracy.
Also, providing at train time intermediate results that would help a human calculator (frequencies for PDE, or Jacobians for stability) does not improve performance.
Understanding how the model ﬁnds solutions would be very interesting, as no simpler solutions than the classical mathematical steps are known. To this eﬀect, we tried to analyze model behavior by looking at the attention heads and the tokens the models focus on when it predicts a speciﬁc sequence (following Clark et al.
Unfortunately, we were not able to extract speciﬁc patterns, and found that each head in the model, from the ﬁrst layer onwards, attends many more tokens than in usual natural language tasks (i.e. attention weights tend to be uniformly distributed). This makes interpretation very diﬃcult. To improve visualization, we tried to reduce the number of hidden states a head can attend by using a top-k on the attention weights, but this deteriorated the performance, and we did not investigate more in this direction. We also ran a sequence-to-sequence model without attention, so that each input equation is mapped to a ﬁxed-sized representation.
We then fed a set of input equations into the model, and used a t-SNE visualization to see whether we can observe clusters of equations. What we observed is mainly that equations with nearby representations have similar length / tokens. However, even embeddings in similar locations can lead to diﬀerent decoded sequences. The relevance of the representations built in the encoder depends on how the computation is split between the encoder and the decoder.
If the decoder does the majority of the work, encoder representations become less meaningful.
These results open many perspectives for transformers in ﬁelds that need both symbolic and numerical computations.
There is even hope that our models could help solve mathematical problems that are still open.
On a more practical level, they sometimes provide fast alternatives to classical solvers.
The algorithmic complexity of transformer inference and classical algorithms for the problems we consider here is discussed in appendix G. However, for the problems in our dataset, the simpler and parallelizable computations used by transformers allow for shorter evaluation times.
Table 7 compares the average time needed to solve one problem, for a trained transformer running on a GPU, and a Python implementation of the algorithms, running on a MacBook Pro.
[Table]
[Table] [Table]  Conclusion In this paper, we show that by training transformers over generated datasets of mathematical problems, advanced and complex computations can be learned, and qualitative and numerical tasks performed with high accuracy.
Our models have no built-in mathematical knowledge, and learn from examples only.
However, solving problems with high accuracy does not mean that our models have learned the techniques we use to compute their solutions.
Problems such as non-autonomous control involve long and complex chains of computations, which some of the smaller models we used could certainly not handle.
Most probably, our models learn shortcuts that allow them to solve speciﬁc problems, without having to learn or understand their theoretical background.
Such a situation is common in everyday life.
Most of us learn and use language without understanding its rules.
On many practical subjects, we have tacit knowledge and know more than we can tell (Polanyi and Sen (2009)).
This may be the way neural networks learn advanced mathematics.
Understanding what these shortcuts are, how neural networks discover them, and how they can impact mathematical practice, is a subject for future research.