Original: O N THE T URING C OMPLETENESS OF M ODERN N EURAL N ETWORK A RCHITECTURES
Revised: O N THE T URING C OMPLETENESS OF M ODERN N EURAL N ETWORK A RCHITECTURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Alternatives to recurrent neural networks, in particular, architectures based on attention or convolutions , have been gaining momentum for processing input sequences.
Revised: Alternatives to recurrent neural networks, in particular, architectures based on attention or convolutions , have been gaining momentum for processing input sequences.

Original: In spite of their relevance , the computational properties of these alternatives have not yet been fully explored.
Revised: In spite of their relevance, the computational properties of these alternatives have not yet been fully explored.

Original: We study the computational power of two of the most paradigmatic architectures exemplifying these mechanisms the
Revised: We study the computational power of two of the most paradigmatic architectures exemplifying these mechanisms: the

Original: Transformer (Vaswani et al., 2017) and the Neural GPU (Kaiser & Sutskever, 2016)
Revised: Transformer (Vaswani et al., 2017) and the Neural GPU (Kaiser & Sutskever, 2016).

Original: We show both models to be Turing complete exclusively based on their capacity to compute and access internal dense representations of the data.
Revised: We show both models to be Turing complete exclusively based on their capacity to compute and access internal dense representations of the data.

Original: In particular, neither the Transformer nor the Neural GPU requires access to an external memory to become Turing complete.
Revised: In particular, neither the Transformer nor the Neural GPU requires access to an external memory to become Turing complete.

Original: Our study also reveals some minimal sets of elements needed to obtain these completeness results
Revised: Our study also reveals some minimal sets of elements needed to obtain these completeness results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is an increasing interest in designing neural network architectures capable of learning algorithms from examples (Graves et al., 2014; Grefenstette et al., 2015; Joulin & Mikolov, 2015; Kaiser & Sutskever, 2016; Kurach et al., 2016; Dehghani et al., 2018).
Revised: There is an increasing interest in designing neural network architectures capable of learning algorithms from examples (Graves et al., 2014; Grefenstette et al., 2015; Joulin & Mikolov, 2015; Kaiser & Sutskever, 2016; Kurach et al., 2016; Dehghani et al., 2018).

Original: A key requirement for any such an architecture is thus to have the capacity of implementing arbitrary algorithms, that is, to be Turing complete .
Revised: A key requirement for any such an architecture is thus to have the capacity of implementing arbitrary algorithms, that is, to be Turing complete .

Original: Turing completeness often follows for these networks as they can be seen as a control unit with access to an unbounded memory; as such, they are capable of simulating any Turing machine.
Revised: Turing completeness often follows for these networks as they can be seen as a control unit with access to an unbounded memory; as such, they are capable of simulating any Turing machine.

Original: On the other hand, the work by Siegelmann & Sontag (1995) has established a different way of looking at the Turing completeness of neural networks.
Revised: On the other hand, the work by Siegelmann & Sontag (1995) has established a different way of looking at the Turing completeness of neural networks.

Original: In particular, their work establishes that recurrent neural networks (RNNs) are Turing complete even if only a bounded number of resources (i.e., neurons and weights) is allowed.
Revised: In particular, their work establishes that recurrent neural networks (RNNs) are Turing complete even if only a bounded number of resources (i.e., neurons and weights) is allowed.

Original: This is based on two conditions: (1) the ability of RNNs to compute internal dense representations of the data, and (2) the mechanisms they use for accessing such representations.
Revised: This is based on two conditions: (1) the ability of RNNs to compute internal dense representations of the data, and (2) the mechanisms they use for accessing such representations.

Original: Hence, the view proposed by Siegelmann & Sontag shows that it is possible to release the full computational power of RNNs without arbitrarily increasing its model complexity.
Revised: Hence, the view proposed by Siegelmann & Sontag shows that it is possible to release the full computational power of RNNs without arbitrarily increasing its model complexity.

Original: Most of the early neural architectures proposed for learning algorithms correspond to extensions of
Revised: Most of the early neural architectures proposed for learning algorithms correspond to extensions of

Original: RNNs – e.g., Neural Turing Machines (Graves et al., 2014) –, and hence they are Turing complete in the sense of Siegelmann & Sontag.
Revised: RNNs – e.g., Neural Turing Machines (Graves et al., 2014) –, and hence they are Turing complete in the sense of Siegelmann & Sontag.

Original: However, a recent trend has shown the beneﬁts of designing networks that manipulate sequences but do not directly apply a recurrence to sequentially process their input symbols.
Revised: However, a recent trend has shown the beneﬁts of designing networks that manipulate sequences but do not directly apply a recurrence to sequentially process their input symbols.

Original: Architectures based on attention or convolutions are two prominent examples of this approach.
Revised: Architectures based on attention or convolutions are two prominent examples of this approach.

Original: In this work we look at the problem of Turing completeness `a la Siegelmann & Sontag for two of the most paradigmatic models exemplifying these features: the Transformer (Vaswani et al., 2017) and the Neural GPU (Kaiser & Sutskever, 2016).
Revised: In this work we look at the problem of Turing completeness `a la Siegelmann & Sontag for two of the most paradigmatic models exemplifying these features: the Transformer (Vaswani et al., 2017) and the Neural GPU (Kaiser & Sutskever, 2016).

Original: The main contribution of our paper is to show that the Transformer and the Neural GPU are Turing completeexclusively based on their capacity to compute and access internal dense representations of the data.
Revised: The main contribution of our paper is to show that the Transformer and the Neural GPU are Turing complete based on their capacity to compute and access internal dense representations of the data.

Original: In particular, neither the Transformer nor the Neural GPU requires access to an external  memory to become Turing complete.
Revised: In particular, neither the Transformer nor the Neural GPU requires access to an external additional memory to become Turing complete.

Original: Thus the completeness holdseven for bounded architectures (bounded number of neurons and parameters).
Revised: Thus the completeness holds for bounded architectures (bounded number of neurons and parameters).

Original: To prove this we assume that internal activations are represented as rational numbers with arbitrary precision, and in both cases the  result follows by simulating standard sequence-to-sequence RNNs.
Revised: To prove this we assume that internal activations are represented as rational numbers with arbitrary precision. For the case of the Transformer we provide a direct simulation of a Turing machine, while for the case of the Neural GPU our result follows by simulating standard sequence-to-sequence RNNs.

Original: Our study also reveals some minimal sets of elements needed to obtain these completeness results.
Revised: Our study also reveals some minimal sets of elements needed to obtain these completeness results.

Original: This might be of help in understanding how such elements can be exploited to obtain full computational power at a minimal computational cost.

Original: Moreover, the computational power of Transformers and of Neural GPUs has been compared in the current literature (Dehghani et al., 2018), but both are only informally used.
Revised: The computational power of Transformers and of Neural GPUs has been compared in the current literature (Dehghani et al., 2018), but both are only informally used.

Original: Our paper provides a formal way of approaching this comparison.
Revised: Our paper provides a formal way of approaching this comparison.

Revised: For the sake of space, we only include sketch of some proofs in the body of the paper.

Original:  Background work The study of the computational power of neural networks can be traced back to McCulloch & Pitts (1943) which established an analogy between neurons with hard-threshold activations and ﬁrst order logic sentences, and Kleene (1956) that draw a connection between neural networks and ﬁnite automata.
Revised: The details for every proof can be found in the appendix. Background work The study of the computational power of neural networks can be traced back to McCulloch & Pitts (1943) which established an analogy between neurons with hard-threshold activations and ﬁrst order logic sentences, and Kleene (1956) that draw a connection between neural networks and ﬁnite automata.

Original: As mentioned earlier, the ﬁrst work showing the Turing completeness of ﬁnite neural networks with linear connections was carried out by Siegelmann & Sontag (1992; 1995).
Revised: As mentioned earlier, the ﬁrst work showing the Turing completeness of ﬁnite neural networks with linear connections was carried out by Siegelmann & Sontag (1992; 1995).

Original: Since being Turing complete does not ensure the ability to actually learn algorithms in practice, there has been an increasing interest in enhancing RNNs with mechanisms for supporting this task.
Revised: Since being Turing complete does not ensure the ability to actually learn algorithms in practice, there has been an increasing interest in enhancing RNNs with mechanisms for supporting this task.

Original: One strategy has been the addition of inductive biases in the form of external memory, being the Neural Turing Machine (NTM) (Graves et al., 2014) a paradigmatic example.
Revised: One strategy has been the addition of inductive biases in the form of external memory, being the Neural Turing Machine (NTM) (Graves et al., 2014) a paradigmatic example.

Original: To ensure that
Revised: To ensure that

Original: NTMs are differentiable, their memory is accessed via a soft attention mechanism (Bahdanau et al., 2014).
Revised: NTMs are differentiable, their memory is accessed via a soft attention mechanism (Bahdanau et al., 2014).

Original: Other examples of architectures that extend RNNs with memory are the Stack-RNN (Joulin & Mikolov, 2015), and the (De)Queue-RNNs (Grefenstette et al., 2015).
Revised: Other examples of architectures that extend RNNs with memory are the Stack-RNN (Joulin & Mikolov, 2015), and the (De)Queue-RNNs (Grefenstette et al., 2015).

Original: By Siegelmann & Sontag’s results, all these architectures are Turing complete.
Revised: By Siegelmann & Sontag’s results, all these architectures are Turing complete.

Original: The Transformer architecture (Vaswani et al., 2017) is almost exclusively based on the attention mechanism, and it has achieved state of the art results on many language-processing tasks.
Revised: The Transformer architecture (Vaswani et al., 2017) is almost exclusively based on the attention mechanism, and it has achieved state of the art results on many language-processing tasks.

Original: While not initially designed to learn general algorithms, a parallel study has advocated the need for enriching its architecture with several new features as a way to learn general procedures in practice (Dehghani et al., 2018).
Revised: While not initially designed to learn general algorithms, Dehghani et al. (2018) have advocated the need for enriching its architecture with several new features as a way to learn general procedures in practice.

Original: This enrichment is motivated by the empirical observation that the original Transformer architecture struggles to generalize to input of lengths not seen during training.
Revised: This enrichment is motivated by the empirical observation that the original Transformer architecture struggles to generalize to input of lengths not seen during training.

Original: We, in contrast, show that the original Transformer architecture is Turing complete, based on different considerations.
Revised: We, in contrast, show that the original Transformer architecture is Turing complete, based on different considerations.

Original: These results do not contradict each other, but show the differences that may arise between theory and practice.
Revised: These results do not contradict each other, but show the differences that may arise between theory and practice.

Revised: For instance, Dehghani et al.

Original: In fact, we  think that both approaches can be complementary as our theoretical results can shed light on what are the intricacies of the original architecture, which aspects of it are candidates for change or improvement, and which others are strictly needed.
Revised: (2018) assume ﬁxed precision, while we allow arbitrary internal precision during computation. We think that both approaches can be complementary as our theoretical results can shed light on what are the intricacies of the original architecture, which aspects of it are candidates for change or improvement, and which others are strictly needed.

Original: For instance, our proof usesadditive hard attention while the Transformer is often trained withmultiplicative soft attention (Vaswani et al., 2017).
Revised: For instance, our proof uses hard attention while the Transformer is often trained with soft attention (Vaswani et al., 2017).

Original: See Section 3.3 for a discussion on these differences.
Revised: See Section 3.3 for a discussion on these differences.

Original: The Neural GPU is an architecture that mixes convolutions and gated recurrences over tridimensional tensors.
Revised: The Neural GPU is an architecture that mixes convolutions and gated recurrences over tridimensional tensors.

Original: It has been shown that NeuralGPUs are powerful enough to learn decimal multiplication from examples (Freivalds & Liepins, 2018), being the ﬁrst neural architecture capable of solving this problem end-to-end.
Revised: It has been shown that NeuralGPUs are powerful enough to learn decimal multiplication from examples (Freivalds & Liepins, 2018), being the ﬁrst neural architecture capable of solving this problem end-to-end.

Original: The similarity of Neural GPUs and cellular automata has been used as an argument to state the Turing completeness of the architecture (Kaiser & Sutskever, 2016;
Revised: The similarity of Neural GPUs and cellular automata has been used as an argument to state the Turing completeness of the architecture (Kaiser & Sutskever, 2016;

Original: Price et al., 2016).
Revised: Price et al., 2016).

Original: Cellular automata are Turing complete (Smith III, 1971; Ollinger, 2012) and their completeness is established assuming an unbounded number of cells.
Revised: Cellular automata are Turing complete (Smith III, 1971; Ollinger, 2012) and their completeness is established assuming an unbounded number of cells.

Original: In the Neural GPU architecture, in contrast, the number of cells that can be used during a computation is proportional to the size of the input sequence (Kaiser & Sutskever, 2016).
Revised: In the Neural GPU architecture, in contrast, the number of cells that can be used during a computation is proportional to the size of the input sequence (Kaiser & Sutskever, 2016).

Original: One can cope with the need for more cells by padding the Neural GPU input with additional (dummy) symbols, as much as needed for a particular computation.
Revised: One can cope with the need for more cells by padding the Neural GPU input with additional (dummy) symbols, as much as needed for a particular computation.

Original: Nevertheless, this is only a partial solution, as for a Turing-complete model of computation, one cannot decide a priori how much memory is needed to solve a particular problem.
Revised: Nevertheless, this is only a partial solution, as for a Turing-complete model of computation, one cannot decide a priori how much memory is needed to solve a particular problem.

Original: Our results in this paper are somehow orthogonal to the previous argument; we show that one can leverage the dense representations of the Neural GPU cells to obtain Turing completeness without requiring to add cells beyond the ones used to store the input.
Revised: Our results in this paper are somehow orthogonal to the previous argument; we show that one can leverage the dense representations of the Neural GPU cells to obtain Turing completeness without requiring to add cells beyond the ones used to store the input.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We assume all weights and activations to be rational numbers of arbitrary precision.
Revised: We assume all weights and activations to be rational numbers of arbitrary precision.

Original: Moreover, we only allow the use of rational functions with rational coefﬁcients.
Revised: Moreover, we only allow the use of rational functions with rational coefﬁcients.

Original: Most of our positive results make use of the piecewise-linear sigmoidal activation function σ : Q → Q , which is deﬁned asσ ( x ) =
Revised: Most of our positive results make use of the piecewise-linear sigmoidal activation function σ : Q → Q , which is deﬁned as

Original: (cid:40)x < 0 , x 0 ≤ x ≤ 1 , x > 1 .
Revised: [Equation]

Original: We are mostly interested in sequence-to-sequence (seq-to-seq) neural network architectures that we next formalize.
Revised: We are mostly interested in sequence-to-sequence (seq-to-seq) neural network architectures that we next formalize.

Original: A seq-to-seq network N receives as input a sequence
Revised: A seq-to-seq network N receives as input a sequence

Original: X = ( x 1 , . . .
Revised: X = ( x 1 , . . .

Original: , x n ) of vectors x i ∈ Q d , for some d > 0 , and produces as output a sequence Y =
Revised: , x n ) of vectors x i ∈ Q d , for some d > 0 , and produces as output a sequence Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y m ) of vectors y i ∈ Q d .
Revised: , y m ) of vectors y i ∈ Q d .

Original: Most of these types of architectures require a seed vector s and some stopping criterion for determining the length of the output.
Revised: Most of these types of architectures require a seed vector s and some stopping criterion for determining the length of the output.

Original: The latter is usually based on the generation of a particular output vector called an end of sequence mark.
Revised: The latter is usually based on the generation of a particular output vector called an end of sequence mark.

Original: In our formalization instead, we allow a network to produce a ﬁxed number r ≥ 0 of output vectors.
Revised: In our formalization instead, we allow a network to produce a ﬁxed number r ≥ 0 of output vectors.

Original: Thus, for convenience we see a general seq-toseq network as a function N such that the value N ( X , s , r ) corresponds to an output sequence of the form Y =
Revised: Thus, for convenience we see a general seq-toseq network as a function N such that the value N ( X , s , r ) corresponds to an output sequence of the form Y =

Original: ( y 1 , y 2 , . . .
Revised: ( y 1 , y 2 , . . .

Original: , y r ) .
Revised: , y r ) .

Original: With this deﬁnition, we can view every seq-to-seq network as a language recognizer of strings as follows.
Revised: With this deﬁnition, we can view every seq-to-seq network as a language recognizer of strings as follows.

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: A seq-to-seq language recognizer is a tuple A
Revised: A seq-to-seq language recognizer is a tuple A

Original: = (Σ , f, N, s , F ) , where Σ is a ﬁnite alphabet, f : Σ → Q d is an embedding function , N is a seq-to-seq network, s ∈ Q d is a seed vector , and F ⊆ Q d is a set of ﬁnal vectors .
Revised: = (Σ , f, N, s , F ) , where Σ is a ﬁnite alphabet, f : Σ → Q d is an embedding function , N is a seq-to-seq network, s ∈ Q d is a seed vector , and F ⊆ Q d is a set of ﬁnal vectors .

Original: We say that A accepts the string w ∈ Σ ∗ , if there exists an integer r ∈ N such that N ( f ( w ) , s , r ) =
Revised: We say that A accepts the string w ∈ Σ ∗ , if there exists an integer r ∈ N such that N ( f ( w ) , s , r ) =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y r ) and y r ∈ F .
Revised: , y r ) and y r ∈ F .

Original: The language accepted by
Revised: The language accepted by

Original: A , denoted by L ( A ) , is the set of all strings accepted by A .
Revised: A , denoted by L ( A ) , is the set of all strings accepted by A .

Original: We impose two additional restrictions over recognizers.
Revised: We impose two additional restrictions over recognizers.

Original: The embedding function f : Σ → Q d should be computed by a Turing machine in time linear w.r.t.
Revised: The embedding function f : Σ → Q d should be computed by a Turing machine in time linear w.r.t.

Original: the size of Σ .
Revised: the size of Σ .

Original: This covers the two most typical ways of computing input embeddings from symbols: the one-hot encoding , and embeddings computed by ﬁxed feed-forward networks.
Revised: This covers the two most typical ways of computing input embeddings from symbols: the one-hot encoding , and embeddings computed by ﬁxed feed-forward networks.

Original: Moreover, the set F should also be recognizable in lineartime; given a vector f , the membership f ∈ F should be decided by a Turing machine working in linear time with respect to the size (in bits) of f .
Revised: Moreover, the set F should also be recognizable in lineartime; given a vector f , the membership f ∈ F should be decided by a Turing machine working in linear time with respect to the size (in bits) of f .

Original: This covers the usual way of checking equality with a ﬁxed end-of-sequence vector.
Revised: This covers the usual way of checking equality with a ﬁxed end-of-sequence vector.

Original: We impose these restrictions to disallow the possibility of cheating by encoding arbitrary computations in the input embedding or the stop condition, while being permissive enough to construct meaningful embeddings and stoping criterions.
Revised: We impose these restrictions to disallow the possibility of cheating by encoding arbitrary computations in the input embedding or the stop condition, while being permissive enough to construct meaningful embeddings and stoping criterions.

Original: Finally, a class N of seq-to-seq neural network architectures deﬁnes the class L N composed of all the languages accepted by language recognizers that use networks in N .
Revised: Finally, a class N of seq-to-seq neural network architectures deﬁnes the class L N composed of all the languages accepted by language recognizers that use networks in N .

Original: From these notions, the formalization of Turing completeness of a class N naturally follows.
Revised: From these notions, the formalization of Turing completeness of a class N naturally follows.

Original: Deﬁnition 2.2.
Revised: Deﬁnition 2.2.

Original: A class N of seq-to-seq neural network architectures is Turing Complete if L N is exactly the class of languages recognized by Turing machines.
Revised: A class N of seq-to-seq neural network architectures is Turing Complete if L N is exactly the class of languages recognized by Turing machines.

Original: Given an input sequence X = ( x 1 , . . .
Revised: Given an input sequence X = ( x 1 , . . .

Original: , x n ) , a seed vector y 0 , and r ∈ N , an encoder-decoder RNN is given by the following two recursions
Revised: , x n ) , a seed vector y 0 , and r ∈ N , an encoder-decoder RNN is given by the following two recursions

Original: [Equation]
Revised: [Equation]

Original: where V , W , U , R are matrices, b 1 and b 2 are vectors, O ( · ) is an output function, and f 1 and fare activations functions.
Revised: where V , W , U , R are matrices, b 1 and b 2 are vectors, O ( · ) is an output function, and f 1 and fare activations functions.

Original: Equation (1) is called the RNN- encoder and (2) the RNN- decoder .
Revised: Equation (2) is called the RNN- encoder and (3) the RNN- decoder .

Original: The next Theorem follows by inspection of the proof by Siegelmann & Sontag (1992; 1995) after adapting it to our formalization of encoder-decoder RNNs.
Revised: The next Theorem follows by inspection of the proof by Siegelmann & Sontag (1992; 1995) after adapting it to our formalization of encoder-decoder RNNs.

Original: Theorem 2.3 (Siegelmann & Sontag (1992; 1995)) .
Revised: Theorem 2.3 (Siegelmann & Sontag (1992; 1995)) .

Original: The class of encoder-decoder RNNs is Turing complete.
Revised: The class of encoder-decoder RNNs is Turing complete.

Original: Turing completeness holds even if we restrict to the class in which R is the zero matrix,
Revised: Turing completeness holds even if we restrict to the class in which R is the zero matrix,

Original: b 1 and b 2 are the zero vector, O ( · ) is the identity function, and f 1 and f 2 are the piecewise-linear sigmoidal activation σ .
Revised: b 1 and b 2 are the zero vector, O ( · ) is the identity function, and f 1 and f 2 are the piecewise-linear sigmoidal activation σ .

Original:  T HE T RANSFORMER ARCHITECTURE
Revised:  T HE T RANSFORMER ARCHITECTURE

Original: In this section we present a formalization of the Transformer architecture (Vaswani et al., 2017), abstracting away from speciﬁc choices of functions and parameters.
Revised: In this section we present a formalization of the Transformer architecture (Vaswani et al., 2017), abstracting away from speciﬁc choices of functions and parameters.

Original: Our formalization is not meant to produce an efﬁcient implementation of the Transformer, but to provide a simple setting over which its mathematical properties can be established in a clean way.
Revised: Our formalization is not meant to produce an efﬁcient implementation of the Transformer, but to provide a simple setting over which its mathematical properties can be established in a formal way.

Original: The Transformer is heavily based on the attention mechanism introduced next.
Revised: The Transformer is heavily based on the attention mechanism introduced next.

Original: Consider a scoring function score : Q d × Q d → Q and a normalization function ρ : Q n → Q n , for d, n > 0 .
Revised: Consider a scoring function score : Q d × Q d → Q and a normalization function ρ : Q n → Q n , for d, n > 0 .

Original: Let us assume that q ∈ Q d , and that K =
Revised: Assume that q ∈ Q d , and that K =

Original: ( k 1 , . . .
Revised: ( k 1 , . . .

Original: , k n ) and V = ( v 1 , . .
Revised: , k n ) and V = ( v 1 , . .

Original: . , v n ) are tuples of elements in Q d .
Revised: . , v n ) are tuples of elements in Q d .

Original: A q -attention over ( K , V ) , denoted by Att( q , K , V ) , is a vector a ∈ Q d deﬁned as follows.
Revised: A q -attention over ( K , V ) , denoted by Att( q , K , V ) , is a vector a ∈ Q d deﬁned as follows.

Original: [Equation]
Revised: [Equation]

Original: Usually, q is called the query , K the keys , and V the values .
Revised: Usually, q is called the query , K the keys , and V the values .

Original: We do not pose any restriction on the scoring and normalization functions, as some of our results hold in general.
Revised: We do not pose any restriction on the scoring and normalization functions, as some of our results hold in general.

Original: We only require the normalization function to satisfy that there is a function f ρ from Q to Q + such that for each x = ( x 1 , . . .
Revised: We only require the normalization function to satisfy that there is a function f ρ from Q to Q + such that for each x = ( x 1 , . . .

Original: , x n ) ∈ Q
Revised: , x n ) ∈ Q

Original: n it is the case that the i -th component ρ i ( x ) of ρ ( x ) is equal to f ρ ( x i ) / (cid:80) nj
Revised: n it is the case that the i -th component ρ i ( x ) of ρ ( x ) is equal to f ρ ( x i ) / (cid:80) nj

Original: =1 f ρ ( x j ) .
Revised: =1 f ρ ( x j ) .

Original: Thus, a in Equation (4) is a convex combination of the vectors in V .
Revised: Thus, a in Equation (5) is a convex combination of the vectors in V .

Original: When proving possibility results,in particular Turing completeness, we will need to pick speciﬁc scoring and normalization functions.
Revised: When proving possibility results, we will need to pick speciﬁc scoring and normalization functions.

Original: A usual choice for the scoring function is a feed forward network with input ( q , k i )and a single neuron as output (Bahdanau et al., 2014), sometimes called additive attention .
Revised: A usual choice for the scoring function is a feed forward network with input ( q , k i ) sometimes called additive attention (Bahdanau et al., 2014).

Original: Another possibility is to use the dot product (cid:104) q , k i (cid:105)  (Vaswani et al., 2017), called multiplicative attention .
Revised: Another possibility is to use the dot product (cid:104) q , k i (cid:105) called multiplicative attention (Vaswani et al., 2017). We use a combination of both: multiplicative attention plus a non linear function.

Original: For the normalization function, softmax is a standard choice.
Revised: For the normalization function, softmax is a standard choice.

Original: Nevertheless, in ourpossibility proofs wemostly use the hardmax function, which is obtained by setting f hardmax ( x i )
Revised: Nevertheless, in our proofs we use the hardmax function, which is obtained by setting f hardmax ( x i )

Original: = 

Revised: =

Original: if x i is the maximum value, and f hardmax ( x i )
Revised: if x i is the maximum value, and f hardmax ( x i )

Original: = 

Revised: =

Original: otherwise.
Revised: otherwise.

Original: Thus, for a vector x in which the maximum value occurs r times, we have that hardmax i ( x ) =
Revised: Thus, for a vector x in which the maximum value occurs r times, we have that hardmax i ( x ) =

Original: 1 r if x i is the maximum value of x , and hardmax i ( x ) = 

Revised: 1 r if x i is the maximum value of x , and hardmax i ( x ) =

Original: otherwise.
Revised: otherwise.

Original: We call it hard attention whenever hardmax is used as normalization function.
Revised: We call it hard attention whenever hardmax is used as normalization function.

Original: As customary, if F : Q d → Q d and X = ( x 1 , x 2 , . . .
Revised: As customary, if F : Q d → Q d and X = ( x 1 , x 2 , . . .

Original: , x n ) , with x i ∈ Q d , we write F ( X ) for the sequence ( F ( x 1 ) , . .
Revised: , x n ) , with x i ∈ Q d , we write F ( X ) for the sequence ( F ( x 1 ) , . .

Original: , F ( x n )) .
Revised: , F ( x n )) .

Original: Transformer Encoder and Decoder A single-layer encoder of the Transformer is a parametric function
Revised: Transformer Encoder and Decoder A single-layer encoder of the Transformer is a parametric function

Original: Enc( X ; θ ) receiving a sequence X =
Revised: Enc( X ; θ ) receiving a sequence X =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x n ) of vectores in Q d and returning a sequence Z = ( z 1 , . . .
Revised: , x n ) of vectores in Q d and returning a sequence Z = ( z 1 , . . .

Original: , z n ) of the same length of vectors in Q d .
Revised: , z n ) of the same length of vectors in Q d .

Original: In general, we consider the parameters in Enc( X ; θ ) as functions Q ( · ) , K ( · ) , V ( · ) , and O ( · ) , all of them from Q d to Q d .
Revised: In general, we consider the parameters in Enc( X ; θ ) as functions Q ( · ) , K ( · ) , V ( · ) , and O ( · ) , all of them from Q d to Q d .

Original: The single-layer encoder is then deﬁned as follows.
Revised: The single-layer encoder is then deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original: In practice Q ( · ) , K ( · ) , V ( · ) are typically matrix multiplications, and O ( · ) a feed-forward network.
Revised: In practice Q ( · ) , K ( · ) , V ( · ) are typically matrix multiplications, and O ( · ) a feed-forward network.

Original: The + x i and + a i summands are usually called residual connections (He et al., 2016a;b).
Revised: The + x i and + a i summands are usually called residual connections (He et al., 2016a;b).

Original: When the particular functions used as parameters are not important, we simply write Z
Revised: When the particular functions used as parameters are not important, we simply write Z

Original: = Enc( X ) .
Revised: = Enc( X ) .

Original: The Transformer encoder is deﬁned simply as the repeated application of single-layer encoders (with independent parameters), plus two ﬁnal transformation functions K ( · ) and V ( · ) applied to every vector in the output sequence of the ﬁnal layer.
Revised: The Transformer encoder is deﬁned simply as the repeated application of single-layer encoders (with independent parameters), plus two ﬁnal transformation functions K ( · ) and V ( · ) applied to every vector in the output sequence of the ﬁnal layer.

Original: Thus the L -layer Transformer encoder is deﬁned by the following recursion (with 1 ≤ (cid:96)
Revised: Thus the L -layer Transformer encoder is deﬁned by the following recursion (with 1 ≤ (cid:96)

Original: ≤ L − 1 and X
Revised: ≤ L − 1 and X

Original: 1 = X ).
Revised: 1 = X ).

Original: [Equation]
Revised: [Equation]

Original: We use ( K , V ) =
Revised: We use ( K , V ) =

Original: TEnc L ( X ) to denote an L -layer Transformer encoder over the sequence X .
Revised: TEnc L ( X ) to denote an L -layer Transformer encoder over the sequence X .

Original: A single-layer decoder is similar to a single-layer encoder but with additional attention to an external pair of key-value vectors ( K e , V e ) .
Revised: A single-layer decoder is similar to a single-layer encoder but with additional attention to an external pair of key-value vectors ( K e , V e ) .

Original: The input for the single-layer decoder is a sequence Y =
Revised: The input for the single-layer decoder is a sequence Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y k ) plus the external pair ( K e , V e ) , and the output is a sequence Z
Revised: , y k ) plus the external pair ( K e , V e ) , and the output is a sequence Z

Original: = ( z 1 , . . .
Revised: = ( z 1 , . . .

Original: , z k ) .
Revised: , z k ) .

Revised: When deﬁning a decoder layer we denote by Y j the sequence ( y 1 , . . .

Original: This layer is also parameterized by four functions Q ( · ) , K ( · ) , V ( · ) and O ( · ) and is deﬁned as follows.
Revised: , y j ) , for 1 ≤ j ≤ k . The layer is also parameterized by four functions Q ( · ) , K ( · ) , V ( · ) and O ( · ) and is deﬁned as follows.

Original: [Equation]
Revised: [Equation]

Original: Notice that the ﬁrst (self) attention over ( K ( Y  ) , V ( Y  ))  is used to generate a query p i to attend the external pair ( K e , V e ) .
Revised: Notice that the ﬁrst (self) attention over ( K ( Y i ) , V ( Y i )) considers the subsequence of Y only until index i and is used to generate a query p i to attend the external pair ( K e , V e ) .

Original: We denote this single-decoder layer by Dec(( K e , V e ) , Y ; θ ) .
Revised: We denote the single-decoder layer by Dec(( K e , V e ) , Y ; θ ) .

Original: The Transformer decoder is a repeated application of single-layer decoders, plus a transformation function F : Q d → Q dwhich is applied over the ﬁnal vector of the decoded sequence.
Revised: The Transformer decoder is a repeated application of single-layer decoders, plus a transformation function F : Q d → Q d applied to the ﬁnal vector of the decoded sequence.

Original: Thus, the output of the decoder is a single vector z ∈ Q d , and not a sequence of them as in the case of the encoder.
Revised: Thus, the output of the decoder is a single vector z ∈ Q d .

Original: Formally, the L -layer Transformer decoder is deﬁned as 
Revised: Formally, the L -layer Transformer decoder is deﬁned as Y [Equation] Dec

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: Y (cid:96)

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: We use z =
Revised: We use z =

Original: TDec L (( K e , V e ), Y ) to denote an L -layer Transformer decoder.
Revised: TDec L (( K e , V e )

Original: The complete Tansformer A Transformer network receives an input sequence X , a seed vector y 0 , and a value r ∈ N .
Revised: The complete Tansformer A Transformer network receives an input sequence X , a seed vector y 0 , and a value r ∈ N .

Original: Its output is a sequence Y =
Revised: Its output is a sequence Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y r ) deﬁned as [Equation] [Equation] [Equation]
Revised: , y r ) deﬁned as y t + = TDec(TEnc( X ) , ( y 0 , y 1 , . . .

Original: [Equation]

Original: . , y t )) , for 0 ≤ t ≤ r − 1 .
Revised: , y t )) , for 0 ≤ t ≤ r − 1 .

Original: We denote the output sequence of the transformer as Y =
Revised: We denote the output sequence of the transformer as Y =

Original: ( y 1 , y 2 , . . .
Revised: ( y 1 , y 2 , . . .

Original: , y r ) =
Revised: , y r ) =

Original: Trans( X , y 0 , r ) .
Revised: Trans( X , y 0 , r ) .

Original: 3. I NVARIANCE UNDER PROPORTIONS
Revised: I NVARIANCE UNDER PROPORTIONS

Original: The Transformer, as deﬁned above, is order-invariant : two input sequences that are permutations of each other produce exactly the same output.
Revised: The Transformer, as deﬁned above, is order-invariant : two input sequences that are permutations of each other produce exactly the same output.

Original: This is a consequence of the followingstraightforward property of the attention function: if K = ( k 1 , . . .
Revised: This is a consequence of the following property of the attention function: if K = ( k 1 , . . .

Original: , k n ) , V = ( v 1 , . . .
Revised: , k n ) , V = ( v 1 , . . .

Original: , v n ) , and π : { 1 , . . . , n } → { 1 , . . .
Revised: , v n ) , and π : { 1 , . . . , n } → { 1 , . . .

Original: , n } is a permutation, then Att( q , K , V ) =
Revised: , n } is a permutation, then Att( q , K , V ) =

Original: Att( q , π ( K ) , π ( V )) for every query q .
Revised: Att( q , π ( K ) , π ( V )) for every query q .

Original: This weakness has motivated the need for including information about the order of the input sequence by other means; in particular, this is often achieved by using the so-called positional encodings (Vaswani et al., 2017; Shaw et al., 2018), which we study below.
Revised: This weakness has motivated the need for including information about the order of the input sequence by other means; in particular, this is often achieved by using the so-called positional encodings (Vaswani et al., 2017; Shaw et al., 2018), which we study below.

Original: But before going into positional encodings, a natural question is what languages the Transformer can recognize without them.
Revised: But before going into positional encodings, a natural question is what languages the Transformer can recognize without them.

Original: As a standard yardstick we use the well-studied class of regular languages, i.e., languages recognized by ﬁnite automata.
Revised: As a standard yardstick we use the well-studied class of regular languages, i.e., languages recognized by ﬁnite automata.

Original: Order-invariance implies that not every regular language can be recognized by a Transformer network.
Revised: Order-invariance implies that not every regular language can be recognized by a Transformer network.

Original: As an example, there is no Transformer network that can recognize the regular language ( ab ) ∗ , as the latter is not order-invariant.
Revised: As an example, there is no Transformer network that can recognize the regular language ( ab ) ∗ , as the latter is not order-invariant.

Original: A reasonable question then is whether the Transformer can express all regular languages which are order-invariant.
Revised: A reasonable question then is whether the Transformer can express all regular languages which are order-invariant.

Original: It is possible to show that this is not the case by proving that the Transformer actually satisﬁes a stronger invariance property, which we call proportion invariance .
Revised: It is possible to show that this is not the case by proving that the Transformer actually satisﬁes a stronger invariance property, which we call proportion invariance .

Original: For a string w ∈ Σ ∗ and a symbol a ∈ Σ , we use prop( a, w ) to denote the ratio between the number of times that a appears in w and the length of w .
Revised: For a string w ∈ Σ ∗ and a symbol a ∈ Σ , we use prop( a, w ) to denote the ratio between the number of times that a appears in w and the length of w .

Original: Consider now the set PropInv( w ) = { u ∈ Σ ∗ | prop( a, w ) =
Revised: Consider now the set PropInv( w ) = { u ∈ Σ ∗ | prop( a, w ) =

Original: prop( a, u ) for every a ∈ Σ } .
Revised: prop( a, u ) for every a ∈ Σ } .

Original: Then:

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Let Trans be a Transformer, s a seed, r ∈ N , and f : Σ → Q d an embedding function.
Revised: Let Trans be a Transformer, s a seed, r ∈ N , and f : Σ → Q d an embedding function.

Original: Then Trans( f ( w ) , s , r ) =
Revised: Then Trans( f ( w ) , s , r ) =

Original: Trans( f ( u ) , s , r ) , for each u, w ∈ Σ ∗ with u ∈ PropInv( w ) .
Revised: Trans( f ( u ) , s , r ) , for each u, w ∈ Σ ∗ with u ∈ PropInv( w ) .

Original: As an immediate corollary we obtain the following.
Revised: As an immediate corollary we obtain the following.

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: Consider the order-invariant regular language L = { w ∈ { a, b } ∗ | w has an even number of a symbols } .
Revised: Consider the order-invariant regular language L = { w ∈ { a, b } ∗ | w has an even number of a symbols } .

Original: Then L cannot be recognized by a Transformer network.
Revised: Then L cannot be recognized by a Transformer network.

Original: On the other hand, languages recognized by Transformer networks are not necessarily regular.
Revised: On the other hand, languages recognized by Transformer networks are not necessarily regular.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: There is a Transformer network that recognizes the non-regular language S = { w ∈ { a, b } ∗ | w has strictly more symbols a than symbols b } .
Revised: There is a Transformer network that recognizes the non-regular language S = { w ∈ { a, b } ∗ | w has strictly more symbols a than symbols b } .

Original: That is, the computational power of Transformer networks without positional encoding is both rather weak (they do not even contain order-invariant regular languages) and not so easy to capture (as they can express counting properties that go beyond regularity).
Revised: That is, the computational power of Transformer networks without positional encoding is both rather weak (they do not even contain order-invariant regular languages) and not so easy to capture (as they can express counting properties that go beyond regularity).

Original: As we show in the next section, the inclusion of positional encodings radically changes the picture.
Revised: As we show in the next section, the inclusion of positional encodings radically changes the picture.

Original: In fact, with such positional encodings the Tranformer is able to recognize all computable languages.

Original: 3. P OSITIONAL E NCODINGS AND C OMPLETENESS OF THE T RANSFORMER
Revised: 3. P OSITIONAL E NCODINGS AND C OMPLETENESS OF THE T RANSFORMER

Original: Order-invariance states that Transformer networks lack information about how the input is ordered.

Original: Positional encodings come to remedy this issue by providing information about the absolute positions of the symbols in the input.
Revised: Positional encodings come to remedy the order-invariance issue by providing information about the absolute positions of the symbols in the input.

Original: A positional encoding is just a function pos : N → Q d .
Revised: A positional encoding is just a function pos : N → Q d .

Original: Function pos combined with an embedding function f : Σ → Q d give rise to a new embedding function f pos : Σ × N → Q d such that f pos ( a, i ) =
Revised: Function pos combined with an embedding function f : Σ → Q d give rise to a new embedding function f pos : Σ × N → Q d such that f pos ( a, i ) =

Original: f ( a ) + pos
Revised: f ( a ) + pos

Original: ( i ) .
Revised: ( i ) .

Original: Thus, given an input string w = a 1 a 2 · · · a n ∈ Σ , the result of the embedding function f pos ( w ) provides a “new” input (cid:0) f pos ( a 1 , 1) , f pos ( a 2 , 2) , . . .
Revised: Thus, given an input string w = a 1 a 2 · · · a n ∈ Σ , the result of the embedding function f pos ( w ) provides a “new” input (cid:0) [Equation] [Equation]

Original: , f pos ( a n , n ) (cid:1) to the Transformer encoder.
Revised: to the Transformer encoder.

Original: Similarly, the Transformer decoder instead of receiving the sequence
Revised: Similarly, the Transformer decoder instead of receiving the sequence

Original: Y =
Revised: Y =

Original: ( y 0 , y 1 , . .
Revised: ( y 0 , y 1 , . .

Original: . , y t ) as input, it receives now the sequence
Revised: . , y t ) as input, it receives now the sequence

Original: [Equation]
Revised: [Equation]

Original: As for the case of the embedding functions, we require the positional encoding pos( i ) to be computable by a Turing machine working in linear time w.r.t.
Revised: As for the case of the embedding functions, we require the positional encoding pos( i ) to be computable by a Turing machine working in linear time w.r.t.

Original: the size (in bits) of i .
Revised: the size (in bits) of i .

Original: The main result of this section is the completeness of Transformers with positional encodings.
Revised: The main result of this section is the completeness of Transformers with positional encodings.

Revised: # q read sq read s n q read # → q read s n → q read s 1 → q read s n n q read s q read # q read sq read s

Original: Theorem 3.4.
Revised:  q init s ( t − 1) ... Att Att (cid:122) (cid:125)(cid:124) (cid:123) [Figure]

Revised: index of cell (cid:122) (cid:125)(cid:124) (cid:123) computes

Revised: [Figure]

Revised: symbol at the

Revised: [Figure]

Revised: step ( t +1 ) written at cell c ( t +1) (cid:122) (cid:125)(cid:124) (cid:123) implements M ’s transition function v ( t − 1) q ( t ) c ( t ) ...

Revised:   attends to the encoder and copies the corresponding symbol uses self attention to compute next state and the next symbol under M ’s head m ( t − 1) q ( t ) c ( t +1) q ( t ) s ( t ) v ( t ) m ( t ) q

Revised: ( t +1)

Revised: s ( t +1) q ( t ) s ( t ) q ( t − 1) q

Original: The class of Transformer networks with positional encodings is Turing complete.
Revised: [Figure] q ( t +1) [Figure] ( t − 1)

Original: Proof Sketch.
Revised: [Figure]

Original: For the sake of space we only present a sketch of the proof. The full proof can be found in the appendix. Let A = (Σ , f, N, s , F ) be an RNN encoder-decoder language recognizer, such that N is of dimension d and its encoder and decoder are deﬁned by the equations
Revised: [Figure]

Original: h i = σ ( x i W + h i − 1 V ) and g t = σ ( g t − 1 U ) , respectively, where g 0 = h n and n is the length of the input.
Revised: [Figure]

Original: We explain how to construct a Transformer network A (cid:48)

Original: = (Σ , f (cid:48) , T, s (cid:48) , F (cid:48) ) such that L ( A ) = L ( A (cid:48) ) .
Revised: [Figure]

Original: By Theorem 2.3 this yields Turing-completeness for Transformer networks.
Revised: Theorem 3.4. The class of Transformer networks with positional encodings is Turing complete.

Original: We use a simple positional encoding pos( i ) =
Revised: Proof Sketch.

Original: [ 0 , 0 , 0 , i ] ∈ Q 3 d +1 , where 0 is the vector composed exclusively by 0 ’s. For each a ∈ Σ we deﬁne f (cid:48) ( a ) =
Revised: We show that for every Turing machine M = ( Q, Σ , δ, q init , F ) there exists a transformer that simulates the complete execution of M .

Original: [ f ( a ) , 0 , 0 , 0] . Therefore, if for an input w ∈ Σ ∗ it is the case that f ( w) = ( x 1 , . . . 
Revised: We represent a string w = s 1 s 2 . . . s n ∈ Σ ∗ as a sequence X of one-hot vectors with their corresponding positional encodings.

Original: , x n ) , then f (cid:48) pos ( w ) = ( x 1 , . . . , x n ) , where x i = [ x i , 0 , 0 , i ] .
Revised: Denote by q ( t ) ∈ Q the state of M at time t when processing w , and s ( t ) ∈ Σ the symbol under M ’s head at time t .

Original:  We  construct a Tranformer encoder which consists of a single-layer that takes each input vector x i = [ x i , 0 , 0 , i ] and produces k i = [ 0 , 0 , 0 , i ] as keys and v i =
Revised: Similarly, v ( t ) ∈ Σ is the symbol written by M and m ( t ) ∈ {← , →} the head direction. We next describe how to construct a transformer Trans M that with input X produces a sequence y 0 , y 1 , y 2 , . . .

Original: [ 0 , x i , 0 , 0] as values. The ﬁrst n steps of the Transformer decoder emulate the work of the RNN encoder. We maintain the following invariant: for every i ≤ n the i -th output of the decoder is y i = [ 0 , h i , 0 , 0] .
Revised: such that y i contains information about q ( i ) and s ( i ) (encoded as one-hot vectors).

Original: We prove this by an inductive argument.
Revised: The construction and proof goes by induction.

Original: Let us assume by induction hypothesis that y i =
Revised: Assume the decoder receives y 0 , . .

Original: [ 0 , h i , 0 , 0] , and thus thatits positional encoding is y i = [ 0 , h i , 0 , i + 1] .
Revised: , y t such that y i contains q ( i ) and s ( i ) .

Original: We omit the ﬁrst self attention (Equation (8)) so that p i = y i , and use p i to attend – exactly – at the  ( i + 1) -th pair (k i [-+
-]
Revised: To construct y ( t +1) , in the ﬁrst layer we just implement M ’s transition function δ ; note that δ ( q ( i ) , s ( i [-+
-]) )

Original: =
Revised: =

Original: [ 0 , 0 , 0 ,  i + 1] , v  i +1 = [ 0 , x i +1 , 0 , 0]) of the Transformer encoder.
Revised: ( q ( i +1) , v ( i ) , m ( i ) ) thus, we use ( q ( i ) , s ( i ) ) to compute ( q ( i +1) , v ( i ) , m ( i ) ) for every i and store them in the sequence z 10 , . . . , z 1 t .

Original: This  can be done with a  feed-forward network with three layers plus hard attention.
Revised: This computation can be done with a two-layer feed-forward network.

Original: Hence, a i in Equation (9) will be v i [-+
 + p-] i 
Revised: For the next layer, lets denote by c ( i [-+
 + p-]) the index of the cell that M is pointing to at time i .

Original: = [ x i +1 , h i , 0 ,i + 1] . We then deﬁne the O ( · ) function in Equation (10) in such a way that O
Revised: It can be proved that given z 10 , . . . , z 1 t one can compute (a representation of) c ( i ) and c

Original: ([ x i +1 , h i , 0 , i + 1]) = [ 0 , 0 , σ ( x i +1 W + h
Revised: ( i +1) for every i ≤ t with a self-attention layer, and store them in z 20 , . . .

Original: i V ) , 0]
Revised: , z 2 t .

Original: = [ 0 , 0 , h i +1 , 0] , and thus z i = O ( a i ) + a i
Revised: In particular, z 2 t contains c ( t +1) which is the index to which M is going to be pointing to in the next time step.

Original: =
Revised: By using the residual connections we also store q

Original: [ x i +1 , h i , h i +1 , i+ 1] .
Revised: ( i +1) and v ( i ) in z 2 i .

Original: We produce y i [-+
-]
Revised: The ﬁnal piece of our construction is to compute the symbol that the tape holds at index c ( t +1) , that is, the symbol under M ’s head at time t + 1 . For this we use the following observation: the symbol at index c ( t +1) in time t + 1 coincides with the last symbol written by M at index c ( t +1) . Thus, we need to ﬁnd the maximum value i [-+
-](cid:63) ≤ t such that c ( i (cid:63) )

Original: =
Revised: =

Original: [ 0 , h i +1 , 0 , 0] as the output of another decoder layer and use function F ( · ) in Equation (11) as the identity to produce the ﬁnal output.
Revised: c ( t +1) and then copy v ( i (cid:63) ) which is the symbol that was written by M at time step i (cid:63) .

Original: Simulating the decoder of the RNN N , on the other hand, is a bit more cumbersome. We need to extend the previous construction so that the Transformer decoder can stop reading the input and begin implementing the recursion g t = σ ( g t − 1 U ) . For this, we maintain a [-state value b i in the decoder such that b i = 
-]
Revised: This last computation can also be done with a [-state value b i in the decoder such that b i = 
-]self-attention layer.

Original: if i = n + 1 , and b i = 1 otherwise. We do this by combining an attention over the key-value pairs ( k i , v  i  ) generated by the Transformer encoder together with a self-attention in the Transformer decoder as follows. The attention we use over the encoder satisﬁes that, whenever one tries to attend over an index bigger than n , it yields the same value, say v ∗ . On the other hand, the self-attention over the decoder decides if the value obtained from the encoder in this iteration is the same as in the previous one.
Revised: Thus, we attend directly to position i (cid:63) (hard attention plus positional encodings) and copy v ( i (cid:63) ) which is exactly s ( t +1) .

Original: In this way we can generate a sequence a 1 , a 2 , a 3 , . . . of the form 000 · · · 0111 · · · that changes from 0 to 1 exactly when i = n + 1 .
Revised: We ﬁnally copy q ( t +1) and s ( t +1) into the output to construct y t +1 .

Original: With a decoder layer we produce the desired b i simply as 1 − ( a i − a i − 1 ) . We maintain another vector δ i and set δ i : = σ ( h i − 1 − b i 1 ) .
Revised: Figure 1 shows a high-level diagram of the decoder computation.

Original: Since 0 ≤ h i ≤ 1 for every i , we have that δ i = h n exactly when i = n + 1 , and δ i = 0 otherwise. Combining this with the idea developed in  the previous paragraph for simulating the encoder of
Revised: There are several other details in the construction, in particular, at the beginning of the computation (ﬁrst n steps), the decoder needs to attend to the encoder and copy the input symbols so they can later be processed as described above.

Original: N , we can maintain yet another vector γ i deﬁned as γ i := σ ( γ i − 1 U ) + δ i . We obtain then that g i =
Revised: Another detail is when M reaches a cell that has not been visited before, then the symbol under the head has to be set as # (the blank symbol).

Original: [-γ n +
 + i , which implies-] that we can also simulate the decoder part of N .
Revised: [-γ n +
 + i , which implies-]We show that all these decisions can be implemented with feed-forward networks plus attention.

Original: Putting it all together, we have that the Transformer needs a vector in Q 6 d +8 to store the vectors that simulate h i and g i , the positional encoding, the values b i , a i , and some additional working space. 
Revised: The complete construction uses one encoder layer, three decoder layers and vectors of dimension d = 2 | Q | +4 | Σ | +11 to store one-hot representations of states, symbols and some additional working space. All details can be found in the appendix.

Original: 3. D IFFERENCES WITH V ASWANI ET AL . (2017)’ S FRAMEWORK
Revised: 3. D IFFERENCES WITH V ASWANI ET AL . (2017)’ S FRAMEWORK

Original: Although the general architecture that we presented closely follows that of Vaswani et al.
Revised: Although the general architecture that we presented closely follows that of Vaswani et al.

Original: (2017), some choices for functions and parameters in our positive results are different to the usual choices in practice.
Revised: (2017), some choices for functions and parameters in our positive results are different to the usual choices in practice.

Original: For instance, we useadditive hard attention which allow us to attend directly to speciﬁc positions.
Revised: For instance, we use hard attention which allow us to attend directly to speciﬁc positions.

Original: In contrast, Vaswani et al.
Revised: In contrast, Vaswani et al.

Original: (2017) usedot-product for scores, softmax to attend, plus sin - cos functions as positional encodings.
Revised: (2017) use softmax to attend, plus sin - cos functions as positional encodings.

Original: The softmax , sin and cos are not rational functions, and thus, are forbidden in our formalization.
Revised: The softmax , sin and cos are not rational functions, and thus, are forbidden in our formalization.

Original: An interesting line for future work is to consider arbitrary functions but with additional restrictions, such as ﬁnite precision as done by Weiss et al.
Revised: An interesting line for future work is to consider arbitrary functions but with additional restrictions, such as ﬁnite precision as done by Weiss et al.

Original: Another difference is that for the function O ( · ) in Equation (10) our proof uses a feed-forward network with various layers, while in Vaswani et al.
Revised: Another difference is that for the function O ( · ) in Equation (11) our proof uses a feed-forward network with various layers, while in Vaswani et al.

Original: (2017) only two layers are used.
Revised: (2017) only two layers are used.

Original: But perhaps the most important difference is the use of function F ( · ) in Equation (11).

Original: Vaswani et al.

Original: (2017) uses a function F from Q d to { 0 , 1 } d ; actually, a one-hot function produced by a softmax .

Original: This one-hot vector is then passed back as input for the next iteration of the decoder.

Original: Although the function we use is simpler (we use the identity), it allows us to pass more information to the next decoder step.

Original: We leave as an open question whether the Transformer remains Turing complete when a one-hot output function is used instead of the identity function that we use in our proof.

Original: The need of arbitrary precision Our Turing-complete proof relies on having arbitrary precision for internal representations.
Revised: The need of arbitrary precision Our Turing-complete proof relies on having arbitrary precision for internal representations, in particular, for storing and manipulating positional encodings.

Original: Although this is a standard assumption when studying the expressive power of neural networks (Cybenko (1989); Siegelmann & Sontag (1995)) practical implementations rely on ﬁxed precision hardware.
Revised: Although having arbitrary precision is a standard assumption when studying the expressive power of neural networks (Cybenko (1989); Siegelmann & Sontag (1995)) practical implementations rely on ﬁxed precision hardware.

Original: If ﬁxed precision is used, then positional encodings can be seen as functions of the form pos : N → A where A is a ﬁnite subset of Q d .
Revised: If ﬁxed precision is used, then positional encodings can be seen as functions of the form pos : N → A where A is a ﬁnite subset of Q d .

Original: Thus, the embedding function f pos can be seen as a regular embedding function f (cid:48) : Σ (cid:48)
Revised: Thus, the embedding function f pos can be seen as a regular embedding function f (cid:48) : Σ (cid:48)

Original: → Q d where Σ (cid:48) =
Revised: → Q d where Σ (cid:48) =

Original: Σ × A .
Revised: Σ × A .

Original: Thus, whenever ﬁxed precision is used, the net effect of having positional encodings is to just increase the size of the input alphabet.
Revised: Thus, whenever ﬁxed precision is used, the net effect of having positional encodings is to just increase the size of the input alphabet.

Original: Then from Proposition 3.1 we obtain that the Transformer with positional encodings and ﬁxed precision is not Turing complete.
Revised: Then from Proposition 3.1 we obtain that the Transformer with positional encodings and ﬁxed precision is not Turing complete.

Original: Although no longer Turing complete, one can still study the computational power of ﬁxed-precision Transformers.
Revised: Although no longer Turing complete, one can still study the computational power of ﬁxed-precision Transformers.

Original: We left this as future work.
Revised: We left this as future work.

Original:  N EURAL GPU S
Revised:  N EURAL GPU S

Original: The Neural GPU (Kaiser & Sutskever, 2016) is an architecture that mixes convolutions and gated recurrences over tridimensional tensors.
Revised: The Neural GPU (Kaiser & Sutskever, 2016) is an architecture that mixes convolutions and gated recurrences over tridimensional tensors.

Original: It is parameterized by three functions U ( · ) (update function), R ( · ) (reset function), and F ( · ) .
Revised: It is parameterized by three functions U ( · ) (update function), R ( · ) (reset function), and F ( · ) .

Original: Given a tensor S ∈ Q h × w × d and a value r ∈ N , it produces a sequence S 1 , S 2 , . . .
Revised: Given a tensor S ∈ Q h × w × d and a value r ∈ N , it produces a sequence S 1 , S 2 , . . .

Original: , S r given by the following recursive deﬁnition (with S 0 = S ).
Revised: , S r given by the following recursive deﬁnition (with S 0 = S ).

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes the element-wise product, and 1 is a tensor with only 1 ’s. Neural GPUs force functions U ( · ) and R ( · ) to produce a tensor of the same shape as its input with all values in [0 , 1] .
Revised: where (cid:12) denotes the element-wise product, and 1 is a tensor with only 1 ’s. Neural GPUs force functions U ( · ) and R ( · ) to produce a tensor of the same shape as its input with all values in [0 , 1] .

Original: Thus, a Neural GPU resembles a gated recurrent unit (Cho et al., 2014), with U working as the update gate and R as the reset gate.
Revised: Thus, a Neural GPU resembles a gated recurrent unit (Cho et al., 2014), with U working as the update gate and R as the reset gate.

Original: Functions U ( · ) , R ( · ) , and F ( · ) are deﬁned as a convolution of its input with a 4-dimensional kernel bank with shape ( k H , k W , d, d ) plus a bias tensor, followed by a point-wise transformation
Revised: Functions U ( · ) , R ( · ) , and F ( · ) are deﬁned as a convolution of its input with a 4-dimensional kernel bank with shape ( k H , k W , d, d ) plus a bias tensor, followed by a point-wise transformation

Original: [Equation]
Revised: [Equation]

Original: 13) with different kernels and biases for U ( · ) , R ( · ) , and F ( · ) .
Revised: with different kernels and biases for U ( · ) , R ( · ) , and F ( · ) .

Original: To have an intuition on how the convolution K ∗ S works, it is illustrative to think of S as an ( h × w ) -grid of (row) vectors and K as a ( k H × k W ) -grid of ( d × d ) matrices.
Revised: To have an intuition on how the convolution K ∗ S works, it is illustrative to think of S as an ( h × w ) -grid of (row) vectors and K as a ( k H × k W ) -grid of ( d × d ) matrices.

Original: More speciﬁcally, let s ij =
Revised: More speciﬁcally, let s ij =

Original: S i,j, : , and K ij =
Revised: S i,j, : , and K ij =

Original: K i,j, : , : , then K ∗ S is a regular two-dimensional convolution in which scalar multiplication has been replaced by vector-matrix multiplication as in the following expression
Revised: K i,j, : , : , then K ∗ S is a regular two-dimensional convolution in which scalar multiplication has been replaced by vector-matrix multiplication as in the following expression

Original: [Equation]
Revised: [Equation]

Original: where ∆ 1 ( u ) = u − (cid:98) k H / 2 (cid:99) − 1 and ∆ 2 ( v ) = v − (cid:98) k W / 2 (cid:99) − 1 .
Revised: where ∆ 1 ( u ) = u − (cid:98) k H / 2 (cid:99) − 1 and ∆ 2 ( v ) = v − (cid:98) k W / 2 (cid:99) − 1 .

Original: This intuition makes evident the similarity between Neural GPUs and cellular automata: S is a grid of cells, and in every iteration each cell is updated considering the values of its neighbors according to a ﬁxed rule given by
Revised: This intuition makes evident the similarity between Neural GPUs and cellular automata: S is a grid of cells, and in every iteration each cell is updated considering the values of its neighbors according to a ﬁxed rule given by

Original: K (Kaiser & Sutskever, 2016).
Revised: K (Kaiser & Sutskever, 2016).

Original: As customary, we assume zero-padding when convolving outside S .
Revised: As customary, we assume zero-padding when convolving outside S .

Original: 4. T HE COMPUTATIONAL POWER OF N EURAL GPU S
Revised: 4. T HE COMPUTATIONAL POWER OF N EURAL GPU S

Original: To study the computational power of Neural GPUs, we cast them as a standard seq-to-seq architecture.
Revised: To study the computational power of Neural GPUs, we cast them as a standard seq-to-seq architecture.

Original: Given an input sequence, we put every vector in the ﬁrst column of the tensor S .
Revised: Given an input sequence, we put every vector in the ﬁrst column of the tensor S .

Original: We also need to pick a special cell of S as the output cell from which we read the output vector in every iteration.
Revised: We also need to pick a special cell of S as the output cell from which we read the output vector in every iteration.

Original: We pick the last cell of the ﬁrst column of S .
Revised: We pick the last cell of the ﬁrst column of S .

Original: Formally, given a sequence X
Revised: Formally, given a sequence X

Original: =
Revised: =

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x n ) with x i ∈ Q d , and a ﬁxed value w ∈ N , we construct the tensor S ∈ Q n × w × d by leting S i, 1 , : = x i and
Revised: , x n ) with x i ∈ Q d , and a ﬁxed value w ∈ N , we construct the tensor S ∈ Q n × w × d by leting S i, 1 , : = x i and

Original: S i,j, : = 0 for j > 1 .
Revised: S i,j, : = 0 for j > 1 .

Original: The output of the Neural GPU, denoted by NG PU ( X , r ) , is the sequence of vectors Y =
Revised: The output of the Neural GPU, denoted by NG PU ( X , r ) , is the sequence of vectors Y =

Original: ( y 1 , y 2 , . . .
Revised: ( y 1 , y 2 , . . .

Original: , y r ) such that y t
Revised: , y r ) such that y t

Original: =
Revised: =

Original: S tn, 1 , : .
Revised: S tn, 1 , : .

Original: Given this deﬁnition, we can naturally view the Neural GPUs as language recognizers (as formalizedthem in Section 2).
Revised: Given this deﬁnition, we can naturally view the Neural GPUs as language recognizers (as formalized in Section 2).

Original: Since the bias tensor B in Equation (13) is of the same size than S , the number of parameters in a
Revised: Since the bias tensor B in Equation (14) is of the same size than S , the number of parameters in a

Original: Neural GPU grows with the size of the input.
Revised: Neural GPU grows with the size of the input.

Original: Thus, a Neural GPU cannot be considered as a ﬁxed architecture.
Revised: Thus, a Neural GPU cannot be considered as a ﬁxed architecture.

Original: To tackle this issue we introduce the notion of uniform Neural GPU , as one in which for every bias B there exists a matrix B ∈ Q w × d such that B i, : , : =
Revised: To tackle this issue we introduce the notion of uniform Neural GPU , as one in which for every bias B there exists a matrix B ∈ Q w × d such that B i, : , : =

Original: B for each i .
Revised: B for each i .

Original: Thus, uniform
Revised: Thus, uniform

Original: Neural GPUs can be ﬁnitely speciﬁed (as they have a constant number of parameters, not depending on the length of the input).
Revised: Neural GPUs can be ﬁnitely speciﬁed (as they have a constant number of parameters, not depending on the length of the input).

Original: We now establish the  completeness of this model.
Revised: We now establish the Turing completeness of this model.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: The class of uniform Neural GPUs is Turing complete.
Revised: The class of uniform Neural GPUs is Turing complete.

Original: Proof sketch.
Revised: Proof sketch.

Original: The proof is based on simulating a seq-to-seq RNN; thus, completeness follows from
Revised: The proof is based on simulating a seq-to-seq RNN; thus, completeness follows from

Original: Theorem 2.3.
Revised: Theorem 2.3.

Original: Consider an RNN encoder-decoder language recognizer, such that N is of dimension d and its encoder and decoder are deﬁned by the equations
Revised: Consider an RNN encoder-decoder language recognizer, such that N is of dimension d and its encoder and decoder are deﬁned by the equations

Original: h i =
Revised: h i =

Original: σ ( x i W
Revised: σ ( x i W

Original: + h
Revised: + h

Original: i − 1 V ) and g t = σ ( g t − 1 U ) , respectively, where g 0 = h n and n is the length of the input.
Revised: i − 1 V ) and g t = σ ( g t − 1 U ) , respectively, where g 0 = h n and n is the length of the input.

Original: We use a Neural GPU with input tensor S ∈ Q n × 1 × 3 d +3 .
Revised: We use a Neural GPU with input tensor S ∈ Q n × 1 × 3 d +3 .

Original: Let E i =
Revised: Let E i =

Original: S i, 1 , 1: d and D i =
Revised: S i, 1 , 1: d and D i =

Original: S i, 1 ,d +1:2 d .
Revised: S i, 1 ,d +1:2 d .

Original: The idea is to use E for the encoder and D for the decoder.
Revised: The idea is to use E for the encoder and D for the decoder.

Original: We use kernel banks of shape (2 , 1 , 3 d + 3 , 3 d + 3) with uniform bias tensors to simulate the following computation.
Revised: We use kernel banks of shape (2 , 1 , 3 d + 3 , 3 d + 3) with uniform bias tensors to simulate the following computation.

Original: In every step t , we ﬁrst compute the value of σ ( E t W + E t − 1 V ) and store it in E t , and then reset E t − 1 to zero.
Revised: In every step t , we ﬁrst compute the value of σ ( E t W + E t − 1 V ) and store it in E t , and then reset E t − 1 to zero.

Original: Similarly, in step t we update the vector in position D t − 1 storing in it the value σ ( D t − 1 U + E t − 1 U ) (for the value of E t − 1 before the reset).
Revised: Similarly, in step t we update the vector in position D t − 1 storing in it the value σ ( D t − 1 U + E t − 1 U ) (for the value of E t − 1 before the reset).

Original: We use the gating mechanism to ensure a sequential update of the cells such that at time t we update only positions E i and D j for i ≤ t and j ≤ t − 1 .
Revised: We use the gating mechanism to ensure a sequential update of the cells such that at time t we update only positions E i and D j for i ≤ t and j ≤ t − 1 .

Original: Thus the updates on the D are always one iteration behind the update of E .
Revised: Thus the updates on the D are always one iteration behind the update of E .

Original: Since the vectors in D are never reset to zero, they keep being updated which allows us to simulate an arbitrary long computation.
Revised: Since the vectors in D are never reset to zero, they keep being updated which allows us to simulate an arbitrary long computation.

Original: In particular we prove that at iteration t it holds that E t
Revised: In particular we prove that at iteration t it holds that E t

Original: = h t , and at iteration n + t it holds that D n = g t .
Revised: = h t , and at iteration n + t it holds that D n = g t .

Original: We require 3 d +components, as we need to implement several gadgets for properly using the update and reset gates.
Revised: We require 3 d +components, as we need to implement several gadgets for properly using the update and reset gates.

Original: In particular, we need to store the value of E t − 1 before we reset it.
Revised: In particular, we need to store the value of E t − 1 before we reset it.

Original: The detailed construction and the correctness proof can be found in the appendix.
Revised: The detailed construction and the correctness proof can be found in the appendix.

Original: The proof above makes use of kernels of shape (2 , 1 , d, d ) to obtain Turing completeness.
Revised: The proof above makes use of kernels of shape (2 , 1 , d, d ) to obtain Turing completeness.

Original: This is, in a sense, optimal, as one can easily prove that Neural GPUs with kernels of shape (1 , 1 , d, d ) are not Turing complete, regardless of the size of d .
Revised: This is, in a sense, optimal, as one can easily prove that Neural GPUs with kernels of shape (1 , 1 , d, d ) are not Turing complete, regardless of the size of d .

Original: In fact, for kernels of this shape the value of a cell of S at time t depends only on the value of the same cell in time t − 1 .
Revised: In fact, for kernels of this shape the value of a cell of S at time t depends only on the value of the same cell in time t − 1 .

Original: Zero padding vs circular convolution The proof of Theorem 4.1 requires the application of zero padding in convolution.
Revised: Zero padding vs circular convolution The proof of Theorem 4.1 requires the application of zero padding in convolution.

Original: This allows us to clearly differentiate internal cells from cells corresponding to the endpoints of the input sequence.
Revised: This allows us to clearly differentiate internal cells from cells corresponding to the endpoints of the input sequence.

Original: Interestingly, Turing-completeness is lost if we replace zero padding with circular convolution .
Revised: Interestingly, Turing-completeness is lost if we replace zero padding with circular convolution .

Original: Formally, given S ∈ Q h × w × d , a circular convolution is obtained by deﬁning S h + n, : , : = S n, : , : for n ∈ Z .
Revised: Formally, given S ∈ Q h × w × d , a circular convolution is obtained by deﬁning S h + n, : , : = S n, : , : for n ∈ Z .

Original: One can prove that uniform Neural GPUs with circular convolutions cannot differentiate among periodic sequences of different length; in particular, they cannot check if a periodic input sequence is of even or odd length.
Revised: One can prove that uniform Neural GPUs with circular convolutions cannot differentiate among periodic sequences of different length; in particular, they cannot check if a periodic input sequence is of even or odd length.

Original: This yields the following:
Revised: This yields the following:

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Uniform Neural GPUs with circular convolutions are not Turing complete.
Revised: Uniform Neural GPUs with circular convolutions are not Turing complete.

Original: Related to this last result is the empirical observation by Price et al.
Revised: Related to this last result is the empirical observation by Price et al.

Original: (2016) that Neural GPUs that learn to solve hard problems, e.g., binary multiplication, and which generalizes to most of the inputs, struggle with highly symmetric (and nearly periodic) inputs.
Revised: (2016) that Neural GPUs that learn to solve hard problems, e.g., binary multiplication, and which generalizes to most of the inputs, struggle with highly symmetric (and nearly periodic) inputs.

Original: Actually, Price et al.
Revised: Actually, Price et al.

Original: (2016) exhibit examples of the form 11111111 × 11111111 failing for all inputs with eight or more 1 s. We leave as future work to explore the implications of our theoretical results on this practical observation.
Revised: (2016) exhibit examples of the form 11111111 × 11111111 failing for all inputs with eight or more 1 s. We leave as future work to explore the implications of our theoretical results on this practical observation.

Original: Bidimensional tensors and piecewise linear activations Freivalds & Liepins (2018) simpliﬁed
Revised: Bidimensional tensors and piecewise linear activations Freivalds & Liepins (2018) simpliﬁed

Original: Neural GPUs and proved that, by considering piecewise linear activations and bidimensional input tensors instead of the original smooth activations and tridimensional tensors used by Kaiser &
Revised: Neural GPUs and proved that, by considering piecewise linear activations and bidimensional input tensors instead of the original smooth activations and tridimensional tensors used by Kaiser &

Original: Sutskever (2016), it is possible to achieve substantially better results in terms of training time and generalization.
Revised: Sutskever (2016), it is possible to achieve substantially better results in terms of training time and generalization.

Original: Our Turing completeness proof also relies on a bidimensional tensor and uses piecewise linear activations, thus providing theoretical evidence that these simpliﬁcations actually retain the full expressiveness of Neural GPUs while simplifying its practical applicability.
Revised: Our Turing completeness proof also relies on a bidimensional tensor and uses piecewise linear activations, thus providing theoretical evidence that these simpliﬁcations actually retain the full expressiveness of Neural GPUs while simplifying its practical applicability.

Original:  F INAL R EMARKS AND F UTURE W ORK
Revised:  F INAL R EMARKS AND F UTURE W ORK

Original: We have presented an analysis of the Turing completeness of two popular neural architectures for sequence-processing tasks; namely, the Transformer, based on attention, and the Neural GPU, based on recurrent convolutions.
Revised: We have presented an analysis of the Turing completeness of two popular neural architectures for sequence-processing tasks; namely, the Transformer, based on attention, and the Neural GPU, based on recurrent convolutions.

Original: We plan to further reﬁne this analysis in the future.
Revised: We plan to further reﬁne this analysis in the future.

Original: For example, our proof of Turing completeness for the Transformer requires the presence of residual connections , i.e., the + x i , + a i , + y i , and + p i summands in Equations (5-10), while our proof for Neural GPUs heavily relies on the gating mechanism.
Revised: For example, our proof of Turing completeness for the Transformer requires the presence of residual connections , i.e., the + x i , + a i , + y i , and + p i summands in Equations (6-11), while our proof for Neural GPUs heavily relies on the gating mechanism.

Original: We will study whether these features are actually essential to obtain completeness.
Revised: We will study whether these features are actually essential to obtain completeness.

Original: We presented general abstract versions of both architectures in order to prove our theoretical results.
Revised: We presented general abstract versions of both architectures in order to prove our theoretical results.

Original: Although we closely follow their original deﬁnitions, some choices for functions and parameters in our positive results are different to the usual choices in practice, most notably, the use of hard attention for the case of the Transformer, and the piecewise linear activation functions for both architectures.
Revised: Although we closely follow their original deﬁnitions, some choices for functions and parameters in our positive results are different to the usual choices in practice, most notably, the use of hard attention for the case of the Transformer, and the piecewise linear activation functions for both architectures.

Original: As we have mentioned, Freivalds & Liepins (2018) showed that for Neural GPUs piecewise linear activations actually help in practice, but for the case of the Transformer architecture more experimentation is needed to have a conclusive response.
Revised: As we have mentioned, Freivalds & Liepins (2018) showed that for Neural GPUs piecewise linear activations actually help in practice, but for the case of the Transformer architecture more experimentation is needed to have a conclusive response.

Original: This is part of our future work.
Revised: This is part of our future work.

Original: Although our results are mostly of theoretical interest, they might lead to observations of practical interest.
Revised: Although our results are mostly of theoretical interest, they might lead to observations of practical interest.

Original: For example, Chen et al.
Revised: For example, Chen et al.

Original: (2018) have established the undecidability of several practical problems related to probabilistic language modeling with RNNs.
Revised: (2018) have established the undecidability of several practical problems related to probabilistic language modeling with RNNs.

Original: This means that such problems can only be approached in practice via heuristics solutions.
Revised: This means that such problems can only be approached in practice via heuristics solutions.

Original: Many of the results in Chen et al.
Revised: Many of the results in Chen et al.

Original: (2018) are, in fact, a consequence of the Turing completeness of RNNs as established by Siegelmann & Sontag (1995).
Revised: (2018) are, in fact, a consequence of the Turing completeness of RNNs as established by Siegelmann & Sontag (1995).

Original: We plan to study to what extent our analogous undecidability results for Transformers and
Revised: We plan to study to what extent our analogous undecidability results for Transformers and

Original: Neural GPUs imply undecidability for language modeling problems based on these architectures.
Revised: Neural GPUs imply undecidability for language modeling problems based on these architectures.

Original: Finally, our results rely on being able to compute internal representations of arbitrary precision.
Revised: Finally, our results rely on being able to compute internal representations of arbitrary precision.

Original: It would be interesting to perform a theoretical study of the main properties of both architectures in a setting in which only ﬁnite precision is allowed, as have been recently carried out for RNNs (Weiss et al., 2018).
Revised: It would be interesting to perform a theoretical study of the main properties of both architectures in a setting in which only ﬁnite precision is allowed, as have been recently carried out for RNNs (Weiss et al., 2018).

Original: We also plan to tackle this problem in our future work.
Revised: We also plan to tackle this problem in our future work.

Original: A P ROOFS FOR S ECTIONA. P ROOF OF T HEOREM 2. 
Revised: A P ROOFS FOR S ECTIONA. P ROOF OF T HEOREM 2.

Original: We ﬁrst sketch the main idea of Siegelmann & Sontag’s proof.
Revised: We ﬁrst sketch the main idea of Siegelmann & Sontag’s proof.

Original: We refer the reader to the original paper for details.
Revised: We refer the reader to the original paper for details.

Original: Siegelmann & Sontag show how to simulate a two-stack machine M (and subsequently, a Turing machine) with a single RNN N with σ as activation.
Revised: Siegelmann & Sontag show how to simulate a two-stack machine M (and subsequently, a Turing machine) with a single RNN N with σ as activation.

Original: They ﬁrst construct a network
Revised: They ﬁrst construct a network

Original: N 1 that, with 0 as initial state ( h N 1= 0 ) and with a binary string w ∈ { 0 , 1 } ∗ as input sequence, produces a representation of w as a rational number and stores it as one of its internal values.
Revised: N 1 that, with 0 as initial state ( h N 1= 0 ) and with a binary string w ∈ { 0 , 1 } ∗ as input sequence, produces a representation of w as a rational number and stores it as one of its internal values.

Original: Their internal representation of strings encodes every w as a rational number between 0 and 1 .
Revised: Their internal representation of strings encodes every w as a rational number between 0 and 1 .

Original: In particular, they use base 4 such that, for example, a string w = 100110 is encoded as (0 . 311331) 4 that is, its encoding is
Revised: In particular, they use base 4 such that, for example, a string w = 100110 is encoded as (0 . 311331) 4 that is, its encoding is

Original: [Equation]
Revised: [Equation]

Original: This representation allows one to easily simulate stack operations as afﬁne transformations plus σ activations.
Revised: This representation allows one to easily simulate stack operations as afﬁne transformations plus σ activations.

Original: For instance, if x w is the value representing string w =
Revised: For instance, if x w is the value representing string w =

Original: b 1 b 2 · · · b n seen as a stack, then the top( w ) operation can be deﬁned as simply y
Revised: b 1 b 2 · · · b n seen as a stack, then the top( w ) operation can be deﬁned as simply y

Original: = σ (4 x w − 2) , since y = 1 if and only if b 1 = 1 , and y = 

Revised: = σ (4 x w − 2) , since y = 1 if and only if b 1 = 1 , and y =

Original: if and only if b 1 = 0 .
Revised: if and only if b 1 = 0 .

Original: Other stack operations can de similarly simulated.
Revised: Other stack operations can de similarly simulated.

Original: Using this representation, they construct a second network N 2 that simulates the two-stacks machine by using one neuron value to simulate each stack.
Revised: Using this representation, they construct a second network N 2 that simulates the two-stacks machine by using one neuron value to simulate each stack.

Original: The input w for the simulated machine M is assumed to be at an internal value given to N 2 as an initial state ( h N 2 0 ) .
Revised: The input w for the simulated machine M is assumed to be at an internal value given to N 2 as an initial state ( h N 2 0 ) .

Original: Thus, N 2 expects only zeros as input.
Revised: Thus, N 2 expects only zeros as input.

Original: Actually, to make N 2 work for r steps, an input of the form 0 r should be provided.
Revised: Actually, to make N 2 work for r steps, an input of the form 0 r should be provided.

Original: Finally, they combine N 1 and N 2 to construct a network N which expects an input of the following form: ( b 1 , 1 , 0)
Revised: Finally, they combine N 1 and N 2 to construct a network N which expects an input of the following form: ( b 1 , 1 , 0)

Original: ( b 2 , 1 , 0) · · · ( b n , 1 , 0)(0 , 0 , 1)(0 , 0 , 0)(0 , 0 , 0) · · · (0 , 0 , 0) .
Revised: ( b 2 , 1 , 0) · · · ( b n , 1 , 0)(0 , 0 , 1)(0 , 0 , 0)(0 , 0 , 0) · · · (0 , 0 , 0) .

Original: The idea is that the ﬁrst component contains the input string w = b 1 b 2 · · · b n , the second component states when the input is active , and the third component is 1 only when the input is inactive for the ﬁrst time.
Revised: The idea is that the ﬁrst component contains the input string w = b 1 b 2 · · · b n , the second component states when the input is active , and the third component is 1 only when the input is inactive for the ﬁrst time.

Original: Before the input vector (0 , 0 , 1) the network N 1 is working.
Revised: Before the input vector (0 , 0 , 1) the network N 1 is working.

Original: The input (0 , 0 , 1) is used to simulate a change from N 1 to N 2 , and the rest of input vectors (0 , 0 , 0) are provided to continue with N 2 for as many steps as needed.
Revised: The input (0 , 0 , 1) is used to simulate a change from N 1 to N 2 , and the rest of input vectors (0 , 0 , 0) are provided to continue with N 2 for as many steps as needed.

Original: The number neurons that this construction needs to simulate a machine M with m states, is 10 m + 30 .
Revised: The number neurons that this construction needs to simulate a machine M with m states, is 10 m + 30 .

Original: It is clear that Siegelmann & Sontag’s proof resembles a modern encoder-decoder RNN architecture, where N 1 is the encoder and N 2 is the decoder, thus it is straightforward to use the same construction to provide an RNN encoder-decoder N (cid:48) and a language recognizer
Revised: It is clear that Siegelmann & Sontag’s proof resembles a modern encoder-decoder RNN architecture, where N 1 is the encoder and N 2 is the decoder, thus it is straightforward to use the same construction to provide an RNN encoder-decoder N (cid:48) and a language recognizer

Original: A that uses N (cid:48)
Revised: A that uses N (cid:48)

Original: and simulates the two-stacks machine M .
Revised: and simulates the two-stacks machine M .

Original: There are some details that is important to notice.
Revised: There are some details that is important to notice.

Original: Assume that N (cid:48) is given by the formulas in Equations (1) and (2).
Revised: Assume that N (cid:48) is given by the formulas in Equations (2) and (3).

Original: First, since N 2 in the above construction expects no input, we can safely assume that R in Equation (2) is the null matrix.
Revised: First, since N 2 in the above construction expects no input, we can safely assume that R in Equation (3) is the null matrix.

Original: Moreover, since A deﬁnes its own embedding function, we can ensure that every vector that we provide for the encoder part of N (cid:48) has a 1 in a ﬁxed component, and thus we do not need the bias b 1 in Equation (1) since it can be simulated with one row of matrix V .
Revised: Moreover, since A deﬁnes its own embedding function, we can ensure that every vector that we provide for the encoder part of N (cid:48) has a 1 in a ﬁxed component, and thus we do not need the bias b 1 in Equation (2) since it can be simulated with one row of matrix V .

Original: We can do a similar construction for the bias b 2 (Equation (2)).
Revised: We can do a similar construction for the bias b 2 (Equation (3)).

Original: Finally,
Revised: Finally,

Original: Siegelmann & Sontag show that its construction can be modiﬁed such that a particular neuron of
Revised: Siegelmann & Sontag show that its construction can be modiﬁed such that a particular neuron of

Original: N 2 , say n (cid:63) , is always 0 except for the ﬁrst time an accepting state of M is reached, in which case n (cid:63) = 1 .
Revised: N 2 , say n (cid:63) , is always 0 except for the ﬁrst time an accepting state of M is reached, in which case n (cid:63) = 1 .

Original: Thus, one can consider O ( · ) (Equation (2)) as the identity function and add to A the stopping criterion that just checks if n (cid:63) is 1 .
Revised: Thus, one can consider O ( · ) (Equation (3)) as the identity function and add to A the stopping criterion that just checks if n (cid:63) is 1 .

Original: This completes the proof sketch of Theorem 2.3.
Revised: This completes the proof sketch of Theorem 2.3.

Original: B P ROOFS FOR S ECTIONB. P ROOF OF P ROPOSITION 3. 
Revised: B P ROOFS FOR S ECTIONB. P ROOF OF P ROPOSITION 3.

Original: Proof.

Original: We extend the deﬁnition of the function PropInv to sequences of vectors.
Revised: We extend the deﬁnition of the function PropInv to sequences of vectors.

Original: Given a sequence 
Revised: Given a sequence X =

Original: X = ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x n ) we use vals( X ) to denote the set of all vectors occurring in X .
Revised: , x n ) we use vals( X ) to denote the set of all vectors occurring in X .

Original: Similarly as for strings, we use prop( v , X ) as the number of times that v occurs in X divided by the length of X .
Revised: Similarly as for strings, we use prop( v , X ) as the number of times that v occurs in X divided by the length of X .

Original: Now we are ready to extend PropInv with the following deﬁnition:
Revised: Now we are ready to extend PropInv with the following deﬁnition:

Original: PropInv
Revised: PropInv

Revised: [Equation]

Revised: [Equation]

Revised: { X

Original: [Equation]
Revised: [Equation]

Original:  and prop( v , X ) =
Revised: X [Equation] ) = vals( X ) and prop( v , X ) =

Original: prop( v , X (cid:48) )
Revised: prop( v , X (cid:48) )

Original: for all v ∈ vals( X ) }
Revised: for all v ∈ vals( X ) }

Original: Notice that for every embedding function f : Σ → Q d and string w ∈ Σ ∗ , we have that if u ∈ PropInv( w ) then f ( u ) ∈ PropInv( f ( w )) .
Revised: Notice that for every embedding function f : Σ → Q d and string w ∈ Σ ∗ , we have that if u ∈ PropInv( w ) then f ( u ) ∈ PropInv( f ( w )) .

Original: Thus in order to prove that Trans( f ( w ) , s , r ) =
Revised: Thus in order to prove that Trans( f ( w ) , s , r ) =

Original: Trans( f ( u ) , s , r ) for every u ∈ PropInv( w ) , it is enough to prove that
Revised: Trans( f ( u ) , s , r ) for every u ∈ PropInv( w ) , it is enough to prove that

Original: [Equation]
Revised: [Equation]

Original: To further simplify the exposition of the proof we introduce another notation.
Revised: To further simplify the exposition of the proof we introduce another notation.

Original: We denote by p Xv as the number of times that vector v occurs in X .
Revised: We denote by p Xv as the number of times that vector v occurs in X .

Original: Thus we have that X (cid:48) ∈ PropInv( X ) if and only if, there exists a value γ ∈ Q + such that for every v ∈ vals( X ) it holds that p X (cid:48) v = γp Xv .
Revised: Thus we have that X (cid:48) ∈ PropInv( X ) if and only if, there exists a value γ ∈ Q + such that for every v ∈ vals( X ) it holds that p X (cid:48) v = γp Xv .

Original: We now have all the necessary to proceed with the proof of Proposition 3.1.
Revised: We now have all the necessary to proceed with the proof of Proposition 3.1.

Original: We will prove it by proving the property in (15).
Revised: We will prove it by proving the property in (16).

Original: Let X =
Revised: Let X =

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x n ) be an arbitrary sequence of vectors, and let X (cid:48)
Revised: , x n ) be an arbitrary sequence of vectors, and let X (cid:48)

Original: =
Revised: =

Original: ( x (cid:48) 1 , . . .
Revised: ( x (cid:48) 1 , . . .

Original: , x (cid:48) m ) ∈ PropInv( X ) .
Revised: , x (cid:48) m ) ∈ PropInv( X ) .

Original: Moreover, let Z
Revised: Moreover, let Z

Original: =
Revised: =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z n ) =
Revised: , z n ) =

Original: Enc( X ; θ ) and
Revised: Enc( X ; θ ) and

Original: Z (cid:48) =
Revised: Z (cid:48) =

Original: ( z (cid:48) 1 , . . .
Revised: ( z (cid:48) 1 , . . .

Original: , z (cid:48) m )
Revised: , z (cid:48) m )

Original: =
Revised: =

Original: Enc( X (cid:48) ; θ ) .
Revised: Enc( X (cid:48) ; θ ) .

Original: We ﬁrst prove the following property:
Revised: We ﬁrst prove the following property:

Original: For every pair of indices ( i, j ) ∈ { 1 , . . . , n } × { 1 , . . .
Revised: For every pair of indices ( i, j ) ∈ { 1 , . . . , n } × { 1 , . . .

Original: , m } , if x i
Revised: , m } , if x i

Original: = x (cid:48)
Revised: = x (cid:48)

Original: j then z i
Revised: j then z i

Original: =
Revised: =

Original: z (cid:48)
Revised: z (cid:48)

Original: j .
Revised: j .

Original: Lets ( i, j ) be a pair of indices such that x i
Revised: Lets ( i, j ) be a pair of indices such that x i

Original: = x (cid:48)
Revised: = x (cid:48)

Original: j .
Revised: j .

Original: From Equations (5-6) we have that z
Revised: From Equations (6-7) we have that z

Original: i
Revised: i

Original: =
Revised: =

Original: O
Revised: O

Original: ( a i )
Revised: ( a i )

Original: + a i where a i =
Revised: + a i where a i =

Original: Att( Q ( x i ) , K ( X ) , V ( X ))
Revised: Att( Q ( x i ) , K ( X ) , V ( X ))

Original: + x i .
Revised: + x i .

Original: Thus, since x i = x (cid:48)
Revised: Thus, since x i = x (cid:48)

Original: j , in order to prove z i =
Revised: j , in order to prove z i =

Original: z (cid:48)
Revised: z (cid:48)

Original: j it is enough to prove that Att( Q ( x i ) , K ( X ) , V ( X ))
Revised: j it is enough to prove that Att( Q ( x i ) , K ( X ) , V ( X ))

Original: =
Revised: =

Original: Att( Q ( x (cid:48) j ) , K ( X (cid:48) ) , V ( X (cid:48) )) .
Revised: Att( Q ( x (cid:48) j ) , K ( X (cid:48) ) , V ( X (cid:48) )) .

Original: By equations (34) and the restriction over the form of normalization functions we have that
Revised: By equations (45) and the restriction over the form of normalization functions we have that

Original: [Equation]
Revised: [Equation]

Original: where α = (cid:80) n(cid:96)
Revised: where α = (cid:80) n(cid:96)

Original: =1 f ρ (score( Q ( x (cid:96) ) , K ( x (cid:96) ))) .
Revised: =1 f ρ (score( Q ( x (cid:96) ) , K ( x (cid:96) ))) .

Original: The above equation can be rewritten as
Revised: The above equation can be rewritten as

Original: [Equation]
Revised: [Equation]

Original:  V ( v ) with α = (cid:80) v ∈ vals( X ) p
Revised: v [Equation] V ( v ) with α = (cid:80) v ∈ vals( X ) p

Original: Xv f ρ (score( Q ( v ) , K ( v ))) .
Revised: Xv f ρ (score( Q ( v ) , K ( v ))) .

Original: By a similar reasoning we can write
Revised: By a similar reasoning we can write

Original: Att( Q ( x (cid:48) j ) , K ( X (cid:48) ) , V ( X (cid:48) ))
Revised: Att( Q ( x (cid:48) j ) , K ( X (cid:48) ) , V ( X (cid:48) ))

Original: = 1 β (cid:88) v ∈ vals( X (cid:48) )
Revised: = 1 β (cid:88) v ∈ vals( X (cid:48) )

Original: p X (cid:48) v f ρ (score( Q ( x (cid:48) j ) , K ( v )))
Revised: p X (cid:48) v f ρ (score( Q ( x (cid:48) j ) , K ( v )))

Original: V ( v ) with β = (cid:80) v ∈ vals( X (cid:48) )
Revised: V ( v ) with β = (cid:80) v ∈ vals( X (cid:48) )

Original: p X (cid:48) v f ρ (score( Q ( v ) , K ( v ))) .
Revised: p X (cid:48) v f ρ (score( Q ( v ) , K ( v ))) .

Original: Now, since X (cid:48) ∈ PropInv( X ) we know that vals( X ) =
Revised: Now, since X (cid:48) ∈ PropInv( X ) we know that vals( X ) =

Original: vals( X (cid:48) )
Revised: vals( X (cid:48) )

Original: and there exists a γ ∈ Q + such that p X (cid:48) v = γp Xv for every v ∈ vals( X ) .
Revised: and there exists a γ ∈ Q + such that p X (cid:48) v = γp Xv for every v ∈ vals( X ) .

Original: Finally, from this last property, plus the fact that x i = x (cid:48)
Revised: Finally, from this last property, plus the fact that x i = x (cid:48)

Original: j we have
Revised: j we have

Original: Att( 
Revised: Att( Q ( x (cid:48) j ) , K ( X (cid:48) ) , V ( X (cid:48) )) = γα (cid:88) v ∈ vals( X )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: f ρ (score( Q ( x (cid:48) j ) , K ( v ))) V ( v ) = α (cid:88) v ∈ vals( X ) p

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: f

Original: [Equation]
Revised: [Equation]

Original: =
Revised: (score( Q ( x i ) , K ( v )))

Original: [Equation]
Revised: [Equation]

Revised: ( v ) =

Revised: Att( Q ( x i ) , K ( X ) , V ( X ))

Original: Which completes the proof of Property (16) above.
Revised: Which completes the proof of Property (17) above.

Original: Consider now the complete encoder TEnc .
Revised: Consider now the complete encoder TEnc .

Original: Let ( K , V ) =
Revised: Let ( K , V ) =

Original: TEnc( X ) and ( K (cid:48) , V (cid:48) )
Revised: TEnc( X ) and ( K (cid:48) , V (cid:48) )

Original: =
Revised: =

Original: TEnc( X (cid:48) ) , and let q be an arbitrary vector.
Revised: TEnc( X (cid:48) ) , and let q be an arbitrary vector.

Original: We will prove now that Att( q , K , V ) =
Revised: We will prove now that Att( q , K , V ) =

Original: Att( q , K (cid:48) , V (cid:48) ) .
Revised: Att( q , K (cid:48) , V (cid:48) ) .

Original: By following a similar reasoning as for proving Property (16) (plus induction on the layers of TEnc ) we obtain that if x i = x (cid:48)
Revised: By following a similar reasoning as for proving Property (17) (plus induction on the layers of TEnc ) we obtain that if x i = x (cid:48)

Original: j then k i = k (cid:48)
Revised: j then k i = k (cid:48)

Original: j and v i = v (cid:48)
Revised: j and v i = v (cid:48)

Original: j , for every i ∈ { 1 , . .
Revised: j , for every i ∈ { 1 , . .

Original: . , n } and j ∈ { 1 , . .
Revised: . , n } and j ∈ { 1 , . .

Original: , m } .
Revised: , m } .

Original: Thus, there exists a mapping M K : vals( X ) → vals( K ) such that M K ( x i )
Revised: Thus, there exists a mapping M K : vals( X ) → vals( K ) such that M K ( x i )

Original: =
Revised: =

Original: k i and M K ( x (cid:48) j ) =
Revised: k i and M K ( x (cid:48) j ) =

Original: k (cid:48)
Revised: k (cid:48)

Original: j and similarly a mapping M V : vals( X ) → vals( V ) such that M V ( x i ) =
Revised: j and similarly a mapping M V : vals( X ) → vals( V ) such that M V ( x i ) =

Original: v i and M V ( x (cid:48)
Revised: v i and M V ( x (cid:48)

Original: j )
Revised: j )

Original: = v (cid:48)
Revised: = v (cid:48)

Original: j , for every i ∈ { 1 , . .
Revised: j , for every i ∈ { 1 , . .

Original: . , n } and j ∈ { 1 , . .
Revised: . , n } and j ∈ { 1 , . .

Original: , m } .
Revised: , m } .

Original: Lets focus now on Att( q , K , V ) .
Revised: Lets focus now on Att( q , K , V ) .

Original: We have:
Revised: We have:

Original: Att( q , K , V ) =
Revised: Att( q , K , V ) =

Original: α n (cid:88) i =
Revised: α n (cid:88) i =

Original: f ρ (score( q , k i )) v i with α = (cid:80) ni
Revised: f ρ (score( q , k i )) v i with α = (cid:80) ni

Original: =

Revised: =

Original: f ρ (score( q , k i )) .
Revised: f ρ (score( q , k i )) .

Original: Similarly as before, we can rewrite this as
Revised: Similarly as before, we can rewrite this as

Original: Att( q , K , V ) =
Revised: Att( q , K , V ) =

Original: α n (cid:88) i =
Revised: α n (cid:88) i =

Original: f ρ (score( q , M K ( x i )))
Revised: f ρ (score( q , M K ( x i )))

Original: M V ( x i ) =
Revised: M V ( x i ) =

Original: α (cid:88) v ∈ vals( X ) p Xv f ρ (score( q , M K ( v )))
Revised: α (cid:88) v ∈ vals( X ) p Xv f ρ (score( q , M K ( v )))

Original: M V ( v ) with α = (cid:80) v ∈ vals( X ) p
Revised: M V ( v ) with α = (cid:80) v ∈ vals( X ) p

Original: Xv f ρ (score( q , M K ( v ))) .
Revised: Xv f ρ (score( q , M K ( v ))) .

Original: Similarly for Att( q , K (cid:48) , V (cid:48) )
Revised: Similarly for Att( q , K (cid:48) , V (cid:48) )

Original: we have
Revised: we have

Original: Att( q , K (cid:48) , V (cid:48) )
Revised: Att( q , K (cid:48) , V (cid:48) )

Original: =
Revised: =

Original: β m (cid:88) j = f ρ (score( q , M K (
Revised: β m (cid:88) j = f ρ (score( q , M K (

Original: [Equation]
Revised: [Equation]

Original: And ﬁnally using that X (cid:48) ∈ PropInv( X ) we obtain
Revised: And ﬁnally using that X (cid:48) ∈ PropInv( X ) we obtain

Original: [Equation]
Revised: [Equation]

Original: which is what we wanted.
Revised: which is what we wanted.

Original: To complete the rest proof, consider Trans( X , y 0 , r ) which is deﬁned by the recursion
Revised: To complete the rest proof, consider Trans( X , y 0 , r ) which is deﬁned by the recursion

Original: [Equation]
Revised: [Equation]

Original: To prove that Trans( X , y 0 , r ) =
Revised: To prove that Trans( X , y 0 , r ) =

Original: Trans( X (cid:48) , y 0 , r ) we use an inductive argument.
Revised: Trans( X (cid:48) , y 0 , r ) we use an inductive argument.

Original: We know that
Revised: We know that

Original: [Equation]
Revised: [Equation]

Original: Now TDec only access ( K , V ) via attentions of the form Att( q , K , V ) and for the case of y 1 the vector q can only depend on y 0 , thus, from Att( q , K , V ) =
Revised: Now TDec only access ( K , V ) via attentions of the form Att( q , K , V ) and for the case of y 1 the vector q can only depend on y 0 , thus, from Att( q , K , V ) =

Original: Att( q , K (cid:48) , V (cid:48) )
Revised: Att( q , K (cid:48) , V (cid:48) )

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: The rest of the steps follow by a simple induction on k .
Revised: The rest of the steps follow by a simple induction on k .

Original: B. P ROOF OF C OROLLARY 3. 
Revised: B. P ROOF OF C OROLLARY 3.

Original: To obtain a contradiction, assume that there is a language recognizer
Revised: To obtain a contradiction, assume that there is a language recognizer

Original: A that uses a Transformer network and such that L =
Revised: A that uses a Transformer network and such that L =

Original: L ( A ) .
Revised: L ( A ) .

Original: Now consider the strings w 1 = aabb and w 2 = aaabbb .
Revised: Now consider the strings w 1 = aabb and w 2 = aaabbb .

Original: Since w 1 ∈ PropInv( w 2 ) by Proposition 3.1 we have that w 1 ∈ L ( A ) if and only if w 2 ∈ L ( A ) which is a contradiction since w 1 ∈ L but w 2 / ∈ L .
Revised: Since w 1 ∈ PropInv( w 2 ) by Proposition 3.1 we have that w 1 ∈ L ( A ) if and only if w 2 ∈ L ( A ) which is a contradiction since w 1 ∈ L but w 2 / ∈ L .

Original: This completes the proof of the corollary.
Revised: This completes the proof of the corollary.

Original: B. P ROOF OF P ROPOSITION 3. 
Revised: B. P ROOF OF P ROPOSITION 3.

Original: We construct a language recognizer
Revised: We construct a language recognizer

Original: A = (Σ , f, Trans , s , F ) with Trans a very simple Transformer network with dimension d = 2 and using just one layer of encoder and one layer of decoder, such that L ( A ) = { w ∈ { a, b } ∗ | w has strictly more symbols a than symbols b } .
Revised: A = (Σ , f, Trans , s , F ) with Trans a very simple Transformer network with dimension d = 2 and using just one layer of encoder and one layer of decoder, such that L ( A ) = { w ∈ { a, b } ∗ | w has strictly more symbols a than symbols b } .

Original: As embedding function, we use f ( a ) =
Revised: As embedding function, we use f ( a ) =

Original: [0 , 1] and f ( b ) =
Revised: [0 , 1] and f ( b ) =

Original: [0 , − 1] .
Revised: [0 , − 1] .

Original: Assume that the output for the encoder part of the transformer is X
Revised: Assume that the output for the encoder part of the transformer is X

Original: = ( x 1 , . . .
Revised: = ( x 1 , . . .

Original: , x n ) .
Revised: , x n ) .

Original: First we use an encoder layer that implements the identity function.
Revised: First we use an encoder layer that implements the identity function.

Original: This can be trivially done using null functions for the self attention and through the residual connections this encoder layer shall preserve the original x i values.
Revised: This can be trivially done using null functions for the self attention and through the residual connections this encoder layer shall preserve the original x i values.

Original: For the ﬁnal V ( · ) and K ( · ) functions of the Transformer encoder (Equation (7)), we use V ( x ) =
Revised: For the ﬁnal V ( · ) and K ( · ) functions of the Transformer encoder (Equation (8)), we use V ( x ) =

Original: x the identity function and K ( x ) =
Revised: x the identity function and K ( x ) =

Original: [0 , 0] , giving V e = X
Revised: [0 , 0] , giving V e = X

Original: and
Revised: and

Original: K e = ([0 , 0] , [0 , 0] , . . .
Revised: K e = ([0 , 0] , [0 , 0] , . . .

Original: For the decoder we use a similar approach.
Revised: For the decoder we use a similar approach.

Original: We consider the identity in the self attention plus the residual (which can be done by just using the null functions for the self attention).
Revised: We consider the identity in the self attention plus the residual (which can be done by just using the null functions for the self attention).

Original: Considering the external attention, that is the attention over ( K e , V e ) , we let score and ρ be arbitrary scoring and normalization functions.
Revised: Considering the external attention, that is the attention over ( K e , V e ) , we let score and ρ be arbitrary scoring and normalization functions.

Original: And ﬁnally for the function O ( · ) (Equation (10)) we use a single layer neural network implementing the afﬁne transformation O ([ x, y ]) =
Revised: And ﬁnally for the function O ( · ) (Equation (11)) we use a single layer neural network implementing the afﬁne transformation O ([ x, y ]) =

Original: [ y − x, − y ] such that
Revised: [ y − x, − y ] such that

Original: O ([ x, y ])
Revised: O ([ x, y ])

Original: + [ x, y ] =
Revised: + [ x, y ] =

Original: [ y, 0] .
Revised: [ y, 0] .

Original: The ﬁnal function F ( · ) is just the identity function.
Revised: The ﬁnal function F ( · ) is just the identity function.

Original: In order to complete the proof we introduce some notation.
Revised: In order to complete the proof we introduce some notation.

Original: Lets denote by # a ( w ) as the number of a ’s in w , and similarly # b ( w ) for the number of b ’s in w .
Revised: Lets denote by # a ( w ) as the number of a ’s in w , and similarly # b ( w ) for the number of b ’s in w .

Original: Lets call c w as the value # a ( w ) − # b ( w ) n .
Revised: Lets call c w as the value # a ( w ) − # b ( w ) n .

Original: We now prove that, for any string w ∈ { a, b } ∗ if we consider f ( w ) =
Revised: We now prove that, for any string w ∈ { a, b } ∗ if we consider f ( w ) =

Original: X =
Revised: X =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x n ) as the input sequence for Trans
Revised: , x n ) as the input sequence for Trans

Original: and we use initial value s =
Revised: and we use initial value s =

Original: [0 , 0] for the decoder, the complete network shall compute a sequence y 1 , y 2 , . .
Revised: [0 , 0] for the decoder, the complete network shall compute a sequence y 1 , y 2 , . .

Original: y r such that:
Revised: y r such that:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: We proceed by induction.
Revised: We proceed by induction.

Original: The base case trivially holds since y 

Revised: The base case trivially holds since y

Original: = s =
Revised: = s =

Original: Assume now that we are at step r and the input for the decoder is ( y 0 , y 1 , . . .
Revised: Assume now that we are at step r and the input for the decoder is ( y 0 , y 1 , . . .

Original: , y r ) .
Revised: , y r ) .

Original: We will show that y r +

Revised: We will show that y r +

Original: =
Revised: =

Original: [ c w , 0] .
Revised: [ c w , 0] .

Original: Since we consider the identity in the self attention (Equation (8)), we have that p i
Revised: Since we consider the identity in the self attention (Equation (9)), we have that p i

Original: =
Revised: =

Original: y i for every i in { 0 , . . .
Revised: y i for every i in { 0 , . . .

Original: , i } .
Revised: , i } .

Original: Now considering the external attention, that is the attention over ( K e , V e ) , Since all key vectors in K e are [0 , 0] , the external attention will produce the same score value for all positions.
Revised: Now considering the external attention, that is the attention over ( K e , V e ) , Since all key vectors in K e are [0 , 0] , the external attention will produce the same score value for all positions.

Original: That is, score( p i , k j 1 ) = score( p i , k j 2 ) for every j 1 , j 2 .
Revised: That is, score( p i , k j 1 ) = score( p i , k j 2 ) for every j 1 , j 2 .

Original: Lets call this value s (cid:63) .
Revised: Lets call this value s (cid:63) .

Original: Thus we have that
Revised: Thus we have that

Original: [Equation]
Revised: [Equation]

Original: Then, since V e
Revised: Then, since V e

Original: =
Revised: =

Original: X we have that
Revised: X we have that

Original: [Equation]
Revised: [Equation]

Original: for every i ∈ { 0 , . . .
Revised: for every i ∈ { 0 , . . .

Original: , r } .
Revised: , r } .

Original: The last equality holds since our embedding are f ( a ) =
Revised: The last equality holds since our embedding are f ( a ) =

Original: [0 , 1] and f ( b ) =
Revised: [0 , 1] and f ( b ) =

Original: [0 , − 1] , and so every a in w sums one and every b subtracts one.
Revised: [0 , − 1] , and so every a in w sums one and every b subtracts one.

Original: Thus, we have that
Revised: Thus, we have that

Original: [Equation]
Revised: [Equation]

Original: for every i ∈ { 0 , . . .
Revised: for every i ∈ { 0 , . . .

Original: , r } .
Revised: , r } .

Original: In the next step, after the external attention plus the residual connection (Equation (9))
Revised: In the next step, after the external attention plus the residual connection (Equation (10))

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Applying function O ( · ) plus the residual connection (Equation (10))
Revised: Applying function O ( · ) plus the residual connection (Equation (11))

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Finally, y r +1 =
Revised: Finally, y r +1 =

Original: F ( z r ) =
Revised: F ( z r ) =

Original: z r = [ c w , 0] which is exactly what we wanted to prove.
Revised: z r = [ c w , 0] which is exactly what we wanted to prove.

Original: To complete the proof, notice that # a ( w ) > # b ( w ) if and only if c w > 0 .
Revised: To complete the proof, notice that # a ( w ) > # b ( w ) if and only if c w > 0 .

Original: If we deﬁne F as Q + × Q , the recognizer
Revised: If we deﬁne F as Q + × Q , the recognizer

Original: A =
Revised: A =

Original: (Σ , f, Trans , s , F ) will accept the string w exactly when c w > 0 , that is, w ∈ L ( A )
Revised: (Σ , f, Trans , s , F ) will accept the string w exactly when c w > 0 , that is, w ∈ L ( A )

Original: if and only if # a ( w ) > # b ( w ) .
Revised: if and only if # a ( w ) > # b ( w ) .

Original: That is exactly the language S , and so the proof is complete.
Revised: That is exactly the language S , and so the proof is complete.

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF T HEOREM 3.

Original: The main idea is to simulate an RNN encoder-decoder architecture using a Transformer network.

Original: Let A =
Revised: Let M =

Original: (Σ , f, N, s , F ) be a recognizer, with N an RNN encoder-decoder of dimension d and composed of the equations
Revised: ( Q, Σ , δ, q init , F ) be a Turing machine with a inﬁnite tape and assume that the special symbol # ∈ Σ is used to mark blank positions in the tape.

Original: [Equation]
Revised: We make the following assumptions about how M works when processing an input string: • M always moves its head either to the left or to the right (it never stays at the same cell). • M begins at state q init pointing to the cell immediately to the left of the input string. • M never makes a transition to the left of the initial position.

Original: [-with h 0 = 
 (the vector in-] Q [-d with only zeros), and g 
-]
Revised: [-with h 0 = 
 (the vector in-]• Q [-d with only zeros), and g 
-]has a special state q read used to read the complete input. • Initially (time 0 )

Original: = h n where n is the length of the input.
Revised: , M makes a transition to state q read and move its head to the right. • While in state q read it moves to the right until symbol # is read. • There are no transitions going out from accepting states (states in F ). It is easy to prove that every general Turing machine is equivalent to one that satisﬁes the above assumptions.

Original: We shall construct a recognizer A (cid:48) = (Σ , f (cid:48) , Trans , s (cid:48) , F (cid:48) ) with Trans a Transformer network such that L ( A ) =
Revised: We prove that one can construct a transformer network Trans M that is able to simulate M on every possible input string. The construction is somehow involved and uses several helping values, sequences and intermediate results.

Original: L ( A (cid:48) ) .
Revised: To make the reading more easy we divide the construction and proof in three parts. We ﬁrst give a high-level view of the strategy we use. Then we give some details on the architecture of the encoder and decoder needed to implement our strategy, and ﬁnally we formally prove that every part of our architecture can be actually implemented. B.4. O VERVIEW OF THE CONSTRUCTION AND HIGH - LEVEL STRATEGY

Original: In our proof we will use vectors in Q 6 d +8 .
Revised: In the encoder part of Trans M we receive as input the string w = s 1 s 2 . . . s n .

Original: The idea of every component of this vector will be clear when we describe the details of the construction.
Revised: We ﬁrst use an embedding function to represent every s i as a one-hot vector and add a positional encoding for every index.

Original: P RELIMINARIES
Revised: The encoder produces output ( K e , V e ) where K e = ( k e 1 , . . . , k e n ) and V e = ( v e 1 , . . . , v e n ) are sequences of keys and values such that v e i contains the information of s i and k e i contains the information of the i -th positional encoding.

Original: We ﬁrst introduce the embedding and positional encoding functions.
Revised: We later show that this allows us to attend to every speciﬁc position and copy every input symbol from the encoder to the decoder (Lemma B.1).

Original: We use an embedding function f (cid:48) :
Revised: In the decoder part of Trans M we simulate a complete execution of M over w = s 1 s 2 · · · s n . For this we deﬁne the following sequences (for i ≥ 0 ): [Equation] symbol under the head of M

Revised: [Equation]

Revised: time

Revised: [Equation]

Revised: symbol written by M at time i m

Revised: [Equation]

Revised: : head direction in the transition of M at time

Revised: i

Revised: For the case of m ( i ) we assume that − 1 represents a movement to the left and 1 represents a movement to the right.

Revised: In our construction we show how to build a decoder that computes all the above values for every time step i using self attention plus attention over the encoder part.

Revised: Since the above values contain all the needed information to reconstruct the complete history of the computation, we can effectively simulate M .

Revised: In particular our construction produces the sequence of output vectors y 1 , y 2 , . . .

Revised: such that, for every i , the vector y i contains information about q ( i ) and s ( i ) encoded as one-hot vectors.

Revised: The construction and proof goes by induction.

Revised: We begin with an initial vector y 0 that represents the state of the computation before it has started, that is q (0) =

Revised: q init and s (0) =

Revised: For the induction step we assume that we have already computed y 1 , . . .

Revised: y r such that y i contains information about q ( i ) and s ( i ) , and we show how with input ( y 0 , y 1 , . . .

Revised: , y r ) the decoder produces the next vector y r

Original: Σ → Q 6 d +8 such that
Revised: + containing q ( r +1) and s ( r +1) .

Revised: The overview of the construction is as follows.

Revised: First notice that the transition function δ relates the above values with the following equation:

Revised: [Equation]

Revised: We prove that we can use a two-layer feed-forward network to mimic the transition function δ (Lemma B.2).

Revised: Thus, given that the input vector y i contains q ( i ) and s ( i ) , we can produce the values q ( i +1) , v ( i ) and m ( i ) (and store them as values in the decoder).

Revised: In particular, since y r is in the input, we can produce q ( r +1) which is part of what we need for y r +1 .

Revised: In order to complete the construction we also need to compute the value s ( r +1) , that is, we need to compute the symbol under the head of machine M at the next time step (time r + 1 ).

Revised: We next describe at a high level, how this symbol can be computed with two additional decoder layers.

Revised: We ﬁrst make some observations about s ( i ) that are fundamental in our computation.

Revised: Assume that at time i the head of M is pointing to the cell at index k .

Revised: Then we have three possibilities:

Revised: If i ≤ n , then s ( i ) =

Revised: s i since M is still reading its input string.

Revised: If i > n and M has never written at index k , then s ( i ) =

Revised: # , the blank symbol.

Revised: In other case, that is, if i > n and time i is not the ﬁrst time that M is pointing to index k , then s ( i ) is the last symbol written by M at index k .

Revised: For the case (1) we can produce s ( i ) by simply attending to position i in the encoder part.

Revised: Thus, if r + 1 ≤ n to produce s ( r +1) we can just attend to index r + 1 in the encoder and copy this value to y r +1 .

Revised: For cases (2) and (3) the solution is a bit more complicated, but almost all the important work is to compute what is the index that M is going to be pointing to in time r + 1 .

Revised: To formalize this computation, lets denote by c ( i )

Revised: ∈

Revised: Z the following value: c ( i ) : the index of the cell to which the head of M is pointing to at time

Revised: i

Revised: Notice that value c ( i ) satisﬁes that c ( i ) =

Revised: c ( i − 1) + m ( i − 1) .

Original: {+If we unroll this equation and assuming that c (0) = 
+}
Revised: {+If we unroll this equation and assuming that c (0) = 
+}

Revised: we obtain that

Revised: [Equation]

Revised: Then, at the step i in the decoder we have all the necessary to compute value c ( i ) but also the necessary to compute c ( i +1) .

Revised: We actually show that the computation (of a representation) of c ( i ) and c ( i +1) can be done by using one layer of self attention (Lemma B.3).

Revised: We still need to deﬁne a ﬁnal notion.

Revised: With c ( i ) one can deﬁne the helping value (cid:96) ( i ) as follows:

Revised: [Equation]

Revised: Thus, (cid:96) ( i ) is a value such that c ( (cid:96) ( i ))

Revised: =

Revised: c ( i ) , which means that at time i and at time (cid:96) ( i ) the head of M was pointing to the same cell.

Revised: Moreover, (cid:96) ( i ) is the maximum value less than i that satisﬁes such condition.

Revised: That is (cid:96) ( i ) is the last time (previous to i ) in which M was pointing to position c ( i ) .

Revised: First notice that in every step, M moves its head either to the right or to the left (it never stays in the same cell).

Revised: This implies that for every i it holds that c ( i ) (cid:54) =

Revised: c ( i − 1) , from which we obtain that (cid:96) ( i ) <

Revised: i − 1 .

Revised: Moreover, in the case that c ( i ) is visited for the ﬁrst time at time step i , the value (cid:96) ( i ) is ill-deﬁned.

Revised: In such a case we let (cid:96) ( i ) =

Revised: i − 1 .

Revised: This makes (cid:96) ( i ) ≤ i − 1 for all i , and allows us to check that c ( i ) is visited for the ﬁrst time at time step i by just checking that (cid:96) ( i ) =

Revised: i − 1 .

Revised: We now have all the necessary to explain how we compute our desired s ( r +1) value.

Revised: Assume that r + 1 > n (the case r + 1 ≤ n was already covered before).

Revised: We ﬁrst note that if (cid:96) ( r + 1) =

Revised: r then s ( r +1) =

Revised: # since this is the ﬁrst time that cell c ( r +1) is visited.

Revised: On the other hand, if (cid:96) ( r + 1) <

Revised: r then s ( r +1) is the value written by M at time (cid:96) ( r + 1) which is exactly v ( (cid:96) ( r +1)) .

Revised: Thus, in this case we only need to attend to position (cid:96) ( r +1) and copy the value v ( (cid:96) ( r +1)) to produce s ( r +1) .

Revised: We show that all this can be done with an additional self-attention decoder layer (Lemma B.4).

Revised: We have described at a high-level a decoder that, with input ( y 0 , y 1 , . . .

Revised: , y r ) , computes the values q ( r +1) and s ( r +1) which is what we need to produce y r +1 .

Revised: We next show all the details of this construction.

Revised: B.4. D ETAILS OF THE ARCHITECTURE OF Trans M

Revised: In this section we give more details on the architecture of the encoder and decoder needed to implement our strategy.

Revised: We let several intermediate claims as lemmas that we formally prove in Section B.4.3.

Revised: A TTENTION MECHANISM

Revised: For our attention mechanism we use the following non-linear function:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: We note that ϕ ( x ) = −| x | and it can be implemented as ϕ ( x ) = − relu( x ) − relu( − x ) . We use ϕ ( · ) to deﬁne a scoring function score ϕ : R d × R d → R such that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Now, let q ∈ Q d , and K = ( k 1 , . . . , k n ) and V = ( v 1 , . . . , v n ) be tuples of elements in Q d .

Original: [Equation]
Revised: We now describe how Att( q , K , V ) is generally computed when hard attention is considered. Assume ﬁrst that there exists a single j (cid:63) ∈ { 1 , . . . , n } that maximizes score ϕ ( q , k j ) . In that case we have that Att( q , K , V ) = v j (cid:63) with

Original: [Equation] 
Revised: [Equation] [Equation]

Original: [Equation]
Revised: Thus, when computing hard attention with the function score ϕ ( · ) we essentially select the vector v j such that the dot product (cid:104) q , k j (cid:105) is as close to 0 as possible. If there is more than one index, say indexes j 1 , j 2 , . . , j r , that minimizes the dot product (cid:104) q , k j (cid:105) then we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Thus, in the extreme case in which all dot products are equal (cid:104) q , k j (cid:105) for every index j , attention behaves just as an average of all value vectors, that is Att( q , K , V ) = 1 n (cid:80) nj =1 v j . We use all these properties of the hard attention in our proof. V ECTORS AND ENCODINGS

Original: [Equation]
Revised: We now describe the vectors that we use in the encoder and decoder parts of Trans M . The vectors that we use in the Trans M layers are of dimension d = 2 | Q | +4 | Σ | +11 . To simplify the exposition, whenever we use a vector v ∈ Q d , we write it arranged in four groups of values as follows

Original: [Equation]
Revised: [Equation]

Original: Our construction uses a very simple positional encoding pos : N → Q 6 d +8 such that pos( i ) is a vector that only has the value i in its last position and 0 everywhere else.{+the length of the corresponding group. Finally, we denote by 
+}
Revised: ] where q i ∈ Q | Q | , s i ∈ Q | Σ | , and x i ∈ Q . Whenever in a vector of the above form any of the four groups of values is composed only of 0 ’s, we just write ‘ 0 , . . . , 0 ’ where the length of this sequence is implicit in {+the length of the corresponding group. Finally, we denote by 
+}

Revised: q the vector in Q | Q | that has only 0 ’s, and similarly 0 s the vector in Q | Σ | that has only 0 ’s.

Original: Thus if f ( a i ) =
Revised: For a symbol s ∈ Σ , we use (cid:74) s (cid:75) to denote a one-hot vector in Q | Σ | that represents s . That is, given an enumeration π : Σ → { 1 , . . . , | Σ |} , the vector (cid:74) s (cid:75) has a 1 in position π ( s ) and a 0 in all other positions. Similarly, for q ∈ Q , we use (cid:74) q (cid:75) to denote a one-hot vector in Q | Q | that represents q .

Original: x i we have that
Revised: We have the necessary to introduce the embedding and positional encoding used in our construction. We use an embedding function f : Σ → Q d deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We denote this last vector by x i .
Revised: Our construction uses the positional encoding pos : N → Q d such that

Original: That is, the inputfor Trans is the sequence ( x 1 , x 2 , . . . , x n ) .
Revised: [Equation] Thus, given an input sequence s 1 s 2 · · · s n ∈ Σ ∗ , we have that

Original: We need a ﬁnal notation to begin with the proof.
Revised: [Equation]

Original: We will usually have the lastcomponent of all our vectors with the positional encoding. Thus, given an arbitrary vectorv ∈ Q 6 d +8 we denote by e ( v ) the value of its last component. That is, for the vector x i above we have e ( x i )
Revised: We denote this last vector by x i . That is, if M receives the input string w =

Original: = i . T HE ATTENTION MECHANISM USED In what follows we will use a scoring function score : Q 6 d +8 × Q 6 d +8 → Q that essentially performs the following computation for a pair of vectors ( q , k ) :
Revised: s

Original: [Equation] This function can be implemented with an additive attention as follows. We use a feed-forward network of three layers that receives as input a vector of the form [ q ,k ] ∈ Q 2(6 d +8) . The ﬁrst layer will be a function f 1 : Q 2(6 d +8)
Revised: 1 s 2 · · · s n , then the input for Trans M is the sequence ( x 1 , x 2 , . . .

Original: → Q 2 that just picks the values at positions 6 d +8 and at 2(6 d +8) , thus, with input [ q , k ] it produces f 1 ([ q , k ]) = [ e ( q ), e ( k )] .It is clear that this transformation can be performed with a single-layer neural network (with an identity activation) since this is just a linear transformation. The next two layers are deﬁned by functions f 2 : Q 2 → Q 2 , and f 3 : Q 2 → Q deﬁned as follows:
Revised: , x n ) .

Original: [Equation] Thus, score( q , k ) will be [Equation] It is not difﬁcult to see that the result will be
Revised: The need for using a positional encoding having values 1 /i and 1 /i 2 will be clear when we formally prove the correctness of our construction.

Original: [Equation] which is exactly what we wanted to obtain. The most important property of this score function is that, when combined with hard attention, it has the effect of attending to a speciﬁc position of the attended sequence of vectors. More formally, assume that q is a vector such that 1 ≤ e ( q ) ≤ n , and k 1 , k 2 , . .
Revised: We need a ﬁnal preliminary notion.

Original: k n are all such that e ( k i ) = i . Then we have that the following holds:
Revised: In the formal construction of Trans M we also use the following helping sequences:

Original: [Equation]
Revised: [Equation]

Original: where the 0 appears exactly in the k -th position of the above vector. Thus we have that [Equation] Where the 1 appears exactly in the k -th position.
Revised: These are used to identify when M is still reading the input string.

Original: Moreover, if we assume that e ( q ) = k such that k > n then [Equation] and thus the maximum value occurs in the last position which implies that
Revised: C ONSTRUCTION OF TEnc M

Original: [Equation] That is, it is the sequence with a 1 in its last position. Putting all this in the Att function, we obtain the following. Assume that we have a sequence
Revised: The encoder part of Trans M is very simple.

Original: K = ( k 1 , . . . , k n ) of key vectors such thate ( k i ) = i for every i . Let V =
Revised: For TEnc M we use a single-layer encoder, such that

Original: ( v 1 , . .

Original: . , v n ) be a sequence of arbitrary value vectors, and assume that we have a query q such that 1 ≤ e ( q ) ≤ n .

Original: Then we have [Equation] Moreover, whenever we have a query vector q such that e ( q ) > n then [Equation]
Revised: TEnc

Original: Given this property and also given that all vectors that we use will have a positional encoding in its last component, to simplify the notation we can use the function Att( i, K , V )

Original: = v i , and thus

Original: [Equation]

Original: Notice that

Original: [Equation]
Revised: [Equation]

Original: We extensively use these properties of the attention function in our construction.

Original: We now have all the necessary to begin with the details of our proof.

Original: C ONSTRUCTION OF TEnc

Original: The encoder part of Trans is very simple.

Original: For TEnc we use a single-layer encoder, such that

Original: TEnc( x 1 , . . . , x n )  = ( K e , V e ) where
Revised: ( K e , V e ) where K e = ( k 1 , . . . , k n ) and V e = ( v 1 , . . .

Original: [Equation] thatis, it implements the identity function for keys, and [Equation] where
Revised: , v n ) such that

Original: [Equation]
Revised: [Equation]

Original: That is, v i is just x i where the last component has been moved to the center component.

Original: It is straightforward  that this can be done with a singlelayer encoder  by just implementing the identity (that can be done by taking advantage of the residual connection in Equations (5) and (6)), and then using afﬁne transformations for V ( · ) and K ( · ) in Equation (7).
Revised: It is straightforward to see that these vectors can be produced with a single encoder layer by using a trivial self attention, taking advantage of the residual connections in Equations (6) and (7), and then using linear transformations for V ( · ) and K ( · ) in Equation (8).

Original: The most important part of the construction is in the decoderthat we next describe.
Revised: When constructing the decoder we use the following property. Lemma B.1.

Original: We base our proof in the construction the following sequences of vectors and values:
Revised: Let q ∈ Q

Original: [Equation] It is easy to see that [-since β 
-]
Revised: d be a vector such that [-since β 
-]q = [ , . . .

Original: = h 0 and α i = x i for 1 ≤ i ≤ n , then β follows the recurrence β i = σ ( β i − 1 W + x i V ) with the same starting value and equations as the h
Revised: , , 1 , j, , ] where j ∈ N and ‘ ’ denotes an arbitrary value.

Original: i ’s , and so we have thatβ i
Revised: Then we have that

Original: [-= h i for i ≤ n . Then, since γ n +1 = β n = h n = g 0 , and after that γ i = σ ( Uγ i − 1 ) , we conclude that γ n +
-]
Revised: [-= h i for i ≤ n . Then, since γ n +1 = β n = h n = g 0 , and after that γ i = σ ( Uγ i − 1 ) , we conclude that γ n +
-][Equation]

Original: + i = g i . We will show that these sequences can be computed in a transformer network, and so we can simulate the RNN N . In what follows we show an overview on how to compute all these sequences with the decoder part of Trans .
Revised: C ONSTRUCTION OF TDec M

Original: In the decoder, we show how to  produce the sequence of outputs( y 1 , y 2 , . . .
Revised: We next show how to construct the decoder part of Trans M to produce the sequence of outputs y 1 , y 2 , . . .

Original: ,y r ) where y i is given by:
Revised: , where y i is given by:

Original: [Equation]
Revised: [Equation]

Revised: That is, y i contains information about the state of M at time i , the symbol under the head of M at time i , and the last direction followed by M (the direction of the head movement at time i − 1 ).

Revised: The need to include m ( i − 1) will be clear in the construction.

Original: We consider as the starting vector for the decoder the vector
Revised: We consider as the starting vector for the decoder the vector

Original: [Equation]
Revised: [Equation]

Original: Thus, what we are actually stating is that [-β 
-] = γ 

Revised: We are assuming that [-β 
-]m ( − 1) = 0 to represent that previous to time

Original: = 0 , a 0 = c 

Revised: there was no head movement. Our construction resembles a proof by induction; we describe the architecture piece by piece and at the same time we show how for every r ≥

Original: = 0 and b 0= 1 , which coincide with the values for the sequences described above.
Revised: our architecture constructs y r +1 from the previous vectors ( y 0 , . . . , y r ) .

Original: Recall that  we are using positional encodings,thus, when producing the output, the actual input for the ﬁrst layer of the decoder will be the sequence
Revised: Thus, assume that y 0 , . . . , y r satisfy the properties stated above. Since we are using positional encodings, the actual input for the ﬁrst layer of the decoder is the sequence

Original: [Equation]
Revised: [Equation]

Original: We denote by y i the vector
Revised: We denote by y i the vector

Original: y i plus its positional encoding (which is always at its last component).
Revised: y i plus its positional encoding. Thus we have that [Equation] [Equation]

Revised: i + 1) , 1 / ( i + 1) , 1 /

Original: We now roughly describe a two-layer decoder network that, with input K e , V e and sequence  ( y 0 ,. . .
Revised: [Equation] i + 1)] For the ﬁrst self attention in Equation (9) we just produce the identity which can be easily implemented with a trivial attention plus the residual connection. Thus, we produce the sequence of vectors ( p 10 , p 11 , . .

Original: , y r ) will produce y r +1 as output (the details are in the following section).{+form [ , . . . , , 1 , i +1 , , ] by Lemma B.
+}
Revised: , p 1 r ) such that p 1 i = y i . Since p 1 i is of the {+form [ , . . . , , 1 , i +1 , , ] by Lemma B.
+}

Original: Our construction will resemble a proof by induction; we show how to construct y r +1 from theprevious vector y r .
Revised: we know that if we use p 1 i to attend over the encoder we obtain [Equation] Thus in Equation (10) we ﬁnally produce the vector a 1 i given by [Equation]

Original: The overview of the construction is very simple.
Revised: As the ﬁnal piece of the ﬁrst decoder layer we use a function O 1 ( · ) (Equation (11)) that satisﬁes the following lemma. Lemma B.2. There exists a two-layer feed-forward network O 1 : Q d → Q d such that with input vector a 1 i (21) produces as output [Equation]

Original: With input K e , V e and sequence ( y 0  , . . . 
Revised: 1) (cid:75) , (cid:74) v ( i ) (cid:75) , m ( i ) , m ( i − 1) , 0 ,0 , . . . That is, function O 1 ( · ) simulates transition δ ( q ( i ) , s ( i ) ) to construct [Equation] and m

Original: , y r ) the ﬁrst decoder layer produces a sequence of vectors z 10 , z 11 , . . 
Revised: [Equation] besides some other linear transformations. We ﬁnally produce as the output of the ﬁrst decoder layer, the sequence ( z 10 , z 11 , . . .

Original: , z 1 r  such that
Revised: , z 1 r ) such that

Original: [Equation]
Revised: [Equation]

Original: Notice that  the ﬁrst decoder layer will do almost all the heavy work.
Revised: Notice that z 1 r already holds info about q ( r +1) and m ( r ) which we need for constructing vector y r +1 . The single piece of information that we still need to construct is s ( r +1) , that is, the symbol under the head of machine M at the next time step (time r + 1 ). We next describe how this symbol can be computed with two additional decoder layers.

Revised: Recall that c ( i ) is the cell to which M is pointing to at time i , and that it satisﬁes that c ( i )

Original:  We actually use a feed-forward network with 3 layers to obtain this output.
Revised: = m (0) + m (1) + · · · + m ( i − 1) . We can take advantage of this property to prove the following lemma. Lemma B.3.

Original: After that, the second decoder layer, just do some cleaning work by moving some parts of the vector and deleting others to produce the sequence of output vectors z 20 , z 21 , . . . 
Revised: Let Z 1 i = ( z 10 , z 11 , . . . , z 1 i ) . There exists functions Q 2 ( · ) , K 2 ( · ) , and V 2 ( · ) deﬁned by feed-forward networks such that

Original: , z
Revised: [Equation] Lemma B.3 essentially shows that one can construct a representation for values c ( i ) and c ( i +1) for every possible index i . In particular we will know the value c ( r +1) that represents the cell to which the machine is pointing to in the next time step.

Revised: Continuing with the decoder layer, when using the self attention above and after adding the residual in Equation (9) we obtain the sequence of vectors ( p 20 , p 21 , . .

Original:  2 r where
Revised: p 2 r ) such that:

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Revised: From vectors ( p 20 , p 21 , . . .

Revised: , p 2 r ) and by using the residual connection in Equation (10) plus the output function O ( · ) in Equation (11) it is not difﬁcult to produce the sequence of vectors ( z 20 , z 21 , . . .

Original: This is exactly what we want asoutput, and thus, we just consider theﬁnal output transformation
Revised: , z 2 r ) such that z 2 i = p 2 i , as the output of the second decoder layer. That is, we have that [Equation]

Revised: i

Revised: [Equation]

Revised: We now describe how can we use a third and ﬁnal decoder layer to produce our desired s ( r +1) value (the symbol under the head of M in the next time step).

Revised: Recall that (cid:96) ( i ) is the last time (previous to i ) in which M was pointing to position c ( i )

Revised: [Equation]

Revised: or it is i − 1 if this is the ﬁrst time that M is pointing to c ( i ) .

Revised: We can prove the following lemma.

Revised: Lemma B.4.

Revised: There exists functions Q 3 ( · ) , K 3 ( · )

Revised: [Equation]

Revised: and V 3 ( · ) deﬁned by feed-forward networks such that

Revised: [Equation]

Revised: We prove Lemma B.4 by just showing that, for every i one can attend exactly to position (cid:96) ( i +1) and then just copy both values.

Revised: We do this by taking advantage of the values c ( i ) and c ( i +1) previously computed for every index i .

Original: F ( · ) as the identity to produce y r +1 =
Revised: Then we have that p 3 i is given by [Equation] + [Equation]

Revised: From vectors ( p 30 , p 31 , . . .

Revised: , p 3 r ) and by using the residual connection in Equation (10) plus the output function O ( · ) in Equation (11)) it is not difﬁcult to produce the sequence of vectors ( z 30 , z 31 , . . .

Revised: , z 3 r ) such that z 3 i

Revised: = p 3 i , as the output of the third and ﬁnal decoder layer, and thus we have that

Original:  F ( z 2 r ) (Equation (11)) which completes our construction.
Revised: [Equation] We ﬁnish our construction by using the ﬁnal transformation function F ( · ) in Equation (12) in the following lemma. Lemma B.5. There exists a function F : Q d → Q d deﬁned by a feed-forward network such that

Revised: [Equation]

Revised: We prove Lemma B.5 as follows (details in the next section).

Original: D ETAILED CONSTRUCTION OF TDec
Revised: We show that one can construct a feedforward network that with input z 3 r implements the following to produce y r +1 . We move (cid:74) q ( r +1) (cid:75) and m ( r ) to its corresponding position in y r +1 . Then

Revised: if β ( r +1)

Revised: =

Original: {+r + 
+}
Revised: {+r + 
+}

Revised: then we let

Revised: [Equation]

Revised: if β ( r +1) < r + 1 and (cid:96) ( r + 1) =

Revised: r , then we let (cid:74) s ( r +1)

Revised: [Equation]

Revised: = (cid:74)

Revised: # (cid:75) , and 3.

Revised: if β ( r +1) < r + 1 and (cid:96) ( r + 1) (cid:54) =

Revised: r , then we let

Revised: [Equation]

Revised: Finally, we move (cid:74) s ( r +1) (cid:75) to its corresponding position in

Original: {+y r +
+}
Revised: {+y r +
+}

Revised: and we make all other positions 0 .

Revised: The correctness of the above rules is given by the following argument.

Revised: If β ( r +1)

Revised: =

Revised: r +1 then by the deﬁnition of β ( r +1)

Revised: we have that r + 1 ≤ n which implies that α ( r +1) =

Original: {+s r +
+}
Revised: {+s r +
+}

Revised: = s ( r +1) and thus rule (1) above is correct.

Revised: If β ( r +1) < r + 1 then we know that r + 1 > n and if (cid:96) ( r + 1) =

Revised: r then by the deﬁnition of (cid:96) ( · ) we know that c ( r +1) is visited by M for the ﬁrst time at time r + 1 , which implies that s ( r +1) =

Revised: # and thus rule (2) is also correct.

Revised: Finally, If β ( r +1) < r +1 and (cid:96) ( r +1)

Revised: (cid:54) =

Revised: r , then we know that c ( r +1) has been visited before at time (cid:96) ( r +1) , and thus s ( r +1)

Revised: = v ( (cid:96) ( r +1)) which implies the correctness of rule (3).

Revised: F INAL STEP

Revised: We now can use our Trans M network to construct the recognizer A =

Revised: (Σ , f, Trans M , y 0 , F ) such that M = ( Q, Σ , δ, q init , F ) accepts w if and only if A accepts w .

Revised: Notice that M accepts w if and only if an accepting state q f ∈ F

Revised: B.4. D ETAILED PROOFS OF INTERMEDIATE LEMMAS

Revised: Proof of Lemma B.1.

Revised: Let q ∈ Q

Revised: d be a vector such that q = [ , . . .

Revised: , , 1 , j, , ] where j ∈ N and ‘ ’ is an arbitrary value.

Revised: We next prove that

Revised: [Equation]

Revised: where α ( j ) and β ( j ) are deﬁned as

Revised: [Equation]

Revised: Recall that K e

Revised: =

Revised: ( k 1 , . . .

Revised: , k n ) is such that k i =

Revised: . , 0 , i, − 1 , 0 , 0 ] .

Revised: Then we have that

Revised: [Equation]

Revised: Notice that, if j ≤ n , then the above expression is maximized when i = j .

Revised: Otherwise, if j > n then the expression is maximized when i = n .

Revised: Then Att( q , K e , V e ) = v i (cid:63) where i (cid:63) =

Revised: j if j ≤ n and i (cid:63) = n if j > n .

Revised: We note that i (cid:63) as just deﬁned is exactly β ( j ) .

Revised: Thus, given that v i is deﬁned as

Revised: [Equation]

Revised: we obtain that

Revised: [Equation]

Original: First, we have the following useful identities for α i , β i and γ i thatcan be obtained by simply inspecting the deﬁnitions above and the attention function that we deﬁned:
Revised: ] which is what we wanted to prove. Proof of Lemma B.2. In order to prove the lemma we need some intermediate notions and properties. Assume that the enumeration π 1 : Σ → { 1 , . .

Revised: . , | Σ |} is the one used to construct the one-hot vectors (cid:74) s (cid:75) for s ∈ Σ , and that π 2 : Q → { 1 , . .

Revised: . , | Q |} is the one used to construct (cid:74) q (cid:75) with q ∈ Q .

Revised: Using π 1 and π 2 one can construct an enumeration for the pairs in Q × Σ and then construct one-hot vectors for pairs in this set.

Revised: Formally, given ( q, s ) ∈ Q × Σ we denote by (cid:74) ( q, s ) (cid:75) a one-hot vector with a 1 in position ( π 1 ( s ) − 1) | Q | + π 2 ( q ) and a 0 in every other position.

Revised: To simplify the notation we use π ( q, s ) to denote ( π 1 ( s ) − 1) | Q | + π 2 ( q ) .

Revised: One can similarly construct an enumeration π (cid:48) for Q × Σ × {− 1 , 1 } such that π (cid:48) ( q, s, m ) =

Revised: π ( q, s ) if m = − 1 and π (cid:48) ( q, s, m ) =

Revised: | Q || Σ | + π ( q, s ) if m = 1 .

Revised: We denote by (cid:74) ( q, s, m ) (cid:75) the corresponding one-hot vector for every ( q, s, m ) ∈ Q × Σ ×{− 1 , 1 } .

Revised: We next prove three helping properties.

Revised: In every case q ∈ Q , s ∈ Σ , m ∈ {− 1 , 1 } , and δ ( · , · ) is the transition function of machine M .

Revised: There exists f 1 : Q | Q | + | Σ | → Q | Q || Σ | such that f 1 ([ (cid:74) q (cid:75) , (cid:74) s (cid:75) ]) =

Revised: (cid:74) ( q, s ) (cid:75) .

Revised: There exists f δ : Q | Q || Σ | → Q 2 | Q || Σ | such that f δ ( (cid:74) ( q, s ) (cid:75) )

Revised: =

Revised: (cid:74) δ

Revised: ( q, s ) (cid:75) .

Revised: There exists f 2 : Q 2 | Q || Σ | → Q | Q | + | Σ | +1 such that f 2 ( (cid:74) ( q, s, m ) (cid:75) )

Revised: =

Revised: [ (cid:74) q (cid:75) , (cid:74) s (cid:75) , m ] .

Revised: To show (1), lets denote by S i , with i ∈ { 1 , . .

Revised: . , | Σ |} , a matrix of dimensions | Σ | × | Q | such that

Revised: S i has its i -th row with 1 ’s and it is 0 everywhere else.

Revised: We note that for every s ∈ Σ it holds that (cid:74) s (cid:75) S i = 1 if and only if i = π 1 ( s ) and it is 0 otherwise.

Revised: Now, consider the vector v ( q,s )

Revised: [Equation]

Revised: q,s

Revised: [Equation]

Revised: [Equation]

Revised: We ﬁrst note that for every i ∈ { 1 , . .

Revised: . , | Σ |} , if i (cid:54) =

Revised: π 1 ( s ) then (cid:74) q (cid:75) + (cid:74) s (cid:75) S

Revised: i =

Revised: (cid:74) q (cid:75)

Revised: + 0 = (cid:74)

Revised: q (cid:75) .

Revised: Moreover (cid:74) q (cid:75) + (cid:74) s (cid:75)

Revised: S

Revised: π 1 ( s ) =

Revised: (cid:74) q (cid:75) + 1 is a vector that has a 2 exactly at index

Revised: π 2 ( q ) , and it isin all other positions.

Revised: Thus, the vector v ( q,s ) has a 2 exactly at position ( π 1 ( s ) − 1)

Revised: | Q | + π 2 ( q ) and it is either 0 or 1 in every other position.

Revised: Now, lets denote by o a vector in Q | Q || Σ | that has a 1 in every position and consider the following afﬁne transformation

Revised: [Equation]

Revised: Vector g 1 ([ (cid:74) q (cid:75) , (cid:74) s (cid:75) ]) has a 1 only at position ( π 1 ( s ) − 1) | Q | + π 2 ( q ) =

Revised: π ( q, s ) and it is less than or equal to 0 in every other position.

Revised: Thus, to construct f 1 ( · ) we apply the piecewise-linear sigmoidal activation σ ( · ) (see Equation (1)) to obtain

Revised: [Equation]

Revised: which is what we wanted.

Revised: Now, to show (2), lets denote by M δ a matrix of dimensions ( | Q || Σ | ) ×

Revised: (2 | Q || Σ | ) constructed as follows.

Revised: For ( q, s ) ∈ Q × Σ , if δ ( q, s ) =

Revised: ( p, r, m ) then M δ has a 1 at position ( π ( q, s ) , π (cid:48)

Revised: ( p, r, m ))

Revised: and it has a 0 in every other position, that is

Original: [Equation]
Revised: [Equation]

Revised: It is straightforward to see that (cid:74) ( q, s ) (cid:75)

Revised: M δ = (cid:74) δ ( q, s ) (cid:75) , and thus we can deﬁne f 2 ( · ) as

Revised: [Equation]

Revised: To show (3), consider the matrix A of dimensions (2 | Q || Σ | ) × ( | Q | + | Σ | + 1) such that

Revised: [Equation]

Revised: Then we deﬁne f 3 ( · ) as

Revised: [Equation]

Revised: We are now ready to begin with the proof of the lemma.

Revised: Recall that a 1 i is given by

Revised: [Equation]

Revised: We need to construct a function O 1 : Q d → Q d such that

Revised: [Equation]

Revised: We ﬁrst use function h 1 ( · ) that works as follows.

Revised: Lets denote by ˆ m ( i − 1) the value 12 m ( i − 1) + 12 .

Revised: Note that ˆ m ( i − 1) is 0 if m ( i − 1) = − 1 , it is 12 if m ( i − 1)

Original: {+= 
+}
Revised: {+= 
+}

Original: {+and it is 
+}
Revised: {+and it is 
+}

Revised: if m ( i − 1) = 1 .

Revised: We use this transformation just to represent m ( i − 1) with a value between 0 and 1 .

Revised: Now, consider h 1 ( a 1 i ) deﬁned by

Revised: [Equation]

Original: We also have some identities for a i  , b i and c i :
Revised: where g 1 ( · ) is the function deﬁned above in Equation (25). It is clear that h 1 ( · ) is an afﬁne transformation. Moreover, we note that except for g 1 ([ (cid:74) q ( i ) (cid:75) , (cid:74) s ( i ) (cid:75) ]) all values in h 1 ( a 1 i ) are betweenand 1 . Thus if we apply function σ ( · ) to h 1 ( a 1 i ) we obtain

Original: [Equation]
Revised: [Equation]

Original: We also note that b i can be rewritten as:
Revised: Then we can deﬁne h 2 ( · ) such that [Equation] Now we can deﬁne h 3 ( · ) as [Equation]

Revised: Finally we can apply a function h 4 ( · ) to just reorder the values and multiply some components by − 1 to complete our construction

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst layer of the decoder, that we call Dec 1 , is deﬁned by the functions Q 1 ( · ) , V 1 ( · ) , K 1 ( · ) , and  O 1 ( · )  that we now describe.
Revised: We note that we applied a single non-linearity and all other functions are afﬁne transformations. Thus O 1 ( · ) can be implemented with a two-layer feed-forward network. Proof of Lemma B.3. Recall that z

Original: We use the function V 1 : Q6 d+8 → Q6 d +8 such that V 1 ( v ) is the vector with only zeroes, thus, we just nullify the self-attention in the ﬁrst decoder layer.
Revised: i is the following vector [Equation] We consider Q 2 : Q d → Q d and K 2 : Q d → Q d as trivial functions that for every input produce an output vector composed of only 0 ’ s. Moreover, we consider V 2 : Q d → Q d such that for every j ∈ { 0 , 1 , . .

Original: This plus the residual connection in Equation (8) has the net effect of producing the identity and thus we obtain p 1 i 
Revised: , i } V 2 ( z 1 j ) = , 0 , 0 q , 0 s , 0 , 0 , m ( j ) , m ( j − 1) , 0 , . . . Then, since K 2 ( z 1 j ) is the vector with only zeros, then score ϕ ( Q 2 ( z 1 i ) , K 2 ( z 1 j ))

Original: = y i{+
+} for every i ∈ { 0 , . . .
Revised: = {+
+} for every j ∈ { 0 , . . .

Original: , r } .
Revised: , i } .

Original: Notice that e ( p 1 i ) = 
Revised: Thus, we have that the attention Att( Q 2 ( z 1 i ) , K 2 ( Z 1 i ) , V 2 ( Z 1 i )) that we need to compute is just the average of all the vectors in V 2 ( Z 1 i ) = ( V 2 ( z 10 , . . .

Original: e ( y i ) 
Revised: , z 1 i ) , that is

Revised: Att( Q 2 ( z 1 i ) , K 2 ( Z 1 i ) , V 2 ( Z 1 i ))

Original: =
Revised: =

Original: i + 1 , and thus, if i + 1 ≤ n we have that [Equation] [Equation] [Equation]{++1) (cid:80) ij =
+}
Revised: 1 ( i {++1) (cid:80) ij =
+}

Original: [Equation] [Equation]
Revised: V 2 ( z 1 j ) =

Original: + [Equation] 0 , 0  , 0  , 0 , 0 , 0] and, otherwise, if i + 1 > n we have [Equation]
Revised: , 0 , 0 q , 0 s , 0 , 0 , 1 ( i +1) (cid:80) ij =0 m ( j ) , 1 ( i +1) (cid:80) ij =0 m ( j − 1) , 0 , . . .

Original: Thus, combining both cases we actually have that
Revised: Then, since m (0) + · · · + m ( i ) =

Original: [Equation] This last result is summed with p 1 i  =
Revised: c ( i +1) and m ( − 1) + m (0) + · · · + m ( i − 1) =

Original: y i (the residual connection in Equation (9)) to obtain a 1 i which is given by: [Equation]
Revised: c ( i ) we have that Att( Q 2 ( z 1 i ) , K 2 ( Z 1 i ) , V 2 ( Z 1 i )) =

Original: [-We now describe the function O 1 ( · ) which is going to be deﬁned by a feed-forward neural network that we next describe. The ﬁrst layer of the network will be given by a function f 1 : Q 6 d +8 → Q 6 d + such that with input a 1 i it will produce [Equation] It is clear that f 1 can be implemented with a single neural-network layer as it only has to apply the afﬁne transformation given by the equations a i +
-]
Revised: [-We now describe the function O 1 ( · ) which is going to be deﬁned by a feed-forward neural network that we next describe. The ﬁrst layer of the network will be given by a function f 1 : Q 6 d +8 → Q 6 d + such that with input a 1 i it will produce [Equation] It is clear that f 1 can be implemented with a single neural-network layer as it only has to apply the afﬁne transformation given by the equations a i +
-], 0 , 0 q , 0 s , 0 , 0 , c

Original: = 1 − (c i +1 − c  i ) and b i [-+
-]
Revised: ( i +1) ( i +1) , c ( i ) ( i [-+
-]+1) , 0 , . . .

Original: =
Revised: ,] which is exactly what we wanted to show. Proof of Lemma B.4.

Original: c i +1 − c
Revised: Recall that z

Original: i + a i , and making some positions equal to 0 . After f 1 we apply another feed-forward layer f 2 : Q 6 d +8 → Q 6 d +8 given by f 2( f 1 ( a 1 i))
Revised: i is the following vector z 2 i

Original: =
Revised: =

Original: σ ([ 0 , β i  , γ i  , 0 , 0 , 0 , 0 , β i W + α i +1 V , γ i U , β i −
Revised: . , 0 , (cid:74) q ( i +1) (cid:75) , (cid:74) v ( i ) (cid:75) , m ( i ) , m ( i − 1) , c ( i +1)

Revised: ( i +1) , c ( i ) ( i +1) , (cid:74) α

Original: b i +1 1 , a i +1 , 
Revised: ( i +1) (cid:75) , β ( i +1) , 0 s , 0 , 1 , ( i + 1) , 1 / ( i + 1) , 1 / ( i + 1)] We need to construct functions Q 3 ( · ) , K 3 ( · ) , and V 3 ( · ) such that [Equation] We ﬁrst deﬁne the query function Q 3 : Q d → Q d such that

Original: b i +1 , 0 ,0]) where 1 is a vector in Q d with only ones.
Revised: [Equation] Now, for every j ∈ { 0 , 1 , . . , i } we deﬁne K 3 : Q d → Q d and V 3 : Q d → Q d such that [Equation]

Original: It is clear that the internal vector can be computed with an afﬁne transformation.
Revised: It is clear that the three functions are linear transformations and thus they can be deﬁned by feedforward networks. Consider now the attention Att( Q 3 ( z 2 i ) , K 3 ( Z 2 i ) , V 3 ( Z 2 i )) . In order to compute this value, and since we are considering hard attention, we need to ﬁnd the value j ∈ { 0 , 1 , . . , i } that maximizes

Revised: [Equation]

Original: Since a i +1 , b i +1 are all rationals between 0 and 1 , and all the values in β i and γ i are also between 0 and 1 , when we apply the function σ we obtain f
Revised: Actually, assumming that such value is unique, lets say j (cid:63) , then we have that [Equation] We next show that given our deﬁnitions above, it always holds that j (cid:63) = (cid:96) ( i + 1) and then V 3 ( z 2 j (cid:63) ) is exactly the vector that we wanted to obtain.

Revised: To simplify the notation, we denote by χ ij the dot product (cid:104) Q 3 ( z 2 i ) , K 3 ( z 2 j ) (cid:105) .

Revised: Thus, we need to ﬁnd j

Revised: [Equation]

Revised: = arg max j ϕ ( χ ij ) .

Revised: Moreover, given the deﬁnition of ϕ (see Equation (20))we have that

Revised: [Equation]

Revised: Then, it is enough to prove that

Revised: [Equation]

Revised: Now, by our deﬁnition of Q 3 ( · ) and K 3 ( · ) we have that

Revised: [Equation]

Revised: where ε k = 1 ( k +1) .

Revised: We next prove the following helping property.

Revised: If j 1 is such that c ( j 1 ) (cid:54)

Revised: =

Revised: c ( i +1)

Revised: and j 2 is such that c ( j 2 )

Revised: =

Revised: c ( i +1) , then | χ ij

Original: [Equation] We note that σ ( β i W + α
Revised: [Equation] [Equation] In order to prove (26), assume ﬁrst that j 1 ∈ { 0 , . .

Original:  i +1 V ) is actually β
Revised: , i } is such that c ( j 1 ) (cid:54)

Original:  i +1 .
Revised: = c ( i +1) .

Original: Moreover, as we next describe, it holds that σ (β i − b
Revised: Then we have that | c ( i +1) − c ( j 1 ) | ≥ 1 since c ( i +1) and c ( j 1 ) are integer values. From this we have two possibilities for χ i j 1 :

Original:  i +1 1 ) = (1 − b i +1 )
Revised: If c ( i +1) − c ( j 1 ) ≤ − 1 , then

Original: β i .
Revised: [Equation]

Original: To see this, notice that β i has all its values between 0and 1 .[-Then, if b i +1 = 
-]
Revised: Notice that 1 ≥ ε j 1 ≥ ε i > 0 .[-Then, if b i +1 = 
-]

Original:  we have thatβ i −
Revised: Then we have that

Original: b i +1 1 is a vector with all its components less than or equal to 0 which implies that σ ( β i − b i +1 1 ) 
Revised: ε i ε j 1 ≥ ( ε i ε j 1 ) 2 > 13 ( ε i ε j 1 ) 2 , and thus | χ ij 1 | ≥ ε i ε j 1 − ( ε i ε

Original: = 0 .
Revised: [Equation]

Original: On the other hand, if b i +1 = 0 then σ ( β i − b i +1 1 ) = σ ( β i )
Revised: [Equation] 2Finally, and using again that 1 ≥ ε j 1 ≥ ε i > 0 , from the above equation we obtain that [Equation]

Original: {+• If c ( i +1) − c ( j 1 ) ≥ 1 , then χ ij 1 ≥ ε i ε j 1 + 
+}
Revised: {+• If c ( i +1) − c ( j 1 ) ≥ 1 , then χ ij 1 ≥ ε i ε j 1 + 
+}

Original: {+ε i ε j 1 ) 2 and since 1 ≥ ε j 1 ≥ ε i > 
+}
Revised: {+ε i ε j 1 ) 2 and since 1 ≥ ε j 1 ≥ ε i > 
+}

Revised: we obtain that | χ ij 1 | ≥ ε i ε j 1 ≥

Revised: ε i ε i ≥ 23 ( ε i ) 2 .

Revised: Thus, we have that if c ( j 1 ) (cid:54) =

Revised: c ( i +1) then | χ ij 1 | ≥ 23 ( ε i ) 2 .

Revised: Now assume j 2 ∈ { 0 , . .

Revised: , i } is such that c ( j 2 )

Revised: =

Revised: c ( i +1) .

Revised: In this case we have that

Revised: [Equation]

Revised: We showed that if c ( j 1 ) (cid:54) =

Revised: c ( i +1) then | χ ij 1 | ≥ 23 ( ε i ) 2 and if c ( j 2 ) =

Revised: c ( i +1) then | χ ij 2 | ≤ 13 ( ε i )which implies that | χ ij 2 | < | χ ij 1 | .

Revised: This completes the proof of the property in (26).

Revised: We have now all the necessary to prove that arg min j | χ ij | =

Revised: (cid:96) ( i + 1) .

Revised: Recall ﬁrst that (cid:96) ( i + 1) is deﬁned as (cid:96) ( i + 1) =

Revised: (cid:26) max { j | j ≤ i and c ( j ) =

Revised: c ( i +1) } if there exists j ≤ i s.t.

Revised: c ( j ) =

Revised: c ( i +1) , i in other case.

Revised: Assume ﬁrst that there exists j ≤ i such that c ( j )

Revised: =

Revised: c ( i +1) .

Revised: By (26) we know that

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: On the contrary, assume that for every j ≤ i it holds that c ( j ) (cid:54)

Original: =
Revised: =

Revised: c ( i +1) .

Revised: We will prove that in this case | χ ji | < | χ ii | for every j < i and thus arg min j ∈{ 0 ,...,i } | χ ij | =

Original: β i . 
Revised: i . Now, since c ( j ) (cid:54) = c ( i +1) for every j ≤ i , then c ( i +1) is a cell that has never been visited before by M .

Revised: Given that M never makes a transition to the left of its initial cell, then cell c ( i +1) is a cell to the right of every other previously visited cell.

Original: Thus we can write the output of this layer as:
Revised: This implies that c ( i +1) > c ( j ) for every j ≤ i . Thus, for every j ≤ i we have c ( i +1) − c ( j ) ≥ 1 . This implies that | χ ij | =

Revised: χ ij ≥ ε i ε j + 13 ( ε i ε j ) 2 .

Revised: Moreover, notice that if j < i then ε j > ε i and thus, if j < i we have that

Original: [Equation]
Revised: [Equation]

Revised: which implies that arg min j ∈{ 0 ,...,i } | χ ij |

Revised: = i .

Revised: Summing it up, we have shown that

Revised: [Equation]

Revised: [Equation]

Revised: i +1) } if there exists j ≤ i s.t.

Revised: c ( j ) =

Revised: c ( i +1) , i in other case.

Revised: which is exactly the deﬁnition of (cid:96) ( i + 1) .

Revised: This completes the proof of the lemma.

Revised: Proof of Lemma B.5.

Revised: Before going to the proof of Lemma-B.5 we prove the following helping result that allows us to implement a particular type of if statement with a feed-forward network.

Revised: Lemma B.6.

Revised: Let x ∈ { 0 , 1 } m

Revised: and y , z ∈ { 0 , 1 } n be binary vectors, and let b ∈ { 0 , 1 } .

Original: We need a ﬁnal linear layer given by a  function f 3 : Q 6 d +8 → Q 6 d
Revised: There exists a two-layer feed-forward network f : Q m +2 n +1 → Q m + n such that [Equation] Proof. Consider the function f 1 : Q m +2 n +1 → Q m +2 n such that

Revised: [Equation]

Revised: where 1 is the n -dimensional vector with only ones.

Original: [-+
-]
Revised: [-+
-]Thus, we have that [Equation] Now, since x , y and z are all binary vectors, it is easy to obtain that [Equation]

Original: that does the following computation
Revised: Finally, consider the function f 2 : Q m +2 n → Q m + n such that f 2 ([ x , y , z ]) = [ x , y + z ] . Then we have that [Equation]

Revised: We note that f 1 ( · ) and f 2 ( · ) are afﬁne transformations, and thus f ( · ) = f 2 ( σ ( f 1 ( · ))) is a two-layer feed-forward network.

Revised: This completes our proof.

Revised: We can now continue with the proof of Lemma B.5.

Revised: Recall that z 3 r is the following vector

Original: [Equation]
Revised: [Equation]

Original: Notice that σ ( γ i U ) + (1 − b
Revised: Lets denote by (cid:74) m ( r ) (cid:75) a vector such that [Equation] We ﬁrst consider the function f 1 : Q → Q such that [Equation]

Revised: ) (cid:75) , (cid:74) # (cid:75) , (cid:96) ( r + 1) + r − 1 ]

Revised: It is straightforward that f 1 ( · ) can be implemented as an afﬁne transformation.

Revised: Just notice that (cid:74) # (cid:75) is a ﬁxed vector, (cid:96) ( r +1)+ r − 1 = (cid:96) ( r +1)+( r +1) − 2 and that (cid:74) m ( r ) (cid:75) =

Revised: [ m ( r ) 2 , − m ( r )]

Original: i +1 )  β 
Revised: +[ 12 , 12 ] . Moreover, all values in f 1 ( z 3 r ) are binary values except for ( r + 1) − β ( r +1) and (cid:96) ( r + 1) + r − 1 .

Original: i is exactly γ i +1 thus the output of f  3 can be written as
Revised: Thus, if we apply function σ ( · ) to f 1 ( z 3 r ) we obtain

Original: [Equation]
Revised: [Equation]

Original:  1 i ) [Equation] [Equation] [Equation]
Revised: where b 1 = σ (( r +1) − β ( r +1) ) and b 2 = σ ( (cid:96) ( r +1)+ r − 1) .

Original: [Equation] [Equation] [Equation]
Revised: By the deﬁnition of β ( r +1)

Original: [Equation] a i +
Revised: we know that β ( r +1)

Original: [Equation]
Revised: =

Original: i +[Equation] This is the output of function O 1  ( · ) .
Revised: r + 1 whenever r + 1 ≤ n , and β ( r +1) = n if r + 1 > n .

Original: Thus. we have that
Revised: Thus we have that

Original: [Equation]
Revised: [Equation]

Original: Now the output of this decoder layer is z 1 i =O 1  ( a 1 i ) + a 1 i (Equation (10))
Revised: For the case of b 2 , since (cid:96) ( r + 1) ≤ r we have that b 2 = 1 if (cid:96) ( r + 1) =

Original:  and thus we have
Revised: r and it is 0 otherwise. Then, we can use the if function in Lemma B.6 to implement a function f 2 ( · ) such that

Original: [Equation]
Revised: [Equation]

Original: For the second decoder layer Dec 2 , it is straightforward to construct a layer that just do some cleaning and thus from z 10 , z 11 , . . .
Revised: We can use again the if function in Lemma B.6 to implement a function f 3 ( · ) such that [Equation] if b 2 = 0 and b 1 = [Equation]

Original:  , z 1 r produces the sequence z 20 , z 21 , . . .
Revised: 1) (cid:75) , (cid:74) m ( r ) (cid:75) , (cid:74) v ( (cid:96) ( r +1)) (cid:75) ] if b 2 = 0 and b 1 = 1 , [ (cid:74) q

Revised: ( r +1) (cid:75) , (cid:74) m

Original:  , z 2  r such that
Revised: ( r ) (cid:75) , (cid:74) α ( r +1) (cid:75) ] if b 2 = 1 and b 1 = 0 , [ (cid:74) q ( r +1) (cid:75) , (cid:74) m

Revised: ( r ) (cid:75) , (cid:74)

Revised: # (cid:75) ]

Revised: if b 2 = 1 and b 1 = 1 , which can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: We ﬁnally use F ( · ) as the identity function to produce our desired output from z 2 r , that is
Revised: ] if r + 1 ≤ n, [ (cid:74)

Original: [Equation]
Revised: [Equation]

Original: This completes the construction and the correctness of our construction.
Revised: (cid:75) , (cid:74)

Original: F INAL STEP
Revised: [Equation] ( r

Original: We now can use our Trans network to construct the recognizer A (cid:48)
Revised: [Equation] (cid:75) , (cid:74) # (cid:75) ] if r + 1 > n and (cid:96) ( r + 1)

Original: = (Σ , f (cid:48) , Trans , s (cid:48) , F (cid:48) ) .
Revised: = r, [ (cid:74) q ( r +1) (cid:75) , (cid:74) m ( r

Original: We use s (cid:48) as simple y 0 in our above construction.
Revised: [Equation] (cid:75) , (cid:74) v ( (cid:96) ( r +1)) (cid:75) ] if r + 1 > n and (cid:96) ( r + 1) (cid:54) =

Original: Finally, notice that in Trans the vector γ i is actually the output of the RNN encoder-decoder that we are simulating but only for i ≥ n +1 .
Revised: r. From this, it is easy to prove that [Equation]

Original: We take advantage of the value a i which is 1 if and only if i ≥ n +1 .{+then α ( r +1) = s r +
+}
Revised: This can be obtained from the following observation. If r + 1 ≤ n {+then α ( r +1) = s r +
+}

Original: Thus we can deﬁne F (cid:48)
Revised: = s ( r +1) . If r + 1 > n and (cid:96) ( r + 1)

Original: =
Revised: =

Original: {+r then we know that c ( r +1) is visited by M for the ﬁrst time at time r + 
+}
Revised: {+r then we know that c ( r +1) is visited by M for the ﬁrst time at time r + 
+}

Original: Q 2 d × F ×{ 1 }× Q 3 d +7 .
Revised: and it is outside the original input , which implies that s ( r +1) = Finally, If r + 1 > n and (cid:96) ( r + 1) (cid:54) = r , then we know that c ( r +1) has been visited before at time (cid:96) ( r + 1) , and thus s ( r +1) = v ( (cid:96) ( r +1)) .

Revised: The ﬁnal piece of the proof is to just convert (cid:74) m ( r ) (cid:75) back to its value m ( r ) , reorder the values and add 0 ’s to obtain y r +1 .

Revised: We do all this with a ﬁnal linear transformation f 4 ( · ) such that

Original: This completes the proof of the Theorem.
Revised: [Equation] which completes our proof.

Original: C P ROOFS FOR S ECTIONC. P ROOF OF T HEOREM 4. 
Revised: C P ROOFS FOR S ECTIONC. P ROOF OF T HEOREM 4.

Original: The formulas of the Neural GPU in detail are as follows (with S 

Revised: The formulas of the Neural GPU in detail are as follows (with S

Original: the initial input tensor):
Revised: the initial input tensor):

Original: [Equation]
Revised: [Equation]

Original: With U ( · ) , R ( · ) , and F ( · ) deﬁned as
Revised: With U ( · ) , R ( · ) , and F ( · ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Consider now an RNN encoder-decoder N of dimension d and composed of the equations
Revised: Consider now an RNN encoder-decoder N of dimension d and composed of the equations

Original: [Equation]
Revised: [Equation]

Original: with h 0 = 0 and g 0 =
Revised: with h 0 = 0 and g 0 =

Original: h n where n is the length of the input.
Revised: h n where n is the length of the input.

Original: C ONSTRUCTING THE N EURAL GPU TO SIMULATE N
Revised: C ONSTRUCTING THE N EURAL GPU TO SIMULATE N

Original: We construct a Neural GPU network NG PU that simulates N as follows.
Revised: We construct a Neural GPU network NG PU that simulates N as follows.

Original: Assume that the input of
Revised: Assume that the input of

Original: N is X
Revised: N is X

Original: = ( x 1 , . . .
Revised: = ( x 1 , . . .

Original: , x n ) .
Revised: , x n ) .

Original: Then we ﬁrst construct the sequence X (cid:48)
Revised: Then we ﬁrst construct the sequence X (cid:48)

Original: =
Revised: =

Original: ( x (cid:48) 1 , . . .
Revised: ( x (cid:48) 1 , . . .

Original: , x (cid:48) n ) such that x (cid:48) i =
Revised: , x (cid:48) n ) such that x (cid:48) i =

Original: [ x i , 0 , 0 , 1 , 1 , 0] with 0 ∈ Q d the vector with all values as 0 .
Revised: [ x i , 0 , 0 , 1 , 1 , 0] with 0 ∈ Q d the vector with all values as 0 .

Original: Notice that x (cid:48) i ∈ Q 3 d +3 , moreover it is straightforward that if x i was constructed from an embedding function f : Σ → Q d applied to a symbol a ∈ Σ , then x (cid:48)
Revised: Notice that x (cid:48) i ∈ Q 3 d +3 , moreover it is straightforward that if x i was constructed from an embedding function f : Σ → Q d applied to a symbol a ∈ Σ , then x (cid:48)

Original: i can also be constructed with an embedding function f (cid:48) :
Revised: i can also be constructed with an embedding function f (cid:48) :

Original: Σ → Q 3 d +3 such that f (cid:48) ( a ) =
Revised: Σ → Q 3 d +3 such that f (cid:48) ( a ) =

Original: [ f ( a ) , 0 , 0 , 1 , 1 , 0] .
Revised: [ f ( a ) , 0 , 0 , 1 , 1 , 0] .

Original: We consider an input tensor S ∈ Q n × 1 × 3 d +3 such that for every i ∈ { 1 , . . .
Revised: We consider an input tensor S ∈ Q n × 1 × 3 d +3 such that for every i ∈ { 1 , . . .

Original: , n } it holds that
Revised: , n } it holds that

Original: S i, 1 , : = x (cid:48) i =
Revised: S i, 1 , : = x (cid:48) i =

Original: [ x i , 0 , 0 , 1 , 1 , 0] .
Revised: [ x i , 0 , 0 , 1 , 1 , 0] .

Original: Notice that since we picked w = 1 , our tensor S is actually a 2 D grid.
Revised: Notice that since we picked w = 1 , our tensor S is actually a 2 D grid.

Original: Our proof shows that a bi-dimensional tensor is enough for simulating an RNN.
Revised: Our proof shows that a bi-dimensional tensor is enough for simulating an RNN.

Original: We now describe how to construct the kernel banks K U , K R and K F of shape (2 , 1 , 3 d + 3 , 3 d + 3) .
Revised: We now describe how to construct the kernel banks K U , K R and K F of shape (2 , 1 , 3 d + 3 , 3 d + 3) .

Original: Notice that for each kernel K X we essentially have to deﬁne two matrices K X 1 , 1 , : , : and K X 2 , 1 , : , : each one of dimension (3 d + 3) × (3 d + 3) .
Revised: Notice that for each kernel K X we essentially have to deﬁne two matrices K X 1 , 1 , : , : and K X 2 , 1 , : , : each one of dimension (3 d + 3) × (3 d + 3) .

Original: We begin by deﬁning every matrix in K F as block matrices.
Revised: We begin by deﬁning every matrix in K F as block matrices.

Original: When deﬁning the matrices, all blank spaces are considered to be 0 .
Revised: When deﬁning the matrices, all blank spaces are considered to be 0 .

Original: [Equation]
Revised: [Equation]

Original: UU
Revised: UU

Original: [Equation]
Revised: [Equation]

Original:  where F 1 and F 2 are 3 × 3 matrices deﬁned by
Revised:  where F 1 and F 2 are 3 × 3 matrices deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Tensors K U and K R are considerable simpler.
Revised: Tensors K U and K R are considerable simpler.

Original: For the case of K U we have
Revised: For the case of K U we have

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:  where U 1 and U 2 are 3 × 3 matrices deﬁned by
Revised:  where U 1 and U 2 are 3 × 3 matrices deﬁned by

Original: [Equation]
Revised: [Equation]

Original: and where A is the 3 × d matrix deﬁned by
Revised: and where A is the 3 × d matrix deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Finally, we deﬁne K R as
Revised: Finally, we deﬁne K R as

Original: [Equation]
Revised: [Equation]

Original:  where R 2 is the 3 × 3 matrix deﬁned by
Revised:  where R 2 is the 3 × 3 matrix deﬁned by

Original: [Equation]
Revised: [Equation]

Original: and where B is the 3 × d matrix deﬁned by
Revised: and where B is the 3 × d matrix deﬁned by

Original: [Equation]
Revised: [Equation]

Original: The bias tensors B U and B F are 0 (the tensor with all values 0 ).
Revised: The bias tensors B U and B F are 0 (the tensor with all values 0 ).

Original: Finally, to construct tensor B R we consider the matrix D of dimension 1 × (3 d + 3) such that
Revised: Finally, to construct tensor B R we consider the matrix D of dimension 1 × (3 d + 3) such that

Original: [Equation]
Revised: [Equation]

Original: Then we let B Ri, : , : = D for all i .
Revised: Then we let B Ri, : , : = D for all i .

Original: Finally, we consider f U
Revised: Finally, we consider f U

Original: = f R =
Revised: = f R =

Original: f F = σ .
Revised: f F = σ .

Original: The constructed Neural
Revised: The constructed Neural

Original: GPU is a uniform Neural GPU.
Revised: GPU is a uniform Neural GPU.

Original: Before continuing with the proof we note that for every kernel K X and tensor S we have that
Revised: Before continuing with the proof we note that for every kernel K X and tensor S we have that

Original: [Equation]
Revised: [Equation]

Original: C ORRECTNESS OF THE CONSTRUCTION
Revised: C ORRECTNESS OF THE CONSTRUCTION

Original: We now prove that the following properties hold for every t ≥ 0 :
Revised: We now prove that the following properties hold for every t ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: where α kj is given by the recurrence α k 

Revised: where α kj is given by the recurrence α k

Original: = h k and α kj = σ ( α kj − 1 U ) .
Revised: = h k and α kj = σ ( α kj − 1 U ) .

Original: Notice that g j = α nj .
Revised: Notice that g j = α nj .

Original: That is, we are going to prove that our construction actually simulates N .
Revised: That is, we are going to prove that our construction actually simulates N .

Original: By (19) one can see that the intuition in our construction is to use the ﬁrst d components to simulate the encoder part, the next d components to communicate data between the encoder and decoder simulation, and the next d components to simulate the decoder part.
Revised: By (27) one can see that the intuition in our construction is to use the ﬁrst d components to simulate the encoder part, the next d components to communicate data between the encoder and decoder simulation, and the next d components to simulate the decoder part.

Original: The last three components are needed as gadgets for the gates to actually simulate a sequencial read of the input, and to ensure that the hidden state of the encoder and decoder are updated properly.
Revised: The last three components are needed as gadgets for the gates to actually simulate a sequencial read of the input, and to ensure that the hidden state of the encoder and decoder are updated properly.

Original: We prove the above statement by induction in t .
Revised: We prove the above statement by induction in t .

Original: First notice that the property trivially holds for S 0 .
Revised: First notice that the property trivially holds for S 0 .

Original: Now assume that this holds for t − 1 and lets prove it for t .
Revised: Now assume that this holds for t − 1 and lets prove it for t .

Original: We know that U t is computed as
Revised: We know that U t is computed as

Original: [Equation]
Revised: [Equation]

Original: Thus we have that:
Revised: Thus we have that:

Original: [Equation]
Revised: [Equation]

Original: By the induction hypothesis we have
Revised: By the induction hypothesis we have

Original: [Equation]
Revised: [Equation]

Original: Now, notice that K U 1 , 1 , : , : and K U 2 , 1 , : , : are not zero only in its three last rows, thus we can focus on the three last components of the vectors in S t − 1 , and then we can compute U ti, 1 , : as
Revised: Now, notice that K U 1 , 1 , : , : and K U 2 , 1 , : , : are not zero only in its three last rows, thus we can focus on the three last components of the vectors in S t − 1 , and then we can compute U ti, 1 , : as

Original: [Equation]
Revised: [Equation]

Original: K U 1 , 1 , : , : + [ , , , 0 , 0 , 0] K U 2 , 1 , : , : ) for i < t −σ ([ , , , 0 , 0 , 0] K U 1 , 1 , : , : + [ , , , 0 , 1 , 0] K U 2 , 1 , : , : ) for i = t −σ ([ , , , 0 , 1 , 0] K U 1 , 1 , : , : + [ , , , 1 , 1 , 0] K U 2 , 1 , : , : ) for i =
Revised: K U 1 , 1 , : , : + [ , , , 0 , 0 , 0] K U 2 , 1 , : , : ) for i < t −σ ([ , , , 0 , 0 , 0] K U 1 , 1 , : , : + [ , , , 0 , 1 , 0] K U 2 , 1 , : , : ) for i = t −σ ([ , , , 0 , 1 , 0] K U 1 , 1 , : , : + [ , , , 1 , 1 , 0] K U 2 , 1 , : , : ) for i =

Original: t σ ([ , , , 1 , 1 , 0] K U 1 , 1 , : , : + [ , , , 1 , 1 , 0] K U 2 , 1 , : , : ) for i > t = 
Revised: t σ ([ , , , 1 , 1 , 0] K U 1 , 1 , : , : + [ , , , 1 , 1 , 0] K U 2 , 1 , : , : ) for i > t = 

Original:  σ ([ 0 , 0 , 0 , 0 , 0 , 0]) for i < t −σ ([ 0 , 0 , 0 , 0 , 0 , 0]) for i = t −σ ([ 0 , 0 , 1 , 0 , 0 , 1]) for i =
Revised:  σ ([ 0 , 0 , 0 , 0 , 0 , 0]) for i < t −σ ([ 0 , 0 , 0 , 0 , 0 , 0]) for i = t −σ ([ 0 , 0 , 1 , 0 , 0 , 1]) for i =

Original: t σ ([ 1 , 1 , 1 , 1 , 1 , 1]) for i > t = (cid:40) [ 0 , 0 , 0 , 0 , 0 , 0] for i < t [ 0 , 0 , 1 , 0 , 0 , 1] for i =
Revised: t σ ([ 1 , 1 , 1 , 1 , 1 , 1]) for i > t = (cid:40) [ 0 , 0 , 0 , 0 , 0 , 0] for i < t [ 0 , 0 , 1 , 0 , 0 , 1] for i =

Original: t [ 1 , 1 , 1 , 1 , 1 , 1] for i > t
Revised: t [ 1 , 1 , 1 , 1 , 1 , 1] for i > t

Original: Now, for R t we have
Revised: Now, for R t we have

Original: R t =
Revised: R t =

Original: σ ( K R ∗ S t − 1 + B R ) and thus for R ti, 1 , : we have
Revised: σ ( K R ∗ S t − 1 + B R ) and thus for R ti, 1 , : we have

Original: R ti, 1 , : = σ (( K R ∗ S t − 1 ) i, 1 , : + B Ri, 1 , : )
Revised: R ti, 1 , : = σ (( K R ∗ S t − 1 ) i, 1 , : + B Ri, 1 , : )

Original: =
Revised: =

Original: σ ( S t − 1 i − 1 , 1 , : K R 1 , 1 , : , : + S t − 1 i, 1 , : K R 2 , 1 , : , : + B Ri, 1 , : )
Revised: σ ( S t − 1 i − 1 , 1 , : K R 1 , 1 , : , : + S t − 1 i, 1 , : K R 2 , 1 , : , : + B Ri, 1 , : )

Original: =
Revised: =

Original: σ ( S t − 1 i, 1 , : K R 2 , 1 , : , : + B Ri, 1 , : )
Revised: σ ( S t − 1 i, 1 , : K R 2 , 1 , : , : + B Ri, 1 , : )

Original: =
Revised: =

Original: σ ( S t − 1 i, 1 , : K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) where we deleted the term with K R 1 , 1 , : , : since it is the null matrix.
Revised: σ ( S t − 1 i, 1 , : K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) where we deleted the term with K R 1 , 1 , : , : since it is the null matrix.

Original: Then by using the deﬁnition of
Revised: Then by using the deﬁnition of

Original: S t − 1 i, 1 , : above (Equation (20))
Revised: S t − 1 i, 1 , : above (Equation (28))

Original: we have
Revised: we have

Original: R ti, 1 , : =   σ ([ , , , 0 , 0 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i < t −σ ([ , , , 0 , 1 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i = t −σ ([ , , , 1 , 1 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i =
Revised: R ti, 1 , : =   σ ([ , , , 0 , 0 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i < t −σ ([ , , , 0 , 1 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i = t −σ ([ , , , 1 , 1 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i =

Original: t σ ([ , , , 1 , 1 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i > t = 
Revised: t σ ([ , , , 1 , 1 , 0] K R 2 , 1 , : , : + [ 0 , 0 , 1 , 0 , 0 , 1]) for i > t = 

Original:  σ ([ 0 , 0 , 1 , 0 , 0 , 1]) for i < t −σ ([ 0 , 1 , 1 , 0 , 1 , 1]) for i = t −σ ([ 1 , 1 , 1 , 1 , 1 , 1]) for i =
Revised:  σ ([ 0 , 0 , 1 , 0 , 0 , 1]) for i < t −σ ([ 0 , 1 , 1 , 0 , 1 , 1]) for i = t −σ ([ 1 , 1 , 1 , 1 , 1 , 1]) for i =

Original: t σ ([ 1 , 1 , 1 , 1 , 1 , 1]) for i > t = (cid:40) [ 0 , 0 , 1 , 0 , 0 , 1] for i < t −[ 0 , 1 , 1 , 0 , 1 , 1] for i = t −[ 1 , 1 , 1 , 1 , 1 , 1] for i ≥ t
Revised: t σ ([ 1 , 1 , 1 , 1 , 1 , 1]) for i > t = (cid:40) [ 0 , 0 , 1 , 0 , 0 , 1] for i < t −[ 0 , 1 , 1 , 0 , 1 , 1] for i = t −[ 1 , 1 , 1 , 1 , 1 , 1] for i ≥ t

Original: We can now compute S ti, 1 , : .
Revised: We can now compute S ti, 1 , : .

Original: By the deﬁnition of S t we have
Revised: By the deﬁnition of S t we have

Original: [Equation]
Revised: [Equation]

Original: where we dropped B F that has only zeros.
Revised: where we dropped B F that has only zeros.

Original: First, by using what we already computed for U ti, 1 , : we have that
Revised: First, by using what we already computed for U ti, 1 , : we have that

Original: S ti, 1 , :
Revised: S ti, 1 , :

Original: [Equation]
Revised: [Equation]

Original: for i < t
Revised: for i < t

Original: [Equation]
Revised: [Equation]

Original: ) for i =
Revised: for i =

Original: t
Revised: t

Original: S t
Revised: S t

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , : for i > t
Revised: , : for i > t

Original: When i =
Revised: When i =

Original: t we have that S t − 1 i, 1 , : =
Revised: t we have that S t − 1 i, 1 , : =

Original: [ x i , 0 , 0 , 1 , 0 , 0] (Equation (20)), thus [ 0 , 0 , 1 , 0 , 0 , 1] (cid:12) S t − 1 i, 1 , : =[ 0 , 0 , 0 , 0 , 0 , 0] .
Revised: [ x i , 0 , 0 , 1 , 0 , 0] (Equation (28)), thus [ 0 , 0 , 1 , 0 , 0 , 1] (cid:12) S t − 1 i, 1 , : =

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: for i < t
Revised: for i < t

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: for i =
Revised: for i =

Original: t [x i
Revised: t [

Original: [Equation]
Revised: [Equation]

Original: ] for i > t (21)
Revised: [Equation] i > t (29)

Original: We are almost done with the inductive step, we only need to compute σ (( K F ∗ ( R t (cid:12) S t − 1 ))
Revised: We are almost done with the inductive step, we only need to compute σ (( K F ∗ ( R t (cid:12) S t − 1 ))

Original: i, 1 , : ) .
Revised: i, 1 , : ) .

Original: Given what we have for R t and S t − 1 we have that R t (cid:12) S t − 1 is ( R t (cid:12) S t − 1 ) i, 1 , : =
Revised: Given what we have for R t and S t − 1 we have that R t (cid:12) S t − 1 is ( R t (cid:12) S t − 1 ) i, 1 , : =

Original:   [ 0 , 0 , 1 , 0 , 0 , 1] (cid:12) [ 0 , 0 ,
Revised:   [ 0 , 0 , 1 , 0 , 0 , 1] (cid:12) [ 0 , 0 ,

Original: α i t − 1 − i , 0 , 0 , 0] for i < t −[ 0 , 1 , 1 , 0 , 1 , 1] (cid:12) [ h i , h i , 0 , 0 , 1 , 0] for i = t −[ 1 , 1 , 1 , 1 , 1 , 1] (cid:12) [ x i , 0 , 0 , 1 , 1 , 0] for i ≥ t =   [ 0 , 0 , α it − 1 − i , 0 , 0 , 0] for i < t −[ 0 , h i , 0 , 0 , 1 , 0] for i =
Revised: α i t − 1 − i , 0 , 0 , 0] for i < t −[ 0 , 1 , 1 , 0 , 1 , 1] (cid:12) [ h i , h i , 0 , 0 , 1 , 0] for i = t −[ 1 , 1 , 1 , 1 , 1 , 1] (cid:12) [ x i , 0 , 0 , 1 , 1 , 0] for i ≥ t =   [ 0 , 0 , α it − 1 − i , 0 , 0 , 0] for i < t −[ 0 , h i , 0 , 0 , 1 , 0] for i =

Original: t −[ x i , 0 , 0 , 1 , 1 , 0] for i ≥ t
Revised: t −[ x i , 0 , 0 , 1 , 1 , 0] for i ≥ t

Original: Lets T t =
Revised: Lets T t =

Original: σ ( K F ∗ ( R t (cid:12) S t − 1 )) .
Revised: σ ( K F ∗ ( R t (cid:12) S t − 1 )) .

Original: Notice that from Equation (21) we actually need to know the values in T ti, 1 , : only for i ≤ t .
Revised: Notice that from Equation (29) we actually need to know the values in T ti, 1 , : only for i ≤ t .

Original: Now we have that
Revised: Now we have that

Original: [Equation]
Revised: [Equation]

Original: for i < t −
Revised: for i < t −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: for i =
Revised: for i =

Original: t −
Revised: t −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 0] K
Revised: 0] K

Original: [Equation]
Revised: [Equation]

Original: , 1 , : , : ) for i =
Revised: , 1 , : , : ) for i =

Original: t =
Revised: t =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for i < t −
Revised: for i < t −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 0]) for i =
Revised: 0]) for i =

Original: t −
Revised: t −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: 0 , 0 , 1 , 0]) for i =
Revised: 0 , 0 , 1 , 0]) for i =

Original: t =  
Revised: t =  

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: it − i , 0 , 0 , 0] for i < t −[ 0 , 0 , α i 1 , 0 , 0 , 0] for i =
Revised: it − i , 0 , 0 , 0] for i < t −[ 0 , 0 , α i 1 , 0 , 0 , 0] for i =

Original: t −[ h i , h i , 0 , 0 , 1 , 0] for i =
Revised: t −[ h i , h i , 0 , 0 , 1 , 0] for i =

Original: t = (cid:26) [ 0 , 0 , α i t − i , 0 , 0 , 0] for i < t [ h i , h i , 0 , 0 , 1 , 0] for i =
Revised: t = (cid:26) [ 0 , 0 , α i t − i , 0 , 0 , 0] for i < t [ h i , h i , 0 , 0 , 1 , 0] for i =

Original: t
Revised: t

Original: Putting the value of T ti, 1 , : in Equation (21) we obtain
Revised: Putting the value of T ti, 1 , : in Equation (29) we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for i < t
Revised: for i < t

Original: [Equation]
Revised: [Equation]

Original: for i =
Revised: for i =

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: for
Revised: for

Original: [Equation]
Revised: [Equation]

Original: which is exactly what we needed to prove (Equation (19)).
Revised: which is exactly what we needed to prove (Equation (27)).

Original: Now, lets focus on S n + t n, 1 , : for t ≥ 1 .
Revised: Now, lets focus on S n + t n, 1 , : for t ≥ 1 .

Original: By what we have just proved, we obtain that
Revised: By what we have just proved, we obtain that

Original: [Equation]
Revised: [Equation]

Original: which is the decoder part of the RNN N .
Revised: which is the decoder part of the RNN N .

Original: Thus, we can simulate the complete network N with a
Revised: Thus, we can simulate the complete network N with a

Original: Neural GPU.
Revised: Neural GPU.

Original: C. P ROOF OF P ROPOSITION 4. 
Revised: C. P ROOF OF P ROPOSITION 4.

Original: We ﬁrst prove the following claim: Assume that S ∈ Q h × w ×
Revised: We ﬁrst prove the following claim: Assume that S ∈ Q h × w ×

Original: d is a tensor that satisﬁes the following property: there exists a p ≥ 1 that divides h and such that, for every i ∈ { 1 , 2 , . .
Revised: d is a tensor that satisﬁes the following property: there exists a p ≥ 1 that divides h and such that, for every i ∈ { 1 , 2 , . .

Original: , h − p } it holds that
Revised: , h − p } it holds that

Original: [Equation]
Revised: [Equation]

Original: Given that we will be considering circular convolutions, then we have that for every (cid:96) ≤ 

Revised: Given that we will be considering circular convolutions, then we have that for every (cid:96) ≤

Original: we have that S (cid:96),j, : =
Revised: we have that S (cid:96),j, : =

Original: S h + (cid:96),j, : and for every (cid:96) >
Revised: S h + (cid:96),j, : and for every (cid:96) >

Original: h we have that S (cid:96),j, : =
Revised: h we have that S (cid:96),j, : =

Original: S h − (cid:96),j, : .
Revised: S h − (cid:96),j, : .

Original: With this we have that for every i ∈ N it holds that S i, : , : = S i + p, : , : that is, we do not need to restrict to values in i ∈ { 1 , 2 , . .
Revised: With this we have that for every i ∈ N it holds that S i, : , : = S i + p, : , : that is, we do not need to restrict to values in i ∈ { 1 , 2 , . .

Original: , h − p } .
Revised: , h − p } .

Original: Now lets K be an arbitrary kernel bank of shape ( k H , k W , d, d ) .
Revised: Now lets K be an arbitrary kernel bank of shape ( k H , k W , d, d ) .

Original: Let T =
Revised: Let T =

Original: K (cid:126) S where (cid:126) denotes the circular convolution.
Revised: K (cid:126) S where (cid:126) denotes the circular convolution.

Original: We prove next that
Revised: We prove next that

Original: T i,: ,
Revised: T i,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: T i
Revised: T i

Original: [Equation]
Revised: [Equation]

Original: p, : , : for every i .
Revised: p, : , : for every i .

Original: This is a simple fact that follows from the way in which the convolution is deﬁned.
Revised: This is a simple fact that follows from the way in which the convolution is deﬁned.

Original: We use the notation in the body of the paper for T =
Revised: We use the notation in the body of the paper for T =

Original: K (cid:126) S , that is, we denote by s ij the vector S
Revised: K (cid:126) S , that is, we denote by s ij the vector S

Original: i,j, : and K ij the matrix K i,j, : , : .
Revised: i,j, : and K ij the matrix K i,j, : , : .

Original: Notice that s ij =
Revised: Notice that s ij =

Original: s i + p,j for every i ∈ N .
Revised: s i + p,j for every i ∈ N .

Original: Now the circular convolution is
Revised: Now the circular convolution is

Original: [Equation]
Revised: [Equation]

Original: where ∆ 1 ( u ) = u − (cid:98) k H / 2 (cid:99) − 1 and ∆ 2 ( v ) = v − (cid:98) k W / 2 (cid:99) − 1 .
Revised: where ∆ 1 ( u ) = u − (cid:98) k H / 2 (cid:99) − 1 and ∆ 2 ( v ) = v − (cid:98) k W / 2 (cid:99) − 1 .

Original: Then, given that s ij = s i + p,j for every i ∈ N we have that
Revised: Then, given that s ij = s i + p,j for every i ∈ N we have that

Original: [Equation]
Revised: [Equation]

Original: , : and then, T i, : , : = T i + p, : , : .
Revised: , : and then, T i, : , : = T i + p, : , : .

Original: Consider now an arbitrary uniform Neural GPU that processes tensor S above, and assume that
Revised: Consider now an arbitrary uniform Neural GPU that processes tensor S above, and assume that

Original: S 1 , S 2 , . .
Revised: S 1 , S 2 , . .

Original: , S r is the sequence produced by it.
Revised: , S r is the sequence produced by it.

Original: Next we prove that for every t and for every i it holds that S ti, : , : = S ti + p, : , : .
Revised: Next we prove that for every t and for every i it holds that S ti, : , : = S ti + p, : , : .

Original: We prove it by induction in t .
Revised: We prove it by induction in t .

Original: For the case S 0 it holds by deﬁnition.
Revised: For the case S 0 it holds by deﬁnition.

Original: Thus assume that S t − 1 satisﬁes the property.
Revised: Thus assume that S t − 1 satisﬁes the property.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Since we are considering uniform Neural GPUs, we know that there exist three matrices B U , B R and B F such that for every i it holds that B Ui, : , : = B U , B Ri, : , : = B R , and B Fi, : , : = B F .
Revised: Since we are considering uniform Neural GPUs, we know that there exist three matrices B U , B R and B F such that for every i it holds that B Ui, : , : = B U , B Ri, : , : = B R , and B Fi, : , : = B F .

Original: It is easy to prove that U ti, : , : = U ti + p, : , : .
Revised: It is easy to prove that U ti, : , : = U ti + p, : , : .

Original: First note that by inductive hypothesis, we have that S t − 1 i, : , : =
Revised: First note that by inductive hypothesis, we have that S t − 1 i, : , : =

Original: S t − 1 i + p, : , : and thus by the property proved above we have that ( K U ∗ S t − 1 )
Revised: S t − 1 i + p, : , : and thus by the property proved above we have that ( K U ∗ S t − 1 )

Original: i, : , : =
Revised: i, : , : =

Original: ( K U ∗ S t − 1 )
Revised: ( K U ∗ S t − 1 )

Original: [Equation]
Revised: [Equation]

Original: Thus we have that
Revised: Thus we have that

Original: [Equation]
Revised: [Equation]

Original: With a similar argument we can prove that R ti, : , : = R ti + p, : , : .
Revised: With a similar argument we can prove that R ti, : , : = R ti + p, : , : .

Original: Moreover, notice that ( R t (cid:12) S t − 1 ) i, : , : =
Revised: Moreover, notice that ( R t (cid:12) S t − 1 ) i, : , : =

Original: ( R t (cid:12) S t − 1 ) i + p, : , : , and thus ( K F ∗ ( R t (cid:12) S t − 1 ))
Revised: ( R t (cid:12) S t − 1 ) i + p, : , : , and thus ( K F ∗ ( R t (cid:12) S t − 1 ))

Original: i, : , : =
Revised: i, : , : =

Original: ( K F ∗ ( R t (cid:12) S t − 1 ))
Revised: ( K F ∗ ( R t (cid:12) S t − 1 ))

Original: i + p, : , : .
Revised: i + p, : , : .

Original: With all this we ﬁnally we have that
Revised: With all this we ﬁnally we have that

Original: [Equation]
Revised: [Equation]

Original: This completes the ﬁrst part of the proof.
Revised: This completes the ﬁrst part of the proof.

Original: We have shown that if the input of a uniform neural GPU is periodic, then the output is also periodic.
Revised: We have shown that if the input of a uniform neural GPU is periodic, then the output is also periodic.

Original: We make a ﬁnal observation.
Revised: We make a ﬁnal observation.

Original: Let N be a uniform Neural GPU, and S ∈ Q kp × w ×
Revised: Let N be a uniform Neural GPU, and S ∈ Q kp × w ×

Original: d be a tensor such that S i, : , : = S i + p, : , : for every i .
Revised: d be a tensor such that S i, : , : = S i + p, : , : for every i .

Original: Moreover, let T ∈ Q k (cid:48)
Revised: Moreover, let T ∈ Q k (cid:48)

Original: p × w ×
Revised: p × w ×

Original: d be a tensor such that T i, : , : = T i + p, : , : for every i , and assume that S 1: p, : , : =
Revised: d be a tensor such that T i, : , : = T i + p, : , : for every i , and assume that S 1: p, : , : =

Original: T 1: p, : , : .
Revised: T 1: p, : , : .

Original: Lets S 1 , S 2 , . . .
Revised: Lets S 1 , S 2 , . . .

Original: and T 1 , T 2 , . .
Revised: and T 1 , T 2 , . .

Original: be the sequences produced by N .
Revised: be the sequences produced by N .

Original: Then with a similar argument as above it is easy to prove that for every t it holds that S t 1: p, : , : =
Revised: Then with a similar argument as above it is easy to prove that for every t it holds that S t 1: p, : , : =

Original: T t 1: p, : , : .
Revised: T t 1: p, : , : .

Original: From this it is easy to prove that uniform Neural GPUs will no be able to recognize the length of periodic inputs.
Revised: From this it is easy to prove that uniform Neural GPUs will no be able to recognize the length of periodic inputs.

Original: Thus assume that there is a language recognizer A deﬁned by of a uniform neural
Revised: Thus assume that there is a language recognizer A deﬁned by of a uniform neural

Original: GPU N such that L ( A ) contains all strings of even length.
Revised: GPU N such that L ( A ) contains all strings of even length.

Original: Assume that u is an arbitrary string in Σ such that | u | = p with p an odd number, and let w = uu and w (cid:48) = uuu .
Revised: Assume that u is an arbitrary string in Σ such that | u | = p with p an odd number, and let w = uu and w (cid:48) = uuu .

Original: Notice that | w | = 2 p and thus w ∈ L ( A ) , but | w (cid:48) | = 3 p and thus w (cid:48) (cid:54)∈ L ( A ) .
Revised: Notice that | w | = 2 p and thus w ∈ L ( A ) , but | w (cid:48) | = 3 p and thus w (cid:48) (cid:54)∈ L ( A ) .

Original: Let f : Σ → Q d and let X = f ( w ) =
Revised: Let f : Σ → Q d and let X = f ( w ) =

Original: ( x 1 , x 2 , . . .
Revised: ( x 1 , x 2 , . . .

Original: , x 2 p ) and X (cid:48) =
Revised: , x 2 p ) and X (cid:48) =

Original: f ( w (cid:48) )
Revised: f ( w (cid:48) )

Original: =
Revised: =

Original: ( x (cid:48) 1 , x (cid:48) 2 , . .
Revised: ( x (cid:48) 1 , x (cid:48) 2 , . .

Original: , x (cid:48) 3 p ) .
Revised: , x (cid:48) 3 p ) .

Original: Consider now the tensor S ∈ Q 2 p × w ×
Revised: Consider now the tensor S ∈ Q 2 p × w ×

Original: d
Revised: d

Original: such that S i, 1 , : = x i for i ∈ { 1 , . . .
Revised: such that S i, 1 , : = x i for i ∈ { 1 , . . .

Original: , 2 p } , thus S i, : , : =
Revised: , 2 p } , thus S i, : , : =

Original: S i + p, : , : .
Revised: S i + p, : , : .

Original: Similarly, consider T ∈ Q 3 p × w ×
Revised: Similarly, consider T ∈ Q 3 p × w ×

Original: d
Revised: d

Original: such that such that T i, 1 , : = x (cid:48)
Revised: such that such that T i, 1 , : = x (cid:48)

Original: i for i ∈ { 1 , . .
Revised: i for i ∈ { 1 , . .

Original: . , 3 p } , and thus T i, : , : =
Revised: . , 3 p } , and thus T i, : , : =

Original: T i + p, : , : .
Revised: T i + p, : , : .

Original: Notice that S 1: p, : , : =
Revised: Notice that S 1: p, : , : =

Original: T 1: p, : , : then by the property above we have that for every t it holds that S t 1: p, : , :
Revised: T 1: p, : , : then by the property above we have that for every t it holds that S t 1: p, : , :

Original: = T t 1: p, : , : .
Revised: = T t 1: p, : , : .

Original: In particular, we have S tp, : , : =
Revised: In particular, we have S tp, : , : =

Original: T tp, : , : .
Revised: T tp, : , : .

Original: We also know that S tp, : , : =
Revised: We also know that S tp, : , : =

Original: S t 2 p, : , : and that T tp, : , : = T t 2 p, : , : = T t 3 p, : , : .
Revised: S t 2 p, : , : and that T tp, : , : = T t 2 p, : , : = T t 3 p, : , : .

Original: Thus we have that for every t it holds that
Revised: Thus we have that for every t it holds that

Original: S t 2 p, 1 , : = T t 3 p, 1 , : .
Revised: S t 2 p, 1 , : = T t 3 p, 1 , : .

Original: From this we conclude that the outputs of N for both inputs X and X (cid:48) are the same, and thus if A accepts w then A accepts w (cid:48) which is a contradiction.
Revised: From this we conclude that the outputs of N for both inputs X and X (cid:48) are the same, and thus if A accepts w then A accepts w (cid:48) which is a contradiction.